We would like to approximate a high dimensional set in a Hilbert space by some lower dimensional manifold of dimension , using an auto-encoder.
We start with an illustrative example. Consider the KdV equation which descibes the propagation of a solition in one dimensional space with periodic boudnary conditions. Let be the set we wish to approximate. In practice the equation is solved on a space=time grid, meaning we take for example a 256-point discretization of the space domain and a 5000-point discretization of the time domain (+1=5001 for the initial condition) to be our solution snapshots, and we wish to approximate this high dimensional set of size . For each point in time, we need to figure out 256 values for u, meaning the solution depends on 256 parameters. The goal is to find an auto-encoder that gives a lower dimensional estimate for u, meaning with much fewer parameters than 256 that encapsulates all the necessary information about the 256 points up to some prescribed tolerance with respect to some error measure - usually /mean-squared or /worst-case.
The idea could be explained by the following.
One basic approach is to use a purely linear approximation. Meaning we approximate (or the discretized set) by a lower dimensional linear space of a large enough dimension such that all the values in are well approximated after going through the auto-encoder. That is, in a Hilbert space, our auto-encoder is just the linear map which projects onto the optimal lower-dimensional subspace of dimension . In practice, this is achieved through principal component analysis (PCA) performed on the matrix discretizing the set and taking as a basis for the linear subspace the first principal components associated with the largest singular values. The problem with this approach is that for highly-nonlinear manifolds, the number of dimensions needed to a get a high precision is often high, and sometimes infeasible computationaly. This is in particular the case for problems dominated by transport (e.g. KdV).
Since using a purely linear approach is not a great idea, we seek to nonlinearize. For this purpose we first acknowledge that the auto-encoder is a two step operation, where the values in M first are encoded in a lower dimensional space by an encoder . Then a decoder maps the set of parameters associated with each value back to the Hilbert space, hoping to get a good enough approximation of . The auto-encoder is associated with the pair . In the case of a linear approximation, both and are linear. If we wish to nonlinearize, we are faced with three choices: we only use a nonlinear encoder, we only use a nonlinear decoder, or we set both to be nonlinear. If we only consider linear decoders, then the image of the decoder will be a linear space in , yielding the same error scenario as in the fully-linear. If we consider a nonlinear endcoder, then imposing stability on the approximation operator yields a difficult-to-implement optimal nonlinear encoder - associated with NP-hard optimization problems. A linear encoder is nevertheless easy to understand: it is simply projecting onto the (optimal) basis elements of the parameter space. This leaves us with the choice of a linear encoder and a nonlinear decoder. The encoder gives us parameters, and we use the optimal basis elements to span an dimensional linear space inside X, and the decoder adds a nonlinear “correction” using the first parameters so that the final approximation set, , is an -dimensional manifold living in a potentially higher dimensional linear space , depending on how many nonlinear corrections are added. The final expression for our decoder would look like: where the are nonlinear functions of the parameter vector.
The question now is how do we decide on ? Where do we get these nice basis elements? How many should we consider for the encoder and how many for the decoder?
To start, should be determined such that the projection error onto is controlled in some way. That is, we pick the first eigenvalues of the operator associated with PCA such that for some error threshold that guarantees the desired precision. Our nice basis will then be the corresponding principal components. Here is where we should pay attention: choosing the first principal components as the basis for the encoder will NOT give the optimal approximation - this was demonestrated. One must instead opt for an adaptive approach, choosing those components whose coefficients won’t be approximated well by our nonlinear approximations - in that case we just use the coefficient that is the constant projection coefficient (i.e. consider that direction in linear approximation). Thus, there will be a learning phase, performed on a training set where we not only construct but also check for which principal component directions should be considered for the nonliear part and which for the linear part. In our earlier example, we can take the values $t_i [0 , 0.2] $ to learn the indiced of directions to be considered in the linear set and the coefficients .
Now comes the contribution of the paper: we wish to choose good functions . Our nice choice comes from an observation.
A coefficient for may have a highly nonlinear
relation with the first coefficients but a much smoother relation when expressed in terms of and additional coefficients with . This observation suggests the following compositional structure of the decoder’s functions
(Bensalah et al. 2025)
The nonlinear functions proposed, as suggested by the titles, are polynomials. There are different options for the space from which these polynomials could come, but ultimetly the final polynomial is that one that minimizes some error from the projections coefficients (on the sample data).
Therefore the coefficients will be given by , where is the set of indices associated with that we constructed adaptively.
That’s all we need to know.
The method is compared to other sota methods and outperforms them all in error and numeber of parameters. For our KdV equation, with degree 5 polynomials form an appropriate polynomial space with only 43 PCA components, an error of is achieved using only two parameters!