Harmonics of learning: a mathematical theory for the emergence of Fourier features in learning systems such as neural networks

Artificial neural networks (ANNs) show a notable pattern when trained on natural data, regardless of the exact initialization, data set, or training goal; Models trained on the same data domain converge on similar learned patterns. For example, for different image models, the weights of the initial layers tend to converge on Gabor filters and color contrast detectors. Many of these features suggest a global representation that goes beyond biological and artificial systems, and these features are observed in the visual cortex. These findings are practical and well established in the field of machines that can interpret literature but lack theoretical explanations.

Localized versions of canonical 2D Fourier basis functions are the most widely observed universal features in image models, for example, Gabor filters or wavelets. When vision models are trained on tasks such as efficient coding, classification, temporal coherence, and next-step prediction goals, these Fourier features appear in the initial layers of the model. Apart from this, non-localized Fourier features have been observed in networks trained to solve tasks where cyclic wrapping is allowed, for example, modular arithmetic, more general group compositions, or invariance to the group of cyclic translations.

Researchers from KTH, the Redwood Center for Theoretical Neuroscience and UC Santa Barbara introduced a mathematical explanation for the increase in Fourier features in learning systems such as neural networks. This increase is due to the posterior invariance of the learner becoming insensitive to certain transformations, for example, plane translation or rotation. The team has obtained theoretical guarantees about the Fourier characteristics in invariant learners that can be used in different machine learning models. This derivation is based on the concept that invariance is a fundamental bias that can be implicitly and sometimes explicitly injected into learning systems due to the symmetries of natural data.

The standard discrete Fourier transform is a special case of more general Fourier transforms on groups, which can be defined by replacing the basis of the harmonics with different representations of unitary groups. A set of previous theoretical works for sparse coding models is formed, deriving the conditions under which sparse linear combinations are used to recover the original bases that generate data with the help of a network. The proposed theory covers various situations and neural network architectures that help lay the foundation for a theory of representation learning in artificial and biological neural systems.

The team gave two informal theorems in this paper, the first stating that if a parametric function of a certain type is invariant in the input variable to the action of a finite group G, then each component of its weights W coincides with a harmonic of G to a linear transformation. The second theorem states that if a parametric function is almost invariant with respect to G under some functional limits and the weights are orthonormal, then the multiplicative table of G can be recovered from W. Furthermore, a model is implemented to satisfy the need of the proposed method. theory and trained through different learning on an objective that supports invariance and the extraction of the multiplicative table of G from its weights.

In conclusion, the researchers introduced a mathematical explanation for the growth of Fourier functions in learning systems such as neural networks. Furthermore, they showed that if a machine learning model of a specific type is invariant for a finite group, then its weights are closely related to the Fourier transform on that group, and the algebraic structure of an unknown group can be recovered from an invariant model. . Future work includes the study of analogues of the proposed real number theory, which is an interesting area that will align more with current practices in this field.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our 42k+ ML SubReddit