Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

The symmetry and geometry of input data are considered to be encoded in the internal data representation inside the neural network, but the specific encoding rule has been less investigated. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform, from a joint group invariant function on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it can describe the arrangement of parameters for the network to represent a target function, which is understood as the encoding rule, and (2) it implies the universality of the network. Based on the group representation theory, we present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks, for example, the original ridgelet transform, formal deep networks, and the dual voice transform. Since traditional universality theorems were demonstrated based on functional analysis, this study sheds light on the group theoretic aspect of the approximation theory, connecting geometric deep learning to abstract harmonic analysis.Comment: NeurReps 202

Probabilistic Invariant Learning with Randomized Linear Classifiers

Designing models that are both expressive and preserve known invariances of tasks is an increasingly hard problem. Existing solutions tradeoff invariance for computational or memory resources. In this work, we show how to leverage randomness and design models that are both expressive and invariant but use less resources. Inspired by randomized algorithms, our key insight is that accepting probabilistic notions of universal approximation and invariance can reduce our resource requirements. More specifically, we propose a class of binary classification models called Randomized Linear Classifiers (RLCs). We give parameter and sample size conditions in which RLCs can, with high probability, approximate any (smooth) function while preserving invariance to compact group transformations. Leveraging this result, we design three RLCs that are provably probabilistic invariant for classification tasks over sets, graphs, and spherical data. We show how these models can achieve probabilistic invariance and universality using less resources than (deterministic) neural networks and their invariant counterparts. Finally, we empirically demonstrate the benefits of this new class of models on invariant tasks where deterministic invariant neural networks are known to struggle

Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systems

A new class of non-homogeneous state-affine systems is introduced for use in reservoir computing. Sufficient conditions are identified that guarantee first, that the associated reservoir computers with linear readouts are causal, time-invariant, and satisfy the fading memory property and second, that a subset of this class is universal in the category of fading memory filters with stochastic almost surely uniformly bounded inputs. This means that any discrete-time filter that satisfies the fading memory property with random inputs of that type can be uniformly approximated by elements in the non-homogeneous state-affine family.Comment: 41 page

Observing scale-invariance in non-critical dynamical systems

Recent observation for scale invariant neural avalanches in the brain have been discussed in details in the scientific literature. We point out, that these results do not necessarily imply that the properties of the underlying neural dynamics are also scale invariant. The reason for this discrepancy lies in the fact that the sampling statistics of observations and experiments is generically biased by the size of the basins of attraction of the processes to be studied. One has hence to precisely define what one means with statements like `the brain is critical'. We recapitulate the notion of criticality, as originally introduced in statistical physics for second order phase transitions, turning then to the discussion of critical dynamical systems. We elucidate in detail the difference between a 'critical system', viz a system on the verge of a phase transition, and a 'critical state', viz state with scale-invariant correlations, stressing the fact that the notion of universality is linked to critical states. We then discuss rigorous results for two classes of critical dynamical systems, the Kauffman net and a vertex routing model, which both have non-critical states. However, an external observer that samples randomly the phase space of these two critical models, would find scale invariance. We denote this phenomenon as 'observational criticality' and discuss its relevance for the response properties of critical dynamical systems