450 research outputs found
Zeros of nonlinear systems with input invariances
A nonlinear system possesses an invariance with respect to a set of transformations if its output dynamics remain invariant when transforming the input, and adjusting the initial condition accordingly. Most research has focused on invariances with respect to time-independent pointwise transformations like translational-invariance (u(t) -> u(t) + p, p in R) or scale-invariance (u(t) -> pu(t), p in R>0). In this article, we introduce the concept of s0-invariances with respect to continuous input transformations exponentially growing/decaying over time. We show that s0-invariant systems not only encompass linear time-invariant (LTI) systems with transfer functions having an irreducible zero at s0 in R, but also that the input/output relationship of nonlinear s0-invariant systems possesses properties well known from their linear counterparts. Furthermore, we extend the concept of s0-invariances to second- and higher-order s0-invariances, corresponding to invariances with respect to transformations of the time-derivatives of the input, and encompassing LTI systems with zeros of multiplicity two or higher. Finally, we show that nth-order 0-invariant systems realize – under mild conditions – nth-order nonlinear differential operators: when excited by an input of a characteristic functional form, the system’s output converges to a constant value only depending on the nth (nonlinear) derivative of the input
Learning with Algebraic Invariances, and the Invariant Kernel Trick
When solving data analysis problems it is important to integrate prior
knowledge and/or structural invariances. This paper contributes by a novel
framework for incorporating algebraic invariance structure into kernels. In
particular, we show that algebraic properties such as sign symmetries in data,
phase independence, scaling etc. can be included easily by essentially
performing the kernel trick twice. We demonstrate the usefulness of our theory
in simulations on selected applications such as sign-invariant spectral
clustering and underdetermined ICA
Understanding Slow Feature Analysis: A Mathematical Framework
Slow feature analysis is an algorithm for unsupervised learning of invariant representations from data with temporal correlations. Here, we present a mathematical analysis of slow feature analysis for the case where the input-output functions are not restricted in complexity. We show that the optimal functions obey a partial differential eigenvalue problem of a type that is common in theoretical physics. This analogy allows the transfer of mathematical techniques and intuitions from physics to concrete applications of slow feature analysis, thereby providing the means for analytical predictions and a better understanding of simulation results. We put particular emphasis on the situation where the input data are generated from a set of statistically independent sources.\ud
The dependence of the optimal functions on the sources is calculated analytically for the cases where the sources have Gaussian or uniform distribution
Neural Networks as Paths through the Space of Representations
Deep neural networks implement a sequence of layer-by-layer operations that
are each relatively easy to understand, but the resulting overall computation
is generally difficult to understand. We consider a simple hypothesis for
interpreting the layer-by-layer construction of useful representations: perhaps
the role of each layer is to reformat information to reduce the "distance" to
the desired outputs. With this framework, the layer-wise computation
implemented by a deep neural network can be viewed as a path through a
high-dimensional representation space. We formalize this intuitive idea of a
"path" by leveraging recent advances in *metric* representational similarity.
We extend existing representational distance methods by computing geodesics,
angles, and projections of representations, going beyond mere layer distances.
We then demonstrate these tools by visualizing and comparing the paths taken by
ResNet and VGG architectures on CIFAR-10. We conclude by sketching additional
ways that this kind of representational geometry can be used to understand and
interpret network training, and to describe novel kinds of similarities between
different models.Comment: 10 pages, submitted to ICLR 202
Decoupling Multivariate Polynomials Using First-Order Information
We present a method to decompose a set of multivariate real polynomials into
linear combinations of univariate polynomials in linear forms of the input
variables. The method proceeds by collecting the first-order information of the
polynomials in a set of operating points, which is captured by the Jacobian
matrix evaluated at the operating points. The polyadic canonical decomposition
of the three-way tensor of Jacobian matrices directly returns the unknown
linear relations, as well as the necessary information to reconstruct the
univariate polynomials. The conditions under which this decoupling procedure
works are discussed, and the method is illustrated on several numerical
examples
- …