582 research outputs found
Input anticipating critical reservoirs show power law forgetting of unexpected input events
Usually, reservoir computing shows an exponential memory decay. This paper
investigates under which circumstances echo state networks can show a power law
forgetting. That means traces of earlier events can be found in the reservoir
for very long time spans. Such a setting requires critical connectivity exactly
at the limit of what is permissible according the echo state condition.
However, for general matrices the limit cannot be determined exactly from
theory. In addition, the behavior of the network is strongly influenced by the
input flow. Results are presented that use certain types of restricted
recurrent connectivity and anticipation learning with regard to the input,
where indeed power law forgetting can be achieved
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
We introduce a novel approach to perform first-order optimization with
orthogonal and unitary constraints. This approach is based on a parametrization
stemming from Lie group theory through the exponential map. The parametrization
transforms the constrained optimization problem into an unconstrained one over
a Euclidean space, for which common first-order optimization methods can be
used. The theoretical results presented are general enough to cover the special
orthogonal group, the unitary group and, in general, any connected compact Lie
group. We discuss how this and other parametrizations can be computed
efficiently through an implementation trick, making numerically complex
parametrizations usable at a negligible runtime cost in neural networks. In
particular, we apply our results to RNNs with orthogonal recurrent weights,
yielding a new architecture called expRNN. We demonstrate how our method
constitutes a more robust approach to optimization with orthogonal constraints,
showing faster, accurate, and more stable convergence in several tasks designed
to test RNNs
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
A recent strategy to circumvent the exploding and vanishing gradient problem
in RNNs, and to allow the stable propagation of signals over long time scales,
is to constrain recurrent connectivity matrices to be orthogonal or unitary.
This ensures eigenvalues with unit norm and thus stable dynamics and training.
However this comes at the cost of reduced expressivity due to the limited
variety of orthogonal transformations. We propose a novel connectivity
structure based on the Schur decomposition and a splitting of the Schur form
into normal and non-normal parts. This allows to parametrize matrices with
unit-norm eigenspectra without orthogonality constraints on eigenbases. The
resulting architecture ensures access to a larger space of spectrally
constrained matrices, of which orthogonal matrices are a subset. This crucial
difference retains the stability advantages and training speed of orthogonal
RNNs while enhancing expressivity, especially on tasks that require
computations over ongoing input sequences
Zhang Neural Networks for Fast and Accurate Computations of the Field of Values
In this paper a new and different neural network, called Zhang Neural Network
(ZNN) is appropriated from discrete time-varying matrix problems and applied to
the angle parameter-varying matrix field of values (FoV) problem. This problem
acts as a test bed for newly discovered convergent 1-step ahead finite
difference formulas of high truncation orders. The ZNN method that uses a
look-ahead finite difference scheme of error order 6 gives us 15+ accurate
digits of the FoV boundary in record time when applied to hermitean matrix
flows
Center Manifold Dynamics in Randomly Coupled Oscillators and in Cochlea
In dynamical systems theory, a fixed point of the activity is called nonhyperbolic if the linearization of the system around the fixed point has at least one eigenvalue with zero real part. The center manifold existence theorem guarantees the local existence of an invariant subspace of the activity, known as a center manifold, around nonhyperbolic fixed points. A growing number of theoretical and experimental studies suggest that neural systems utilize dynamics on center manifolds to display complex, nonlinear behavior and to flexibly adapt to wide-ranging sensory input parameters. In this thesis, I will present two lines of research exploring nonhyperbolicity in neural dynamics
Learning Transformation Synchronization
Reconstructing the 3D model of a physical object typically requires us to
align the depth scans obtained from different camera poses into the same
coordinate system. Solutions to this global alignment problem usually proceed
in two steps. The first step estimates relative transformations between pairs
of scans using an off-the-shelf technique. Due to limited information presented
between pairs of scans, the resulting relative transformations are generally
noisy. The second step then jointly optimizes the relative transformations
among all input depth scans. A natural constraint used in this step is the
cycle-consistency constraint, which allows us to prune incorrect relative
transformations by detecting inconsistent cycles. The performance of such
approaches, however, heavily relies on the quality of the input relative
transformations. Instead of merely using the relative transformations as the
input to perform transformation synchronization, we propose to use a neural
network to learn the weights associated with each relative transformation. Our
approach alternates between transformation synchronization using weighted
relative transformations and predicting new weights of the input relative
transformations using a neural network. We demonstrate the usefulness of this
approach across a wide range of datasets
Lipschitz Recurrent Neural Networks
Viewing recurrent neural networks (RNNs) as continuous-time dynamical
systems, we propose a recurrent unit that describes the hidden state's
evolution with two parts: a well-understood linear component plus a Lipschitz
nonlinearity. This particular functional form facilitates stability analysis of
the long-term behavior of the recurrent unit using tools from nonlinear systems
theory. In turn, this enables architectural design decisions before
experimentation. Sufficient conditions for global stability of the recurrent
unit are obtained, motivating a novel scheme for constructing hidden-to-hidden
matrices. Our experiments demonstrate that the Lipschitz RNN can outperform
existing recurrent units on a range of benchmark tasks, including computer
vision, language modeling and speech prediction tasks. Finally, through
Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more
robust with respect to input and parameter perturbations as compared to other
continuous-time RNNs
Magnetic Eigenmaps for the Visualization of Directed Networks
We propose a framework for the visualization of directed networks relying on
the eigenfunctions of the magnetic Laplacian, called here Magnetic Eigenmaps.
The magnetic Laplacian is a complex deformation of the well-known combinatorial
Laplacian. Features such as density of links and directionality patterns are
revealed by plotting the phases of the first magnetic eigenvectors. An
interpretation of the magnetic eigenvectors is given in connection with the
angular synchronization problem. Illustrations of our method are given for both
artificial and real networks
Applications of Structural Balance in Signed Social Networks
We present measures, models and link prediction algorithms based on the
structural balance in signed social networks. Certain social networks contain,
in addition to the usual 'friend' links, 'enemy' links. These networks are
called signed social networks. A classical and major concept for signed social
networks is that of structural balance, i.e., the tendency of triangles to be
'balanced' towards including an even number of negative edges, such as
friend-friend-friend and friend-enemy-enemy triangles. In this article, we
introduce several new signed network analysis methods that exploit structural
balance for measuring partial balance, for finding communities of people based
on balance, for drawing signed social networks, and for solving the problem of
link prediction. Notably, the introduced methods are based on the signed graph
Laplacian and on the concept of signed resistance distances. We evaluate our
methods on a collection of four signed social network datasets.Comment: 37 page
LieDetect: Detection of representation orbits of compact Lie groups from point clouds
We suggest a new algorithm to estimate representations of compact Lie groups
from finite samples of their orbits. Different from other reported techniques,
our method allows the retrieval of the precise representation type as a direct
sum of irreducible representations. Moreover, the knowledge of the
representation type permits the reconstruction of its orbit, which is useful to
identify the Lie group that generates the action. Our algorithm is general for
any compact Lie group, but only instantiations for SO(2), T^d, SU(2) and SO(3)
are considered. Theoretical guarantees of robustness in terms of Hausdorff and
Wasserstein distances are derived. Our tools are drawn from geometric measure
theory, computational geometry, and optimization on matrix manifolds. The
algorithm is tested for synthetic data up to dimension 16, as well as real-life
applications in image analysis, harmonic analysis, and classical mechanics
systems, achieving very accurate results.Comment: 84 pages, 16 figure
- …