7 research outputs found
A contrastive rule for meta-learning
Meta-learning algorithms leverage regularities that are present on a set of tasks to speed up and improve the performance of a subsidiary learning process. Recent work on deep neural networks has shown that prior gradient-based learning of meta-parameters can greatly improve the efficiency of subsequent learning. Here, we present a biologically plausible meta-learning algorithm based on equilibrium propagation. Instead of explicitly differentiating the learning process, our contrastive meta-learning rule estimates meta-parameter gradients by executing the subsidiary process more than once. This avoids reversing the learning dynamics in time and computing second-order derivatives. In spite of this, and unlike previous first-order methods, our rule recovers an arbitrarily accurate meta-parameter update given enough compute. We establish theoretical bounds on its performance and present experiments on a set of standard benchmarks and neural network architectures
Gated recurrent neural networks discover attention
Recent architectural developments have enabled recurrent neural networks
(RNNs) to reach and even surpass the performance of Transformers on certain
sequence modeling tasks. These modern RNNs feature a prominent design pattern:
linear recurrent layers interconnected by feedforward paths with multiplicative
gating. Here, we show how RNNs equipped with these two design elements can
exactly implement (linear) self-attention, the main building block of
Transformers. By reverse-engineering a set of trained RNNs, we find that
gradient descent in practice discovers our construction. In particular, we
examine RNNs trained to solve simple in-context learning tasks on which
Transformers are known to excel and find that gradient descent instills in our
RNNs the same attention-based in-context learning algorithm used by
Transformers. Our findings highlight the importance of multiplicative
interactions in neural networks and suggest that certain RNNs might be
unexpectedly implementing attention under the hood
Random initialisations performing above chance and how to find them
Neural networks trained with stochastic gradient descent (SGD) starting from
different random initialisations typically find functionally very similar
solutions, raising the question of whether there are meaningful differences
between different SGD solutions. Entezari et al.\ recently conjectured that
despite different initialisations, the solutions found by SGD lie in the same
loss valley after taking into account the permutation invariance of neural
networks. Concretely, they hypothesise that any two solutions found by SGD can
be permuted such that the linear interpolation between their parameters forms a
path without significant increases in loss. Here, we use a simple but powerful
algorithm to find such permutations that allows us to obtain direct empirical
evidence that the hypothesis is true in fully connected networks. Strikingly,
we find that two networks already live in the same loss valley at the time of
initialisation and averaging their random, but suitably permuted initialisation
performs significantly above chance. In contrast, for convolutional
architectures, our evidence suggests that the hypothesis does not hold.
Especially in a large learning rate regime, SGD seems to discover diverse
modes.Comment: NeurIPS 2022, 14th Annual Workshop on Optimization for Machine
Learning (OPT2022
Beyond Backpropagation: Bilevel Optimization Through Implicit Differentiation and Equilibrium Propagation
This review examines gradient-based techniques to solve bilevel optimization problems. Bilevel optimization extends the loss minimization framework underlying statistical learning to systems that are implicitly defined through a quantity they minimize. This characterization can be applied to neural networks, optimizers, algorithmic solvers, and even physical systems and allows for greater modeling flexibility compared to the usual explicit definition of such systems. We focus on solving learning problems of this kind through gradient descent, leveraging the toolbox of implicit differentiation and, for the first time applied to this setting, the equilibrium propagation theorem. We present the mathematical foundations behind such methods, introduce the gradient estimation algorithms in detail, and compare the competitive advantages of the different approaches
Beyond backpropagation: implicit gradients for bilevel optimization
This paper reviews gradient-based techniques to solve bilevel optimization
problems. Bilevel optimization is a general way to frame the learning of
systems that are implicitly defined through a quantity that they minimize. This
characterization can be applied to neural networks, optimizers, algorithmic
solvers and even physical systems, and allows for greater modeling flexibility
compared to an explicit definition of such systems. Here we focus on
gradient-based approaches that solve such problems. We distinguish them in two
categories: those rooted in implicit differentiation, and those that leverage
the equilibrium propagation theorem. We present the mathematical foundations
that are behind such methods, introduce the gradient-estimation algorithms in
detail and compare the competitive advantages of the different approaches
The least-control principle for local learning at equilibrium
Equilibrium systems are a powerful way to express neural computations. As special cases, they include models of great current interest in both neuroscience and machine learning, such as deep neural networks, equilibrium recurrent neural networks, deep equilibrium models, or meta-learning. Here, we present a new principle for learning such systems with a temporally- and spatially-local rule. Our principle casts learning as a \emph{least-control} problem, where we first introduce an optimal controller to lead the system towards a solution state, and then define learning as reducing the amount of control needed to reach such a state. We show that incorporating learning signals within a dynamics as an optimal control enables transmitting activity-dependent credit assignment information, avoids storing intermediate states in memory, and does not rely on infinitesimal learning signals. In practice, our principle leads to strong performance matching that of leading gradient-based learning methods when applied to an array of problems involving recurrent neural networks and meta-learning. Our results shed light on how the brain might learn and offer new ways of approaching a broad class of machine learning problems
Learning where to learn: Gradient sparsity in meta and continual learning
Finding neural network weights that generalize well from small datasets is difficult. A promising approach is to learn a weight initialization such that a small number of weight changes results in low generalization error. We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn. We find that patterned sparsity emerges from this process, with the pattern of sparsity varying on a problem-by-problem basis. This selective sparsity results in better generalization and less interference in a range of few-shot and continual learning problems. Moreover, we find that sparse learning also emerges in a more expressive model where learning rates are meta-learned. Our results shed light on an ongoing debate on whether meta-learning can discover adaptable features and suggest that learning by sparse gradient descent is a powerful inductive bias for meta-learning systems