60 research outputs found
Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
Lack of performance when it comes to continual learning over non-stationary
distributions of data remains a major challenge in scaling neural network
learning to more human realistic settings. In this work we propose a new
conceptualization of the continual learning problem in terms of a temporally
symmetric trade-off between transfer and interference that can be optimized by
enforcing gradient alignment across examples. We then propose a new algorithm,
Meta-Experience Replay (MER), that directly exploits this view by combining
experience replay with optimization based meta-learning. This method learns
parameters that make interference based on future gradients less likely and
transfer based on future gradients more likely. We conduct experiments across
continual lifelong supervised learning benchmarks and non-stationary
reinforcement learning environments demonstrating that our approach
consistently outperforms recently proposed baselines for continual learning.
Our experiments show that the gap between the performance of MER and baseline
algorithms grows both as the environment gets more non-stationary and as the
fraction of the total experiences stored gets smaller.Comment: ICLR 201
Is Fast Adaptation All You Need?
Gradient-based meta-learning has proven to be highly effective at learning
model initializations, representations, and update rules that allow fast
adaptation from a few samples. The core idea behind these approaches is to use
fast adaptation and generalization -- two second-order metrics -- as training
signals on a meta-training dataset. However, little attention has been given to
other possible second-order metrics. In this paper, we investigate a different
training signal -- robustness to catastrophic interference -- and demonstrate
that representations learned by directing minimizing interference are more
conducive to incremental learning than those learned by just maximizing fast
adaptation.Comment: Meta Learning Workshop, NeurIPS 2019, 2 figures, MRCL, MAM
Batch-level Experience Replay with Review for Continual Learning
Continual learning is a branch of deep learning that seeks to strike a
balance between learning stability and plasticity. The CVPR 2020 CLVision
Continual Learning for Computer Vision challenge is dedicated to evaluating and
advancing the current state-of-the-art continual learning methods using the
CORe50 dataset with three different continual learning scenarios. This paper
presents our approach, called Batch-level Experience Replay with Review, to
this challenge. Our team achieved the 1'st place in all three scenarios out of
79 participated teams. The codebase of our implementation is publicly available
at https://github.com/RaptorMai/CVPR20_CLVision_challeng
Meta-Learning Representations for Continual Learning
A continual learning agent should be able to build on top of existing
knowledge to learn on new data quickly while minimizing forgetting. Current
intelligent systems based on neural network function approximators arguably do
the opposite---they are highly prone to forgetting and rarely trained to
facilitate future learning. One reason for this poor behavior is that they
learn from a representation that is not explicitly trained for these two goals.
In this paper, we propose OML, an objective that directly minimizes
catastrophic interference by learning representations that accelerate future
learning and are robust to forgetting under online updates in continual
learning. We show that it is possible to learn naturally sparse representations
that are more effective for online updating. Moreover, our algorithm is
complementary to existing continual learning strategies, such as MER and GEM.
Finally, we demonstrate that a basic online updating strategy on
representations learned by OML is competitive with rehearsal based methods for
continual learning. We release an implementation of our method at
https://github.com/khurramjaved96/mrcl .Comment: Accepted at NeurIPS19, 15 pages, 10 figures, open-source,
representation learning, continual learning, online learnin
Routing Networks with Co-training for Continual Learning
The core challenge with continual learning is catastrophic forgetting, the
phenomenon that when neural networks are trained on a sequence of tasks they
rapidly forget previously learned tasks. It has been observed that catastrophic
forgetting is most severe when tasks are dissimilar to each other. We propose
the use of sparse routing networks for continual learning. For each input,
these network architectures activate a different path through a network of
experts. Routing networks have been shown to learn to route similar tasks to
overlapping sets of experts and dissimilar tasks to disjoint sets of experts.
In the continual learning context this behaviour is desirable as it minimizes
interference between dissimilar tasks while allowing positive transfer between
related tasks. In practice, we find it is necessary to develop a new training
method for routing networks, which we call co-training which avoids poorly
initialized experts when new tasks are presented. When combined with a small
episodic memory replay buffer, sparse routing networks with co-training
outperform densely connected networks on the MNIST-Permutations and
MNIST-Rotations benchmarks.Comment: Presented at ICML Workshop on Continual Learning 202
Meta-learnt priors slow down catastrophic forgetting in neural networks
Current training regimes for deep learning usually involve exposure to a
single task / dataset at a time. Here we start from the observation that in
this context the trained model is not given any knowledge of anything outside
its (single-task) training distribution, and has thus no way to learn
parameters (i.e., feature detectors or policies) that could be helpful to solve
other tasks, and to limit future interference with the acquired knowledge, and
thus catastrophic forgetting. Here we show that catastrophic forgetting can be
mitigated in a meta-learning context, by exposing a neural network to multiple
tasks in a sequential manner during training. Finally, we present SeqFOMAML, a
meta-learning algorithm that implements these principles, and we evaluate it on
sequential learning problems composed by Omniglot and MiniImageNet
classification tasks
On Tiny Episodic Memories in Continual Learning
In continual learning (CL), an agent learns from a stream of tasks leveraging
prior experience to transfer knowledge to future tasks. It is an ideal
framework to decrease the amount of supervision in the existing learning
algorithms. But for a successful knowledge transfer, the learner needs to
remember how to perform previous tasks. One way to endow the learner the
ability to perform tasks seen in the past is to store a small memory, dubbed
episodic memory, that stores few examples from previous tasks and then to
replay these examples when training for future tasks. In this work, we
empirically analyze the effectiveness of a very small episodic memory in a CL
setup where each training example is only seen once. Surprisingly, across four
rather different supervised learning benchmarks adapted to CL, a very simple
baseline, that jointly trains on both examples from the current task as well as
examples stored in the episodic memory, significantly outperforms specifically
designed CL approaches with and without episodic memory. Interestingly, we find
that repetitive training on even tiny memories of past tasks does not harm
generalization, on the contrary, it improves it, with gains between 7\% and
17\% when the memory is populated with a single example per class.Comment: Making the main point of the paper more clea
Overcoming Catastrophic Interference in Online Reinforcement Learning with Dynamic Self-Organizing Maps
Using neural networks in the reinforcement learning (RL) framework has
achieved notable successes. Yet, neural networks tend to forget what they
learned in the past, especially when they learn online and fully incrementally,
a setting in which the weights are updated after each sample is received and
the sample is then discarded. Under this setting, an update can lead to overly
global generalization by changing too many weights. The global generalization
interferes with what was previously learned and deteriorates performance, a
phenomenon known as catastrophic interference. Many previous works use
mechanisms such as experience replay (ER) buffers to mitigate interference by
performing minibatch updates, ensuring the data distribution is approximately
independent-and-identically-distributed (i.i.d.). But using ER would become
infeasible in terms of memory as problem complexity increases. Thus, it is
crucial to look for more memory-efficient alternatives. Interference can be
averted if we replace global updates with more local ones, so only weights
responsible for the observed data sample are updated. In this work, we propose
the use of dynamic self-organizing map (DSOM) with neural networks to induce
such locality in the updates without ER buffers. Our method learns a DSOM to
produce a mask to reweigh each hidden unit's output, modulating its degree of
use. It prevents interference by replacing global updates with local ones,
conditioned on the agent's state. We validate our method on standard RL
benchmarks including Mountain Car and Lunar Lander, where existing methods
often fail to learn without ER. Empirically, we show that our online and fully
incremental method is on par with and in some cases, better than
state-of-the-art in terms of final performance and learning speed. We provide
visualizations and quantitative measures to show that our method indeed
mitigates interference.Comment: 9 Pages, 7 Figures, NeurIPS Workshop on Biological and Artificial
Reinforcement Learning, 201
Learning to Continually Learn Rapidly from Few and Noisy Data
Neural networks suffer from catastrophic forgetting and are unable to
sequentially learn new tasks without guaranteed stationarity in data
distribution. Continual learning could be achieved via replay -- by
concurrently training externally stored old data while learning a new task.
However, replay becomes less effective when each past task is allocated with
less memory. To overcome this difficulty, we supplemented replay mechanics with
meta-learning for rapid knowledge acquisition. By employing a meta-learner,
which \textit{learns a learning rate per parameter per past task}, we found
that base learners produced strong results when less memory was available.
Additionally, our approach inherited several meta-learning advantages for
continual learning: it demonstrated strong robustness to continually learn
under the presence of noises and yielded base learners to higher accuracy in
less updates.Comment: Accepted to the Meta-Learning and Co-Hosted Competition of AAAI 2021.
See https://aaai.org/Conferences/AAAI-21/ws21workshops/ and see
https://sites.google.com/chalearn.org/metalearning?pli=1#h.kt23ep5wleh
BI-MAML: Balanced Incremental Approach for Meta Learning
We present a novel Balanced Incremental Model Agnostic Meta Learning system
(BI-MAML) for learning multiple tasks. Our method implements a meta-update rule
to incrementally adapt its model to new tasks without forgetting old tasks.
Such a capability is not possible in current state-of-the-art MAML approaches.
These methods effectively adapt to new tasks, however, suffer from
'catastrophic forgetting' phenomena, in which new tasks that are streamed into
the model degrade the performance of the model on previously learned tasks. Our
system performs the meta-updates with only a few-shots and can successfully
accomplish them. Our key idea for achieving this is the design of balanced
learning strategy for the baseline model. The strategy sets the baseline model
to perform equally well on various tasks and incorporates time efficiency. The
balanced learning strategy enables BI-MAML to both outperform other
state-of-the-art models in terms of classification accuracy for existing tasks
and also accomplish efficient adaption to similar new tasks with less required
shots. We evaluate BI-MAML by conducting comparisons on two common benchmark
datasets with multiple number of image classification tasks. BI-MAML
performance demonstrates advantages in both accuracy and efficiency.Comment: Please see associated video at: https://youtu.be/4qlb-iG5SF
- …