5 research outputs found
Class-Incremental Continual Learning into the eXtended DER-verse
The staple of human intelligence is the capability of acquiring knowledge in
a continuous fashion. In stark contrast, Deep Networks forget catastrophically
and, for this reason, the sub-field of Class-Incremental Continual Learning
fosters methods that learn a sequence of tasks incrementally, blending
sequentially-gained knowledge into a comprehensive prediction.
This work aims at assessing and overcoming the pitfalls of our previous
proposal Dark Experience Replay (DER), a simple and effective approach that
combines rehearsal and Knowledge Distillation. Inspired by the way our minds
constantly rewrite past recollections and set expectations for the future, we
endow our model with the abilities to i) revise its replay memory to welcome
novel information regarding past data ii) pave the way for learning yet unseen
classes.
We show that the application of these strategies leads to remarkable
improvements; indeed, the resulting method - termed eXtended-DER (X-DER) -
outperforms the state of the art on both standard benchmarks (such as CIFAR-100
and miniImagenet) and a novel one here introduced. To gain a better
understanding, we further provide extensive ablation studies that corroborate
and extend the findings of our previous research (e.g. the value of Knowledge
Distillation and flatter minima in continual learning setups).Comment: 23 pages, 22 figures. To appear in IEEE TPAM
Continual semi-supervised learning through contrastive interpolation consistency
Continual Learning (CL) investigates how to train Deep Networks on a stream of tasks without incurring forgetting. CL settings proposed in literature assume that every incoming example is paired with ground-truth annotations. However, this clashes with many real-world applications: gathering labeled data, which is in itself tedious and expensive, becomes infeasible when data flow as a stream. This work explores Continual Semi-Supervised Learning (CSSL): here, only a small fraction of labeled input examples are shown to the learner. We assess how current CL methods (e.g.: EWC, LwF, iCaRL, ER, GDumb, DER) perform in this novel and challenging scenario, where overfitting entangles forgetting. Subsequently, we design a novel CSSL method that exploits metric learning and consistency regularization to leverage unlabeled examples while learning. We show that our proposal exhibits higher resilience to diminishing supervision and, even more surprisingly, relying only on supervision suffices to outperform SOTA methods trained under full supervision
Rethinking Experience Replay: a Bag of Tricks for Continual Learning
In Continual Learning, a Neural Network is trained on a stream of data whose distribution shifts over time. Under these assumptions, it is especially challenging to improve on classes appearing later in the stream while remaining accurate on previous ones. This is due to the infamous problem of catastrophic forgetting, which causes a quick performance degradation when the classifier focuses on learning new categories. Recent literature proposed various approaches to tackle this issue, often resorting to very sophisticated techniques. In this work, we show that naïve rehearsal can be patched to achieve similar performance. We point out some shortcomings that restrain Experience Replay (ER) and propose five tricks to mitigate them. Experiments show that ER, thus enhanced, displays an accuracy gain of 51.2 and 26.9 percentage points on the CIFAR-10 and CIFAR-100 datasets respectively (memory buffer size 1000). As a result, it surpasses current state-of-the-art rehearsal-based methods
Dark Experience for General Continual Learning: a Strong, Simple Baseline
Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance