131 research outputs found
Bayesian Flow Networks in Continual Learning
Bayesian Flow Networks (BFNs) has been recently proposed as one of the most
promising direction to universal generative modelling, having ability to learn
any of the data type. Their power comes from the expressiveness of neural
networks and Bayesian inference which make them suitable in the context of
continual learning. We delve into the mechanics behind BFNs and conduct the
experiments to empirically verify the generative capabilities on non-stationary
data.Comment: Submitted to NeurIPS 2023 Workshop on Diffusion Model
Omnidirectional Transfer for Quasilinear Lifelong Learning
In biological learning, data are used to improve performance not only on the
current task, but also on previously encountered and as yet unencountered
tasks. In contrast, classical machine learning starts from a blank slate, or
tabula rasa, using data only for the single task at hand. While typical
transfer learning algorithms can improve performance on future tasks, their
performance on prior tasks degrades upon learning new tasks (called
catastrophic forgetting). Many recent approaches for continual or lifelong
learning have attempted to maintain performance given new tasks. But striving
to avoid forgetting sets the goal unnecessarily low: the goal of lifelong
learning, whether biological or artificial, should be to improve performance on
all tasks (including past and future) with any new data. We propose
omnidirectional transfer learning algorithms, which includes two special cases
of interest: decision forests and deep networks. Our key insight is the
development of the omni-voter layer, which ensembles representations learned
independently on all tasks to jointly decide how to proceed on any given new
data point, thereby improving performance on both past and future tasks. Our
algorithms demonstrate omnidirectional transfer in a variety of simulated and
real data scenarios, including tabular data, image data, spoken data, and
adversarial tasks. Moreover, they do so with quasilinear space and time
complexity
Segmentation of Multiple Sclerosis Lesions across Hospitals: Learn Continually or Train from Scratch?
Segmentation of Multiple Sclerosis (MS) lesions is a challenging problem.
Several deep-learning-based methods have been proposed in recent years.
However, most methods tend to be static, that is, a single model trained on a
large, specialized dataset, which does not generalize well. Instead, the model
should learn across datasets arriving sequentially from different hospitals by
building upon the characteristics of lesions in a continual manner. In this
regard, we explore experience replay, a well-known continual learning method,
in the context of MS lesion segmentation across multi-contrast data from 8
different hospitals. Our experiments show that replay is able to achieve
positive backward transfer and reduce catastrophic forgetting compared to
sequential fine-tuning. Furthermore, replay outperforms the multi-domain
training, thereby emerging as a promising solution for the segmentation of MS
lesions. The code is available at this link:
https://github.com/naga-karthik/continual-learning-msComment: Accepted at the Medical Imaging Meets NeurIPS (MedNeurIPS) Workshop
202
LifeLonger: A Benchmark for Continual Disease Classification
Deep learning models have shown a great effectiveness in recognition of
findings in medical images. However, they cannot handle the ever-changing
clinical environment, bringing newly annotated medical data from different
sources. To exploit the incoming streams of data, these models would benefit
largely from sequentially learning from new samples, without forgetting the
previously obtained knowledge. In this paper we introduce LifeLonger, a
benchmark for continual disease classification on the MedMNIST collection, by
applying existing state-of-the-art continual learning methods. In particular,
we consider three continual learning scenarios, namely, task and class
incremental learning and the newly defined cross-domain incremental learning.
Task and class incremental learning of diseases address the issue of
classifying new samples without re-training the models from scratch, while
cross-domain incremental learning addresses the issue of dealing with datasets
originating from different institutions while retaining the previously obtained
knowledge. We perform a thorough analysis of the performance and examine how
the well-known challenges of continual learning, such as the catastrophic
forgetting exhibit themselves in this setting. The encouraging results
demonstrate that continual learning has a major potential to advance disease
classification and to produce a more robust and efficient learning framework
for clinical settings. The code repository, data partitions and baseline
results for the complete benchmark will be made publicly available
The Importance of Robust Features in Mitigating Catastrophic Forgetting
Continual learning (CL) is an approach to address catastrophic forgetting,
which refers to forgetting previously learned knowledge by neural networks when
trained on new tasks or data distributions. The adversarial robustness has
decomposed features into robust and non-robust types and demonstrated that
models trained on robust features significantly enhance adversarial robustness.
However, no study has been conducted on the efficacy of robust features from
the lens of the CL model in mitigating catastrophic forgetting in CL. In this
paper, we introduce the CL robust dataset and train four baseline models on
both the standard and CL robust datasets. Our results demonstrate that the CL
models trained on the CL robust dataset experienced less catastrophic
forgetting of the previously learned tasks than when trained on the standard
dataset. Our observations highlight the significance of the features provided
to the underlying CL models, showing that CL robust features can alleviate
catastrophic forgetting
- …