3,759 research outputs found
A Survey on Dataset Distillation: Approaches, Applications and Future Directions
Dataset distillation is attracting more attention in machine learning as
training sets continue to grow and the cost of training state-of-the-art models
becomes increasingly high. By synthesizing datasets with high information
density, dataset distillation offers a range of potential applications,
including support for continual learning, neural architecture search, and
privacy protection. Despite recent advances, we lack a holistic understanding
of the approaches and applications. Our survey aims to bridge this gap by first
proposing a taxonomy of dataset distillation, characterizing existing
approaches, and then systematically reviewing the data modalities, and related
applications. In addition, we summarize the challenges and discuss future
directions for this field of research
Dataset Distillation: A Comprehensive Review
Recent success of deep learning is largely attributed to the sheer amount of
data used for training deep neural networks.Despite the unprecedented success,
the massive data, unfortunately, significantly increases the burden on storage
and transmission and further gives rise to a cumbersome model training process.
Besides, relying on the raw data for training \emph{per se} yields concerns
about privacy and copyright. To alleviate these shortcomings, dataset
distillation~(DD), also known as dataset condensation (DC), was introduced and
has recently attracted much research attention in the community. Given an
original dataset, DD aims to derive a much smaller dataset containing synthetic
samples, based on which the trained models yield performance comparable with
those trained on the original dataset. In this paper, we give a comprehensive
review and summary of recent advances in DD and its application. We first
introduce the task formally and propose an overall algorithmic framework
followed by all existing DD methods. Next, we provide a systematic taxonomy of
current methodologies in this area, and discuss their theoretical
interconnections. We also present current challenges in DD through extensive
experiments and envision possible directions for future works.Comment: 23 pages, 168 references, 8 figures, under revie
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning
Current deep learning research is dominated by benchmark evaluation. A method
is regarded as favorable if it empirically performs well on the dedicated test
set. This mentality is seamlessly reflected in the resurfacing area of
continual learning, where consecutively arriving sets of benchmark data are
investigated. The core challenge is framed as protecting previously acquired
representations from being catastrophically forgotten due to the iterative
parameter updates. However, comparison of individual methods is nevertheless
treated in isolation from real world application and typically judged by
monitoring accumulated test set performance. The closed world assumption
remains predominant. It is assumed that during deployment a model is guaranteed
to encounter data that stems from the same distribution as used for training.
This poses a massive challenge as neural networks are well known to provide
overconfident false predictions on unknown instances and break down in the face
of corrupted data. In this work we argue that notable lessons from open set
recognition, the identification of statistically deviating data outside of the
observed dataset, and the adjacent field of active learning, where data is
incrementally queried such that the expected performance gain is maximized, are
frequently overlooked in the deep learning era. Based on these forgotten
lessons, we propose a consolidated view to bridge continual learning, active
learning and open set recognition in deep neural networks. Our results show
that this not only benefits each individual paradigm, but highlights the
natural synergies in a common framework. We empirically demonstrate
improvements when alleviating catastrophic forgetting, querying data in active
learning, selecting task orders, while exhibiting robust open world application
where previously proposed methods fail.Comment: 32 page
Few-Shot Continual Learning for Conditional Generative Adversarial Networks
In few-shot continual learning for generative models, a target mode must be
learned with limited samples without adversely affecting the previously learned
modes. In this paper, we propose a new continual learning approach for
conditional generative adversarial networks (cGAN) based on a new mode-affinity
measure for generative modeling. Our measure is entirely based on the cGAN's
discriminator and can identify the existing modes that are most similar to the
target. Subsequently, we expand the continual learning model by including the
target mode using a weighted label derived from those of the closest modes. To
prevent catastrophic forgetting, we first generate labeled data samples using
the cGAN's generator, and then train the cGAN model for the target mode while
memory replaying with the generated data. Our experimental results demonstrate
the efficacy of our approach in improving the generation performance over the
baselines and the state-of-the-art approaches for various standard datasets
while utilizing fewer training samples
- …