208 research outputs found
A New Probabilistic Distance Metric With Application In Gaussian Mixture Reduction
This paper presents a new distance metric to compare two continuous
probability density functions. The main advantage of this metric is that,
unlike other statistical measurements, it can provide an analytic, closed-form
expression for a mixture of Gaussian distributions while satisfying all metric
properties. These characteristics enable fast, stable, and efficient
calculations, which are highly desirable in real-world signal processing
applications. The application in mind is Gaussian Mixture Reduction (GMR),
which is widely used in density estimation, recursive tracking, and belief
propagation. To address this problem, we developed a novel algorithm dubbed the
Optimization-based Greedy GMR (OGGMR), which employs our metric as a criterion
to approximate a high-order Gaussian mixture with a lower order. Experimental
results show that the OGGMR algorithm is significantly faster and more
efficient than state-of-the-art GMR algorithms while retaining the geometric
shape of the original mixture
DataDAM: Efficient Dataset Distillation with Attention Matching
Researchers have long tried to minimize training costs in deep learning while
maintaining strong generalization across diverse datasets. Emerging research on
dataset distillation aims to reduce training costs by creating a small
synthetic set that contains the information of a larger real dataset and
ultimately achieves test accuracy equivalent to a model trained on the whole
dataset. Unfortunately, the synthetic data generated by previous methods are
not guaranteed to distribute and discriminate as well as the original training
data, and they incur significant computational costs. Despite promising
results, there still exists a significant performance gap between models
trained on condensed synthetic sets and those trained on the whole dataset. In
this paper, we address these challenges using efficient Dataset Distillation
with Attention Matching (DataDAM), achieving state-of-the-art performance while
reducing training costs. Specifically, we learn synthetic images by matching
the spatial attention maps of real and synthetic data generated by different
layers within a family of randomly initialized neural networks. Our method
outperforms the prior methods on several datasets, including CIFAR10/100,
TinyImageNet, ImageNet-1K, and subsets of ImageNet-1K across most of the
settings, and achieves improvements of up to 6.5% and 4.1% on CIFAR100 and
ImageNet-1K, respectively. We also show that our high-quality distilled images
have practical benefits for downstream applications, such as continual learning
and neural architecture search.Comment: Accepted in International Conference in Computer Vision (ICCV) 202
- …