1,827 research outputs found
Semi-Supervised Learning under Class Distribution Mismatch
Semi-supervised learning (SSL) aims to avoid the need for collecting prohibitively expensive labelled training data. Whilst
demonstrating impressive performance boost, existing SSL
methods artificially assume that small labelled data and large
unlabelled data are drawn from the same class distribution. In
a more realistic scenario with class distribution mismatch between the two sets, they often suffer severe performance degradation due to error propagation introduced by irrelevant unlabelled samples. Our work addresses this under-studied and realistic SSL problem by a novel algorithm named UncertaintyAware Self-Distillation (UASD). Specifically, UASD produces
soft targets that avoid catastrophic error propagation, and empower learning effectively from unconstrained unlabelled data
with out-of-distribution (OOD) samples. This is based on joint
Self-Distillation and OOD filtering in a unified formulation.
Without bells and whistles, UASD significantly outperforms
six state-of-the-art methods in more realistic SSL under class
distribution mismatch on three popular image classification
datasets: CIFAR10, CIFAR100, and TinyImageNet
Anytime Inference with Distilled Hierarchical Neural Ensembles
Inference in deep neural networks can be computationally expensive, and
networks capable of anytime inference are important in mscenarios where the
amount of compute or quantity of input data varies over time. In such networks
the inference process can interrupted to provide a result faster, or continued
to obtain a more accurate result. We propose Hierarchical Neural Ensembles
(HNE), a novel framework to embed an ensemble of multiple networks in a
hierarchical tree structure, sharing intermediate layers. In HNE we control the
complexity of inference on-the-fly by evaluating more or less models in the
ensemble. Our second contribution is a novel hierarchical distillation method
to boost the prediction accuracy of small ensembles. This approach leverages
the nested structure of our ensembles, to optimally allocate accuracy and
diversity across the individual models. Our experiments show that, compared to
previous anytime inference models, HNE provides state-of-the-art
accuracy-computate trade-offs on the CIFAR-10/100 and ImageNet datasets
- …