10 research outputs found
Learning to Purify Noisy Labels via Meta Soft Label Corrector
Recent deep neural networks (DNNs) can easily overfit to biased training data
with noisy labels. Label correction strategy is commonly used to alleviate this
issue by designing a method to identity suspected noisy labels and then correct
them. Current approaches to correcting corrupted labels usually need certain
pre-defined label correction rules or manually preset hyper-parameters. These
fixed settings make it hard to apply in practice since the accurate label
correction usually related with the concrete problem, training data and the
temporal information hidden in dynamic iterations of training process. To
address this issue, we propose a meta-learning model which could estimate soft
labels through meta-gradient descent step under the guidance of noise-free meta
data. By viewing the label correction procedure as a meta-process and using a
meta-learner to automatically correct labels, we could adaptively obtain
rectified soft labels iteratively according to current training problems
without manually preset hyper-parameters. Besides, our method is model-agnostic
and we can combine it with any other existing model with ease. Comprehensive
experiments substantiate the superiority of our method in both synthetic and
real-world problems with noisy labels compared with current SOTA label
correction strategies.Comment: 12 pages,6 figure
Towards Robust Learning with Different Label Noise Distributions
Noisy labels are an unavoidable consequence of labeling processes and
detecting them is an important step towards preventing performance degradations
in Convolutional Neural Networks. Discarding noisy labels avoids a harmful
memorization, while the associated image content can still be exploited in a
semi-supervised learning (SSL) setup. Clean samples are usually identified
using the small loss trick, i.e. they exhibit a low loss. However, we show that
different noise distributions make the application of this trick less
straightforward and propose to continuously relabel all images to reveal a
discriminative loss against multiple distributions. SSL is then applied twice,
once to improve the clean-noisy detection and again for training the final
model. We design an experimental setup based on ImageNet32/64 for better
understanding the consequences of representation learning with differing label
noise distributions and find that non-uniform out-of-distribution noise better
resembles real-world noise and that in most cases intermediate features are not
affected by label noise corruption. Experiments in CIFAR-10/100, ImageNet32/64
and WebVision (real-world noise) demonstrate that the proposed label noise
Distribution Robust Pseudo-Labeling (DRPL) approach gives substantial
improvements over recent state-of-the-art. Code is available at
https://git.io/JJ0PV
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
Image classification systems recently made a giant leap with the advancement
of deep neural networks. However, these systems require an excessive amount of
labeled data to be adequately trained. Gathering a correctly annotated dataset
is not always feasible due to several factors, such as the expensiveness of the
labeling process or difficulty of correctly classifying data, even for the
experts. Because of these practical challenges, label noise is a common problem
in real-world datasets, and numerous methods to train deep neural networks with
label noise are proposed in the literature. Although deep neural networks are
known to be relatively robust to label noise, their tendency to overfit data
makes them vulnerable to memorizing even random noise. Therefore, it is crucial
to consider the existence of label noise and develop counter algorithms to fade
away its adverse effects to train deep neural networks efficiently. Even though
an extensive survey of machine learning techniques under label noise exists,
the literature lacks a comprehensive survey of methodologies centered
explicitly around deep learning in the presence of noisy labels. This paper
aims to present these algorithms while categorizing them into one of the two
subgroups: noise model based and noise model free methods. Algorithms in the
first group aim to estimate the noise structure and use this information to
avoid the adverse effects of noisy labels. Differently, methods in the second
group try to come up with inherently noise robust algorithms by using
approaches like robust losses, regularizers or other learning paradigms
Safeguarded Dynamic Label Regression for Noisy Supervision
Learning with noisy labels is imperative in the Big Data era since it reduces expensive labor on accurate annotations. Previous method, learning with noise transition, has enjoyed theoretical guarantees when it is applied to the scenario with the class-conditional noise. However, this approach critically depends on an accurate pre-estimated noise transition, which is usually impractical. Subsequent improvement adapts the preestimation in the form of a Softmax layer along with the training progress. However, the parameters in the Softmax layer are highly tweaked for the fragile performance and easily get stuck into undesired local minimums. To overcome this issue, we propose a Latent Class-Conditional Noise model (LCCN) that models the noise transition in a Bayesian form. By projecting the noise transition into a Dirichlet-distributed space, the learning is constrained on a simplex instead of some adhoc parametric space. Furthermore, we specially deduce a dynamic label regression method for LCCN to iteratively infer the latent true labels and jointly train the classifier and model the noise. Our approach theoretically safeguards the bounded update of the noise transition, which avoids arbitrarily tuning via a batch of samples. Extensive experiments have been conducted on controllable noise data with CIFAR10 and CIFAR-100 datasets, and the agnostic noise data with Clothing1M and WebVision17 datasets. Experimental results have demonstrated that the proposed model outperforms several state-of-the-art methods
Safeguarded Dynamic Label Regression for Noisy Supervision
Learning with noisy labels is imperative in the Big Data era since it reduces expensive labor on accurate annotations. Previous method, learning with noise transition, has enjoyed theoretical guarantees when it is applied to the scenario with the class-conditional noise. However, this approach critically depends on an accurate pre-estimated noise transition, which is usually impractical. Subsequent improvement adapts the preestimation in the form of a Softmax layer along with the training progress. However, the parameters in the Softmax layer are highly tweaked for the fragile performance and easily get stuck into undesired local minimums. To overcome this issue, we propose a Latent Class-Conditional Noise model (LCCN) that models the noise transition in a Bayesian form. By projecting the noise transition into a Dirichlet-distributed space, the learning is constrained on a simplex instead of some adhoc parametric space. Furthermore, we specially deduce a dynamic label regression method for LCCN to iteratively infer the latent true labels and jointly train the classifier and model the noise. Our approach theoretically safeguards the bounded update of the noise transition, which avoids arbitrarily tuning via a batch of samples. Extensive experiments have been conducted on controllable noise data with CIFAR10 and CIFAR-100 datasets, and the agnostic noise data with Clothing1M and WebVision17 datasets. Experimental results have demonstrated that the proposed model outperforms several state-of-the-art methods