433 research outputs found
Robust Training under Label Noise by Over-parameterization
Recently, over-parameterized deep networks, with increasingly more network
parameters than training samples, have dominated the performances of modern
machine learning. However, when the training data is corrupted, it has been
well-known that over-parameterized networks tend to overfit and do not
generalize. In this work, we propose a principled approach for robust training
of over-parameterized deep networks in classification tasks where a proportion
of training labels are corrupted. The main idea is yet very simple: label noise
is sparse and incoherent with the network learned from clean data, so we model
the noise and learn to separate it from the data. Specifically, we model the
label noise via another sparse over-parameterization term, and exploit implicit
algorithmic regularizations to recover and separate the underlying corruptions.
Remarkably, when trained using such a simple method in practice, we demonstrate
state-of-the-art test accuracy against label noise on a variety of real
datasets. Furthermore, our experimental results are corroborated by theory on
simplified linear models, showing that exact separation between sparse noise
and low-rank data can be achieved under incoherent conditions. The work opens
many interesting directions for improving over-parameterized models by using
sparse over-parameterization and implicit regularization.Comment: 25 pages, 4 figures and 6 tables. Code is available at
https://github.com/shengliu66/SO
PipeNet: Selective Modal Pipeline of Fusion Network for Multi-Modal Face Anti-Spoofing
Face anti-spoofing has become an increasingly important and critical security
feature for authentication systems, due to rampant and easily launchable
presentation attacks. Addressing the shortage of multi-modal face dataset,
CASIA recently released the largest up-to-date CASIA-SURF Cross-ethnicity Face
Anti-spoofing(CeFA) dataset, covering 3 ethnicities, 3 modalities, 1607
subjects, and 2D plus 3D attack types in four protocols, and focusing on the
challenge of improving the generalization capability of face anti-spoofing in
cross-ethnicity and multi-modal continuous data. In this paper, we propose a
novel pipeline-based multi-stream CNN architecture called PipeNet for
multi-modal face anti-spoofing. Unlike previous works, Selective Modal Pipeline
(SMP) is designed to enable a customized pipeline for each data modality to
take full advantage of multi-modal data. Limited Frame Vote (LFV) is designed
to ensure stable and accurate prediction for video classification. The proposed
method wins the third place in the final ranking of Chalearn Multi-modal
Cross-ethnicity Face Anti-spoofing Recognition Challenge@CVPR2020. Our final
submission achieves the Average Classification Error Rate (ACER) of 2.21 with
Standard Deviation of 1.26 on the test set.Comment: Accepted to appear in CVPR2020 WM
Generalized Neural Collapse for a Large Number of Classes
Neural collapse provides an elegant mathematical characterization of learned
last layer representations (a.k.a. features) and classifier weights in deep
classification models. Such results not only provide insights but also motivate
new techniques for improving practical deep models. However, most of the
existing empirical and theoretical studies in neural collapse focus on the case
that the number of classes is small relative to the dimension of the feature
space. This paper extends neural collapse to cases where the number of classes
are much larger than the dimension of feature space, which broadly occur for
language models, retrieval systems, and face recognition applications. We show
that the features and classifier exhibit a generalized neural collapse
phenomenon, where the minimum one-vs-rest margins is maximized.We provide
empirical study to verify the occurrence of generalized neural collapse in
practical deep neural networks. Moreover, we provide theoretical study to show
that the generalized neural collapse provably occurs under unconstrained
feature model with spherical constraint, under certain technical conditions on
feature dimension and number of classes.Comment: 32 pages, 12 figure
- …