1,803 research outputs found
On the Implicit Bias in Deep-Learning Algorithms
Gradient-based deep-learning algorithms exhibit remarkable performance in
practice, but it is not well-understood why they are able to generalize despite
having more parameters than training examples. It is believed that implicit
bias is a key factor in their ability to generalize, and hence it was widely
studied in recent years. In this short survey, we explain the notion of
implicit bias, review main results and discuss their implications.Comment: Some minor edit
Generator Born from Classifier
In this paper, we make a bold attempt toward an ambitious task: given a
pre-trained classifier, we aim to reconstruct an image generator, without
relying on any data samples. From a black-box perspective, this challenge seems
intractable, since it inevitably involves identifying the inverse function for
a classifier, which is, by nature, an information extraction process. As such,
we resort to leveraging the knowledge encapsulated within the parameters of the
neural network. Grounded on the theory of Maximum-Margin Bias of gradient
descent, we propose a novel learning paradigm, in which the generator is
trained to ensure that the convergence conditions of the network parameters are
satisfied over the generated distribution of the samples. Empirical validation
from various image generation tasks substantiates the efficacy of our strategy
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activation and confirm the statistical benefits of this implicit bias
- …