31 research outputs found
Benign Oscillation of Stochastic Gradient Descent with Large Learning Rates
In this work, we theoretically investigate the generalization properties of
neural networks (NN) trained by stochastic gradient descent (SGD) algorithm
with large learning rates. Under such a training regime, our finding is that,
the oscillation of the NN weights caused by the large learning rate SGD
training turns out to be beneficial to the generalization of the NN, which
potentially improves over the same NN trained by SGD with small learning rates
that converges more smoothly. In view of this finding, we call such a
phenomenon "benign oscillation". Our theory towards demystifying such a
phenomenon builds upon the feature learning perspective of deep learning.
Specifically, we consider a feature-noise data generation model that consists
of (i) weak features which have a small -norm and appear in each data
point; (ii) strong features which have a larger -norm but only appear
in a certain fraction of all data points; and (iii) noise. We prove that NNs
trained by oscillating SGD with a large learning rate can effectively learn the
weak features in the presence of those strong features. In contrast, NNs
trained by SGD with a small learning rate can only learn the strong features
but makes little progress in learning the weak features. Consequently, when it
comes to the new testing data which consist of only weak features, the NN
trained by oscillating SGD with a large learning rate could still make correct
predictions consistently, while the NN trained by small learning rate SGD
fails. Our theory sheds light on how large learning rate training benefits the
generalization of NNs. Experimental results demonstrate our finding on "benign
oscillation".Comment: 63 pages, 10 figure
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
We study the implicit bias of batch normalization trained by gradient
descent. We show that when learning a linear model with batch normalization for
binary classification, gradient descent converges to a uniform margin
classifier on the training data with an convergence
rate. This distinguishes linear models with batch normalization from those
without batch normalization in terms of both the type of implicit bias and the
convergence rate. We further extend our result to a class of two-layer,
single-filter linear convolutional neural networks, and show that batch
normalization has an implicit bias towards a patch-wise uniform margin. Based
on two examples, we demonstrate that patch-wise uniform margin classifiers can
outperform the maximum margin classifiers in certain learning problems. Our
results contribute to a better theoretical understanding of batch
normalization.Comment: 53 pages, 2 figure