59,651 research outputs found
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks
We study the implicit bias of batch normalization trained by gradient
descent. We show that when learning a linear model with batch normalization for
binary classification, gradient descent converges to a uniform margin
classifier on the training data with an convergence
rate. This distinguishes linear models with batch normalization from those
without batch normalization in terms of both the type of implicit bias and the
convergence rate. We further extend our result to a class of two-layer,
single-filter linear convolutional neural networks, and show that batch
normalization has an implicit bias towards a patch-wise uniform margin. Based
on two examples, we demonstrate that patch-wise uniform margin classifiers can
outperform the maximum margin classifiers in certain learning problems. Our
results contribute to a better theoretical understanding of batch
normalization.Comment: 53 pages, 2 figure
Orthogonal Recurrent Neural Networks and Batch Normalization in Deep Neural Networks
Despite the recent success of various machine learning techniques, there are still numerous obstacles that must be overcome. One obstacle is known as the vanishing/exploding gradient problem. This problem refers to gradients that either become zero or unbounded. This is a well known problem that commonly occurs in Recurrent Neural Networks (RNNs). In this work we describe how this problem can be mitigated, establish three different architectures that are designed to avoid this issue, and derive update schemes for each architecture. Another portion of this work focuses on the often used technique of batch normalization. Although found to be successful in decreasing training times and in preventing overfitting, it is still unknown why this technique works. In this paper we describe batch normalization and provide a potential alternative with the end goal of improving our understanding of how batch normalization works
Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift
This paper first answers the question "why do the two most powerful
techniques Dropout and Batch Normalization (BN) often lead to a worse
performance when they are combined together?" in both theoretical and
statistical aspects. Theoretically, we find that Dropout would shift the
variance of a specific neural unit when we transfer the state of that network
from train to test. However, BN would maintain its statistical variance, which
is accumulated from the entire learning procedure, in the test phase. The
inconsistency of that variance (we name this scheme as "variance shift") causes
the unstable numerical behavior in inference that leads to more erroneous
predictions finally, when applying Dropout before BN. Thorough experiments on
DenseNet, ResNet, ResNeXt and Wide ResNet confirm our findings. According to
the uncovered mechanism, we next explore several strategies that modifies
Dropout and try to overcome the limitations of their combination by avoiding
the variance shift risks.Comment: 9 pages, 7 figure
- …