29,465 research outputs found
New Interpretations of Normalization Methods in Deep Learning
In recent years, a variety of normalization methods have been proposed to
help train neural networks, such as batch normalization (BN), layer
normalization (LN), weight normalization (WN), group normalization (GN), etc.
However, mathematical tools to analyze all these normalization methods are
lacking. In this paper, we first propose a lemma to define some necessary
tools. Then, we use these tools to make a deep analysis on popular
normalization methods and obtain the following conclusions: 1) Most of the
normalization methods can be interpreted in a unified framework, namely
normalizing pre-activations or weights onto a sphere; 2) Since most of the
existing normalization methods are scaling invariant, we can conduct
optimization on a sphere with scaling symmetry removed, which can help
stabilize the training of network; 3) We prove that training with these
normalization methods can make the norm of weights increase, which could cause
adversarial vulnerability as it amplifies the attack. Finally, a series of
experiments are conducted to verify these claims.Comment: Accepted by AAAI 202
Recommended from our members
Data and Computation Efficient Meta-Learning
In order to make predictions with high accuracy, conventional deep learning systems require large training datasets consisting of thousands or millions of examples and long training times measured in hours or days, consuming high levels of electricity with a negative impact on our environment. It is desirable to have have machine learning systems that can emulate human behavior such that they can quickly learn new concepts from only a few examples. This is especially true if we need to quickly customize or personalize machine learning models to specific scenarios where it would be impractical to acquire a large amount of training data and where a mobile device is the means for computation. We define a data efficient machine learning system to be one that can learn a new concept from only a few examples (or shots) and a computation efficient machine learning system to be one that can learn a new concept rapidly without retraining on an everyday computing device such as a smart phone.
In this work, we design, develop, analyze, and extend the theory of machine learning systems that are both data efficient and computation efficient. We present systems that are trained using multiple tasks such that it "learns how to learn" to solve new tasks from only a few examples. These systems can efficiently solve new, unseen tasks drawn from a broad range of data distributions, in both the low and high data regimes, without the need for costly retraining. Adapting to a new task requires only a forward pass of the example task data through the trained network making the learning of new tasks possible on mobile devices. In particular, we focus on few-shot image classification systems, i.e. machine learning systems that can distinguish between numerous classes of objects depicted in digital images given only a few examples of each class of object to learn from.
To accomplish this, we first develop ML-PIP, a general framework for Meta-Learning approximate Probabilistic Inference for Prediction. ML-PIP extends existing probabilistic interpretations of meta-learning to cover a broad class of methods. We then introduce Versa, an instance of the framework employing a fast, flexible and versatile amortization network that takes few-shot learning datasets as inputs, with arbitrary numbers of training examples, and outputs a distribution over task-specific parameters in a single forward pass of the network. We evaluate Versa on benchmark datasets, where at the time, the method achieved state-of-the-art results when compared to meta-learning approaches using similar training regimes and feature extractor capacity.
Next, we build on Versa and add a second amortized network to adapt key parameters in the feature extractor to the current task. To accomplish this, we introduce CNAPs, a conditional neural process based approach to multi-task classification. We demonstrate that, at the time, CNAPs achieved state-of-the-art results on the challenging Meta-Dataset benchmark indicating high-quality transfer-learning. Timing experiments reveal that CNAPs is computationally efficient when adapting to an unseen task as it does not involve gradient back propagation computations. We show that trained models are immediately deployable to continual learning and active learning where they can outperform existing approaches that do not leverage transfer learning.
Finally, we investigate the effects of different methods of batch normalization on meta-learning systems. Batch normalization has become an essential component of deep learning systems as it significantly accelerates the training of neural networks by allowing the use of higher learning rates and decreasing the sensitivity to network initialization. We show that the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective. We evaluate a range of approaches to batch normalization for few-shot learning scenarios, and develop a novel approach that we call TaskNorm. Experiments demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient based- and gradient-free meta-learning approaches and that TaskNorm consistently improves performance
Learn to Interpret Atari Agents
Deep Reinforcement Learning (DeepRL) agents surpass human-level performances
in a multitude of tasks. However, the direct mapping from states to actions
makes it hard to interpret the rationale behind the decision making of agents.
In contrast to previous a-posteriori methods of visualizing DeepRL policies, we
propose an end-to-end trainable framework based on Rainbow, a representative
Deep Q-Network (DQN) agent. Our method automatically learns important regions
in the input domain, which enables characterizations of the decision making and
interpretations for non-intuitive behaviors. Hence we name it Region Sensitive
Rainbow (RS-Rainbow). RS-Rainbow utilizes a simple yet effective mechanism to
incorporate visualization ability into the learning model, not only improving
model interpretability, but leading to improved performance. Extensive
experiments on the challenging platform of Atari 2600 demonstrate the
superiority of RS-Rainbow. In particular, our agent achieves state of the art
at just 25% of the training frames. Demonstrations and code are available at
https://github.com/yz93/Learn-to-Interpret-Atari-Agents
Utilizing Class Information for Deep Network Representation Shaping
Statistical characteristics of deep network representations, such as sparsity
and correlation, are known to be relevant to the performance and
interpretability of deep learning. When a statistical characteristic is
desired, often an adequate regularizer can be designed and applied during the
training phase. Typically, such a regularizer aims to manipulate a statistical
characteristic over all classes together. For classification tasks, however, it
might be advantageous to enforce the desired characteristic per class such that
different classes can be better distinguished. Motivated by the idea, we design
two class-wise regularizers that explicitly utilize class information:
class-wise Covariance Regularizer (cw-CR) and class-wise Variance Regularizer
(cw-VR). cw-CR targets to reduce the covariance of representations calculated
from the same class samples for encouraging feature independence. cw-VR is
similar, but variance instead of covariance is targeted to improve feature
compactness. For the sake of completeness, their counterparts without using
class information, Covariance Regularizer (CR) and Variance Regularizer (VR),
are considered together. The four regularizers are conceptually simple and
computationally very efficient, and the visualization shows that the
regularizers indeed perform distinct representation shaping. In terms of
classification performance, significant improvements over the baseline and
L1/L2 weight regularization methods were found for 21 out of 22 tasks over
popular benchmark datasets. In particular, cw-VR achieved the best performance
for 13 tasks including ResNet-32/110.Comment: Published in AAAI 201
- …