20,008 research outputs found
On the Depth of Deep Neural Networks: A Theoretical View
People believe that depth plays an important role in success of deep neural
networks (DNN). However, this belief lacks solid theoretical justifications as
far as we know. We investigate role of depth from perspective of margin bound.
In margin bound, expected error is upper bounded by empirical margin error plus
Rademacher Average (RA) based capacity term. First, we derive an upper bound
for RA of DNN, and show that it increases with increasing depth. This indicates
negative impact of depth on test performance. Second, we show that deeper
networks tend to have larger representation power (measured by Betti numbers
based complexity) than shallower networks in multi-class setting, and thus can
lead to smaller empirical margin error. This implies positive impact of depth.
The combination of these two results shows that for DNN with restricted number
of hidden units, increasing depth is not always good since there is a tradeoff
between positive and negative impacts. These results inspire us to seek
alternative ways to achieve positive impact of depth, e.g., imposing
margin-based penalty terms to cross entropy loss so as to reduce empirical
margin error without increasing depth. Our experiments show that in this way,
we achieve significantly better test performance.Comment: AAAI 201
Generalization Error in Deep Learning
Deep learning models have lately shown great performance in various fields
such as computer vision, speech recognition, speech translation, and natural
language processing. However, alongside their state-of-the-art performance, it
is still generally unclear what is the source of their generalization ability.
Thus, an important question is what makes deep neural networks able to
generalize well from the training set to new data. In this article, we provide
an overview of the existing theory and bounds for the characterization of the
generalization error of deep neural networks, combining both classical and more
recent theoretical and empirical results
OL\'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning
Deep neural networks trained using a softmax layer at the top and the
cross-entropy loss are ubiquitous tools for image classification. Yet, this
does not naturally enforce intra-class similarity nor inter-class margin of the
learned deep representations. To simultaneously achieve these two goals,
different solutions have been proposed in the literature, such as the pairwise
or triplet losses. However, such solutions carry the extra task of selecting
pairs or triplets, and the extra computational burden of computing and learning
for many combinations of them. In this paper, we propose a plug-and-play loss
term for deep networks that explicitly reduces intra-class variance and
enforces inter-class margin simultaneously, in a simple and elegant geometric
manner. For each class, the deep features are collapsed into a learned linear
subspace, or union of them, and inter-class subspaces are pushed to be as
orthogonal as possible. Our proposed Orthogonal Low-rank Embedding (OL\'E) does
not require carefully crafting pairs or triplets of samples for training, and
works standalone as a classification loss, being the first reported deep metric
learning framework of its kind. Because of the improved margin between features
of different classes, the resulting deep networks generalize better, are more
discriminative, and more robust. We demonstrate improved classification
performance in general object recognition, plugging the proposed loss term into
existing off-the-shelf architectures. In particular, we show the advantage of
the proposed loss in the small data/model scenario, and we significantly
advance the state-of-the-art on the Stanford STL-10 benchmark
- …