617 research outputs found
Empirical Study of Easy and Hard Examples in CNN Training
Deep Neural Networks (DNNs) generalize well despite their massive size and
capability of memorizing all examples. There is a hypothesis that DNNs start
learning from simple patterns and the hypothesis is based on the existence of
examples that are consistently well-classified at the early training stage
(i.e., easy examples) and examples misclassified (i.e., hard examples). Easy
examples are the evidence that DNNs start learning from specific patterns and
there is a consistent learning process. It is important to know how DNNs learn
patterns and obtain generalization ability, however, properties of easy and
hard examples are not thoroughly investigated (e.g., contributions to
generalization and visual appearances). In this work, we study the similarities
of easy and hard examples respectively for different Convolutional Neural
Network (CNN) architectures, assessing how those examples contribute to
generalization. Our results show that easy examples are visually similar to
each other and hard examples are visually diverse, and both examples are
largely shared across different CNN architectures. Moreover, while hard
examples tend to contribute more to generalization than easy examples, removing
a large number of easy examples leads to poor generalization. By analyzing
those results, we hypothesize that biases in a dataset and Stochastic Gradient
Descent (SGD) are the reasons why CNNs have consistent easy and hard examples.
Furthermore, we show that large scale classification datasets can be
efficiently compressed by using easiness proposed in this work.Comment: Accepted to ICONIP 201
- …