124,319 research outputs found
Learning from Minimum Entropy Queries in a Large Committee Machine
In supervised learning, the redundancy contained in random examples can be
avoided by learning from queries. Using statistical mechanics, we study
learning from minimum entropy queries in a large tree-committee machine. The
generalization error decreases exponentially with the number of training
examples, providing a significant improvement over the algebraic decay for
random examples. The connection between entropy and generalization error in
multi-layer networks is discussed, and a computationally cheap algorithm for
constructing queries is suggested and analysed.Comment: 4 pages, REVTeX, multicol, epsf, two postscript figures. To appear in
Physical Review E (Rapid Communications
Hints and the VC Dimension
Learning from hints is a generalization of learning from examples that allows for a variety of information about the unknown function to be used in the learning process. In this paper, we use the VC dimension, an established tool for analyzing learning from examples, to analyze learning from hints. In particular, we show how the VC dimension is affected by the introduction of a hint. We also derive a new quantity that defines a VC dimension for the hint itself. This quantity is used to estimate the number of examples needed to "absorb" the hint. We carry out the analysis for two types of hints, invariances and catalysts. We also describe how the same method can be applied to other types of hints
Empirical Study of Easy and Hard Examples in CNN Training
Deep Neural Networks (DNNs) generalize well despite their massive size and
capability of memorizing all examples. There is a hypothesis that DNNs start
learning from simple patterns and the hypothesis is based on the existence of
examples that are consistently well-classified at the early training stage
(i.e., easy examples) and examples misclassified (i.e., hard examples). Easy
examples are the evidence that DNNs start learning from specific patterns and
there is a consistent learning process. It is important to know how DNNs learn
patterns and obtain generalization ability, however, properties of easy and
hard examples are not thoroughly investigated (e.g., contributions to
generalization and visual appearances). In this work, we study the similarities
of easy and hard examples respectively for different Convolutional Neural
Network (CNN) architectures, assessing how those examples contribute to
generalization. Our results show that easy examples are visually similar to
each other and hard examples are visually diverse, and both examples are
largely shared across different CNN architectures. Moreover, while hard
examples tend to contribute more to generalization than easy examples, removing
a large number of easy examples leads to poor generalization. By analyzing
those results, we hypothesize that biases in a dataset and Stochastic Gradient
Descent (SGD) are the reasons why CNNs have consistent easy and hard examples.
Furthermore, we show that large scale classification datasets can be
efficiently compressed by using easiness proposed in this work.Comment: Accepted to ICONIP 201
A Unified View on PAC-Bayes Bounds for Meta-Learning
Meta learning automatically infers an inductive bias, that includes the hyperparameter of the baselearning algorithm, by observing data from a finite number of related tasks. This paper studies PAC-Bayes bounds on meta generalization gap. The meta-generalization gap comprises two sources of generalization gaps: the environmentlevel and task-level gaps resulting from observation of a finite number of tasks and data samples per task, respectively. In this paper, by upper bounding arbitrary convex functions, which link the expected and empirical losses at the environment and also per-task levels, we obtain new PACBayes bounds. Using these bounds, we develop new PAC-Bayes meta-learning algorithms. Numerical examples demonstrate the merits of the proposed novel bounds and algorithm in comparison to prior PAC-Bayes bounds for meta-learning
- …