241 research outputs found
Learning a spin glass: determining Hamiltonians from metastable states
We study the problem of determining the Hamiltonian of a fully connected
Ising Spin Glass of units from a set of measurements, whose sizes needs to
be bits. The student-teacher scenario, used to study learning
in feed-forward neural networks, is here extended to spin systems with
arbitrary couplings. The set of measurements consists of data about the local
minima of the rugged energy landscape. We compare simulations and analytical
approximations for the resulting learning curves obtained by using different
algorithms.Comment: 5 pages, 1 figure, to appear in Physica
Statistical physics and practical training of soft-committee machines
Equilibrium states of large layered neural networks with differentiable
activation function and a single, linear output unit are investigated using the
replica formalism. The quenched free energy of a student network with a very
large number of hidden units learning a rule of perfectly matching complexity
is calculated analytically. The system undergoes a first order phase transition
from unspecialized to specialized student configurations at a critical size of
the training set. Computer simulations of learning by stochastic gradient
descent from a fixed training set demonstrate that the equilibrium results
describe quantitatively the plateau states which occur in practical training
procedures at sufficiently small but finite learning rates.Comment: 11 pages, 4 figure
Pruning training sets for learning of object categories
Training datasets for learning of object categories are often contaminated or imperfect. We explore an approach to automatically identify examples that are noisy or troublesome for learning and exclude them from the training set. The problem is relevant to learning in semi-supervised or unsupervised setting, as well as to learning when the training data is contaminated with wrongly labeled examples or when correctly labeled, but hard to learn examples, are present. We propose a fully automatic mechanism for noise cleaning, called âdata pruningâ, and demonstrate its success on learning of human faces. It is not assumed that the data or the noise can be modeled or that additional training examples are available. Our experiments show that data pruning can improve on generalization performance for algorithms with various robustness to noise. It outperforms methods with regularization properties and is superior to commonly applied aggregation methods, such as bagging
Noise-Tolerant Learning, the Parity Problem, and the Statistical Query Model
We describe a slightly sub-exponential time algorithm for learning parity
functions in the presence of random classification noise. This results in a
polynomial-time algorithm for the case of parity functions that depend on only
the first O(log n log log n) bits of input. This is the first known instance of
an efficient noise-tolerant algorithm for a concept class that is provably not
learnable in the Statistical Query model of Kearns. Thus, we demonstrate that
the set of problems learnable in the statistical query model is a strict subset
of those problems learnable in the presence of noise in the PAC model.
In coding-theory terms, what we give is a poly(n)-time algorithm for decoding
linear k by n codes in the presence of random noise for the case of k = c log n
loglog n for some c > 0. (The case of k = O(log n) is trivial since one can
just individually check each of the 2^k possible messages and choose the one
that yields the closest codeword.)
A natural extension of the statistical query model is to allow queries about
statistical properties that involve t-tuples of examples (as opposed to single
examples). The second result of this paper is to show that any class of
functions learnable (strongly or weakly) with t-wise queries for t = O(log n)
is also weakly learnable with standard unary queries. Hence this natural
extension to the statistical query model does not increase the set of weakly
learnable functions
Crowdsourced PAC Learning under Classification Noise
In this paper, we analyze PAC learnability from labels produced by
crowdsourcing. In our setting, unlabeled examples are drawn from a distribution
and labels are crowdsourced from workers who operate under classification
noise, each with their own noise parameter. We develop an end-to-end
crowdsourced PAC learning algorithm that takes unlabeled data points as input
and outputs a trained classifier. Our three-step algorithm incorporates
majority voting, pure-exploration bandits, and noisy-PAC learning. We prove
several guarantees on the number of tasks labeled by workers for PAC learning
in this setting and show that our algorithm improves upon the baseline by
reducing the total number of tasks given to workers. We demonstrate the
robustness of our algorithm by exploring its application to additional
realistic crowdsourcing settings.Comment: 14 page
- âŚ