96 research outputs found
Modular Autoencoders for Ensemble Feature Extraction
We introduce the concept of a Modular Autoencoder (MAE), capable of learning
a set of diverse but complementary representations from unlabelled data, that
can later be used for supervised tasks. The learning of the representations is
controlled by a trade off parameter, and we show on six benchmark datasets the
optimum lies between two extremes: a set of smaller, independent autoencoders
each with low capacity, versus a single monolithic encoding, outperforming an
appropriate baseline. In the present paper we explore the special case of
linear MAE, and derive an SVD-based algorithm which converges several orders of
magnitude faster than gradient descent.Comment: 18 pages, 8 figures, to appear in a special issue of The Journal Of
Machine Learning Research (vol.44, Dec 2015
Classification with unknown class-conditional label noise on non-compact feature spaces
We investigate the problem of classification in the presence of unknown
class-conditional label noise in which the labels observed by the learner have
been corrupted with some unknown class dependent probability. In order to
obtain finite sample rates, previous approaches to classification with unknown
class-conditional label noise have required that the regression function is
close to its extrema on sets of large measure. We shall consider this problem
in the setting of non-compact metric spaces, where the regression function need
not attain its extrema.
In this setting we determine the minimax optimal learning rates (up to
logarithmic factors). The rate displays interesting threshold behaviour: When
the regression function approaches its extrema at a sufficient rate, the
optimal learning rates are of the same order as those obtained in the
label-noise free setting. If the regression function approaches its extrema
more gradually then classification performance necessarily degrades. In
addition, we present an adaptive algorithm which attains these rates without
prior knowledge of either the distributional parameters or the local density.
This identifies for the first time a scenario in which finite sample rates are
achievable in the label noise setting, but they differ from the optimal rates
without label noise
Optimistic Bounds for Multi-output Prediction
We investigate the challenge of multi-output learning, where the goal is to
learn a vector-valued function based on a supervised data set. This includes a
range of important problems in Machine Learning including multi-target
regression, multi-class classification and multi-label classification. We begin
our analysis by introducing the self-bounding Lipschitz condition for
multi-output loss functions, which interpolates continuously between a
classical Lipschitz condition and a multi-dimensional analogue of a smoothness
condition. We then show that the self-bounding Lipschitz condition gives rise
to optimistic bounds for multi-output learning, which are minimax optimal up to
logarithmic factors. The proof exploits local Rademacher complexity combined
with a powerful minoration inequality due to Srebro, Sridharan and Tewari. As
an application we derive a state-of-the-art generalization bound for
multi-class gradient boosting
Fast rates for a kNN classifier robust to unknown asymmetric label noise
We consider classification in the presence of class-dependent asymmetric
label noise with unknown noise probabilities. In this setting, identifiability
conditions are known, but additional assumptions were shown to be required for
finite sample rates, and so far only the parametric rate has been obtained.
Assuming these identifiability conditions, together with a measure-smoothness
condition on the regression function and Tsybakov's margin condition, we show
that the Robust kNN classifier of Gao et al. attains, the minimax optimal rates
of the noise-free setting, up to a log factor, even when trained on data with
unknown asymmetric label noise. Hence, our results provide a solid theoretical
backing for this empirically successful algorithm. By contrast the standard kNN
is not even consistent in the setting of asymmetric label noise. A key idea in
our analysis is a simple kNN based method for estimating the maximum of a
function that requires far less assumptions than existing mode estimators do,
and which may be of independent interest for noise proportion estimation and
randomised optimisation problems.Comment: ICML 201
- …