598 research outputs found
Local Rademacher complexities
We propose new bounds on the error of learning algorithms in terms of a
data-dependent notion of complexity. The estimates we establish give optimal
rates and are based on a local and empirical version of Rademacher averages, in
the sense that the Rademacher averages are computed from the data, on a subset
of functions with small empirical error. We present some applications to
classification and prediction with convex function classes, and with kernel
classes in particular.Comment: Published at http://dx.doi.org/10.1214/009053605000000282 in the
Annals of Statistics (http://www.imstat.org/aos/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Relax and Localize: From Value to Algorithms
We show a principled way of deriving online learning algorithms from a
minimax analysis. Various upper bounds on the minimax value, previously thought
to be non-constructive, are shown to yield algorithms. This allows us to
seamlessly recover known methods and to derive new ones. Our framework also
captures such "unorthodox" methods as Follow the Perturbed Leader and the R^2
forecaster. We emphasize that understanding the inherent complexity of the
learning problem leads to the development of algorithms.
We define local sequential Rademacher complexities and associated algorithms
that allow us to obtain faster rates in online learning, similarly to
statistical learning theory. Based on these localized complexities we build a
general adaptive method that can take advantage of the suboptimality of the
observed sequence.
We present a number of new algorithms, including a family of randomized
methods that use the idea of a "random playout". Several new versions of the
Follow-the-Perturbed-Leader algorithms are presented, as well as methods based
on the Littlestone's dimension, efficient methods for matrix completion with
trace norm, and algorithms for the problems of transductive learning and
prediction with static experts
Complexity regularization via localized random penalties
In this article, model selection via penalized empirical loss minimization in
nonparametric classification problems is studied. Data-dependent penalties are
constructed, which are based on estimates of the complexity of a small subclass
of each model class, containing only those functions with small empirical loss.
The penalties are novel since those considered in the literature are typically
based on the entire model class. Oracle inequalities using these penalties are
established, and the advantage of the new penalties over those based on the
complexity of the whole model class is demonstrated.Comment: Published by the Institute of Mathematical Statistics
(http://www.imstat.org) in the Annals of Statistics
(http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/00905360400000046
- …