22,965 research outputs found
Boosting Simple Learners
Boosting is a celebrated machine learning approach which is based on the idea
of combining weak and moderately inaccurate hypotheses to a strong and accurate
one. We study boosting under the assumption that the weak hypotheses belong to
a class of bounded capacity. This assumption is inspired by the common
convention that weak hypotheses are "rules-of-thumbs" from an "easy-to-learn
class". (Schapire and Freund '12, Shalev-Shwartz and Ben-David '14.) Formally,
we assume the class of weak hypotheses has a bounded VC dimension. We focus on
two main questions: (i) Oracle Complexity: How many weak hypotheses are needed
in order to produce an accurate hypothesis? We design a novel boosting
algorithm and demonstrate that it circumvents a classical lower bound by Freund
and Schapire ('95, '12). Whereas the lower bound shows that
weak hypotheses with -margin are sometimes
necessary, our new method requires only weak
hypothesis, provided that they belong to a class of bounded VC dimension.
Unlike previous boosting algorithms which aggregate the weak hypotheses by
majority votes, the new boosting algorithm uses more complex ("deeper")
aggregation rules. We complement this result by showing that complex
aggregation rules are in fact necessary to circumvent the aforementioned lower
bound. (ii) Expressivity: Which tasks can be learned by boosting weak
hypotheses from a bounded VC class? Can complex concepts that are "far away"
from the class be learned? Towards answering the first question we identify a
combinatorial-geometric parameter which captures the expressivity of
base-classes in boosting. As a corollary we provide an affirmative answer to
the second question for many well-studied classes, including half-spaces and
decision stumps. Along the way, we establish and exploit connections with
Discrepancy Theory.Comment: A minor revision according to STOC review
Generalized additive modelling with implicit variable selection by likelihood based boosting
The use of generalized additive models in statistical data analysis suffers from the restriction to few explanatory variables and the problems of selection of smoothing parameters. Generalized additive model boosting circumvents these problems by means of stagewise fitting of weak learners. A fitting procedure is derived which works for all simple exponential family distributions, including binomial, Poisson and normal response variables. The procedure combines the selection of variables and the determination of the appropriate amount of smoothing. As weak learners penalized regression splines and the newly introduced penalized stumps are considered. Estimates of standard deviations and stopping criteria which are notorious problems in iterative procedures are based on an approximate hat matrix. The method is shown to outperform common procedures for the fitting of generalized additive models. In particular in high dimensional settings it is the only method that works properly
Vote-boosting ensembles
Vote-boosting is a sequential ensemble learning method in which the
individual classifiers are built on different weighted versions of the training
data. To build a new classifier, the weight of each training instance is
determined in terms of the degree of disagreement among the current ensemble
predictions for that instance. For low class-label noise levels, especially
when simple base learners are used, emphasis should be made on instances for
which the disagreement rate is high. When more flexible classifiers are used
and as the noise level increases, the emphasis on these uncertain instances
should be reduced. In fact, at sufficiently high levels of class-label noise,
the focus should be on instances on which the ensemble classifiers agree. The
optimal type of emphasis can be automatically determined using
cross-validation. An extensive empirical analysis using the beta distribution
as emphasis function illustrates that vote-boosting is an effective method to
generate ensembles that are both accurate and robust
Model-based Boosting in R: A Hands-on Tutorial Using the R Package mboost
We provide a detailed hands-on tutorial for the R add-on package mboost. The package implements boosting for optimizing general risk functions utilizing component-wise (penalized) least squares estimates as base-learners for fitting various kinds of generalized linear and generalized additive models to potentially high-dimensional data. We give a theoretical background and demonstrate how mboost can be used to fit interpretable models of different complexity. As an example we use mboost to predict the body fat based on anthropometric measurements throughout the tutorial
Improved Multi-Class Cost-Sensitive Boosting via Estimation of the Minimum-Risk Class
We present a simple unified framework for multi-class cost-sensitive boosting.
The minimum-risk class is estimated directly, rather than via an approximation
of the posterior distribution. Our method jointly optimizes binary weak learners
and their corresponding output vectors, requiring classes to share features at each
iteration. By training in a cost-sensitive manner, weak learners are invested in separating
classes whose discrimination is important, at the expense of less relevant
classification boundaries. Additional contributions are a family of loss functions
along with proof that our algorithm is Boostable in the theoretical sense, as well
as an efficient procedure for growing decision trees for use as weak learners. We
evaluate our method on a variety of datasets: a collection of synthetic planar data,
common UCI datasets, MNIST digits, SUN scenes, and CUB-200 birds. Results
show state-of-the-art performance across all datasets against several strong baselines,
including non-boosting multi-class approaches
Proximal boosting and its acceleration
Gradient boosting is a prediction method that iteratively combines weak
learners to produce a complex and accurate model. From an optimization point of
view, the learning procedure of gradient boosting mimics a gradient descent on
a functional variable. This paper proposes to build upon the proximal point
algorithm when the empirical risk to minimize is not differentiable to
introduce a novel boosting approach, called proximal boosting. Besides being
motivated by non-differentiable optimization, the proposed algorithm benefits
from Nesterov's acceleration in the same way as gradient boosting [Biau et al.,
2018]. This leads to a variant, called accelerated proximal boosting.
Advantages of leveraging proximal methods for boosting are illustrated by
numerical experiments on simulated and real-world data. In particular, we
exhibit a favorable comparison over gradient boosting regarding convergence
rate and prediction accuracy
Infinite Ensemble Learning with Support Vector Machines
Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of base learners. However, existing algorithms are limited to combining only a finite number of base learners, and the generated ensemble is usually sparse. It is not clear whether we should construct an ensemble classifier with a larger or even an infinite number of base learners.
In addition, constructing an infinite ensemble itself is a challenging task. In this paper, we formulate an infinite ensemble learning framework based on SVM. The framework could output an infinite and nonsparse ensemble, and can be applied to construct new kernels for SVM as well as to interpret existing ones. We demonstrate the framework with a concrete application, the stump kernel, which embodies infinitely many decision stumps. The stump kernel is simple, yet powerful.
Experimental results show that SVM with the stump kernel usually achieves better performance than boosting, even with noisy data.</p
- …