3,014 research outputs found
Generalization error for multi-class margin classification
In this article, we study rates of convergence of the generalization error of
multi-class margin classifiers. In particular, we develop an upper bound theory
quantifying the generalization error of various large margin classifiers. The
theory permits a treatment of general margin losses, convex or nonconvex, in
presence or absence of a dominating class. Three main results are established.
First, for any fixed margin loss, there may be a trade-off between the ideal
and actual generalization performances with respect to the choice of the class
of candidate decision functions, which is governed by the trade-off between the
approximation and estimation errors. In fact, different margin losses lead to
different ideal or actual performances in specific cases. Second, we
demonstrate, in a problem of linear learning, that the convergence rate can be
arbitrarily fast in the sample size depending on the joint distribution of
the input/output pair. This goes beyond the anticipated rate .
Third, we establish rates of convergence of several margin classifiers in
feature selection with the number of candidate variables allowed to greatly
exceed the sample size but no faster than .Comment: Published at http://dx.doi.org/10.1214/07-EJS069 in the Electronic
Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of
Mathematical Statistics (http://www.imstat.org
Comment on "Support Vector Machines with Applications"
Comment on ``Support Vector Machines with Applications'' [math.ST/0612817]Comment: Published at http://dx.doi.org/10.1214/088342306000000484 in the
Statistical Science (http://www.imstat.org/sts/) by the Institute of
Mathematical Statistics (http://www.imstat.org
On Reject and Refine Options in Multicategory Classification
In many real applications of statistical learning, a decision made from
misclassification can be too costly to afford; in this case, a reject option,
which defers the decision until further investigation is conducted, is often
preferred. In recent years, there has been much development for binary
classification with a reject option. Yet, little progress has been made for the
multicategory case. In this article, we propose margin-based multicategory
classification methods with a reject option. In addition, and more importantly,
we introduce a new and unique refine option for the multicategory problem,
where the class of an observation is predicted to be from a set of class
labels, whose cardinality is not necessarily one. The main advantage of both
options lies in their capacity of identifying error-prone observations.
Moreover, the refine option can provide more constructive information for
classification by effectively ruling out implausible classes. Efficient
implementations have been developed for the proposed methods. On the
theoretical side, we offer a novel statistical learning theory and show a fast
convergence rate of the excess -risk of our methods with emphasis on
diverging dimensionality and number of classes. The results can be further
improved under a low noise assumption. A set of comprehensive simulation and
real data studies has shown the usefulness of the new learning tools compared
to regular multicategory classifiers. Detailed proofs of theorems and extended
numerical results are included in the supplemental materials available online.Comment: A revised version of this paper was accepted for publication in the
Journal of the American Statistical Association Theory and Methods Section.
52 pages, 6 figure
The Support Vector Machine and Mixed Integer Linear Programming: Ramp Loss SVM with L1-Norm Regularization
The support vector machine (SVM) is a flexible classification method that accommodates a kernel trick to learn nonlinear decision rules. The traditional formulation as an optimization problem is a quadratic program. In efforts to reduce computational complexity, some have proposed using an L1-norm regularization to create a linear program (LP). In other efforts aimed at increasing the robustness to outliers, investigators have proposed using the ramp loss which results in what may be expressed as a quadratic integer programming problem (QIP). In this paper, we consider combining these ideas for ramp loss SVM with L1-norm regularization. The result is four formulations for SVM that each may be expressed as a mixed integer linear program (MILP). We observe that ramp loss SVM with L1-norm regularization provides robustness to outliers with the linear kernel. We investigate the time required to find good solutions to the various formulations using a branch and bound solver
- …