18 research outputs found
Calibration of One-Class SVM for MV set estimation
A general approach for anomaly detection or novelty detection consists in
estimating high density regions or Minimum Volume (MV) sets. The One-Class
Support Vector Machine (OCSVM) is a state-of-the-art algorithm for estimating
such regions from high dimensional data. Yet it suffers from practical
limitations. When applied to a limited number of samples it can lead to poor
performance even when picking the best hyperparameters. Moreover the solution
of OCSVM is very sensitive to the selection of hyperparameters which makes it
hard to optimize in an unsupervised setting. We present a new approach to
estimate MV sets using the OCSVM with a different choice of the parameter
controlling the proportion of outliers. The solution function of the OCSVM is
learnt on a training set and the desired probability mass is obtained by
adjusting the offset on a test set to prevent overfitting. Models learnt on
different train/test splits are then aggregated to reduce the variance induced
by such random splits. Our approach makes it possible to tune the
hyperparameters automatically and obtain nested set estimates. Experimental
results show that our approach outperforms the standard OCSVM formulation while
suffering less from the curse of dimensionality than kernel density estimates.
Results on actual data sets are also presented.Comment: IEEE DSAA' 2015, Oct 2015, Paris, Franc
An Exponential Lower Bound on the Complexity of Regularization Paths
For a variety of regularized optimization problems in machine learning,
algorithms computing the entire solution path have been developed recently.
Most of these methods are quadratic programs that are parameterized by a single
parameter, as for example the Support Vector Machine (SVM). Solution path
algorithms do not only compute the solution for one particular value of the
regularization parameter but the entire path of solutions, making the selection
of an optimal parameter much easier.
It has been assumed that these piecewise linear solution paths have only
linear complexity, i.e. linearly many bends. We prove that for the support
vector machine this complexity can be exponential in the number of training
points in the worst case. More strongly, we construct a single instance of n
input points in d dimensions for an SVM such that at least \Theta(2^{n/2}) =
\Theta(2^d) many distinct subsets of support vectors occur as the
regularization parameter changes.Comment: Journal version, 28 Pages, 5 Figure
Classification with Margin Constraints: A Unification with Applications to Optimization
This paper introduces Classification with Margin Constraints (CMC), a simple generalization of cost-sensitive classification that unifies several learning settings. In particular, we show that a CMC classifier can be used, out of the box, to solve regression, quantile estimation, and several anomaly detection formulations. On the one hand, our reductions to CMC are at the loss level: the optimization problem to solve under the equivalent CMC setting is exactly the same as the optimization problem under the original (e.g. regression) setting. On the other hand, due to the close relationship between CMC and standard binary classification, the ideas proposed for efficient optimization in binary classification naturally extend to CMC. As such, any improvement in CMC optimization immediately transfers to the domains reduced to CMC, without the need for new derivations or programs. To our knowledge, this unified view has been overlooked by the existing practice in the literature, where an optimization technique (such as SMO or PEGASOS) is first developed for binary classification and then extended to other problem domains on a case-by-case basis. We demonstrate the flexibility of CMC by reducing two recent anomaly detection and quantile learning methods to CMC