497 research outputs found
Feature Selection for Linear SVM with Provable Guarantees
We give two provably accurate feature-selection techniques for the linear
SVM. The algorithms run in deterministic and randomized time respectively. Our
algorithms can be used in an unsupervised or supervised setting. The supervised
approach is based on sampling features from support vectors. We prove that the
margin in the feature space is preserved to within -relative error of
the margin in the full feature space in the worst-case. In the unsupervised
setting, we also provide worst-case guarantees of the radius of the minimum
enclosing ball, thereby ensuring comparable generalization as in the full
feature space and resolving an open problem posed in Dasgupta et al. We present
extensive experiments on real-world datasets to support our theory and to
demonstrate that our method is competitive and often better than prior
state-of-the-art, for which there are no known provable guarantees.Comment: Appearing in Proceedings of 18th AISTATS, JMLR W&CP, vol 38, 201
Training Support Vector Machines Using Frank-Wolfe Optimization Methods
Training a Support Vector Machine (SVM) requires the solution of a quadratic
programming problem (QP) whose computational complexity becomes prohibitively
expensive for large scale datasets. Traditional optimization methods cannot be
directly applied in these cases, mainly due to memory restrictions.
By adopting a slightly different objective function and under mild conditions
on the kernel used within the model, efficient algorithms to train SVMs have
been devised under the name of Core Vector Machines (CVMs). This framework
exploits the equivalence of the resulting learning problem with the task of
building a Minimal Enclosing Ball (MEB) problem in a feature space, where data
is implicitly embedded by a kernel function.
In this paper, we improve on the CVM approach by proposing two novel methods
to build SVMs based on the Frank-Wolfe algorithm, recently revisited as a fast
method to approximate the solution of a MEB problem. In contrast to CVMs, our
algorithms do not require to compute the solutions of a sequence of
increasingly complex QPs and are defined by using only analytic optimization
steps. Experiments on a large collection of datasets show that our methods
scale better than CVMs in most cases, sometimes at the price of a slightly
lower accuracy. As CVMs, the proposed methods can be easily extended to machine
learning problems other than binary classification. However, effective
classifiers are also obtained using kernels which do not satisfy the condition
required by CVMs and can thus be used for a wider set of problems
Direct L2 Support Vector Machine
This dissertation introduces a novel model for solving the L2 support vector machine dubbed Direct L2 Support Vector Machine (DL2 SVM). DL2 SVM represents a new classification model that transforms the SVM\u27s underlying quadratic programming problem into a system of linear equations with nonnegativity constraints. The devised system of linear equations has a symmetric positive definite matrix and a solution vector has to be nonnegative.
Furthermore, this dissertation introduces a novel algorithm dubbed Non-Negative Iterative Single Data Algorithm (NN ISDA) which solves the underlying DL2 SVM\u27s constrained system of equations. This solver shows significant speedup compared to several other state-of-the-art algorithms. The training time improvement is achieved at no cost, in other words, the accuracy is kept at the same level. All the experiments that support this claim were conducted on various datasets within the strict double cross-validation scheme. DL2 SVM solved with NN ISDA has faster training time on both medium and large datasets.
In addition to a comprehensive DL2 SVM model we introduce and derive its three variants. Three different solvers for the DL2\u27s system of linear equations with nonnegativity constraints were implemented, presented and compared in this dissertation
Dataset Condensation via Generative Model
Dataset condensation aims to condense a large dataset with a lot of training
samples into a small set. Previous methods usually condense the dataset into
the pixels format. However, it suffers from slow optimization speed and large
number of parameters to be optimized. When increasing image resolutions and
classes, the number of learnable parameters grows accordingly, prohibiting
condensation methods from scaling up to large datasets with diverse classes.
Moreover, the relations among condensed samples have been neglected and hence
the feature distribution of condensed samples is often not diverse. To solve
these problems, we propose to condense the dataset into another format, a
generative model. Such a novel format allows for the condensation of large
datasets because the size of the generative model remains relatively stable as
the number of classes or image resolution increases. Furthermore, an
intra-class and an inter-class loss are proposed to model the relation of
condensed samples. Intra-class loss aims to create more diverse samples for
each class by pushing each sample away from the others of the same class.
Meanwhile, inter-class loss increases the discriminability of samples by
widening the gap between the centers of different classes. Extensive
comparisons with state-of-the-art methods and our ablation studies confirm the
effectiveness of our method and its individual component. To our best
knowledge, we are the first to successfully conduct condensation on
ImageNet-1k.Comment: old work,done in 202
- …