A study on multi-scale kernel optimisation via centered kernel-target alignment

Kernel mapping is one of the most widespread approaches to intrinsically deriving nonlinear classifiers. With the aim of better suiting a given dataset, different kernels have been proposed and different bounds and methodologies have been studied to optimise them. We focus on the optimisation of a multi-scale kernel, where a different width is chosen for each feature. This idea has been barely studied in the literature, although it has been shown to achieve better performance in the presence of heterogeneous attributes. The large number of parameters in multi-scale kernels makes it computationally unaffordable to optimise them by applying traditional cross-validation. Instead, an analytical measure known as centered kernel-target alignment (CKTA) can be used to align the kernel to the so-called ideal kernel matrix. This paper analyses and compares this and other alternatives, providing a review of the literature in kernel optimisation and some insights into the usefulness of multi-scale kernel optimisation via CKTA. When applied to the binary support vector machine paradigm (SVM), the results using 24 datasets show that CKTA with a multi-scale kernel leads to the construction of a well-defined feature space and simpler SVM models, provides an implicit filtering of non-informative features and achieves robust and comparable performance to other methods even when using random initialisations. Finally, we derive some considerations about when a multi-scale approach could be, in general, useful and propose a distance-based initialisation technique for the gradient-ascent method, which shows promising results

Analysis and convergence of SMO-like decomposition and geometrical algorithms for support vector machines

Tesis doctoral inédita. Universidad Autónoma de Madrid, Escuela Politécnica Superior, noviembre de 201

Gradient based optimization of support vector machines

In dieser Arbeit werden zwei unabhängige Probleme aus dem Bereich des Lernens mit Support-Vektor-Maschinen (SVM) mit Hilfe gradientenbasierter Optimierungstechniken gelöst. Für das erste Problem, das Training der Maschine, werden effiziente Algorithmen zur Lösung hochdimensionaler quadratischer Programme benötigt. Das zweite Problem, das sogenannte Modellselektionsproblem, erfordert die Konstruktion einer geeigneten Zielfunktion und Methoden zur nicht-konvexen Optimierung. Es werden neue bzw. verbesserte iterative Algorithmen zum Training von Support-Vektor-Maschinen vorgestellt und die Konvergenz der Lösung zur optimalen Lösung bewiesen. Das Modellsekeltionsproblem wird mit Hilfe völlig neuartiger Ansätze bearbeitet. Die Ergebnisse werden durch eine Reihe theoretischer Ergebnisse sowie empirischer Untersuchungen gestützt.Gradient-based optimization techniques are applied to two independent problems in the area of support vector machine (SVM) learning. The first problem, machine training, requires efficient high-dimensional quadratic programming. The second problem, model selection, amounts to the construction of an objective function and non-convex optimization. The main contributions to SVM training are in the area of algorithm design and corresponding convergence proofs. The model selection problem is assessed with completely new approaches. We highlight these achievements with a number of basic theoretical results and empirical evaluations