Algorithms for Sparse Linear Classifiers in the Massive Data Setting

Classifiers favoring sparse solutions, such as support vector machines, relevance vector machines, LASSO-regression based classifiers, etc., provide competitive methods for classification problems in high dimensions. However, current algorithms for training sparse classifiers typically scale quite unfavorably with respect to the number of training examples. This paper proposes online and multi-pass algorithms for training sparse linear classifiers for high dimensional data. These algorithms have computational complexity and memory requirements that make learning on massive data sets feasible. The central idea that makes this possible is a straightforward quadratic approximation to the likelihood function

Parameter Detection of Thin Films From Their X-Ray Reflectivity by Support Vector Machines

Reflectivity measurements are used in thin film investigations for determining the desity and the thickness of layered structures and the roughness of external and internal surfaces. From the mathematical point of view the deduction of these parameters from a measured reflectivity curve represents an inverSe ptoblem. At present, curve fitting procedures, based to a large extent on expert knowledge are commonly used in practice. These techniques suffer from a low degree of automation. In this paper we present a new approach to the evaluation of reflectivity measurements using support vector machines. For the estimation of the different thin film parameters we provide sparse approximations of vector-valued functions, where we work in parallel on the same data sets. Our support vector machines were trained by simulated reflectivity curves generated by the optical matrix method. The solution of the corresponding quadratic programming problem makes use of the SVMTorch algorithm. We present numerical investigations to assess the performance of our method using models of practical relevance. It is concluded that the approximation by support vector machines represents a very promising tool in X-ray reflectivity investigations and seems also to be applicable for a much broader range of parameter detection problems in X-ray analysis

Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval

Relevance feedback schemes based on support vector machines (SVM) have been widely used in content-based image retrieval (CBIR). However, the performance of SVM-based relevance feedback is often poor when the number of labeled positive feedback samples is small. This is mainly due to three reasons: 1) an SVM classifier is unstable on a small-sized training set, 2) SVM's optimal hyperplane may be biased when the positive feedback samples are much less than the negative feedback samples, and 3) overfitting happens because the number of feature dimensions is much higher than the size of the training set. In this paper, we develop a mechanism to overcome these problems. To address the first two problems, we propose an asymmetric bagging-based SVM (AB-SVM). For the third problem, we combine the random subspace method and SVM for relevance feedback, which is named random subspace SVM (RS-SVM). Finally, by integrating AB-SVM and RS-SVM, an asymmetric bagging and random subspace SVM (ABRS-SVM) is built to solve these three problems and further improve the relevance feedback performance

Gaussian Processes for Machine Learning

A comprehensive and self-contained introduction to Gaussian processes, which provide a principled, practical, probabilistic approach to learning in kernel machines. Gaussian processes (GPs) provide a principled, practical, probabilistic approach to learning in kernel machines. GPs have received increased attention in the machine-learning community over the past decade, and this book provides a long-needed systematic and unified treatment of theoretical and practical aspects of GPs in machine learning. The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics. The book deals with the supervised-learning problem for both regression and classification, and includes detailed algorithms. A wide variety of covariance (kernel) functions are presented and their properties discussed. Model selection is discussed both from a Bayesian and a classical perspective. Many connections to other well-known techniques from machine learning and statistics are discussed, including support-vector machines, neural networks, splines, regularization networks, relevance vector machines and others. Theoretical issues including learning curves and the PAC-Bayesian framework are treated, and several approximation methods for learning with large datasets are discussed. The book contains illustrative examples and exercises, and code and datasets are available on the Web. Appendixes provide mathematical background and a discussion of Gaussian Markov processes

Real-time classification of aggregated traffic conditions using relevance vector machines

This paper examines the theory and application of a recently developed machine learning technique namely Relevance Vector Machines (RVMs) in the task of traffic conditions classification. Traffic conditions are labelled as dangerous (i.e. probably leading to a collision) and safe (i.e. a normal driving) based on 15-minute measurements of average speed and volume. Two different RVM algorithms are trained with two real-world datasets and validated with one real-world dataset describing traffic conditions of a motorway and two A-class roads in the UK. The performance of these classifiers is compared to the popular and successfully applied technique of Support vector machines (SVMs). The main findings indicate that RVMs could successfully be employed in real-time classification of traffic conditions. They rely on a fewer number of decision vectors, their training time could be reduced to the level of seconds and their classification rates are similar to those of SVMs. However, RVM algorithms with a larger training dataset consisting of highly disaggregated traffic data, as well as the incorporation of other traffic or network variables so as to better describe traffic dynamics, may lead to higher classification accuracy than the one presented in this paper

Customizing kernel functions for SVM-based hyperspectral image classification

Previous research applying kernel methods such as support vector machines (SVMs) to hyperspectral image classification has achieved performance competitive with the best available algorithms. However, few efforts have been made to extend SVMs to cover the specific requirements of hyperspectral image classification, for example, by building tailor-made kernels. Observation of real-life spectral imagery from the AVIRIS hyperspectral sensor shows that the useful information for classification is not equally distributed across bands, which provides potential to enhance the SVM's performance through exploring different kernel functions. Spectrally weighted kernels are, therefore, proposed, and a set of particular weights is chosen by either optimizing an estimate of generalization error or evaluating each band's utility level. To assess the effectiveness of the proposed method, experiments are carried out on the publicly available 92AV3C dataset collected from the 220-dimensional AVIRIS hyperspectral sensor. Results indicate that the method is generally effective in improving performance: spectral weighting based on learning weights by gradient descent is found to be slightly better than an alternative method based on estimating ";relevance"; between band information and ground trut