3,918 research outputs found
The Geometry of Nonlinear Embeddings in Kernel Discriminant Analysis
Fisher's linear discriminant analysis is a classical method for
classification, yet it is limited to capturing linear features only. Kernel
discriminant analysis as an extension is known to successfully alleviate the
limitation through a nonlinear feature mapping. We study the geometry of
nonlinear embeddings in discriminant analysis with polynomial kernels and
Gaussian kernel by identifying the population-level discriminant function that
depends on the data distribution and the kernel. In order to obtain the
discriminant function, we solve a generalized eigenvalue problem with
between-class and within-class covariance operators. The polynomial
discriminants are shown to capture the class difference through the population
moments explicitly. For approximation of the Gaussian discriminant, we use a
particular representation of the Gaussian kernel by utilizing the exponential
generating function for Hermite polynomials. We also show that the Gaussian
discriminant can be approximated using randomized projections of the data. Our
results illuminate how the data distribution and the kernel interact in
determination of the nonlinear embedding for discrimination, and provide a
guideline for choice of the kernel and its parameters
Benchmarking least squares support vector machine classifiers.
In Support Vector Machines (SVMs), the solution of the classification problem is characterized by a ( convex) quadratic programming (QP) problem. In a modified version of SVMs, called Least Squares SVM classifiers (LS-SVMs), a least squares cost function is proposed so as to obtain a linear set of equations in the dual space. While the SVM classifier has a large margin interpretation, the LS-SVM formulation is related in this paper to a ridge regression approach for classification with binary targets and to Fisher's linear discriminant analysis in the feature space. Multiclass categorization problems are represented by a set of binary classifiers using different output coding schemes. While regularization is used to control the effective number of parameters of the LS-SVM classifier, the sparseness property of SVMs is lost due to the choice of the 2-norm. Sparseness can be imposed in a second stage by gradually pruning the support value spectrum and optimizing the hyperparameters during the sparse approximation procedure. In this paper, twenty public domain benchmark datasets are used to evaluate the test set performance of LS-SVM classifiers with linear, polynomial and radial basis function (RBF) kernels. Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances. These SVM and LS-SVM performances are consistently very good when compared to a variety of methods described in the literature including decision tree based algorithms, statistical algorithms and instance based learning methods. We show on ten UCI datasets that the LS-SVM sparse approximation procedure can be successfully applied.least squares support vector machines; multiclass support vector machines; sparse approximation; discriminant-analysis; sparse approximation; learning algorithms; classification; framework; kernels; time; SISTA;
Scalable Randomized Kernel Methods for Multiview Data Integration and Prediction
We develop scalable randomized kernel methods for jointly associating data
from multiple sources and simultaneously predicting an outcome or classifying a
unit into one of two or more classes. The proposed methods model nonlinear
relationships in multiview data together with predicting a clinical outcome and
are capable of identifying variables or groups of variables that best
contribute to the relationships among the views. We use the idea that random
Fourier bases can approximate shift-invariant kernel functions to construct
nonlinear mappings of each view and we use these mappings and the outcome
variable to learn view-independent low-dimensional representations. Through
simulation studies, we show that the proposed methods outperform several other
linear and nonlinear methods for multiview data integration. When the proposed
methods were applied to gene expression, metabolomics, proteomics, and
lipidomics data pertaining to COVID-19, we identified several molecular
signatures forCOVID-19 status and severity. Results from our real data
application and simulations with small sample sizes suggest that the proposed
methods may be useful for small sample size problems. Availability: Our
algorithms are implemented in Pytorch and interfaced in R and would be made
available at: https://github.com/lasandrall/RandMVLearn.Comment: 24 pages, 5 figures, 4 table
- ā¦