6,348 research outputs found
Sparse least squares support vector regression for nonstationary systems
A new adaptive sparse least squares support vector regression algorithm, referred to as SLSSVR has been introduced for the adaptive modeling of nonstationary systems. Using a sliding window of recent data set of size N to track t he non-stationary characteristics of the incoming data, our adaptive model is initially formulated based on least squares support vector regression with forgetting factor (without bias term). In order to obtain a sparse model in which some parameters are exactly zeros, a l 1 penalty was applied in parameter estimation in the dual problem. Furthermore we exploit the fact that since the associated system/kernel matrix in positive definite, the dual solution of least squares support vector machine without bias term, can be solved iteratively with guaranteed convergence. Furthermore since the models between two consecutive time steps there are (N-1) shared kernels/parameters, the online solution can be obtained efficiently using coordinate descent algorithm in the form of Gauss-Seidel algorithm with minimal number of iterations. This allows a very sparse model per time step to be obtained very efficiently, avoiding expensive matrix inversion. The real stock market dataset and simulated examples have shown that the proposed approaches can lead to superior performances in comparison with the linear recursive least algorithm and a number of online non-linear approaches in terms of modelling performance and model size
The Graphical Lasso: New Insights and Alternatives
The graphical lasso \citep{FHT2007a} is an algorithm for learning the
structure in an undirected Gaussian graphical model, using
regularization to control the number of zeros in the precision matrix
{\B\Theta}={\B\Sigma}^{-1} \citep{BGA2008,yuan_lin_07}. The {\texttt R}
package \GL\ \citep{FHT2007a} is popular, fast, and allows one to efficiently
build a path of models for different values of the tuning parameter.
Convergence of \GL\ can be tricky; the converged precision matrix might not be
the inverse of the estimated covariance, and occasionally it fails to converge
with warm starts. In this paper we explain this behavior, and propose new
algorithms that appear to outperform \GL.
By studying the "normal equations" we see that, \GL\ is solving the {\em
dual} of the graphical lasso penalized likelihood, by block coordinate ascent;
a result which can also be found in \cite{BGA2008}.
In this dual, the target of estimation is \B\Sigma, the covariance matrix,
rather than the precision matrix \B\Theta. We propose similar primal
algorithms \PGL\ and \DPGL, that also operate by block-coordinate descent,
where \B\Theta is the optimization target. We study all of these algorithms,
and in particular different approaches to solving their coordinate
sub-problems. We conclude that \DPGL\ is superior from several points of view.Comment: This is a revised version of our previous manuscript with the same
name ArXiv id: http://arxiv.org/abs/1111.547
Practical Inexact Proximal Quasi-Newton Method with Global Complexity Analysis
Recently several methods were proposed for sparse optimization which make
careful use of second-order information [10, 28, 16, 3] to improve local
convergence rates. These methods construct a composite quadratic approximation
using Hessian information, optimize this approximation using a first-order
method, such as coordinate descent and employ a line search to ensure
sufficient descent. Here we propose a general framework, which includes
slightly modified versions of existing algorithms and also a new algorithm,
which uses limited memory BFGS Hessian approximations, and provide a novel
global convergence rate analysis, which covers methods that solve subproblems
via coordinate descent
L0 Sparse Inverse Covariance Estimation
Recently, there has been focus on penalized log-likelihood covariance
estimation for sparse inverse covariance (precision) matrices. The penalty is
responsible for inducing sparsity, and a very common choice is the convex
norm. However, the best estimator performance is not always achieved with this
penalty. The most natural sparsity promoting "norm" is the non-convex
penalty but its lack of convexity has deterred its use in sparse maximum
likelihood estimation. In this paper we consider non-convex penalized
log-likelihood inverse covariance estimation and present a novel cyclic descent
algorithm for its optimization. Convergence to a local minimizer is proved,
which is highly non-trivial, and we demonstrate via simulations the reduced
bias and superior quality of the penalty as compared to the
penalty
- β¦