398 research outputs found
Optimization with Sparsity-Inducing Penalties
Sparse estimation methods are aimed at using or obtaining parsimonious
representations of data or models. They were first dedicated to linear variable
selection but numerous extensions have now emerged such as structured sparsity
or kernel selection. It turns out that many of the related estimation problems
can be cast as convex optimization problems by regularizing the empirical risk
with appropriate non-smooth norms. The goal of this paper is to present from a
general perspective optimization tools and techniques dedicated to such
sparsity-inducing penalties. We cover proximal methods, block-coordinate
descent, reweighted -penalized techniques, working-set and homotopy
methods, as well as non-convex formulations and extensions, and provide an
extensive set of experiments to compare various algorithms from a computational
point of view
Conjugate gradient acceleration of iteratively re-weighted least squares methods
Iteratively Re-weighted Least Squares (IRLS) is a method for solving
minimization problems involving non-quadratic cost functions, perhaps
non-convex and non-smooth, which however can be described as the infimum over a
family of quadratic functions. This transformation suggests an algorithmic
scheme that solves a sequence of quadratic problems to be tackled efficiently
by tools of numerical linear algebra. Its general scope and its usually simple
implementation, transforming the initial non-convex and non-smooth minimization
problem into a more familiar and easily solvable quadratic optimization
problem, make it a versatile algorithm. However, despite its simplicity,
versatility, and elegant analysis, the complexity of IRLS strongly depends on
the way the solution of the successive quadratic optimizations is addressed.
For the important special case of and sparse
recovery problems in signal processing, we investigate theoretically and
numerically how accurately one needs to solve the quadratic problems by means
of the (CG) method in each iteration in order to
guarantee convergence. The use of the CG method may significantly speed-up the
numerical solution of the quadratic subproblems, in particular, when fast
matrix-vector multiplication (exploiting for instance the FFT) is available for
the matrix involved. In addition, we study convergence rates. Our modified IRLS
method outperforms state of the art first order methods such as Iterative Hard
Thresholding (IHT) or Fast Iterative Soft-Thresholding Algorithm (FISTA) in
many situations, especially in large dimensions. Moreover, IRLS is often able
to recover sparse vectors from fewer measurements than required for IHT and
FISTA.Comment: 40 page
Enhancing Sparsity by Reweighted â„“(1) Minimization
It is now well understood that (1) it is possible to reconstruct sparse signals exactly from what appear to be highly incomplete sets of linear measurements and (2) that this can be done by constrained ℓ1 minimization. In this paper, we study a novel method for sparse signal recovery that in many situations outperforms ℓ1 minimization in the sense that substantially fewer measurements are needed for exact recovery. The algorithm consists of solving a sequence of weighted ℓ1-minimization problems where the weights used for the next iteration are computed from the value of the current solution. We present a series of experiments demonstrating the remarkable performance and broad applicability of this algorithm in the areas of sparse signal recovery, statistical estimation, error correction and image processing. Interestingly, superior gains are also achieved when our method is applied to recover signals with assumed near-sparsity in overcomplete representations—not by reweighting the ℓ1 norm of the coefficient sequence as is common, but by reweighting the ℓ1 norm of the transformed object. An immediate consequence is the possibility of highly efficient data acquisition protocols by improving on a technique known as Compressive Sensing
Super-Linear Convergence of Dual Augmented-Lagrangian Algorithm for Sparsity Regularized Estimation
We analyze the convergence behaviour of a recently proposed algorithm for
regularized estimation called Dual Augmented Lagrangian (DAL). Our analysis is
based on a new interpretation of DAL as a proximal minimization algorithm. We
theoretically show under some conditions that DAL converges super-linearly in a
non-asymptotic and global sense. Due to a special modelling of sparse
estimation problems in the context of machine learning, the assumptions we make
are milder and more natural than those made in conventional analysis of
augmented Lagrangian algorithms. In addition, the new interpretation enables us
to generalize DAL to wide varieties of sparse estimation problems. We
experimentally confirm our analysis in a large scale -regularized
logistic regression problem and extensively compare the efficiency of DAL
algorithm to previously proposed algorithms on both synthetic and benchmark
datasets.Comment: 51 pages, 9 figure
- …