203 research outputs found
Enhanced Lasso Recovery on Graph
This work aims at recovering signals that are sparse on graphs. Compressed
sensing offers techniques for signal recovery from a few linear measurements
and graph Fourier analysis provides a signal representation on graph. In this
paper, we leverage these two frameworks to introduce a new Lasso recovery
algorithm on graphs. More precisely, we present a non-convex, non-smooth
algorithm that outperforms the standard convex Lasso technique. We carry out
numerical experiments on three benchmark graph datasets
Augmented L1 and Nuclear-Norm Models with a Globally Linearly Convergent Algorithm
This paper studies the long-existing idea of adding a nice smooth function to
"smooth" a non-differentiable objective function in the context of sparse
optimization, in particular, the minimization of
, where is a vector, as well as the
minimization of , where is a matrix and
and are the nuclear and Frobenius norms of ,
respectively. We show that they can efficiently recover sparse vectors and
low-rank matrices. In particular, they enjoy exact and stable recovery
guarantees similar to those known for minimizing and under
the conditions on the sensing operator such as its null-space property,
restricted isometry property, spherical section property, or RIPless property.
To recover a (nearly) sparse vector , minimizing
returns (nearly) the same solution as minimizing
almost whenever . The same relation also
holds between minimizing and minimizing
for recovering a (nearly) low-rank matrix , if . Furthermore, we show that the linearized Bregman algorithm for
minimizing subject to enjoys global
linear convergence as long as a nonzero solution exists, and we give an
explicit rate of convergence. The convergence property does not require a
solution solution or any properties on . To our knowledge, this is the best
known global convergence result for first-order sparse optimization algorithms.Comment: arXiv admin note: text overlap with arXiv:1207.5326 by other author
Homotopy based algorithms for -regularized least-squares
Sparse signal restoration is usually formulated as the minimization of a
quadratic cost function , where A is a dictionary and x is an
unknown sparse vector. It is well-known that imposing an constraint
leads to an NP-hard minimization problem. The convex relaxation approach has
received considerable attention, where the -norm is replaced by the
-norm. Among the many efficient solvers, the homotopy
algorithm minimizes with respect to x for a
continuum of 's. It is inspired by the piecewise regularity of the
-regularization path, also referred to as the homotopy path. In this
paper, we address the minimization problem for a
continuum of 's and propose two heuristic search algorithms for
-homotopy. Continuation Single Best Replacement is a forward-backward
greedy strategy extending the Single Best Replacement algorithm, previously
proposed for -minimization at a given . The adaptive search of
the -values is inspired by -homotopy. Regularization
Path Descent is a more complex algorithm exploiting the structural properties
of the -regularization path, which is piecewise constant with respect
to . Both algorithms are empirically evaluated for difficult inverse
problems involving ill-conditioned dictionaries. Finally, we show that they can
be easily coupled with usual methods of model order selection.Comment: 38 page
A Stochastic Majorize-Minimize Subspace Algorithm for Online Penalized Least Squares Estimation
Stochastic approximation techniques play an important role in solving many
problems encountered in machine learning or adaptive signal processing. In
these contexts, the statistics of the data are often unknown a priori or their
direct computation is too intensive, and they have thus to be estimated online
from the observed signals. For batch optimization of an objective function
being the sum of a data fidelity term and a penalization (e.g. a sparsity
promoting function), Majorize-Minimize (MM) methods have recently attracted
much interest since they are fast, highly flexible, and effective in ensuring
convergence. The goal of this paper is to show how these methods can be
successfully extended to the case when the data fidelity term corresponds to a
least squares criterion and the cost function is replaced by a sequence of
stochastic approximations of it. In this context, we propose an online version
of an MM subspace algorithm and we study its convergence by using suitable
probabilistic tools. Simulation results illustrate the good practical
performance of the proposed algorithm associated with a memory gradient
subspace, when applied to both non-adaptive and adaptive filter identification
problems
Doubly Robust Smoothing of Dynamical Processes via Outlier Sparsity Constraints
Coping with outliers contaminating dynamical processes is of major importance
in various applications because mismatches from nominal models are not uncommon
in practice. In this context, the present paper develops novel fixed-lag and
fixed-interval smoothing algorithms that are robust to outliers simultaneously
present in the measurements {\it and} in the state dynamics. Outliers are
handled through auxiliary unknown variables that are jointly estimated along
with the state based on the least-squares criterion that is regularized with
the -norm of the outliers in order to effect sparsity control. The
resultant iterative estimators rely on coordinate descent and the alternating
direction method of multipliers, are expressed in closed form per iteration,
and are provably convergent. Additional attractive features of the novel doubly
robust smoother include: i) ability to handle both types of outliers; ii)
universality to unknown nominal noise and outlier distributions; iii)
flexibility to encompass maximum a posteriori optimal estimators with reliable
performance under nominal conditions; and iv) improved performance relative to
competing alternatives at comparable complexity, as corroborated via simulated
tests.Comment: Submitted to IEEE Trans. on Signal Processin
MAGMA: Multi-level accelerated gradient mirror descent algorithm for large-scale convex composite minimization
Composite convex optimization models arise in several applications, and are
especially prevalent in inverse problems with a sparsity inducing norm and in
general convex optimization with simple constraints. The most widely used
algorithms for convex composite models are accelerated first order methods,
however they can take a large number of iterations to compute an acceptable
solution for large-scale problems. In this paper we propose to speed up first
order methods by taking advantage of the structure present in many applications
and in image processing in particular. Our method is based on multi-level
optimization methods and exploits the fact that many applications that give
rise to large scale models can be modelled using varying degrees of fidelity.
We use Nesterov's acceleration techniques together with the multi-level
approach to achieve convergence rate, where
denotes the desired accuracy. The proposed method has a better
convergence rate than any other existing multi-level method for convex
problems, and in addition has the same rate as accelerated methods, which is
known to be optimal for first-order methods. Moreover, as our numerical
experiments show, on large-scale face recognition problems our algorithm is
several times faster than the state of the art
Implicit Regularization in Over-Parameterized Support Vector Machine
In this paper, we design a regularization-free algorithm for high-dimensional
support vector machines (SVMs) by integrating over-parameterization with
Nesterov's smoothing method, and provide theoretical guarantees for the induced
implicit regularization phenomenon. In particular, we construct an
over-parameterized hinge loss function and estimate the true parameters by
leveraging regularization-free gradient descent on this loss function. The
utilization of Nesterov's method enhances the computational efficiency of our
algorithm, especially in terms of determining the stopping criterion and
reducing computational complexity. With appropriate choices of initialization,
step size, and smoothness parameter, we demonstrate that unregularized gradient
descent achieves a near-oracle statistical convergence rate. Additionally, we
verify our theoretical findings through a variety of numerical experiments and
compare the proposed method with explicit regularization. Our results
illustrate the advantages of employing implicit regularization via gradient
descent in conjunction with over-parameterization in sparse SVMs
- …