32,064 research outputs found
Activity Identification and Local Linear Convergence of Forward--Backward-type methods
In this paper, we consider a class of Forward--Backward (FB) splitting
methods that includes several variants (e.g. inertial schemes, FISTA) for
minimizing the sum of two proper convex and lower semi-continuous functions,
one of which has a Lipschitz continuous gradient, and the other is partly
smooth relatively to a smooth active manifold . We propose a
unified framework, under which we show that, this class of FB-type algorithms
(i) correctly identifies the active manifolds in a finite number of iterations
(finite activity identification), and (ii) then enters a local linear
convergence regime, which we characterize precisely in terms of the structure
of the underlying active manifolds. For simpler problems involving polyhedral
functions, we show finite termination. We also establish and explain why FISTA
(with convergent sequences) locally oscillates and can be slower than FB. These
results may have numerous applications including in signal/image processing,
sparse recovery and machine learning. Indeed, the obtained results explain the
typical behaviour that has been observed numerically for many problems in these
fields such as the Lasso, the group Lasso, the fused Lasso and the nuclear norm
regularization to name only a few.Comment: Full length version of the previous short on
Local Linear Convergence Analysis of Primal-Dual Splitting Methods
In this paper, we study the local linear convergence properties of a
versatile class of Primal-Dual splitting methods for minimizing composite
non-smooth convex optimization problems. Under the assumption that the
non-smooth components of the problem are partly smooth relative to smooth
manifolds, we present a unified local convergence analysis framework for these
methods. More precisely, in our framework we first show that (i) the sequences
generated by Primal-Dual splitting methods identify a pair of primal and dual
smooth manifolds in a finite number of iterations, and then (ii) enter a local
linear convergence regime, which is characterized based on the structure of the
underlying active smooth manifolds. We also show how our results for
Primal-Dual splitting can be specialized to cover existing ones on
Forward-Backward splitting and Douglas-Rachford splitting/ADMM (alternating
direction methods of multipliers). Moreover, based on these obtained local
convergence analysis result, several practical acceleration techniques are
discussed. To exemplify the usefulness of the obtained result, we consider
several concrete numerical experiments arising from fields including
signal/image processing, inverse problems and machine learning, etc. The
demonstration not only verifies the local linear convergence behaviour of
Primal-Dual splitting methods, but also the insights on how to accelerate them
in practice
A Multi-step Inertial Forward--Backward Splitting Method for Non-convex Optimization
In this paper, we propose a multi-step inertial Forward--Backward splitting
algorithm for minimizing the sum of two non-necessarily convex functions, one
of which is proper lower semi-continuous while the other is differentiable with
a Lipschitz continuous gradient. We first prove global convergence of the
scheme with the help of the Kurdyka-{\L}ojasiewicz property. Then, when the
non-smooth part is also partly smooth relative to a smooth submanifold, we
establish finite identification of the latter and provide sharp local linear
convergence analysis. The proposed method is illustrated on a few problems
arising from statistics and machine learning.Comment: This paper is in company with our recent work on
Forward--Backward-type splitting methods http://arxiv.org/abs/1503.0370
Sensitivity Analysis for Mirror-Stratifiable Convex Functions
This paper provides a set of sensitivity analysis and activity identification
results for a class of convex functions with a strong geometric structure, that
we coined "mirror-stratifiable". These functions are such that there is a
bijection between a primal and a dual stratification of the space into
partitioning sets, called strata. This pairing is crucial to track the strata
that are identifiable by solutions of parametrized optimization problems or by
iterates of optimization algorithms. This class of functions encompasses all
regularizers routinely used in signal and image processing, machine learning,
and statistics. We show that this "mirror-stratifiable" structure enjoys a nice
sensitivity theory, allowing us to study stability of solutions of optimization
problems to small perturbations, as well as activity identification of
first-order proximal splitting-type algorithms. Existing results in the
literature typically assume that, under a non-degeneracy condition, the active
set associated to a minimizer is stable to small perturbations and is
identified in finite time by optimization schemes. In contrast, our results do
not require any non-degeneracy assumption: in consequence, the optimal active
set is not necessarily stable anymore, but we are able to track precisely the
set of identifiable strata.We show that these results have crucial implications
when solving challenging ill-posed inverse problems via regularization, a
typical scenario where the non-degeneracy condition is not fulfilled. Our
theoretical results, illustrated by numerical simulations, allow to
characterize the instability behaviour of the regularized solutions, by
locating the set of all low-dimensional strata that can be potentially
identified by these solutions
Proximal Multitask Learning over Networks with Sparsity-inducing Coregularization
In this work, we consider multitask learning problems where clusters of nodes
are interested in estimating their own parameter vector. Cooperation among
clusters is beneficial when the optimal models of adjacent clusters have a good
number of similar entries. We propose a fully distributed algorithm for solving
this problem. The approach relies on minimizing a global mean-square error
criterion regularized by non-differentiable terms to promote cooperation among
neighboring clusters. A general diffusion forward-backward splitting strategy
is introduced. Then, it is specialized to the case of sparsity promoting
regularizers. A closed-form expression for the proximal operator of a weighted
sum of -norms is derived to achieve higher efficiency. We also provide
conditions on the step-sizes that ensure convergence of the algorithm in the
mean and mean-square error sense. Simulations are conducted to illustrate the
effectiveness of the strategy
Activity Identification and Local Linear Convergence of Douglas--Rachford/ADMM under Partial Smoothness
Convex optimization has become ubiquitous in most quantitative disciplines of
science, including variational image processing. Proximal splitting algorithms
are becoming popular to solve such structured convex optimization problems.
Within this class of algorithms, Douglas--Rachford (DR) and alternating
direction method of multipliers (ADMM) are designed to minimize the sum of two
proper lower semi-continuous convex functions whose proximity operators are
easy to compute. The goal of this work is to understand the local convergence
behaviour of DR (resp. ADMM) when the involved functions (resp. their
Legendre-Fenchel conjugates) are moreover partly smooth. More precisely, when
both of the two functions (resp. their conjugates) are partly smooth relative
to their respective manifolds, we show that DR (resp. ADMM) identifies these
manifolds in finite time. Moreover, when these manifolds are affine or linear,
we prove that DR/ADMM is locally linearly convergent. When and are
locally polyhedral, we show that the optimal convergence radius is given in
terms of the cosine of the Friedrichs angle between the tangent spaces of the
identified manifolds. This is illustrated by several concrete examples and
supported by numerical experiments.Comment: 17 pages, 1 figure, published in the proceedings of the Fifth
International Conference on Scale Space and Variational Methods in Computer
Visio
- …