36 research outputs found
From error bounds to the complexity of first-order descent methods for convex functions
This paper shows that error bounds can be used as effective tools for
deriving complexity results for first-order descent methods in convex
minimization. In a first stage, this objective led us to revisit the interplay
between error bounds and the Kurdyka-\L ojasiewicz (KL) inequality. One can
show the equivalence between the two concepts for convex functions having a
moderately flat profile near the set of minimizers (as those of functions with
H\"olderian growth). A counterexample shows that the equivalence is no longer
true for extremely flat functions. This fact reveals the relevance of an
approach based on KL inequality. In a second stage, we show how KL inequalities
can in turn be employed to compute new complexity bounds for a wealth of
descent methods for convex problems. Our approach is completely original and
makes use of a one-dimensional worst-case proximal sequence in the spirit of
the famous majorant method of Kantorovich. Our result applies to a very simple
abstract scheme that covers a wide class of descent methods. As a byproduct of
our study, we also provide new results for the globalization of KL inequalities
in the convex framework.
Our main results inaugurate a simple methodology: derive an error bound,
compute the desingularizing function whenever possible, identify essential
constants in the descent method and finally compute the complexity using the
one-dimensional worst case proximal sequence. Our method is illustrated through
projection methods for feasibility problems, and through the famous iterative
shrinkage thresholding algorithm (ISTA), for which we show that the complexity
bound is of the form where the constituents of the bound only depend
on error bound constants obtained for an arbitrary least squares objective with
regularization
Retraction-based first-order feasible methods for difference-of-convex programs with smooth inequality and simple geometric constraints
In this paper, we propose first-order feasible methods for
difference-of-convex (DC) programs with smooth inequality and simple geometric
constraints. Our strategy for maintaining feasibility of the iterates is based
on a "retraction" idea adapted from the literature of manifold optimization.
When the constraints are convex, we establish the global subsequential
convergence of the sequence generated by our algorithm under strict feasibility
condition, and analyze its convergence rate when the objective is in addition
convex according to the Kurdyka-Lojasiewicz (KL) exponent of the extended
objective (i.e., sum of the objective and the indicator function of the
constraint set). We also show that the extended objective of a large class of
Euclidean norm (and more generally, group LASSO penalty) regularized convex
optimization problems is a KL function with exponent ; consequently,
our algorithm is locally linearly convergent when applied to these problems. We
then extend our method to solve DC programs with a single specially structured
nonconvex constraint. Finally, we discuss how our algorithms can be applied to
solve two concrete optimization problems, namely, group-structured compressed
sensing problems with Gaussian measurement noise and compressed sensing
problems with Cauchy measurement noise, and illustrate the empirical
performance of our algorithms
Control Barrier Function Based Design of Gradient Flows for Constrained Nonlinear Programming
This paper considers the problem of designing a continuous time dynamical
system to solve constrained nonlinear optimization problems such that the
feasible set is forward invariant and asymptotically stable. The invariance of
the feasible set makes the dynamics anytime, when viewed as an algorithm,
meaning that it is guaranteed to return a feasible solution regardless of when
it is terminated. The system is obtained by augmenting the gradient flow of the
objective function with inputs, then designing a feedback controller to keep
the state evolution within the feasible set using techniques from the theory of
control barrier functions. The equilibria of the system correspond exactly to
critical points of the optimization problem. Since the state of the system
corresponds to the primal optimizer, and the steady-state input at equilibria
corresponds to the dual optimizer, the method can be interpreted as a
primal-dual approach. The resulting closed-loop system is locally Lipschitz
continuous, so classical solutions to the system exist. We characterize
conditions under which local minimizers are Lyapunov stable, drawing
connections between various constraint qualification conditions and the
stability of the local minimizer. The algorithm is compared to other continuous
time methods for optimization