6,200 research outputs found
Forward-backward algorithms with different inertial terms for structured non-convex minimization problems
We investigate two inertial forward-backward algorithms in connection with
the minimization of the sum of a non-smooth and possibly non-convex and a
non-convex differentiable function. The algorithms are formulated in the spirit
of the famous FISTA method, however the setting is non-convex and we allow
different inertial terms. Moreover, the inertial parameters in our algorithms
can take negative values too. We also treat the case when the non-smooth
function is convex and we show that in this case a better step size can be
allowed. We prove some abstract convergence results which applied to our
numerical schemes allow us to show that the generated sequences converge to a
critical point of the objective function, provided a regularization of the
objective function satisfies the Kurdyka-Lojasiewicz property. Further, we
obtain a general result that applied to our numerical schemes ensures
convergence rates for the generated sequences and for the objective function
values formulated in terms of the KL exponent of a regularization of the
objective function. Finally, we apply our results to image restoration.Comment: 34 page
On inexact relative-error hybrid proximal extragradient, forward-backward and Tseng's modified forward-backward methods with inertial effects
In this paper, we propose and study the asymptotic convergence and
nonasymptotic global convergence rates (iteration-complexity) of an inertial
under-relaxed version of the relative-error hybrid proximal extragradient (HPE)
method for solving monotone inclusion problems. We analyze the proposed method
under more flexible assumptions than existing ones on the extrapolation and
relative-error parameters. As applications, we propose and/or study inertial
under-relaxed forward-backward and Tseng's modified forward-backward type
methods for solving structured monotone inclusions
An inertial three-operator splitting algorithm with applications to image inpainting
The three-operators splitting algorithm is a popular operator splitting
method for finding the zeros of the sum of three maximally monotone operators,
with one of which is cocoercive operator. In this paper, we propose a class of
inertial three-operator splitting algorithm. The convergence of the proposed
algorithm is proved by applying the inertial Krasnoselskii-Mann iteration under
certain conditions on the iterative parameters in real Hilbert spaces. As
applications, we develop an inertial three-operator splitting algorithm to
solve the convex minimization problem of the sum of three convex functions,
where one of them is differentiable with Lipschitz continuous gradient.
Finally, we conduct numerical experiments on a constrained image inpainting
problem with nuclear norm regularization. Numerical results demonstrate the
advantage of the proposed inertial three-operator splitting algorithms.Comment: 26 pages, 14 figure
Local Convergence of the Heavy-ball Method and iPiano for Non-convex Optimization
A local convergence result for abstract descent methods is proved. The
sequence of iterates is attracted by a local (or global) minimum, stays in its
neighborhood and converges within this neighborhood. This result allows
algorithms to exploit local properties of the objective function. In
particular, the abstract theory in this paper applies to the inertial
forward--backward splitting method: iPiano---a generalization of the Heavy-ball
method. Moreover, it reveals an equivalence between iPiano and inertial
averaged/alternating proximal minimization and projection methods. Key for this
equivalence is the attraction to a local minimum within a neighborhood and the
fact that, for a prox-regular function, the gradient of the Moreau envelope is
locally Lipschitz continuous and expressible in terms of the proximal mapping.
In a numerical feasibility problem, the inertial alternating projection method
significantly outperforms its non-inertial variants
Stochastic inertial primal-dual algorithms
We propose and study a novel stochastic inertial primal-dual approach to
solve composite optimization problems. These latter problems arise naturally
when learning with penalized regularization schemes. Our analysis provide
convergence results in a general setting, that allows to analyze in a unified
framework a variety of special cases of interest. Key in our analysis is
considering the framework of splitting algorithm for solving a monotone
inclusions in suitable product spaces and for a specific choice of
preconditioning operators
Local and Global Convergence of a General Inertial Proximal Splitting Scheme
This paper is concerned with convex composite minimization problems in a
Hilbert space. In these problems, the objective is the sum of two closed,
proper, and convex functions where one is smooth and the other admits a
computationally inexpensive proximal operator. We analyze a general family of
inertial proximal splitting algorithms (GIPSA) for solving such problems. We
establish finiteness of the sum of squared increments of the iterates and
optimality of the accumulation points. Weak convergence of the entire sequence
then follows if the minimum is attained. Our analysis unifies and extends
several previous results.
We then focus on -regularized optimization, which is the ubiquitous
special case where the nonsmooth term is the -norm. For certain
parameter choices, GIPSA is amenable to a local analysis for this problem. For
these choices we show that GIPSA achieves finite "active manifold
identification", i.e. convergence in a finite number of iterations to the
optimal support and sign, after which GIPSA reduces to minimizing a local
smooth function. Local linear convergence then holds under certain conditions.
We determine the rate in terms of the inertia, stepsize, and local curvature.
Our local analysis is applicable to certain recent variants of the Fast
Iterative Shrinkage-Thresholding Algorithm (FISTA), for which we establish
active manifold identification and local linear convergence. Our analysis
motivates the use of a momentum restart scheme in these FISTA variants to
obtain the optimal local linear convergence rate.Comment: 33 pages 1 figur
An Inertial Parallel and Asynchronous Fixed-Point Iteration for Convex Optimization
Two characteristics that make convex decomposition algorithms attractive are
simplicity of operations and generation of parallelizable structures. In
principle, these schemes require that all coordinates update at the same time,
i.e., they are synchronous by construction. Introducing asynchronicity in the
updates can resolve several issues that appear in the synchronous case, like
load imbalances in the computations or failing communication links. However,
and to the best of our knowledge, there are no instances of asynchronous
versions of commonly-known algorithms combined with inertial acceleration
techniques. In this work we propose an inertial asynchronous and parallel
fixed-point iteration from which several new versions of existing convex
optimization algorithms emanate. Departing from the norm that the frequency of
the coordinates' updates should comply to some prior distribution, we propose a
scheme where the only requirement is that the coordinates update within a
bounded interval. We prove convergence of the sequence of iterates generated by
the scheme at a linear rate. One instance of the proposed scheme is implemented
to solve a distributed optimization load sharing problem in a smart grid
setting and its superiority with respect to the non-accelerated version is
illustrated
Rate of convergence of the Nesterov accelerated gradient method in the subcritical case
In a Hilbert space setting , given a convex continuously differentiable function, and a positive
parameter, we consider the inertial system with Asymptotic Vanishing Damping
\begin{equation*}
\mbox{(AVD)}_{\alpha} \quad \quad \ddot{x}(t) + \frac{\alpha}{t} \dot{x}(t) +
\nabla \Phi (x(t)) =0. \end{equation*}
Depending on the value of with respect to 3, we give a complete
picture of the convergence properties as of the trajectories
generated by \mbox{(AVD)}_{\alpha}, as well as iterations of the
corresponding algorithms. Our main result concerns the subcritical case , where we show that .
Then we examine the convergence of trajectories to optimal solutions. As a
new result, in the one-dimensional framework, for the critical value , we prove the convergence of the trajectories without any restrictive
hypothesis on the convex function . In the second part of this paper, we
study the convergence properties of the associated forward-backward inertial
algorithms. They aim to solve structured convex minimization problems of the
form , with smooth
and nonsmooth. The continuous dynamics serves as a guideline for this
study. We obtain a similar rate of convergence for the sequence of iterates
: for we have for all \ . We conclude this study by showing that the
results are robust with respect to external perturbations.Comment: 23 page
A block inertial Bregman proximal algorithm for nonsmooth nonconvex problems with application to symmetric nonnegative matrix tri-factorization
We propose BIBPA, a block inertial Bregman proximal algorithm for minimizing
the sum of a block relatively smooth function (that is, relatively smooth
concerning each block) and block separable nonsmooth nonconvex functions. We
prove that the sequence generated by BIBPA subsequentially converges to
critical points of the objective under standard assumptions, and globally
converges when the objective function is additionally assumed to satisfy the
Kurdyka-{\L}ojasiewicz (K{\L}) property. We also provide the convergence rate
when the objective satisfies the {\L}ojasiewicz inequality. We apply BIBPA to
the symmetric nonnegative matrix tri-factorization (SymTriNMF) problem, where
we propose kernel functions for SymTriNMF and provide closed-form solutions for
subproblems of BIBPA.Comment: 18 page
Non-ergodic Complexity of Convex Proximal Inertial Gradient Descents
The proximal inertial gradient descent is efficient for the composite
minimization and applicable for broad of machine learning problems. In this
paper, we revisit the computational complexity of this algorithm and present
other novel results, especially on the convergence rates of the objective
function values. The non-ergodic O(1/k) rate is proved for proximal inertial
gradient descent with constant stepzise when the objective function is
coercive. When the objective function fails to promise coercivity, we prove the
sublinear rate with diminishing inertial parameters. In the case that the
objective function satisfies optimal strong convexity condition (which is much
weaker than the strong convexity), the linear convergence is proved with much
larger and general stepsize than previous literature. We also extend our
results to the multi-block version and present the computational complexity.
Both cyclic and stochastic index selection strategies are considered
- …