The rate of convergence of Nesterov's accelerated forward-backward method is actually faster than 1/k21/k^{2}

The {\it forward-backward algorithm} is a powerful tool for solving optimization problems with a {\it additively separable} and {\it smooth} + {\it nonsmooth} structure. In the convex setting, a simple but ingenious acceleration scheme developed by Nesterov has been proved useful to improve the theoretical rate of convergence for the function values from the standard $\mathcal O(k^{-1})$ down to $\mathcal O(k^{-2})$. In this short paper, we prove that the rate of convergence of a slight variant of Nesterov's accelerated forward-backward method, which produces {\it convergent} sequences, is actually $o(k^{-2})$, rather than $\mathcal O(k^{-2})$. Our arguments rely on the connection between this algorithm and a second-order differential inclusion with vanishing damping

Splitting methods with variable metric for KL functions

We study the convergence of general abstract descent methods applied to a lower semicontinuous nonconvex function f that satisfies the Kurdyka-Lojasiewicz inequality in a Hilbert space. We prove that any precompact sequence converges to a critical point of f and obtain new convergence rates both for the values and the iterates. The analysis covers alternating versions of the forward-backward method with variable metric and relative errors. As an example, a nonsmooth and nonconvex version of the Levenberg-Marquardt algorithm is detailled

Fast convex optimization via inertial dynamics with Hessian driven damping

We first study the fast minimization properties of the trajectories of the second-order evolution equation $$\ddot{x}(t) + \frac{\alpha}{t} \dot{x}(t) + \beta \nabla^2 \Phi (x(t))\dot{x} (t) + \nabla \Phi (x(t)) = 0,$$ where $\Phi:\mathcal H\to\mathbb R$ is a smooth convex function acting on a real Hilbert space $\mathcal H$, and $\alpha$, $\beta$ are positive parameters. This inertial system combines an isotropic viscous damping which vanishes asymptotically, and a geometrical Hessian driven damping, which makes it naturally related to Newton's and Levenberg-Marquardt methods. For $\alpha\geq 3$, $\beta >0$, along any trajectory, fast convergence of the values $$\Phi(x(t))- \min_{\mathcal H}\Phi =\mathcal O\left(t^{-2}\right)$$ is obtained, together with rapid convergence of the gradients $\nabla\Phi(x(t))$ to zero. For $\alpha>3$, just assuming that $\Phi$ has minimizers, we show that any trajectory converges weakly to a minimizer of $\Phi$, and $\Phi(x(t))-\min_{\mathcal H}\Phi = o(t^{-2})$. Strong convergence is established in various practical situations. For the strongly convex case, convergence can be arbitrarily fast depending on the choice of $\alpha$. More precisely, we have $\Phi(x(t))- \min_{\mathcal H}\Phi = \mathcal O(t^{-\frac{2}{3}\alpha})$. We extend the results to the case of a general proper lower-semicontinuous convex function $\Phi : \mathcal H \rightarrow \mathbb R \cup \{+\infty \}$. This is based on the fact that the inertial dynamic with Hessian driven damping can be written as a first-order system in time and space. By explicit-implicit time discretization, this opens a gate to new $-$ possibly more rapid $-$ inertial algorithms, expanding the field of FISTA methods for convex structured optimization problems

Forward-backward approximation of nonlinear semigroups in finite and infinite horizon

International audienc

Inertial Krasnoselskii-Mann Iterations

We establish the weak convergence of inertial Krasnoselskii-Mann iterations towards a common fixed point of a family of quasi-nonexpansive operators, along with worst case estimates for the rate at which the residuals vanish. Strong and linear convergence are obtained in the quasi-contractive setting. In both cases, we highlight the relationship with the non-inertial case, and show that passing from one regime to the other is a continuous process in terms of parameter hypotheses and convergence rates. Numerical illustrations for an inertial primaldual method and an inertial three-operator splitting algorithm, whose performance is superior to that of their non-inertial counterparts