The rate of convergence of Nesterov's accelerated forward-backward method is actually faster than 1/k21/k^{2}

The {\it forward-backward algorithm} is a powerful tool for solving optimization problems with a {\it additively separable} and {\it smooth} + {\it nonsmooth} structure. In the convex setting, a simple but ingenious acceleration scheme developed by Nesterov has been proved useful to improve the theoretical rate of convergence for the function values from the standard $\mathcal O(k^{-1})$ down to $\mathcal O(k^{-2})$. In this short paper, we prove that the rate of convergence of a slight variant of Nesterov's accelerated forward-backward method, which produces {\it convergent} sequences, is actually $o(k^{-2})$, rather than $\mathcal O(k^{-2})$. Our arguments rely on the connection between this algorithm and a second-order differential inclusion with vanishing damping

Dynamical systems and forward-backward algorithms associated with the sum of a convex subdifferential and a monotone cocoercive operator

In a Hilbert framework, we introduce continuous and discrete dynamical systems which aim at solving inclusions governed by structured monotone operators $A=\partial\Phi+B$, where $\partial\Phi$ is the subdifferential of a convex lower semicontinuous function $\Phi$, and $B$ is a monotone cocoercive operator. We first consider the extension to this setting of the regularized Newton dynamic with two potentials. Then, we revisit some related dynamical systems, namely the semigroup of contractions generated by $A$, and the continuous gradient projection dynamic. By a Lyapunov analysis, we show the convergence properties of the orbits of these systems. The time discretization of these dynamics gives various forward-backward splitting methods (some new) for solving structured monotone inclusions involving non-potential terms. The convergence of these algorithms is obtained under classical step size limitation. Perspectives are given in the field of numerical splitting methods for optimization, and multi-criteria decision processes.Comment: 25 page

Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects

In a Hilbert space $\mathcal H$, we study the asymptotic behaviour, as time variable $t$ goes to $+\infty$, of nonautonomous gradient-like dynamical systems involving inertia and multiscale features. Given $\mathcal H$ a general Hilbert space, $\Phi: \mathcal H \rightarrow \mathbb R$ and $\Psi: \mathcal H \rightarrow \mathbb R$ two convex differentiable functions, $\gamma$ a positive damping parameter, and $\epsilon (t)$ a function of $t$ which tends to zero as $t$ goes to $+\infty$, we consider the second-order differential equation $$\ddot{x}(t) + \gamma \dot{x}(t) + \nabla \Phi (x(t)) + \epsilon (t) \nabla \Psi (x(t)) = 0.$$ This system models the emergence of various collective behaviors in game theory, as well as the asymptotic control of coupled nonlinear oscillators. Assuming that $\epsilon(t)$ tends to zero moderately slowly as $t$ goes to infinity, we show that the trajectories converge weakly in $\mathcal H$. The limiting equilibria are solutions of the hierarchical minimization problem which consists in minimizing $\Psi$ over the set $C$ of minimizers of $\Phi$. As key assumptions, we suppose that $\int_{0}^{+\infty}\epsilon (t) dt = + \infty$ and that, for every $p$ belonging to a convex cone $\mathcal C$ depending on the data $\Phi$ and $\Psi$ $$\int_{0}^{+\infty} \left[\Phi^* \left(\epsilon (t)p\right) -\sigma_C \left(\epsilon (t)p\right)\right]dt < + \infty$$ where $\Phi^*$ is the Fenchel conjugate of $\Phi$, and $\sigma_C$ is the support function of $C$. An application is given to coupled oscillators

Asymptotic behavior of coupled dynamical systems with multiscale aspects

We study the asymptotic behavior, as time t goes to infinity, of nonautonomous dynamical systems involving multiscale features. These systems model the emergence of various collective behaviors in game theory, as well as the asymptotic control of coupled sytems.Comment: 20 page

Fast convex optimization via inertial dynamics with Hessian driven damping

We first study the fast minimization properties of the trajectories of the second-order evolution equation $$\ddot{x}(t) + \frac{\alpha}{t} \dot{x}(t) + \beta \nabla^2 \Phi (x(t))\dot{x} (t) + \nabla \Phi (x(t)) = 0,$$ where $\Phi:\mathcal H\to\mathbb R$ is a smooth convex function acting on a real Hilbert space $\mathcal H$, and $\alpha$, $\beta$ are positive parameters. This inertial system combines an isotropic viscous damping which vanishes asymptotically, and a geometrical Hessian driven damping, which makes it naturally related to Newton's and Levenberg-Marquardt methods. For $\alpha\geq 3$, $\beta >0$, along any trajectory, fast convergence of the values $$\Phi(x(t))- \min_{\mathcal H}\Phi =\mathcal O\left(t^{-2}\right)$$ is obtained, together with rapid convergence of the gradients $\nabla\Phi(x(t))$ to zero. For $\alpha>3$, just assuming that $\Phi$ has minimizers, we show that any trajectory converges weakly to a minimizer of $\Phi$, and $\Phi(x(t))-\min_{\mathcal H}\Phi = o(t^{-2})$. Strong convergence is established in various practical situations. For the strongly convex case, convergence can be arbitrarily fast depending on the choice of $\alpha$. More precisely, we have $\Phi(x(t))- \min_{\mathcal H}\Phi = \mathcal O(t^{-\frac{2}{3}\alpha})$. We extend the results to the case of a general proper lower-semicontinuous convex function $\Phi : \mathcal H \rightarrow \mathbb R \cup \{+\infty \}$. This is based on the fact that the inertial dynamic with Hessian driven damping can be written as a first-order system in time and space. By explicit-implicit time discretization, this opens a gate to new $-$ possibly more rapid $-$ inertial algorithms, expanding the field of FISTA methods for convex structured optimization problems