78 research outputs found
On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization
Extrapolation is a well-known technique for solving convex optimization and
variational inequalities and recently attracts some attention for non-convex
optimization. Several recent works have empirically shown its success in some
machine learning tasks. However, it has not been analyzed for non-convex
minimization and there still remains a gap between the theory and the practice.
In this paper, we analyze gradient descent and stochastic gradient descent with
extrapolation for finding an approximate first-order stationary point in smooth
non-convex optimization problems. Our convergence upper bounds show that the
algorithms with extrapolation can be accelerated than without extrapolation
Principled Analyses and Design of First-Order Methods with Inexact Proximal Operators
Proximal operations are among the most common primitives appearing in both
practical and theoretical (or high-level) optimization methods. This basic
operation typically consists in solving an intermediary (hopefully simpler)
optimization problem. In this work, we survey notions of inaccuracies that can
be used when solving those intermediary optimization problems. Then, we show
that worst-case guarantees for algorithms relying on such inexact proximal
operations can be systematically obtained through a generic procedure based on
semidefinite programming. This methodology is primarily based on the approach
introduced by Drori and Teboulle (Mathematical Programming, 2014) and on convex
interpolation results, and allows producing non-improvable worst-case analyzes.
In other words, for a given algorithm, the methodology generates both
worst-case certificates (i.e., proofs) and problem instances on which those
bounds are achieved.
Relying on this methodology, we provide three new methods with conceptually
simple proofs: (i) an optimized relatively inexact proximal point method, (ii)
an extension of the hybrid proximal extragradient method of Monteiro and
Svaiter (SIAM Journal on Optimization, 2013), and (iii) an inexact accelerated
forward-backward splitting supporting backtracking line-search, and both (ii)
and (iii) supporting possibly strongly convex objectives. Finally, we use the
methodology for studying a recent inexact variant of the Douglas-Rachford
splitting due to Eckstein and Yao (Mathematical Programming, 2018).
We showcase and compare the different variants of the accelerated inexact
forward-backward method on a factorization and a total variation problem.Comment: Minor modifications including acknowledgments and references. Code
available at https://github.com/mathbarre/InexactProximalOperator
A note on approximate accelerated forward-backward methods with absolute and relative errors, and possibly strongly convex objectives
In this short note, we provide a simple version of an accelerated
forward-backward method (a.k.a. Nesterov's accelerated proximal gradient
method) possibly relying on approximate proximal operators and allowing to
exploit strong convexity of the objective function. The method supports both
relative and absolute errors, and its behavior is illustrated on a set of
standard numerical experiments. Using the same developments, we further provide
a version of the accelerated proximal hybrid extragradient method of Monteiro
and Svaiter (2013) possibly exploiting strong convexity of the objective
function.Comment: Minor modifications in notations and acknowledgments. These methods
were originally presented in arXiv:2006.06041v2. Code available at
https://github.com/mathbarre/StronglyConvexForwardBackwar
Iterative Methods for the Elasticity Imaging Inverse Problem
Cancers of the soft tissue reign among the deadliest diseases throughout the world and effective treatments for such cancers rely on early and accurate detection of tumors within the interior of the body. One such diagnostic tool, known as elasticity imaging or elastography, uses measurements of tissue displacement to reconstruct the variable elasticity between healthy and unhealthy tissue inside the body. This gives rise to a challenging parameter identification inverse problem, that of identifying the Lamé parameter μ in a system of partial differential equations in linear elasticity. Due to the near incompressibility of human tissue, however, common techniques for solving the direct and inverse problems are rendered ineffective due to a phenomenon known as the “locking effect”. Alternative methods, such as mixed finite element methods, must be applied to overcome this complication. Using these methods, this work reposes the problem as a generalized saddle point problem along with a presentation of several optimization formulations, including the modified output least squares (MOLS), energy output least squares (EOLS), and equation error (EE) frameworks, for solving the elasticity imaging inverse problem. Subsequently, numerous iterative optimization methods, including gradient, extragradient, and proximal point methods, are explored and applied to solve the related optimization problem. Implementations of all of the iterative techniques under consideration are applied to all of the developed optimization frameworks using a representative numerical example in elasticity imaging. A thorough analysis and comparison of the methods is subsequently presented
- …