173 research outputs found
On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization
Extrapolation is a well-known technique for solving convex optimization and
variational inequalities and recently attracts some attention for non-convex
optimization. Several recent works have empirically shown its success in some
machine learning tasks. However, it has not been analyzed for non-convex
minimization and there still remains a gap between the theory and the practice.
In this paper, we analyze gradient descent and stochastic gradient descent with
extrapolation for finding an approximate first-order stationary point in smooth
non-convex optimization problems. Our convergence upper bounds show that the
algorithms with extrapolation can be accelerated than without extrapolation
The Forward-Backward-Forward Method from continuous and discrete perspective for pseudo-monotone variational inequalities in Hilbert spaces
Tseng's forward-backward-forward algorithm is a valuable alternative for
Korpelevich's extragradient method when solving variational inequalities over a
convex and closed set governed by monotone and Lipschitz continuous operators,
as it requires in every step only one projection operation. However, it is
well-known that Korpelevich's method converges and can therefore be used also
for solving variational inequalities governed by pseudo-monotone and Lipschitz
continuous operators. In this paper, we first associate to a pseudo-monotone
variational inequality a forward-backward-forward dynamical system and carry
out an asymptotic analysis for the generated trajectories. The explicit time
discretization of this system results into Tseng's forward-backward-forward
algorithm with relaxation parameters, which we prove to converge also when it
is applied to pseudo-monotone variational inequalities. In addition, we show
that linear convergence is guaranteed under strong pseudo-monotonicity.
Numerical experiments are carried out for pseudo-monotone variational
inequalities over polyhedral sets and fractional programming problems
- …