1 research outputs found
Nonlinear Acceleration of Momentum and Primal-Dual Algorithms
We describe convergence acceleration schemes for multistep optimization
algorithms. The extrapolated solution is written as a nonlinear average of the
iterates produced by the original optimization method. Our analysis does not
need the underlying fixed-point operator to be symmetric, hence handles e.g.
algorithms with momentum terms such as Nesterov's accelerated method, or
primal-dual methods. The weights are computed via a simple linear system and we
analyze performance in both online and offline modes. We use Crouzeix's
conjecture to show that acceleration performance is controlled by the solution
of a Chebyshev problem on the numerical range of a non-symmetric operator
modeling the behavior of iterates near the optimum. Numerical experiments are
detailed on logistic regression problems