Article thumbnail

A Generic Acceleration Framework for Stochastic Composite Optimization

By Andrei Kulunchakov and Julien Mairal

Abstract

International audienceIn this paper, we introduce various mechanisms to obtain accelerated first-order stochastic optimization algorithms when the objective function is convex or strongly convex. Specifically, we extend the Catalyst approach originally designed for deterministic objectives to the stochastic setting. Given an optimization method with mild convergence guarantees for strongly convex problems, the challenge is to accelerate convergence to a noise-dominated region, and then achieve convergence with an optimal worst-case complexity depending on the noise variance of the gradients. A side contribution of our work is also a generic analysis that can handle inexact proximal operators, providing new insights about the robustness of stochastic algorithms when the proximal operator cannot be exactly computed

Topics: [STAT.ML]Statistics [stat]/Machine Learning [stat.ML], [INFO.INFO-LG]Computer Science [cs]/Machine Learning [cs.LG]
Publisher: Curran Associates
Year: 2019
OAI identifier: oai:HAL:hal-02139489v3
Download PDF:
Sorry, we are unable to provide the full text but you may find it at the following location(s):
  • https://hal.inria.fr/hal-02139... (external link)
  • https://hal.inria.fr/hal-02139... (external link)
  • https://hal.inria.fr/hal-02139... (external link)
  • Suggested articles


    To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.