25,588 research outputs found
Accelerated linearized alternating direction method of multipliers with Nesterov extrapolation
The alternating direction method of multipliers (ADMMs) has found widespread
use in solving separable convex optimization problems. In this paper, by
employing Nesterov extrapolation technique, we propose two families of
accelerated linearized ADMMsfor addressing two-block linearly constrained
separable convex optimization problems where each block of the objective
function exhibits a `nonsmooth' plus `smooth' composite structure. Our proposed
accelerated linearized ADMMs extend two classical Nesterov acceleration methods
designed for unconstrained composite optimization problems to linearly
constrained problems. These methods are capable of achieving non-ergodic
convergence rates of O(1/k^2) provided that one block of the objective function
exhibits strong convexity and the gradients of smooth terms are Lipschitz
continuous. We show that the proposed methods can reduce to accelerated
linearized augmented Lagrangian methods (ALMs) with non-ergodic O(1/k2 )
convergence rates for solving one-block linearly constrained convex
optimization problems. Furthermore, we explore hybrid versions of the proposed
linearized accelerated ADMMs, which also demonstrate non-ergodic convergence
rates of O(1/k2) under the same assumption. We also investigate inexact
versions of the proposed methods. Under mild assumptions regarding error
sequences, these methods still maintain O(1/k2) non-ergodic convergence rates.
By choosing different extrapolation parameters, we explore the relationship
between the proposed methods and existing accelerated methods
Randomized Smoothing for Stochastic Optimization
We analyze convergence rates of stochastic optimization procedures for
non-smooth convex optimization problems. By combining randomized smoothing
techniques with accelerated gradient methods, we obtain convergence rates of
stochastic optimization procedures, both in expectation and with high
probability, that have optimal dependence on the variance of the gradient
estimates. To the best of our knowledge, these are the first variance-based
rates for non-smooth optimization. We give several applications of our results
to statistical estimation problems, and provide experimental results that
demonstrate the effectiveness of the proposed algorithms. We also describe how
a combination of our algorithm with recent work on decentralized optimization
yields a distributed stochastic optimization algorithm that is order-optimal.Comment: 39 pages, 3 figure
- …