3 research outputs found
Asynchronous Coordinate Descent under More Realistic Assumptions
Asynchronous-parallel algorithms have the potential to vastly speed up
algorithms by eliminating costly synchronization. However, our understanding to
these algorithms is limited because the current convergence of asynchronous
(block) coordinate descent algorithms are based on somewhat unrealistic
assumptions. In particular, the age of the shared optimization variables being
used to update a block is assumed to be independent of the block being updated.
Also, it is assumed that the updates are applied to randomly chosen blocks. In
this paper, we argue that these assumptions either fail to hold or will imply
less efficient implementations. We then prove the convergence of
asynchronous-parallel block coordinate descent under more realistic
assumptions, in particular, always without the independence assumption. The
analysis permits both the deterministic (essentially) cyclic and random rules
for block choices. Because a bound on the asynchronous delays may or may not be
available, we establish convergence for both bounded delays and unbounded
delays. The analysis also covers nonconvex, weakly convex, and strongly convex
functions. We construct Lyapunov functions that directly model both objective
progress and delays, so delays are not treated errors or noise. A
continuous-time ODE is provided to explain the construction at a high level
Asynchronous parallel primal-dual block coordinate update methods for affinely constrained convex programs
Recent several years have witnessed the surge of asynchronous (async-)
parallel computing methods due to the extremely big data involved in many
modern applications and also the advancement of multi-core machines and
computer clusters. In optimization, most works about async-parallel methods are
on unconstrained problems or those with block separable constraints.
In this paper, we propose an async-parallel method based on block coordinate
update (BCU) for solving convex problems with nonseparable linear constraint.
Running on a single node, the method becomes a novel randomized primal-dual BCU
with adaptive stepsize for multi-block affinely constrained problems. For these
problems, Gauss-Seidel cyclic primal-dual BCU needs strong convexity to have
convergence. On the contrary, merely assuming convexity, we show that the
objective value sequence generated by the proposed algorithm converges in
probability to the optimal value and also the constraint residual to zero. In
addition, we establish an ergodic convergence result, where is the
number of iterations. Numerical experiments are performed to demonstrate the
efficiency of the proposed method and significantly better speed-up performance
than its sync-parallel counterpart
First-order methods for constrained convex programming based on linearized augmented Lagrangian function
First-order methods have been popularly used for solving large-scale
problems. However, many existing works only consider unconstrained problems or
those with simple constraint. In this paper, we develop two first-order methods
for constrained convex programs, for which the constraint set is represented by
affine equations and smooth nonlinear inequalities. Both methods are based on
the classic augmented Lagrangian function. They update the multipliers in the
same way as the augmented Lagrangian method (ALM) but employ different primal
variable updates. The first method, at each iteration, performs a single
proximal gradient step to the primal variable, and the second method is a block
update version of the first one.
For the first method, we establish its global iterate convergence as well as
global sublinear and local linear convergence, and for the second method, we
show a global sublinear convergence result in expectation. Numerical
experiments are carried out on the basis pursuit denoising and a convex
quadratically constrained quadratic program to show the empirical performance
of the proposed methods. Their numerical behaviors closely match the
established theoretical results