1,437 research outputs found
Alternating Randomized Block Coordinate Descent
Block-coordinate descent algorithms and alternating minimization methods are
fundamental optimization algorithms and an important primitive in large-scale
optimization and machine learning. While various block-coordinate-descent-type
methods have been studied extensively, only alternating minimization -- which
applies to the setting of only two blocks -- is known to have convergence time
that scales independently of the least smooth block. A natural question is
then: is the setting of two blocks special?
We show that the answer is "no" as long as the least smooth block can be
optimized exactly -- an assumption that is also needed in the setting of
alternating minimization. We do so by introducing a novel algorithm AR-BCD,
whose convergence time scales independently of the least smooth (possibly
non-smooth) block. The basic algorithm generalizes both alternating
minimization and randomized block coordinate (gradient) descent, and we also
provide its accelerated version -- AAR-BCD. As a special case of AAR-BCD, we
obtain the first nontrivial accelerated alternating minimization algorithm.Comment: Version 1 appeared Proc. ICML'18. v1 -> v2: added remarks about how
accelerated alternating minimization follows directly from the results that
appeared in ICML'18; no new technical results were needed for thi
An almost cyclic 2-coordinate descent method for singly linearly constrained problems
A block decomposition method is proposed for minimizing a (possibly
non-convex) continuously differentiable function subject to one linear equality
constraint and simple bounds on the variables. The proposed method iteratively
selects a pair of coordinates according to an almost cyclic strategy that does
not use first-order information, allowing us not to compute the whole gradient
of the objective function during the algorithm. Using first-order search
directions to update each pair of coordinates, global convergence to stationary
points is established for different choices of the stepsize under an
appropriate assumption on the level set. In particular, both inexact and exact
line search strategies are analyzed. Further, linear convergence rate is proved
under standard additional assumptions. Numerical results are finally provided
to show the effectiveness of the proposed method.Comment: Computational Optimization and Application
Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimization
Consider the problem of minimizing the sum of a smooth (possibly non-convex)
and a convex (possibly nonsmooth) function involving a large number of
variables. A popular approach to solve this problem is the block coordinate
descent (BCD) method whereby at each iteration only one variable block is
updated while the remaining variables are held fixed. With the recent advances
in the developments of the multi-core parallel processing technology, it is
desirable to parallelize the BCD method by allowing multiple blocks to be
updated simultaneously at each iteration of the algorithm. In this work, we
propose an inexact parallel BCD approach where at each iteration, a subset of
the variables is updated in parallel by minimizing convex approximations of the
original objective function. We investigate the convergence of this parallel
BCD method for both randomized and cyclic variable selection rules. We analyze
the asymptotic and non-asymptotic convergence behavior of the algorithm for
both convex and non-convex objective functions. The numerical experiments
suggest that for a special case of Lasso minimization problem, the cyclic block
selection rule can outperform the randomized rule
- …