324 research outputs found
Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting
The Augmented Lagragian Method (ALM) and Alternating Direction Method of
Multiplier (ADMM) have been powerful optimization methods for general convex
programming subject to linear constraint. We consider the convex problem whose
objective consists of a smooth part and a nonsmooth but simple part. We propose
the Fast Proximal Augmented Lagragian Method (Fast PALM) which achieves the
convergence rate , compared with by the traditional PALM. In
order to further reduce the per-iteration complexity and handle the
multi-blocks problem, we propose the Fast Proximal ADMM with Parallel Splitting
(Fast PL-ADMM-PS) method. It also partially improves the rate related to the
smooth part of the objective function. Experimental results on both synthesized
and real world data demonstrate that our fast methods significantly improve the
previous PALM and ADMM.Comment: AAAI 201
Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning
Many problems in machine learning and other fields can be (re)for-mulated as
linearly constrained separable convex programs. In most of the cases, there are
multiple blocks of variables. However, the traditional alternating direction
method (ADM) and its linearized version (LADM, obtained by linearizing the
quadratic penalty term) are for the two-block case and cannot be naively
generalized to solve the multi-block case. So there is great demand on
extending the ADM based methods for the multi-block case. In this paper, we
propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve
multi-block separable convex programs efficiently. When all the component
objective functions have bounded subgradients, we obtain convergence results
that are stronger than those of ADM and LADM, e.g., allowing the penalty
parameter to be unbounded and proving the sufficient and necessary conditions}
for global convergence. We further propose a simple optimality measure and
reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with
extra convex set constraints, with refined parameter estimation we devise a
practical version of LADMPSAP for faster convergence. Finally, we generalize
LADMPSAP to handle programs with more difficult objective functions by
linearizing part of the objective function as well. LADMPSAP is particularly
suitable for sparse representation and low-rank recovery problems because its
subproblems have closed form solutions and the sparsity and low-rankness of the
iterates can be preserved during the iteration. It is also highly
parallelizable and hence fits for parallel or distributed computing. Numerical
experiments testify to the advantages of LADMPSAP in speed and numerical
accuracy.Comment: Preliminary version published on Asian Conference on Machine Learning
201
- …