22 research outputs found
Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning
Many problems in machine learning and other fields can be (re)for-mulated as
linearly constrained separable convex programs. In most of the cases, there are
multiple blocks of variables. However, the traditional alternating direction
method (ADM) and its linearized version (LADM, obtained by linearizing the
quadratic penalty term) are for the two-block case and cannot be naively
generalized to solve the multi-block case. So there is great demand on
extending the ADM based methods for the multi-block case. In this paper, we
propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve
multi-block separable convex programs efficiently. When all the component
objective functions have bounded subgradients, we obtain convergence results
that are stronger than those of ADM and LADM, e.g., allowing the penalty
parameter to be unbounded and proving the sufficient and necessary conditions}
for global convergence. We further propose a simple optimality measure and
reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with
extra convex set constraints, with refined parameter estimation we devise a
practical version of LADMPSAP for faster convergence. Finally, we generalize
LADMPSAP to handle programs with more difficult objective functions by
linearizing part of the objective function as well. LADMPSAP is particularly
suitable for sparse representation and low-rank recovery problems because its
subproblems have closed form solutions and the sparsity and low-rankness of the
iterates can be preserved during the iteration. It is also highly
parallelizable and hence fits for parallel or distributed computing. Numerical
experiments testify to the advantages of LADMPSAP in speed and numerical
accuracy.Comment: Preliminary version published on Asian Conference on Machine Learning
201
Relaxed Majorization-Minimization for Non-smooth and Non-convex Optimization
We propose a new majorization-minimization (MM) method for non-smooth and
non-convex programs, which is general enough to include the existing MM
methods. Besides the local majorization condition, we only require that the
difference between the directional derivatives of the objective function and
its surrogate function vanishes when the number of iterations approaches
infinity, which is a very weak condition. So our method can use a surrogate
function that directly approximates the non-smooth objective function. In
comparison, all the existing MM methods construct the surrogate function by
approximating the smooth component of the objective function. We apply our
relaxed MM methods to the robust matrix factorization (RMF) problem with
different regularizations, where our locally majorant algorithm shows
advantages over the state-of-the-art approaches for RMF. This is the first
algorithm for RMF ensuring, without extra assumptions, that any limit point of
the iterates is a stationary point.Comment: AAAI1