2,024 research outputs found
Iteratively Linearized Reweighted Alternating Direction Method of Multipliers for a Class of Nonconvex Problems
In this paper, we consider solving a class of nonconvex and nonsmooth
problems frequently appearing in signal processing and machine learning
research. The traditional alternating direction method of multipliers
encounters troubles in both mathematics and computations in solving the
nonconvex and nonsmooth subproblem. In view of this, we propose a reweighted
alternating direction method of multipliers. In this algorithm, all subproblems
are convex and easy to solve. We also provide several guarantees for the
convergence and prove that the algorithm globally converges to a critical point
of an auxiliary function with the help of the Kurdyka-{\L}ojasiewicz property.
Several numerical results are presented to demonstrate the efficiency of the
proposed algorithm
Faster and Non-ergodic O(1/K) Stochastic Alternating Direction Method of Multipliers
We study stochastic convex optimization subjected to linear equality
constraints. Traditional Stochastic Alternating Direction Method of Multipliers
and its Nesterov's acceleration scheme can only achieve ergodic O(1/\sqrt{K})
convergence rates, where K is the number of iteration. By introducing Variance
Reduction (VR) techniques, the convergence rates improve to ergodic O(1/K). In
this paper, we propose a new stochastic ADMM which elaborately integrates
Nesterov's extrapolation and VR techniques. We prove that our algorithm can
achieve a non-ergodic O(1/K) convergence rate which is optimal for separable
linearly constrained non-smooth convex problems, while the convergence rates of
VR based ADMM methods are actually tight O(1/\sqrt{K}) in non-ergodic sense. To
the best of our knowledge, this is the first work that achieves a truly
accelerated, stochastic convergence rate for constrained convex problems. The
experimental results demonstrate that our algorithm is significantly faster
than the existing state-of-the-art stochastic ADMM methods
Linearized ADMM for Non-convex Non-smooth Optimization with Convergence Analysis
Linearized alternating direction method of multipliers (ADMM) as an extension
of ADMM has been widely used to solve linearly constrained problems in signal
processing, machine leaning, communications, and many other fields. Despite its
broad applications in nonconvex optimization, for a great number of nonconvex
and nonsmooth objective functions, its theoretical convergence guarantee is
still an open problem. In this paper, we propose a two-block linearized ADMM
and a multi-block parallel linearized ADMM for problems with nonconvex and
nonsmooth objectives. Mathematically, we present that the algorithms can
converge for a broader class of objective functions under less strict
assumptions compared with previous works. Furthermore, our proposed algorithm
can update coupled variables in parallel and work for less restrictive
nonconvex problems, where the traditional ADMM may have difficulties in solving
subproblems.Comment: 29 pages, 2 tables, 2 figure
Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming
Motivated by big data applications, first-order methods have been extremely
popular in recent years. However, naive gradient methods generally converge
slowly. Hence, much efforts have been made to accelerate various first-order
methods. This paper proposes two accelerated methods towards solving structured
linearly constrained convex programming, for which we assume composite convex
objective.
The first method is the accelerated linearized augmented Lagrangian method
(LALM). At each update to the primal variable, it allows linearization to the
differentiable function and also the augmented term, and thus it enables easy
subproblems. Assuming merely weak convexity, we show that LALM owns
convergence if parameters are kept fixed during all the iterations and can be
accelerated to if the parameters are adapted, where is the
number of total iterations.
The second method is the accelerated linearized alternating direction method
of multipliers (LADMM). In addition to the composite convexity, it further
assumes two-block structure on the objective. Different from classic ADMM, our
method allows linearization to the objective and also augmented term to make
the update simple. Assuming strong convexity on one block variable, we show
that LADMM also enjoys convergence with adaptive parameters. This
result is a significant improvement over that in [Goldstein et. al, SIIMS'14],
which requires strong convexity on both block variables and no linearization to
the objective or augmented term.
Numerical experiments are performed on quadratic programming, image
denoising, and support vector machine. The proposed accelerated methods are
compared to nonaccelerated ones and also existing accelerated methods. The
results demonstrate the validness of acceleration and superior performance of
the proposed methods over existing ones
Iteratively reweighted penalty alternating minimization methods with continuation for image deblurring
In this paper, we consider a class of nonconvex problems with linear
constraints appearing frequently in the area of image processing. We solve this
problem by the penalty method and propose the iteratively reweighted
alternating minimization algorithm. To speed up the algorithm, we also apply
the continuation strategy to the penalty parameter. A convergence result is
proved for the algorithm. Compared with the nonconvex ADMM, the proposed
algorithm enjoys both theoretical and computational advantages like weaker
convergence requirements and faster speed. Numerical results demonstrate the
efficiency of the proposed algorithm
New Analysis of Linear Convergence of Gradient-type Methods via Unifying Error Bound Conditions
This paper reveals that a common and central role, played in many error bound
(EB) conditions and a variety of gradient-type methods, is a residual measure
operator. On one hand, by linking this operator with other optimality measures,
we define a group of abstract EB conditions, and then analyze the interplay
between them; on the other hand, by using this operator as an ascent direction,
we propose an abstract gradient-type method, and then derive EB conditions that
are necessary and sufficient for its linear convergence. The former provides a
unified framework that not only allows us to find new connections between many
existing EB conditions, but also paves a way to construct new EB conditions.
The latter allows us to claim the weakest conditions guaranteeing linear
convergence for a number of fundamental algorithms, including the gradient
method, the proximal point algorithm, and the forward-backward splitting
algorithm. In addition, we show linear convergence for the proximal alternating
linearized minimization algorithm under a group of equivalent EB conditions,
which are strictly weaker than the traditional strongly convex condition.
Moreover, by defining a new EB condition, we show Q-linear convergence of
Nesterov's accelerated forward-backward algorithm without strong convexity.
Finally, we verify EB conditions for a class of dual objective functions.Comment: 40 papes; incorporating the referee's comments, the presentation has
been further improve
Iteration-complexity of a Jacobi-type non-Euclidean ADMM for multi-block linearly constrained nonconvex programs
This paper establishes the iteration-complexity of a Jacobi-type
non-Euclidean proximal alternating direction method of multipliers (ADMM) for
solving multi-block linearly constrained nonconvex programs. The subproblems of
this ADMM variant can be solved in parallel and hence the method has great
potential to solve large scale multi-block linearly constrained nonconvex
programs. Moreover, our analysis allows the Lagrange multiplier to be updated
with a relaxation parameter in the interval (0, 2)
Extended ADMM and BCD for Nonseparable Convex Minimization Models with Quadratic Coupling Terms: Convergence Analysis and Insights
In this paper, we establish the convergence of the proximal alternating
direction method of multipliers (ADMM) and block coordinate descent (BCD) for
nonseparable minimization models with quadratic coupling terms. The novel
convergence results presented in this paper answer several open questions that
have been the subject of considerable discussion. We firstly extend the 2-block
proximal ADMM to linearly constrained convex optimization with a coupled
quadratic objective function, an area where theoretical understanding is
currently lacking, and prove that the sequence generated by the proximal ADMM
converges in point-wise manner to a primal-dual solution pair. Moreover, we
apply randomly permuted ADMM (RPADMM) to nonseparable multi-block convex
optimization, and prove its expected convergence for a class of nonseparable
quadratic programming problems. When the linear constraint vanishes, the
2-block proximal ADMM and RPADMM reduce to the 2-block cyclic proximal BCD
method and randomly permuted BCD (RPBCD). Our study provides the first iterate
convergence result for 2-block cyclic proximal BCD without assuming the
boundedness of the iterates. We also theoretically establish the expected
iterate convergence result concerning multi-block RPBCD for convex quadratic
optimization. In addition, we demonstrate that RPBCD may have a worse
convergence rate than cyclic proximal BCD for 2-block convex quadratic
minimization problems. Although the results on RPADMM and RPBCD are restricted
to quadratic minimization models, they provide some interesting insights: 1)
random permutation makes ADMM and BCD more robust for multi-block convex
minimization problems; 2) cyclic BCD may outperform RPBCD for "nice" problems,
and therefore RPBCD should be applied with caution when solving general convex
optimization problems
A general inertial proximal point method for mixed variational inequality problem
In this paper, we first propose a general inertial proximal point method for
the mixed variational inequality (VI) problem. Based on our knowledge, without
stronger assumptions, convergence rate result is not known in the literature
for inertial type proximal point methods. Under certain conditions, we are able
to establish the global convergence and a convergence rate result
(under certain measure) of the proposed general inertial proximal point method.
We then show that the linearized alternating direction method of multipliers
(ADMM) for separable convex optimization with linear constraints is an
application of a general proximal point method, provided that the algorithmic
parameters are properly chosen. As byproducts of this finding, we establish
global convergence and convergence rate results of the linearized ADMM
in both ergodic and nonergodic sense. In particular, by applying the proposed
inertial proximal point method for mixed VI to linearly constrained separable
convex optimization, we obtain an inertial version of the linearized ADMM for
which the global convergence is guaranteed. We also demonstrate the effect of
the inertial extrapolation step via experimental results on the compressive
principal component pursuit problem.Comment: 21 pages, two figures, 4 table
Block-Simultaneous Direction Method of Multipliers: A proximal primal-dual splitting algorithm for nonconvex problems with multiple constraints
We introduce a generalization of the linearized Alternating Direction Method
of Multipliers to optimize a real-valued function of multiple arguments
with potentially multiple constraints on each of them. The function
may be nonconvex as long as it is convex in every argument, while the
constraints need to be convex but not smooth. If is smooth, the
proposed Block-Simultaneous Direction Method of Multipliers (bSDMM) can be
interpreted as a proximal analog to inexact coordinate descent methods under
constraints. Unlike alternative approaches for joint solvers of
multiple-constraint problems, we do not require linear operators of a
constraint function to be invertible or linked between each
other. bSDMM is well-suited for a range of optimization problems, in particular
for data analysis, where is the likelihood function of a model and
could be a transformation matrix describing e.g. finite differences or basis
transforms. We apply bSDMM to the Non-negative Matrix Factorization task of a
hyperspectral unmixing problem and demonstrate convergence and effectiveness of
multiple constraints on both matrix factors. The algorithms are implemented in
python and released as an open-source package.Comment: 13 pages, 4 figure
- β¦