338 research outputs found
On the Convergence of Alternating Direction Lagrangian Methods for Nonconvex Structured Optimization Problems
Nonconvex and structured optimization problems arise in many engineering
applications that demand scalable and distributed solution methods. The study
of the convergence properties of these methods is in general difficult due to
the nonconvexity of the problem. In this paper, two distributed solution
methods that combine the fast convergence properties of augmented
Lagrangian-based methods with the separability properties of alternating
optimization are investigated. The first method is adapted from the classic
quadratic penalty function method and is called the Alternating Direction
Penalty Method (ADPM). Unlike the original quadratic penalty function method,
in which single-step optimizations are adopted, ADPM uses an alternating
optimization, which in turn makes it scalable. The second method is the
well-known Alternating Direction Method of Multipliers (ADMM). It is shown that
ADPM for nonconvex problems asymptotically converges to a primal feasible point
under mild conditions and an additional condition ensuring that it
asymptotically reaches the standard first order necessary conditions for local
optimality are introduced. In the case of the ADMM, novel sufficient conditions
under which the algorithm asymptotically reaches the standard first order
necessary conditions are established. Based on this, complete convergence of
ADMM for a class of low dimensional problems are characterized. Finally, the
results are illustrated by applying ADPM and ADMM to a nonconvex localization
problem in wireless sensor networks.Comment: 13 pages, 6 figure
Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problems
Optimization methods are at the core of many problems in signal/image
processing, computer vision, and machine learning. For a long time, it has been
recognized that looking at the dual of an optimization problem may drastically
simplify its solution. Deriving efficient strategies which jointly brings into
play the primal and the dual problems is however a more recent idea which has
generated many important new contributions in the last years. These novel
developments are grounded on recent advances in convex analysis, discrete
optimization, parallel processing, and non-smooth optimization with emphasis on
sparsity issues. In this paper, we aim at presenting the principles of
primal-dual approaches, while giving an overview of numerical methods which
have been proposed in different contexts. We show the benefits which can be
drawn from primal-dual algorithms both for solving large-scale convex
optimization problems and discrete ones, and we provide various application
examples to illustrate their usefulness
Convolutional Dictionary Learning: Acceleration and Convergence
Convolutional dictionary learning (CDL or sparsifying CDL) has many
applications in image processing and computer vision. There has been growing
interest in developing efficient algorithms for CDL, mostly relying on the
augmented Lagrangian (AL) method or the variant alternating direction method of
multipliers (ADMM). When their parameters are properly tuned, AL methods have
shown fast convergence in CDL. However, the parameter tuning process is not
trivial due to its data dependence and, in practice, the convergence of AL
methods depends on the AL parameters for nonconvex CDL problems. To moderate
these problems, this paper proposes a new practically feasible and convergent
Block Proximal Gradient method using a Majorizer (BPG-M) for CDL. The
BPG-M-based CDL is investigated with different block updating schemes and
majorization matrix designs, and further accelerated by incorporating some
momentum coefficient formulas and restarting techniques. All of the methods
investigated incorporate a boundary artifacts removal (or, more generally,
sampling) operator in the learning model. Numerical experiments show that,
without needing any parameter tuning process, the proposed BPG-M approach
converges more stably to desirable solutions of lower objective values than the
existing state-of-the-art ADMM algorithm and its memory-efficient variant do.
Compared to the ADMM approaches, the BPG-M method using a multi-block updating
scheme is particularly useful in single-threaded CDL algorithm handling large
datasets, due to its lower memory requirement and no polynomial computational
complexity. Image denoising experiments show that, for relatively strong
additive white Gaussian noise, the filters learned by BPG-M-based CDL
outperform those trained by the ADMM approach.Comment: 21 pages, 7 figures, submitted to IEEE Transactions on Image
Processin
Nonconvex Generalization of ADMM for Nonlinear Equality Constrained Problems
The ever-increasing demand for efficient and distributed optimization
algorithms for large-scale data has led to the growing popularity of the
Alternating Direction Method of Multipliers (ADMM). However, although the use
of ADMM to solve linear equality constrained problems is well understood, we
lacks a generic framework for solving problems with nonlinear equality
constraints, which are common in practical applications (e.g., spherical
constraints). To address this problem, we are proposing a new generic ADMM
framework for handling nonlinear equality constraints, neADMM. After
introducing the generalized problem formulation and the neADMM algorithm, the
convergence properties of neADMM are discussed, along with its sublinear
convergence rate , where is the number of iterations. Next, two
important applications of neADMM are considered and the paper concludes by
describing extensive experiments on several synthetic and real-world datasets
to demonstrate the convergence and effectiveness of neADMM compared to existing
state-of-the-art methods
- …