753 research outputs found
A Distributed Asynchronous Method of Multipliers for Constrained Nonconvex Optimization
This paper presents a fully asynchronous and distributed approach for
tackling optimization problems in which both the objective function and the
constraints may be nonconvex. In the considered network setting each node is
active upon triggering of a local timer and has access only to a portion of the
objective function and to a subset of the constraints. In the proposed
technique, based on the method of multipliers, each node performs, when it
wakes up, either a descent step on a local augmented Lagrangian or an ascent
step on the local multiplier vector. Nodes realize when to switch from the
descent step to the ascent one through an asynchronous distributed logic-AND,
which detects when all the nodes have reached a predefined tolerance in the
minimization of the augmented Lagrangian. It is shown that the resulting
distributed algorithm is equivalent to a block coordinate descent for the
minimization of the global augmented Lagrangian. This allows one to extend the
properties of the centralized method of multipliers to the considered
distributed framework. Two application examples are presented to validate the
proposed approach: a distributed source localization problem and the parameter
estimation of a neural network.Comment: arXiv admin note: substantial text overlap with arXiv:1803.0648
An Alternating Trust Region Algorithm for Distributed Linearly Constrained Nonlinear Programs, Application to the AC Optimal Power Flow
A novel trust region method for solving linearly constrained nonlinear
programs is presented. The proposed technique is amenable to a distributed
implementation, as its salient ingredient is an alternating projected gradient
sweep in place of the Cauchy point computation. It is proven that the algorithm
yields a sequence that globally converges to a critical point. As a result of
some changes to the standard trust region method, namely a proximal
regularisation of the trust region subproblem, it is shown that the local
convergence rate is linear with an arbitrarily small ratio. Thus, convergence
is locally almost superlinear, under standard regularity assumptions. The
proposed method is successfully applied to compute local solutions to
alternating current optimal power flow problems in transmission and
distribution networks. Moreover, the new mechanism for computing a Cauchy point
compares favourably against the standard projected search as for its activity
detection properties
Dual Descent ALM and ADMM
Classical primal-dual algorithms attempt to solve by alternatively minimizing over the primal variable
through primal descent and maximizing the dual variable through dual
ascent. However, when is highly nonconvex with complex
constraints in , the minimization over may not achieve global
optimality, and hence the dual ascent step loses its valid intuition. This
observation motivates us to propose a new class of primal-dual algorithms for
nonconvex constrained optimization with the key feature to reverse dual ascent
to a conceptually new dual descent, in a sense, elevating the dual variable to
the same status as the primal variable. Surprisingly, this new dual scheme
achieves some best iteration complexities for solving nonconvex optimization
problems. In particular, when the dual descent step is scaled by a fractional
constant, we name it scaled dual descent (SDD), otherwise, unscaled dual
descent (UDD). For nonconvex multiblock optimization with nonlinear equality
constraints, we propose SDD-ADMM and show that it finds an
-stationary solution in iterations. The
complexity is further improved to and
under proper conditions. We also propose UDD-ALM,
combining UDD with ALM, for weakly convex minimization over affine constraints.
We show that UDD-ALM finds an -stationary solution in
iterations. These complexity bounds for both
algorithms either achieve or improve the best-known results in the ADMM and ALM
literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing
ADMM frameworks
Optimization Methods for Inverse Problems
Optimization plays an important role in solving many inverse problems.
Indeed, the task of inversion often either involves or is fully cast as a
solution of an optimization problem. In this light, the mere non-linear,
non-convex, and large-scale nature of many of these inversions gives rise to
some very challenging optimization problems. The inverse problem community has
long been developing various techniques for solving such optimization tasks.
However, other, seemingly disjoint communities, such as that of machine
learning, have developed, almost in parallel, interesting alternative methods
which might have stayed under the radar of the inverse problem community. In
this survey, we aim to change that. In doing so, we first discuss current
state-of-the-art optimization methods widely used in inverse problems. We then
survey recent related advances in addressing similar challenges in problems
faced by the machine learning community, and discuss their potential advantages
for solving inverse problems. By highlighting the similarities among the
optimization challenges faced by the inverse problem and the machine learning
communities, we hope that this survey can serve as a bridge in bringing
together these two communities and encourage cross fertilization of ideas.Comment: 13 page
Algorithms for Difference-of-Convex (DC) Programs Based on Difference-of-Moreau-Envelopes Smoothing
In this paper we consider minimization of a difference-of-convex (DC)
function with and without linear constraints. We first study a smooth
approximation of a generic DC function, termed difference-of-Moreau-envelopes
(DME) smoothing, where both components of the DC function are replaced by their
respective Moreau envelopes. The resulting smooth approximation is shown to be
Lipschitz differentiable, capture stationary points, local, and global minima
of the original DC function, and enjoy some growth conditions, such as
level-boundedness and coercivity, for broad classes of DC functions. We then
develop four algorithms for solving DC programs with and without linear
constraints based on the DME smoothing. In particular, for a smoothed DC
program without linear constraints, we show that the classic gradient descent
method as well as an inexact variant can obtain a stationary solution in the
limit with a convergence rate of , where is the
number of proximal evaluations of both components. Furthermore, when the DC
program is explicitly constrained in an affine subspace, we combine the
smoothing technique with the augmented Lagrangian function and derive two
variants of the augmented Lagrangian method (ALM), named LCDC-ALM and composite
LCDC-ALM, focusing on different structures of the DC objective function. We
show that both algorithms find an -approximate stationary solution of
the original DC program in iterations. Comparing
to existing methods designed for linearly constrained weakly convex
minimization, the proposed ALM-based algorithms can be applied to a broader
class of problems, where the objective contains a nonsmooth concave component.
Finally, numerical experiments are presented to demonstrate the performance of
the proposed algorithms
- …