12 research outputs found
Asynchronous Optimization Over Heterogeneous Networks via Consensus ADMM
This paper considers the distributed optimization of a sum of locally
observable, non-convex functions. The optimization is performed over a
multi-agent networked system, and each local function depends only on a subset
of the variables. An asynchronous and distributed alternating directions method
of multipliers (ADMM) method that allows the nodes to defer or skip the
computation and transmission of updates is proposed in the paper. The proposed
algorithm utilizes different approximations in the update step, resulting in
proximal and majorized ADMM variants. Both variants are shown to converge to a
local minimum, under certain regularity conditions. The proposed asynchronous
algorithms are also applied to the problem of cooperative localization in
wireless ad hoc networks, where it is shown to outperform the other
state-of-the-art localization algorithms.Comment: Submitted to Transactions on signal and information processing over
Network
Impact of Communication Delay on Asynchronous Distributed Optimal Power Flow Using ADMM
Distributed optimization has attracted lots of attention in the operation of
power systems in recent years, where a large area is decomposed into smaller
control regions each solving a local optimization problem with periodic
information exchange with neighboring regions. However, most distributed
optimization methods are iterative and require synchronization of all regions
at each iteration, which is hard to achieve without a centralized coordinator
and might lead to under-utilization of computation resources due to the
heterogeneity of the regions. To address such limitations of synchronous
schemes, this paper investigates the applicability of asynchronous distributed
optimization methods to power system optimization. Particularly, we focus on
solving the AC Optimal Power Flow problem and propose an algorithmic framework
based on the Alternating Direction Method of Multipliers (ADMM) method that
allows the regions to perform local updates with information received from a
subset of but not all neighbors. Through experimental studies, we demonstrate
that the convergence performance of the proposed asynchronous scheme is
dependent on the communication delay of passing messages among the regions.
Under mild communication delays, the proposed scheme can achieve comparable or
even faster convergence compared with its synchronous counterpart, which can be
used as a good alternative to centralized or synchronous distributed
optimization approaches.Comment: SmartGridComm 201
Asynchronous ADMM for Distributed Non-Convex Optimization in Power Systems
Large scale, non-convex optimization problems arising in many complex
networks such as the power system call for efficient and scalable distributed
optimization algorithms. Existing distributed methods are usually iterative and
require synchronization of all workers at each iteration, which is hard to
scale and could result in the under-utilization of computation resources due to
the heterogeneity of the subproblems. To address those limitations of
synchronous schemes, this paper proposes an asynchronous distributed
optimization method based on the Alternating Direction Method of Multipliers
(ADMM) for non-convex optimization. The proposed method only requires local
communications and allows each worker to perform local updates with information
from a subset of but not all neighbors. We provide sufficient conditions on the
problem formulation, the choice of algorithm parameter and network delay, and
show that under those mild conditions, the proposed asynchronous ADMM method
asymptotically converges to the KKT point of the non-convex problem. We
validate the effectiveness of asynchronous ADMM by applying it to the Optimal
Power Flow problem in multiple power systems and show that the convergence of
the proposed asynchronous scheme could be faster than its synchronous
counterpart in large-scale applications
Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
Alternating Direction Method of Multipliers (ADMM) has recently been proposed
as a potential alternative optimizer to the Stochastic Gradient Descent(SGD)
for deep learning problems. This is because ADMM can solve gradient vanishing
and poor conditioning problems. Moreover, it has shown good scalability in many
large-scale deep learning applications. However, there still lacks a parallel
ADMM computational framework for deep neural networks because of layer
dependency among variables. In this paper, we propose a novel parallel deep
learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in
each layer of neural networks can be updated independently in parallel. The
convergence of the proposed pdADMM to a critical point is theoretically proven
under mild conditions. The convergence rate of the pdADMM is proven to be
where is the number of iterations. Extensive experiments on six
benchmark datasets demonstrated that our proposed pdADMM can lead to more than
10 times speedup for training large-scale deep neural networks, and
outperformed most of the comparison methods. Our code is available at:
https://github.com/xianggebenben/pdADMM.Comment: ICDM202
Distributed Optimization with Coupling Constraints via Dual Proximal Gradient Method with Applications to Asynchronous Networks
In this paper, we consider solving a distributed optimization problem (DOP)
with coupling constraints in a multi-agent network based on proximal gradient
method. In this problem, each agent aims to minimize an individual cost
function composed of both smooth and non-smooth parts. To this end, we derive
the dual problem by the concept of Fenchel conjugate, which results in two
kinds of dual problems: consensus based constrained and augmented unconstrained
problems. In the first scenario, we propose a fully distributed dual proximal
gradient (D-DPG) algorithm, where the agents can make updates only with the
dual information of their neighbours and local step-sizes. Moreover, if the
non-smooth parts of the objective functions are with certain simple structures,
the agents only need to update dual variables with some simple operations,
which can reduce the overall computational complexity. In the second scenario,
an augmented dual proximal gradient (A-DPG) algorithm is proposed, which allows
for the asymmetric interpretations of the global constraints for the agents and
can be more efficient than D-DGP algorithm in some special-structured DOPs.
Based on A-DPG algorithm, an asynchronous dual proximal gradient (Asyn-DPG)
algorithm is proposed for the asynchronous networks where each agent updates
its strategy with heterogenous step-size and possible outdated dual information
of others. In all the discussed scenarios, analytical (ergodic) convergence
rates are derived. The effectiveness of the proposed algorithms is verified by
solving a social welfare optimization problem in the electricity market.Comment: 20 pages, 11 figure
Distributed Inexact Successive Convex Approximation ADMM: Analysis-Part I
In this two-part work, we propose an algorithmic framework for solving
non-convex problems whose objective function is the sum of a number of smooth
component functions plus a convex (possibly non-smooth) or/and smooth (possibly
non-convex) regularization function. The proposed algorithm incorporates ideas
from several existing approaches such as alternate direction method of
multipliers (ADMM), successive convex approximation (SCA), distributed and
asynchronous algorithms, and inexact gradient methods. Different from a number
of existing approaches, however, the proposed framework is flexible enough to
incorporate a class of non-convex objective functions, allow distributed
operation with and without a fusion center, and include variance reduced
methods as special cases. Remarkably, the proposed algorithms are robust to
uncertainties arising from random, deterministic, and adversarial sources. The
part I of the paper develops two variants of the algorithm under very mild
assumptions and establishes first-order convergence rate guarantees. The proof
developed here allows for generic errors and delays, paving the way for
different variance-reduced, asynchronous, and stochastic implementations,
outlined and evaluated in part II
Asynchronous Decentralized Successive Convex Approximation
We study decentralized asynchronous multiagent optimization over networks,
modeled as static (possibly directed) graphs. The optimization problem consists
of minimizing a (possibly nonconvex) smooth function--the sum of the agents'
local costs--plus a convex (possibly nonsmooth) regularizer, subject to convex
constraints. Agents can perform their local computations as well as communicate
with their immediate neighbors at any time, without any form of coordination or
centralized scheduling; furthermore, when solving their local subproblems, they
can use outdated information from their neighbors. We propose the first
distributed asynchronous algorithm, termed ASY-DSCA, that converges at an
R-linear rate to the optimal solution of convex problems whose objective
function satisfies a general error bound condition; this condition is weaker
than the more frequently used strong convexity, and it is satisfied by several
empirical risk functions that are not strongly convex; examples include LASSO
and logistic regression problems. When the objective function is nonconvex,
ASY-DSCA converges to a stationary solution of the problem at a sublinear rate
Asynchronous Gradient-Push
We consider a multi-agent framework for distributed optimization where each
agent has access to a local smooth strongly convex function, and the collective
goal is to achieve consensus on the parameters that minimize the sum of the
agents' local functions. We propose an algorithm wherein each agent operates
asynchronously and independently of the other agents. When the local functions
are strongly-convex with Lipschitz-continuous gradients, we show that the
iterates at each agent converge to a neighborhood of the global minimum, where
the neighborhood size depends on the degree of asynchrony in the multi-agent
network. When the agents work at the same rate, convergence to the global
minimizer is achieved. Numerical experiments demonstrate that Asynchronous
Gradient-Push can minimize the global objective faster than state-of-the-art
synchronous first-order methods, is more robust to failing or stalling agents,
and scales better with the network size.Comment: 33 pages, 9 figures, accepted to IEEE Transactions on Automatic
Contro
Composite Optimization with Coupling Constraints via Penalized Proximal Gradient Method in Partially Asynchronous Networks
In this paper, we consider a composite optimization problem with linear
coupling constraints in a multi-agent network. In this problem, all the agents
jointly optimize a global composite cost function which is the linear sum of
individual cost functions composed of both smooth and non-smooth components. To
solve this problem, we propose an asynchronous penalized proximal gradient
(Asyn-PPG) algorithm, a variant of classical proximal gradient method, by
considering the asynchronous update instants of the agents and communication
delays in the network. Specifically, we consider a slot-based asynchronous
network (SAN), where the whole time domain is split into sequential time slots
and each agent is permitted to make multiple updates during a slot by accessing
the historical state information of others. Moreover, we consider a set of
global linear constraints and impose some violation penalties on the updating
algorithms. By the Asyn-PPG algorithm, we will show that a periodically
convergence with rate O(1/K) (K is the index of time slots) can be guaranteed
if the coefficient of the penalties for all agents is synchronized at the end
of the time slots and the step-size of the Asyn-PPG algorithm is properly
determined. The feasibility of the proposed algorithm is verified by solving a
consensus based distributed LASSO problem and a social welfare optimization
problem in the electricity market respectively.Comment: 16 pages, 8 figure
Distributed Optimization for Smart Cyber-Physical Networks
The presence of embedded electronics and communication capabilities as well as sensing and control in smart devices has given rise to the novel concept of cyber-physical networks, in which agents aim at cooperatively solving complex tasks by local computation and communication. Numerous estimation, learning, decision and control tasks in smart networks involve the solution of large-scale, structured optimization problems in which network agents have only a partial knowledge of the whole problem. Distributed optimization aims at designing local computation and communication rules for the network processors allowing them to cooperatively solve the global optimization problem without relying on any central unit. The purpose of this survey is to provide an introduction to distributed optimization methodologies. Principal approaches, namely (primal) consensus-based, duality-based and constraint exchange methods, are formalized. An analysis of the basic schemes is supplied, and state-of-the-art extensions are reviewed