402 research outputs found
A Block-wise, Asynchronous and Distributed ADMM Algorithm for General Form Consensus Optimization
Many machine learning models, including those with non-smooth regularizers,
can be formulated as consensus optimization problems, which can be solved by
the alternating direction method of multipliers (ADMM). Many recent efforts
have been made to develop asynchronous distributed ADMM to handle large amounts
of training data. However, all existing asynchronous distributed ADMM methods
are based on full model updates and require locking all global model parameters
to handle concurrency, which essentially serializes the updates from different
workers. In this paper, we present a novel block-wise, asynchronous and
distributed ADMM algorithm, which allows different blocks of model parameters
to be updated in parallel. The lock-free block-wise algorithm may greatly
speedup sparse optimization problems, a common scenario in reality, in which
most model updates only modify a subset of all decision variables. We
theoretically prove the convergence of our proposed algorithm to stationary
points for non-convex general form consensus problems with possibly non-smooth
regularizers. We implement the proposed ADMM algorithm on the Parameter Server
framework and demonstrate its convergence and near-linear speedup performance
as the number of workers increases
Asynchronous ADMM for Distributed Non-Convex Optimization in Power Systems
Large scale, non-convex optimization problems arising in many complex
networks such as the power system call for efficient and scalable distributed
optimization algorithms. Existing distributed methods are usually iterative and
require synchronization of all workers at each iteration, which is hard to
scale and could result in the under-utilization of computation resources due to
the heterogeneity of the subproblems. To address those limitations of
synchronous schemes, this paper proposes an asynchronous distributed
optimization method based on the Alternating Direction Method of Multipliers
(ADMM) for non-convex optimization. The proposed method only requires local
communications and allows each worker to perform local updates with information
from a subset of but not all neighbors. We provide sufficient conditions on the
problem formulation, the choice of algorithm parameter and network delay, and
show that under those mild conditions, the proposed asynchronous ADMM method
asymptotically converges to the KKT point of the non-convex problem. We
validate the effectiveness of asynchronous ADMM by applying it to the Optimal
Power Flow problem in multiple power systems and show that the convergence of
the proposed asynchronous scheme could be faster than its synchronous
counterpart in large-scale applications
Asynchronous Distributed ADMM for Large-Scale Optimization- Part I: Algorithm and Convergence Analysis
Aiming at solving large-scale learning problems, this paper studies
distributed optimization methods based on the alternating direction method of
multipliers (ADMM). By formulating the learning problem as a consensus problem,
the ADMM can be used to solve the consensus problem in a fully parallel fashion
over a computer network with a star topology. However, traditional synchronized
computation does not scale well with the problem size, as the speed of the
algorithm is limited by the slowest workers. This is particularly true in a
heterogeneous network where the computing nodes experience different
computation and communication delays. In this paper, we propose an asynchronous
distributed ADMM (AD-AMM) which can effectively improve the time efficiency of
distributed optimization. Our main interest lies in analyzing the convergence
conditions of the AD-ADMM, under the popular partially asynchronous model,
which is defined based on a maximum tolerable delay of the network.
Specifically, by considering general and possibly non-convex cost functions, we
show that the AD-ADMM is guaranteed to converge to the set of
Karush-Kuhn-Tucker (KKT) points as long as the algorithm parameters are chosen
appropriately according to the network delay. We further illustrate that the
asynchrony of the ADMM has to be handled with care, as slightly modifying the
implementation of the AD-ADMM can jeopardize the algorithm convergence, even
under a standard convex setting.Comment: 37 page
A Partition-Based Implementation of the Relaxed ADMM for Distributed Convex Optimization over Lossy Networks
In this paper we propose a distributed implementation of the relaxed
Alternating Direction Method of Multipliers algorithm (R-ADMM) for optimization
of a separable convex cost function, whose terms are stored by a set of
interacting agents, one for each agent. Specifically the local cost stored by
each node is in general a function of both the state of the node and the states
of its neighbors, a framework that we refer to as `partition-based'
optimization. This framework presents a great flexibility and can be adapted to
a large number of different applications. We show that the partition-based
R-ADMM algorithm we introduce is linked to the relaxed Peaceman-Rachford
Splitting (R-PRS) operator which, historically, has been introduced in the
literature to find the zeros of sum of functions. Interestingly, making use of
non expansive operator theory, the proposed algorithm is shown to be provably
robust against random packet losses that might occur in the communication
between neighboring nodes. Finally, the effectiveness of the proposed algorithm
is confirmed by a set of compelling numerical simulations run over random
geometric graphs subject to i.i.d. random packet losses.Comment: Full version of the paper to be presented at Conference on Decision
and Control (CDC) 201
Impact of Communication Delay on Asynchronous Distributed Optimal Power Flow Using ADMM
Distributed optimization has attracted lots of attention in the operation of
power systems in recent years, where a large area is decomposed into smaller
control regions each solving a local optimization problem with periodic
information exchange with neighboring regions. However, most distributed
optimization methods are iterative and require synchronization of all regions
at each iteration, which is hard to achieve without a centralized coordinator
and might lead to under-utilization of computation resources due to the
heterogeneity of the regions. To address such limitations of synchronous
schemes, this paper investigates the applicability of asynchronous distributed
optimization methods to power system optimization. Particularly, we focus on
solving the AC Optimal Power Flow problem and propose an algorithmic framework
based on the Alternating Direction Method of Multipliers (ADMM) method that
allows the regions to perform local updates with information received from a
subset of but not all neighbors. Through experimental studies, we demonstrate
that the convergence performance of the proposed asynchronous scheme is
dependent on the communication delay of passing messages among the regions.
Under mild communication delays, the proposed scheme can achieve comparable or
even faster convergence compared with its synchronous counterpart, which can be
used as a good alternative to centralized or synchronous distributed
optimization approaches.Comment: SmartGridComm 201
ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates
Finding a fixed point to a nonexpansive operator, i.e., , abstracts
many problems in numerical linear algebra, optimization, and other areas of
scientific computing. To solve fixed-point problems, we propose ARock, an
algorithmic framework in which multiple agents (machines, processors, or cores)
update in an asynchronous parallel fashion. Asynchrony is crucial to
parallel computing since it reduces synchronization wait, relaxes communication
bottleneck, and thus speeds up computing significantly. At each step of ARock,
an agent updates a randomly selected coordinate based on possibly
out-of-date information on . The agents share through either global
memory or communication. If writing is atomic, the agents can read and
write without memory locks.
Theoretically, we show that if the nonexpansive operator has a fixed
point, then with probability one, ARock generates a sequence that converges to
a fixed points of . Our conditions on and step sizes are weaker than
comparable work. Linear convergence is also obtained.
We propose special cases of ARock for linear systems, convex optimization,
machine learning, as well as distributed and decentralized consensus problems.
Numerical experiments of solving sparse logistic regression problems are
presented.Comment: updated the linear convergence proof
An Asynchronous Distributed Proximal Gradient Method for Composite Convex Optimization
We propose a distributed first-order augmented Lagrangian (DFAL) algorithm to
minimize the sum of composite convex functions, where each term in the sum is a
private cost function belonging to a node, and only nodes connected by an edge
can directly communicate with each other. This optimization model abstracts a
number of applications in distributed sensing and machine learning. We show
that any limit point of DFAL iterates is optimal; and for any , an
-optimal and -feasible solution can be computed within
DFAL iterations, which require
proximal
gradient computations and communications per node in total, where
denotes the largest eigenvalue of the graph Laplacian, and is the
minimum degree of the graph. We also propose an asynchronous version of DFAL by
incorporating randomized block coordinate descent methods; and demonstrate the
efficiency of DFAL on large scale sparse-group LASSO problems.Comment: The manuscript will appear in the Proceedings of the 32nd
International Conference on Machine Learning, Lille, France, 2015. JMLR: W&CP
volume 37. Copyright 2015 by the author(s
Cloud-Assisted Remote Sensor Network Virtualization for Distributed Consensus Estimation
We develop cloud-assisted remote sensing techniques for enabling distributed
consensus estimation of unknown parameters in a given geographic area. We first
propose a distributed sensor network virtualization algorithm that searches
for, selects, and coordinates Internet-accessible sensors to perform a sensing
task in a specific region. The algorithm converges in linearithmic time for
large-scale networks, and requires exchanging a number of messages that is at
most linear in the number of sensors. Second, we design an uncoordinated,
distributed algorithm that relies on the selected sensors to estimate a set of
parameters without requiring synchronization among the sensors. Our simulation
results show that the proposed algorithm, when compared to conventional ADMM
(Alternating Direction Method of Multipliers), reduces communication overhead
significantly without compromising the estimation error. In addition, the
convergence time, though increases slightly, is still linear as in the case of
conventional ADMM.Comment: 11 pages, double column, pre-submissio
Newton-Raphson Consensus under asynchronous and lossy communications for peer-to-peer networks
In this work we study the problem of unconstrained convex-optimization in a
fully distributed multi-agent setting which includes asynchronous computation
and lossy communication. In particular, we extend a recently proposed algorithm
named Newton-Raphson Consensus by integrating it with a broadcast-based average
consensus algorithm which is robust to packet losses. We show via the
separation of time scales principle that under mild conditions (i.e.,
persistency of the agents activation and bounded consecutive communication
failures) the proposed algorithm is proved to be locally exponentially stable
with respect to the optimal global solution. Finally, we complement the
theoretical analysis with numerical simulations that are based on real
datasets
Distributed Partitioned Big-Data Optimization via Asynchronous Dual Decomposition
In this paper we consider a novel partitioned framework for distributed
optimization in peer-to-peer networks. In several important applications the
agents of a network have to solve an optimization problem with two key
features: (i) the dimension of the decision variable depends on the network
size, and (ii) cost function and constraints have a sparsity structure related
to the communication graph. For this class of problems a straightforward
application of existing consensus methods would show two inefficiencies: poor
scalability and redundancy of shared information. We propose an asynchronous
distributed algorithm, based on dual decomposition and coordinate methods, to
solve partitioned optimization problems. We show that, by exploiting the
problem structure, the solution can be partitioned among the nodes, so that
each node just stores a local copy of a portion of the decision variable
(rather than a copy of the entire decision vector) and solves a small-scale
local problem
- …