1,638 research outputs found
On the Convergence Analysis of Asynchronous Distributed Quadratic Programming via Dual Decomposition
In this paper, we analyze the convergence as well as the rate of convergence
of asynchronous distributed quadratic programming (QP) with dual decomposition
technique. In general, distributed optimization requires synchronization of
data at each iteration step due to the interdependency of data. This
synchronization latency may incur a large amount of waiting time caused by an
idle process during computation. We aim to attack this synchronization penalty
in distributed QP problems by implementing asynchronous update of dual
variable. The price to pay for adopting asynchronous computing algorithms is
unpredictability of the solution, resulting in a tradeoff between speedup and
accuracy. Thus, the convergence to an optimal solution is not guaranteed owing
to the stochastic behavior of asynchrony. In this paper, we employ the switched
system framework as an analysis tool to investigate the convergence of
asynchronous distributed QP. This switched system will facilitate analysis on
asynchronous distributed QP with dual decomposition, providing necessary and
sufficient conditions for the mean square convergence. Also, we provide an
analytic expression for the rate of convergence through the switched system,
which enables performance analysis of asynchronous algorithms as compared with
synchronous case. To verify the validity of the proposed methods, numerical
examples are presented with an implementation of asynchronous parallel QP using
OpenMP
The Sound of APALM Clapping: Faster Nonsmooth Nonconvex Optimization with Stochastic Asynchronous PALM
We introduce the Stochastic Asynchronous Proximal Alternating Linearized
Minimization (SAPALM) method, a block coordinate stochastic proximal-gradient
method for solving nonconvex, nonsmooth optimization problems. SAPALM is the
first asynchronous parallel optimization method that provably converges on a
large class of nonconvex, nonsmooth problems. We prove that SAPALM matches the
best known rates of convergence --- among synchronous or asynchronous methods
--- on this problem class. We provide upper bounds on the number of workers for
which we can expect to see a linear speedup, which match the best bounds known
for less complex problems, and show that in practice SAPALM achieves this
linear speedup. We demonstrate state-of-the-art performance on several matrix
factorization problems
An Asynchronous Distributed Framework for Large-scale Learning Based on Parameter Exchanges
In many distributed learning problems, the heterogeneous loading of computing
machines may harm the overall performance of synchronous strategies. In this
paper, we propose an effective asynchronous distributed framework for the
minimization of a sum of smooth functions, where each machine performs
iterations in parallel on its local function and updates a shared parameter
asynchronously. In this way, all machines can continuously work even though
they do not have the latest version of the shared parameter. We prove the
convergence of the consistency of this general distributed asynchronous method
for gradient iterations then show its efficiency on the matrix factorization
problem for recommender systems and on binary classification.Comment: 16 page
An Inertial Parallel and Asynchronous Fixed-Point Iteration for Convex Optimization
Two characteristics that make convex decomposition algorithms attractive are
simplicity of operations and generation of parallelizable structures. In
principle, these schemes require that all coordinates update at the same time,
i.e., they are synchronous by construction. Introducing asynchronicity in the
updates can resolve several issues that appear in the synchronous case, like
load imbalances in the computations or failing communication links. However,
and to the best of our knowledge, there are no instances of asynchronous
versions of commonly-known algorithms combined with inertial acceleration
techniques. In this work we propose an inertial asynchronous and parallel
fixed-point iteration from which several new versions of existing convex
optimization algorithms emanate. Departing from the norm that the frequency of
the coordinates' updates should comply to some prior distribution, we propose a
scheme where the only requirement is that the coordinates update within a
bounded interval. We prove convergence of the sequence of iterates generated by
the scheme at a linear rate. One instance of the proposed scheme is implemented
to solve a distributed optimization load sharing problem in a smart grid
setting and its superiority with respect to the non-accelerated version is
illustrated
DISROPT: a Python Framework for Distributed Optimization
In this paper we introduce DISROPT, a Python package for distributed
optimization over networks. We focus on cooperative set-ups in which an
optimization problem must be solved by peer-to-peer processors (without central
coordinators) that have access only to partial knowledge of the entire problem.
To reflect this, agents in DISROPT are modeled as entities that are initialized
with their local knowledge of the problem. Agents then run local routines and
communicate with each other to solve the global optimization problem. A simple
syntax has been designed to allow for an easy modeling of the problems. The
package comes with many distributed optimization algorithms that are already
embedded. Moreover, the package provides full-fledged functionalities for
communication and local computation, which can be used to design and implement
new algorithms. DISROPT is available at github.com/disropt/disropt under the
GPL license, with a complete documentation and many examples
Stochastic Primal-Dual Coordinate Method with Large Step Size for Composite Optimization with Composite Cone-constraints
We introduce a stochastic coordinate extension of the first-order primal-dual
method studied by Cohen and Zhu (1984) and Zhao and Zhu (2018) to solve
Composite Optimization with Composite Cone-constraints (COCC). In this method,
we randomly choose a block of variables based on the uniform distribution. The
linearization and Bregman-like function (core function) to that randomly
selected block allow us to get simple parallel primal-dual decomposition for
COCC. We obtain almost surely convergence and O(1/t) expected convergence rate
in this work. The high probability complexity bound is also derived in this
paper.Comment: arXiv admin note: substantial text overlap with arXiv:1804.0080
A Class of Parallel Doubly Stochastic Algorithms for Large-Scale Learning
We consider learning problems over training sets in which both, the number of
training examples and the dimension of the feature vectors, are large. To solve
these problems we propose the random parallel stochastic algorithm (RAPSA). We
call the algorithm random parallel because it utilizes multiple parallel
processors to operate on a randomly chosen subset of blocks of the feature
vector. We call the algorithm stochastic because processors choose training
subsets uniformly at random. Algorithms that are parallel in either of these
dimensions exist, but RAPSA is the first attempt at a methodology that is
parallel in both the selection of blocks and the selection of elements of the
training set. In RAPSA, processors utilize the randomly chosen functions to
compute the stochastic gradient component associated with a randomly chosen
block. The technical contribution of this paper is to show that this minimally
coordinated algorithm converges to the optimal classifier when the training
objective is convex. Moreover, we present an accelerated version of RAPSA
(ARAPSA) that incorporates the objective function curvature information by
premultiplying the descent direction by a Hessian approximation matrix. We
further extend the results for asynchronous settings and show that if the
processors perform their updates without any coordination the algorithms are
still convergent to the optimal argument. RAPSA and its extensions are then
numerically evaluated on a linear estimation problem and a binary image
classification task using the MNIST handwritten digit dataset.Comment: arXiv admin note: substantial text overlap with arXiv:1603.0678
Asynchronous Distributed Optimization Via Randomized Dual Proximal Gradient
In this paper we consider distributed optimization problems in which the cost function is separable, i.e., a sum of possibly non-smooth functions all sharing a common variable, and can be split into a strongly convex term and a convex one. The second term is typically used to encode constraints or to regularize the solution. We propose a class of distributed optimization algorithms based on proximal gradient methods applied to the dual problem. We show that, by choosing suitable primal variable copies, the dual problem is itself separable when written in terms of conjugate functions, and the dual variables can be stacked into non-overlapping blocks associated to the computing nodes. We first show that a weighted proximal gradient on the dual function leads to a synchronous distributed algorithm with local dual proximal gradient updates at each node. Then, as main paper contribution, we develop asynchronous versions of the algorithm in which the node updates are triggered by local timers without any global iteration counter. The algorithms are shown to be proper randomized block-coordinate proximal gradient updates on the dual function
Asynchronous Parallel Algorithms for Nonconvex Optimization
We propose a new asynchronous parallel block-descent algorithmic framework
for the minimization of the sum of a smooth nonconvex function and a nonsmooth
convex one, subject to both convex and nonconvex constraints. The proposed
framework hinges on successive convex approximation techniques and a novel
probabilistic model that captures key elements of modern computational
architectures and asynchronous implementations in a more faithful way than
current state-of-the-art models. Other key features of the framework are: i) it
covers in a unified way several specific solution methods; ii) it accommodates
a variety of possible parallel computing architectures; and iii) it can deal
with nonconvex constraints. Almost sure convergence to stationary solutions is
proved, and theoretical complexity results are provided, showing nearly ideal
linear speedup when the number of workers is not too large.Comment: This is the first part of a two-paper work. The second part can be
found at: arXiv:1701.0490
Stochastic Primal-Dual Coordinate Method for Nonlinear Convex Cone Programs
Block coordinate descent (BCD) methods and their variants have been widely
used in coping with large-scale nonconstrained optimization problems in many
fields such as imaging processing, machine learning, compress sensing and so
on. For problem with coupling constraints, Nonlinear convex cone programs
(NCCP) are important problems with many practical applications, but these
problems are hard to solve by using existing block coordinate type methods.
This paper introduces a stochastic primal-dual coordinate (SPDC) method for
solving large-scale NCCP. In this method, we randomly choose a block of
variables based on the uniform distribution. The linearization and Bregman-like
function (core function) to that randomly selected block allow us to get simple
parallel primal-dual decomposition for NCCP. The sequence generated by our
algorithm is proved almost surely converge to an optimal solution of primal
problem. Two types of convergence rate with different probability (almost
surely and expected) are also obtained. The probability complexity bound is
also derived in this paper
- …