299,244 research outputs found
Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
In this work, we consider the distributed optimization of non-smooth convex
functions using a network of computing units. We investigate this problem under
two regularity assumptions: (1) the Lipschitz continuity of the global
objective function, and (2) the Lipschitz continuity of local individual
functions. Under the local regularity assumption, we provide the first optimal
first-order decentralized algorithm called multi-step primal-dual (MSPD) and
its corresponding optimal convergence rate. A notable aspect of this result is
that, for non-smooth functions, while the dominant term of the error is in
, the structure of the communication network only impacts a
second-order term in , where is time. In other words, the error due
to limits in communication resources decreases at a fast rate even in the case
of non-strongly-convex objective functions. Under the global regularity
assumption, we provide a simple yet efficient algorithm called distributed
randomized smoothing (DRS) based on a local smoothing of the objective
function, and show that DRS is within a multiplicative factor of the
optimal convergence rate, where is the underlying dimension.Comment: 17 page
Reciprocity of Algorithms Solving Distributed Consensus-Based Optimization and Distributed Resource Allocation
This paper aims at proposing a procedure to derive distributed algorithms for
distributed consensus-based optimization by using distributed algorithms for
network resource allocation and vice versa over switching networks with/without
synchronous protocol. It is shown that first-order gradient distributed
consensus-based optimization algorithms can be used for finding an optimal
solution of distributed resource allocation with synchronous protocol under
weaker assumptions than those given in the literature for non-switching
(static) networks. It is shown that first-order gradient distributed resource
allocation algorithms can be utilized for finding an optimal solution of
distributed consensus-based optimization. The results presented here can be
applied to time-varying and random directed networks with or without
synchronous protocol with arbitrary initialization. As a result, several
algorithms can now be used to derive distributed algorithms for both
consensus-based optimization and resource allocation, that can overcome
limitations of the existing results. While the focus of this paper is on the
first-order gradient algorithms, it is to be noted that the results also work
with second-order gradient algorithms.Comment: 8 page
Optimal decentralized distributed algorithms for stochastic convex optimization
We consider stochastic convex optimization problems with affine constraints and develop several methods using either primal or dual approach to solve it. In the primal case we use special penalization technique to make the initial problem more convenient for using optimization methods. We propose algorithms to solve it based on Similar Triangles Method with Inexact Proximal Step for the convex smooth and strongly convex smooth objective functions and methods based on Gradient Sliding algorithm to solve the same problems in the non-smooth case. We prove the convergence guarantees in smooth convex case with deterministic first-order oracle. We propose and analyze three novel methods to handle stochastic convex optimization problems with affine constraints: SPDSTM, R-RRMA-AC-SA and SSTM_sc. All methods use stochastic dual oracle. SPDSTM is the stochastic primal-dual modification of STM and it is applied for the dual problem when the primal functional is strongly convex and Lipschitz continuous on some ball. R-RRMA-AC-SA is an accelerated stochastic method based on the restarts of RRMA-AC-SA and SSTM_sc is just stochastic STM for strongly convex problems. Both methods are applied to the dual problem when the primal functional is strongly convex, smooth and Lipschitz continuous on some ball and use stochastic dual first-order oracle. We develop convergence analysis for these methods for the unbiased and biased oracles respectively. Finally, we apply all aforementioned results and approaches to solve decentralized distributed optimization problem and discuss optimality of the obtained results in terms of communication rounds and number of oracle calls per node
Quantization Design for Distributed Optimization
We consider the problem of solving a distributed optimization problem using a
distributed computing platform, where the communication in the network is
limited: each node can only communicate with its neighbours and the channel has
a limited data-rate. A common technique to address the latter limitation is to
apply quantization to the exchanged information. We propose two distributed
optimization algorithms with an iteratively refining quantization design based
on the inexact proximal gradient method and its accelerated variant. We show
that if the parameters of the quantizers, i.e. the number of bits and the
initial quantization intervals, satisfy certain conditions, then the
quantization error is bounded by a linearly decreasing function and the
convergence of the distributed algorithms is guaranteed. Furthermore, we prove
that after imposing the quantization scheme, the distributed algorithms still
exhibit a linear convergence rate, and show complexity upper-bounds on the
number of iterations to achieve a given accuracy. Finally, we demonstrate the
performance of the proposed algorithms and the theoretical findings for solving
a distributed optimal control problem
- …