3 research outputs found
Linear Convergent Decentralized Optimization with Compression
Communication compression has become a key strategy to speed up distributed
optimization. However, existing decentralized algorithms with compression
mainly focus on compressing DGD-type algorithms. They are unsatisfactory in
terms of convergence rate, stability, and the capability to handle
heterogeneous data. Motivated by primal-dual algorithms, this paper proposes
the first \underline{L}in\underline{EA}r convergent \underline{D}ecentralized
algorithm with compression, LEAD. Our theory describes the coupled dynamics of
the inexact primal and dual update as well as compression error, and we provide
the first consensus error bound in such settings without assuming bounded
gradients. Experiments on convex problems validate our theoretical analysis,
and empirical study on deep neural nets shows that LEAD is applicable to
non-convex problems.Comment: ICLR 2021 (International Conference on Learning Representations
Compressed Gradient Tracking Methods for Decentralized Optimization with Linear Convergence
Communication compression techniques are of growing interests for solving the
decentralized optimization problem under limited communication, where the
global objective is to minimize the average of local cost functions over a
multi-agent network using only local computation and peer-to-peer
communication. In this paper, we first propose a novel compressed gradient
tracking algorithm (C-GT) that combines gradient tracking technique with
communication compression. In particular, C-GT is compatible with a general
class of compression operators that unifies both unbiased and biased
compressors. We show that C-GT inherits the advantages of gradient
tracking-based algorithms and achieves linear convergence rate for strongly
convex and smooth objective functions. In the second part of this paper, we
propose an error feedback based compressed gradient tracking algorithm
(EF-C-GT) to further improve the algorithm efficiency for biased compression
operators. Numerical examples complement the theoretical findings and
demonstrate the efficiency and flexibility of the proposed algorithms
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine
learning models without a central coordinator. In many scenarios communication
between devices is energy demanding and time consuming and forms the bottleneck
of the entire system.
We propose a new randomized first-order method which tackles the
communication bottleneck by applying randomized compression operators to the
communicated messages. By combining our scheme with a new variance reduction
technique that progressively throughout the iterations reduces the adverse
effect of the injected quantization noise, we obtain the first scheme that
converges linearly on strongly convex decentralized problems while using
compressed communication only.
We prove that our method can solve the problems without any increase in the
number of communications compared to the baseline which does not perform any
communication compression while still allowing for a significant compression
factor which depends on the conditioning of the problem and the topology of the
network. Our key theoretical findings are supported by numerical experiments