Towards Tight Communication Lower Bounds for Distributed Optimisation

We consider a standard distributed optimisation setting where $N$ machines, each holding a $d$-dimensional function $f_i$, aim to jointly minimise the sum of the functions $\sum_{i = 1}^N f_i (x)$. This problem arises naturally in large-scale distributed optimisation, where a standard solution is to apply variants of (stochastic) gradient descent. We focus on the communication complexity of this problem: our main result provides the first fully unconditional bounds on total number of bits which need to be sent and received by the $N$ machines to solve this problem under point-to-point communication, within a given error-tolerance. Specifically, we show that $\Omega( Nd \log d / N\varepsilon)$ total bits need to be communicated between the machines to find an additive $\epsilon$-approximation to the minimum of $\sum_{i = 1}^N f_i (x)$. The result holds for both deterministic and randomised algorithms, and, importantly, requires no assumptions on the algorithm structure. The lower bound is tight under certain restrictions on parameter values, and is matched within constant factors for quadratic objectives by a new variant of quantised gradient descent, which we describe and analyse. Our results bring over tools from communication complexity to distributed optimisation, which has potential for further applications

Communication-Efficient Distributed Optimization with Quantized Preconditioners

We investigate fast and communication-efficient algorithms for the classic problem of minimizing a sum of strongly convex and smooth functions that are distributed among $n$ different nodes, which can communicate using a limited number of bits. Most previous communication-efficient approaches for this problem are limited to first-order optimization, and therefore have \emph{linear} dependence on the condition number in their communication complexity. We show that this dependence is not inherent: communication-efficient methods can in fact have sublinear dependence on the condition number. For this, we design and analyze the first communication-efficient distributed variants of preconditioned gradient descent for Generalized Linear Models, and for Newton's method. Our results rely on a new technique for quantizing both the preconditioner and the descent direction at each step of the algorithms, while controlling their convergence rate. We also validate our findings experimentally, showing fast convergence and reduced communication

Robust Lower Bounds for Graph Problems in the Blackboard Model of Communication

We give lower bounds on the communication complexity of graph problems in the multi-party blackboard model. In this model, the edges of an $n$-vertex input graph are partitioned among $k$ parties, who communicate solely by writing messages on a shared blackboard that is visible to every party. We show that any non-trivial graph problem on $n$-vertex graphs has blackboard communication complexity $\Omega(n)$ bits, even if the edges of the input graph are randomly assigned to the $k$ parties. We say that a graph problem is non-trivial if the output cannot be computed in a model where every party holds at most one edge and no communication is allowed. Our lower bound thus holds for essentially all key graph problems relevant to distributed computing, including Maximal Independent Set (MIS), Maximal Matching, ($\Delta+1$)-coloring, and Dominating Set. In many cases, e.g., MIS, Maximal Matching, and $(\Delta+1)$-coloring, our lower bounds are optimal, up to poly-logarithmic factors