Federated Learning (FL) enables large-scale distributed training of machine
learning models, while still allowing individual nodes to maintain data
locally.
However, executing FL at scale comes with inherent practical challenges:
1) heterogeneity of the local node data distributions,
2) heterogeneity of node computational speeds (asynchrony),
but also 3) constraints in the amount of communication between the clients
and the server.
In this work, we present the first variant of the classic federated averaging
(FedAvg) algorithm
which, at the same time, supports data heterogeneity, partial client
asynchrony, and communication compression.
Our algorithm comes with a rigorous analysis showing that, in spite of these
system relaxations,
it can provide similar convergence to FedAvg in interesting parameter
regimes.
Experimental results in the rigorous LEAF benchmark on setups of up to 300
nodes show that our algorithm ensures fast convergence for standard federated
tasks, improving upon prior quantized and asynchronous approaches