12 research outputs found
Distributed Learning with Sparse Communications by Identification
In distributed optimization for large-scale learning, a major performance
limitation comes from the communications between the different entities. When
computations are performed by workers on local data while a coordinator machine
coordinates their updates to minimize a global loss, we present an asynchronous
optimization algorithm that efficiently reduces the communications between the
coordinator and workers. This reduction comes from a random sparsification of
the local updates. We show that this algorithm converges linearly in the
strongly convex case and also identifies optimal strongly sparse solutions. We
further exploit this identification to propose an automatic dimension
reduction, aptly sparsifying all exchanges between coordinator and workers.Comment: v2 is a significant improvement over v1 (titled "Asynchronous
Distributed Learning with Sparse Communications and Identification") with new
algorithms, results, and discussion