144 research outputs found
Computation Scheduling for Distributed Machine Learning with Straggling Workers
We study scheduling of computation tasks across n workers in a large scale
distributed learning problem with the help of a master. Computation and
communication delays are assumed to be random, and redundant computations are
assigned to workers in order to tolerate stragglers. We consider sequential
computation of tasks assigned to a worker, while the result of each computation
is sent to the master right after its completion. Each computation round, which
can model an iteration of the stochastic gradient descent (SGD) algorithm, is
completed once the master receives k distinct computations, referred to as the
computation target. Our goal is to characterize the average completion time as
a function of the computation load, which denotes the portion of the dataset
available at each worker, and the computation target. We propose two
computation scheduling schemes that specify the tasks assigned to each worker,
as well as their computation schedule, i.e., the order of execution. Assuming a
general statistical model for computation and communication delays, we derive
the average completion time of the proposed schemes. We also establish a lower
bound on the minimum average completion time by assuming prior knowledge of the
random delays. Experimental results carried out on Amazon EC2 cluster show a
significant reduction in the average completion time over existing coded and
uncoded computing schemes. It is also shown numerically that the gap between
the proposed scheme and the lower bound is relatively small, confirming the
efficiency of the proposed scheduling design.Comment: Submitted for publicatio
Latency Analysis of Coded Computation Schemes over Wireless Networks
Large-scale distributed computing systems face two major bottlenecks that
limit their scalability: straggler delay caused by the variability of
computation times at different worker nodes and communication bottlenecks
caused by shuffling data across many nodes in the network. Recently, it has
been shown that codes can provide significant gains in overcoming these
bottlenecks. In particular, optimal coding schemes for minimizing latency in
distributed computation of linear functions and mitigating the effect of
stragglers was proposed for a wired network, where the workers can
simultaneously transmit messages to a master node without interference. In this
paper, we focus on the problem of coded computation over a wireless
master-worker setup with straggling workers, where only one worker can transmit
the result of its local computation back to the master at a time. We consider 3
asymptotic regimes (determined by how the communication and computation times
are scaled with the number of workers) and precisely characterize the total
run-time of the distributed algorithm and optimum coding strategy in each
regime. In particular, for the regime of practical interest where the
computation and communication times of the distributed computing algorithm are
comparable, we show that the total run-time approaches a simple lower bound
that decouples computation and communication, and demonstrate that coded
schemes are times faster than uncoded schemes
- …