Improving Distributed Gradient Descent Using Reed-Solomon Codes

Today's massively-sized datasets have made it necessary to often perform computations on them in a distributed manner. In principle, a computational task is divided into subtasks which are distributed over a cluster operated by a taskmaster. One issue faced in practice is the delay incurred due to the presence of slow machines, known as \emph{stragglers}. Several schemes, including those based on replication, have been proposed in the literature to mitigate the effects of stragglers and more recently, those inspired by coding theory have begun to gain traction. In this work, we consider a distributed gradient descent setting suitable for a wide class of machine learning problems. We adapt the framework of Tandon et al. (arXiv:1612.03301) and present a deterministic scheme that, for a prescribed per-machine computational effort, recovers the gradient from the least number of machines $f$ theoretically permissible, via an $O(f^2)$ decoding algorithm. We also provide a theoretical delay model which can be used to minimize the expected waiting time per computation by optimally choosing the parameters of the scheme. Finally, we supplement our theoretical findings with numerical results that demonstrate the efficacy of the method and its advantages over competing schemes

On the Performance of Short Block Codes over Finite-State Channels in the Rare-Transition Regime

As the mobile application landscape expands, wireless networks are tasked with supporting different connection profiles, including real-time traffic and delay-sensitive communications. Among many ensuing engineering challenges is the need to better understand the fundamental limits of forward error correction in non-asymptotic regimes. This article characterizes the performance of random block codes over finite-state channels and evaluates their queueing performance under maximum-likelihood decoding. In particular, classical results from information theory are revisited in the context of channels with rare transitions, and bounds on the probabilities of decoding failure are derived for random codes. This creates an analysis framework where channel dependencies within and across codewords are preserved. Such results are subsequently integrated into a queueing problem formulation. For instance, it is shown that, for random coding on the Gilbert-Elliott channel, the performance analysis based on upper bounds on error probability provides very good estimates of system performance and optimum code parameters. Overall, this study offers new insights about the impact of channel correlation on the performance of delay-aware, point-to-point communication links. It also provides novel guidelines on how to select code rates and block lengths for real-time traffic over wireless communication infrastructures

V2X Content Distribution Based on Batched Network Coding with Distributed Scheduling

Content distribution is an application in intelligent transportation system to assist vehicles in acquiring information such as digital maps and entertainment materials. In this paper, we consider content distribution from a single roadside infrastructure unit to a group of vehicles passing by it. To combat the short connection time and the lossy channel quality, the downloaded contents need to be further shared among vehicles after the initial broadcasting phase. To this end, we propose a joint infrastructure-to-vehicle (I2V) and vehicle-to-vehicle (V2V) communication scheme based on batched sparse (BATS) coding to minimize the traffic overhead and reduce the total transmission delay. In the I2V phase, the roadside unit (RSU) encodes the original large-size file into a number of batches in a rateless manner, each containing a fixed number of coded packets, and sequentially broadcasts them during the I2V connection time. In the V2V phase, vehicles perform the network coded cooperative sharing by re-encoding the received packets. We propose a utility-based distributed algorithm to efficiently schedule the V2V cooperative transmissions, hence reducing the transmission delay. A closed-form expression for the expected rank distribution of the proposed content distribution scheme is derived, which is used to design the optimal BATS code. The performance of the proposed content distribution scheme is evaluated by extensive simulations that consider multi-lane road and realistic vehicular traffic settings, and shown to significantly outperform the existing content distribution protocols.Comment: 12 pages and 9 figure

Expander Chunked Codes

Chunked codes are efficient random linear network coding (RLNC) schemes with low computational cost, where the input packets are encoded into small chunks (i.e., subsets of the coded packets). During the network transmission, RLNC is performed within each chunk. In this paper, we first introduce a simple transfer matrix model to characterize the transmission of chunks, and derive some basic properties of the model to facilitate the performance analysis. We then focus on the design of overlapped chunked codes, a class of chunked codes whose chunks are non-disjoint subsets of input packets, which are of special interest since they can be encoded with negligible computational cost and in a causal fashion. We propose expander chunked (EC) codes, the first class of overlapped chunked codes that have an analyzable performance,where the construction of the chunks makes use of regular graphs. Numerical and simulation results show that in some practical settings, EC codes can achieve rates within 91 to 97 percent of the optimum and outperform the state-of-the-art overlapped chunked codes significantly.Comment: 26 pages, 3 figures, submitted for journal publicatio