A New Data Processing Inequality and Its Applications in Distributed Source and Channel Coding

In the distributed coding of correlated sources, the problem of characterizing the joint probability distribution of a pair of random variables satisfying an n-letter Markov chain arises. The exact solution of this problem is intractable. In this paper, we seek a single-letter necessary condition for this n-letter Markov chain. To this end, we propose a new data processing inequality on a new measure of correlation by means of spectrum analysis. Based on this new data processing inequality, we provide a single-letter necessary condition for the required joint probability distribution. We apply our results to two specific examples involving the distributed coding of correlated sources: multi-terminal rate-distortion region and multiple access channel with correlated sources, and propose new necessary conditions for these two problems.Comment: 45 pages, 3 figures, submitted to IEEE Trans. Information Theor

Block wavelet transforms for image coding

Cataloged from PDF version of article.In this paper, a new class of block transforms is presented. These transforms are constructed from subband decomposition filter banks corresponding to regular wavelets. New transforms are compared to the discrete cosine transform (DCT). Image coding schemes that employ the block wavelet transform (BWT) are developed. BWT's can be implemented by fast (O(N log N)) algorithms

Multi-user interference mitigation under limited feedback requirements for WCDMA systems with base station cooperation

One of the techniques that has been recently identified for dealing with multi-user interference (MUI) in future communications systems is base station (BS) cooperation or joint processing. However, perfect MUI cancellation with this technique demands severe synchronization requirements, perfect and global channel state information (CSI), and an increased backhaul and signaling overhead. In this paper, we consider a more realistic layout with the aim of mitigating the MUI, where only local CSI is available at the BSs. Due to synchronization inaccuracies and errors in the channel estimation, the system becomes partially asynchronous. In the downlink of wideband code division multiple access based systems, this asynchronism stands for the loss of the orthogonality of the spreading codes allocated to users and thus, for an increase in the MUI level of the system. In this contribution, we propose a framework for mitigating the MUI which builds in three main steps: definition of a cooperation area based on the channel characteristics, statistical modeling of the average MUI power experienced by each user and a specific spreading code allocation scheme for users served with joint processing. This code allocation assigns spreading codes to users in such a way that minimum average cross-correlation between active users can be achieved. Interestingly, these steps can be performed with a limited amount of extra feedback from the user's side

Coded distributed computing with partial recovery

Coded computation techniques provide robustness against straggling workers in distributed computing. However, most of the existing schemes require exact provisioning of the straggling behavior and ignore the computations carried out by straggling workers. Moreover, these schemes are typically designed to recover the desired computation results accurately, while in many machine learning and iterative optimization algorithms, faster approximate solutions are known to result in an improvement in the overall convergence time. In this paper, we first introduce a novel coded matrix-vector multiplication scheme, called coded computation with partial recovery (CCPR), which benefits from the advantages of both coded and uncoded computation schemes, and reduces both the computation time and the decoding complexity by allowing a trade-off between the accuracy and the speed of computation. We then extend this approach to distributed implementation of more general computation tasks by proposing a coded communication scheme with partial recovery, where the results of subtasks computed by the workers are coded before being communicated. Numerical simulations on a large linear regression task confirm the benefits of the proposed scheme in terms of the trade-off between the computation accuracy and latency

Gradient coding with dynamic clustering for straggler-tolerant distributed learning

Distributed implementations are crucial in speeding up large scale machine learning applications. Distributed gradient descent (GD) is widely employed to parallelize the learning task by distributing the dataset across multiple workers. A significant performance bottleneck for the per-iteration completion time in distributed synchronous GD is straggling workers. Coded distributed computation techniques have been introduced recently to mitigate stragglers and to speed up GD iterations by assigning redundant computations to workers. In this paper, we introduce a novel paradigm of dynamic coded computation, which assigns redundant data to workers to acquire the flexibility to dynamically choose from among a set of possible codes depending on the past straggling behavior. In particular, we propose gradient coding (GC) with dynamic clustering, called GC-DC, and regulate the number of stragglers in each cluster by dynamically forming the clusters at each iteration. With time-correlated straggling behavior, GC-DC adapts to the straggling behavior over time; in particular, at each iteration, GC-DC aims at distributing the stragglers across clusters as uniformly as possible based on the past straggler behavior. For both homogeneous and heterogeneous worker models, we numerically show that GC-DC provides significant improvements in the average per-iteration completion time without an increase in the communication load compared to the original GC scheme