Analog Coding Frame-work

Analog coding is a low-complexity method to combat erasures, based on linear redundancy in the signal space domain. Previous work examined "band-limited discrete Fourier transform (DFT)" codes for Gaussian channels with erasures or impulses. We extend this concept to source coding with "erasure side-information" at the encoder and show that the performance of band-limited DFT can be significantly improved using irregular spectrum, and more generally, using equiangular tight frames (ETF). Frames are overcomplete bases and are widely used in mathematics, computer science, engineering, and statistics since they provide a stable and robust decomposition. Design of frames with favorable properties of random subframes is motivated in variety of applications, including code-devision multiple access (CDMA), compressed sensing and analog coding. We present a novel relation between deterministic frames and random matrix theory. We show empirically that the MANOVA ensemble offers a universal description of the spectra of randomly selected subframes with constant aspect ratios, taken from deterministic near-ETFs. Moreover, we derive an analytic framework and bring a formal validation for some of the empirical results, specifically that the asymptotic form for the moments of high orders of subsets of ETF agree with that of MANOVA. Finally, when exploring over-complete bases, the Welch bound is a lower bound on the root mean square cross correlation between vectors. We extend the Welch bound to an erasure setting, in which a reduced frame, composed of a random subset of Bernoulli selected vectors, is of interest. The lower bound involves moment of the reduced frame, and it is tight for ETFs and asymptotically coincides with the MANOVA moments. This result offers a novel perspective on the superiority of ETFs over other frames

An ETF view of Dropout regularization

Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.Comment: Accepted to BMVC 202

Coding Method for Parallel Iterative Linear Solver

Computationally intensive distributed and parallel computing is often bottlenecked by a small set of slow workers known as stragglers. In this paper, we utilize the emerging idea of "coded computation" to design a novel error-correcting-code inspired technique for solving linear inverse problems under specific iterative methods in a parallelized implementation affected by stragglers. Example applications include inverse problems in machine learning on graphs, such as personalized PageRank and sampling on graphs. We provably show that our coded-computation technique can reduce the mean-squared error under a computational deadline constraint. In fact, the ratio of mean-squared error of replication-based and coded techniques diverges to infinity as the deadline increases. Our experiments for personalized PageRank performed on real systems and real social networks show that this ratio can be as large as $10^4$. Further, unlike coded-computation techniques proposed thus far, our strategy combines outputs of all workers, including the stragglers, to produce more accurate estimates at the computational deadline. This also ensures that the accuracy degrades "gracefully" in the event that the number of stragglers is large.Comment: submitte