Gradient Coding from Cyclic MDS Codes and Expander Graphs

Gradient coding is a technique for straggler mitigation in distributed learning. In this paper we design novel gradient codes using tools from classical coding theory, namely, cyclic MDS codes, which compare favorably with existing solutions, both in the applicable range of parameters and in the complexity of the involved algorithms. Second, we introduce an approximate variant of the gradient coding problem, in which we settle for approximate gradient computation instead of the exact one. This approach enables graceful degradation, i.e., the $\ell_2$ error of the approximate gradient is a decreasing function of the number of stragglers. Our main result is that normalized adjacency matrices of expander graphs yield excellent approximate gradient codes, which enable significantly less computation compared to exact gradient coding, and guarantee faster convergence than trivial solutions under standard assumptions. We experimentally test our approach on Amazon EC2, and show that the generalization error of approximate gradient coding is very close to the full gradient while requiring significantly less computation from the workers

Matrix Completion in Colocated MIMO Radar: Recoverability, Bounds & Theoretical Guarantees

It was recently shown that low rank matrix completion theory can be employed for designing new sampling schemes in the context of MIMO radars, which can lead to the reduction of the high volume of data typically required for accurate target detection and estimation. Employing random samplers at each reception antenna, a partially observed version of the received data matrix is formulated at the fusion center, which, under certain conditions, can be recovered using convex optimization. This paper presents the theoretical analysis regarding the performance of matrix completion in colocated MIMO radar systems, exploiting the particular structure of the data matrix. Both Uniform Linear Arrays (ULAs) and arbitrary 2-dimensional arrays are considered for transmission and reception. Especially for the ULA case, under some mild assumptions on the directions of arrival of the targets, it is explicitly shown that the coherence of the data matrix is both asymptotically and approximately optimal with respect to the number of antennas of the arrays involved and further, the data matrix is recoverable using a subset of its entries with minimal cardinality. Sufficient conditions guaranteeing low matrix coherence and consequently satisfactory matrix completion performance are also presented, including the arbitrary 2-dimensional array case.Comment: 19 pages, 7 figures, under review in Transactions on Signal Processing (2013

Sparsity Order Estimation from a Single Compressed Observation Vector

We investigate the problem of estimating the unknown degree of sparsity from compressive measurements without the need to carry out a sparse recovery step. While the sparsity order can be directly inferred from the effective rank of the observation matrix in the multiple snapshot case, this appears to be impossible in the more challenging single snapshot case. We show that specially designed measurement matrices allow to rearrange the measurement vector into a matrix such that its effective rank coincides with the effective sparsity order. In fact, we prove that matrices which are composed of a Khatri-Rao product of smaller matrices generate measurements that allow to infer the sparsity order. Moreover, if some samples are used more than once, one of the matrices needs to be Vandermonde. These structural constraints reduce the degrees of freedom in choosing the measurement matrix which may incur in a degradation in the achievable coherence. We thus also address suitable choices of the measurement matrices. In particular, we analyze Khatri-Rao and Vandermonde matrices in terms of their coherence and provide a new design for Vandermonde matrices that achieves a low coherence

Decimated generalized Prony systems

We continue studying robustness of solving algebraic systems of Prony type (also known as the exponential fitting systems), which appear prominently in many areas of mathematics, in particular modern "sub-Nyquist" sampling theories. We show that by considering these systems at arithmetic progressions (or "decimating" them), one can achieve better performance in the presence of noise. We also show that the corresponding lower bounds are closely related to well-known estimates, obtained for similar problems but in different contexts

Some upper and lower bounds on PSD-rank

Positive semidefinite rank (PSD-rank) is a relatively new quantity with applications to combinatorial optimization and communication complexity. We first study several basic properties of PSD-rank, and then develop new techniques for showing lower bounds on the PSD-rank. All of these bounds are based on viewing a positive semidefinite factorization of a matrix $M$ as a quantum communication protocol. These lower bounds depend on the entries of the matrix and not only on its support (the zero/nonzero pattern), overcoming a limitation of some previous techniques. We compare these new lower bounds with known bounds, and give examples where the new ones are better. As an application we determine the PSD-rank of (approximations of) some common matrices.Comment: 21 page

Stable super-resolution limit and smallest singular value of restricted Fourier matrices

Super-resolution refers to the process of recovering the locations and amplitudes of a collection of point sources, represented as a discrete measure, given $M+1$ of its noisy low-frequency Fourier coefficients. The recovery process is highly sensitive to noise whenever the distance $\Delta$ between the two closest point sources is less than $1/M$. This paper studies the {\it fundamental difficulty of super-resolution} and the {\it performance guarantees of a subspace method called MUSIC} in the regime that $\Delta<1/M$. The most important quantity in our theory is the minimum singular value of the Vandermonde matrix whose nodes are specified by the source locations. Under the assumption that the nodes are closely spaced within several well-separated clumps, we derive a sharp and non-asymptotic lower bound for this quantity. Our estimate is given as a weighted $\ell^2$ sum, where each term only depends on the configuration of each individual clump. This implies that, as the noise increases, the super-resolution capability of MUSIC degrades according to a power law where the exponent depends on the cardinality of the largest clump. Numerical experiments validate our theoretical bounds for the minimum singular value and the resolution limit of MUSIC. When there are $S$ point sources located on a grid with spacing $1/N$, the fundamental difficulty of super-resolution can be quantitatively characterized by a min-max error, which is the reconstruction error incurred by the best possible algorithm in the worst-case scenario. We show that the min-max error is closely related to the minimum singular value of Vandermonde matrices, and we provide a non-asymptotic and sharp estimate for the min-max error, where the dominant term is $(N/M)^{2S-1}$.Comment: 47 pages, 8 figure