Structure Adaptive Lasso

Lasso is of fundamental importance in high-dimensional statistics and has been routinely used to regress a response on a high-dimensional set of predictors. In many scientific applications, there exists external information that encodes the predictive power and sparsity structure of the predictors. In this article, we develop a new method, called the Structure Adaptive Lasso (SA-Lasso), to incorporate these potentially useful side information into a penalized regression. The basic idea is to translate the external information into different penalization strengths for the regression coefficients. We study the risk properties of the resulting estimator. In particular, we generalize the state evolution framework recently introduced for the analysis of the approximate message-passing algorithm to the SA-Lasso setting. We show that the finite sample risk of the SA-Lasso estimator is consistent with the theoretical risk predicted by the state evolution equation. Our theory suggests that the SA-Lasso with an informative group or covariate structure can significantly outperform the Lasso, Adaptive Lasso, and Sparse Group Lasso. This evidence is further confirmed in our numerical studies. We also demonstrate the usefulness and the superiority of our method in a real data application.Comment: 42 pages, 24 figure

Information-theoretically Optimal Sparse PCA

Sparse Principal Component Analysis (PCA) is a dimensionality reduction technique wherein one seeks a low-rank representation of a data matrix with additional sparsity constraints on the obtained representation. We consider two probabilistic formulations of sparse PCA: a spiked Wigner and spiked Wishart (or spiked covariance) model. We analyze an Approximate Message Passing (AMP) algorithm to estimate the underlying signal and show, in the high dimensional limit, that the AMP estimates are information-theoretically optimal. As an immediate corollary, our results demonstrate that the posterior expectation of the underlying signal, which is often intractable to compute, can be obtained using a polynomial-time scheme. Our results also effectively provide a single-letter characterization of the sparse PCA problem.Comment: 5 pages, 1 figure, conferenc

An Overview of Multi-Processor Approximate Message Passing

Approximate message passing (AMP) is an algorithmic framework for solving linear inverse problems from noisy measurements, with exciting applications such as reconstructing images, audio, hyper spectral images, and various other signals, including those acquired in compressive signal acquisiton systems. The growing prevalence of big data systems has increased interest in large-scale problems, which may involve huge measurement matrices that are unsuitable for conventional computing systems. To address the challenge of large-scale processing, multiprocessor (MP) versions of AMP have been developed. We provide an overview of two such MP-AMP variants. In row-MP-AMP, each computing node stores a subset of the rows of the matrix and processes corresponding measurements. In column- MP-AMP, each node stores a subset of columns, and is solely responsible for reconstructing a portion of the signal. We will discuss pros and cons of both approaches, summarize recent research results for each, and explain when each one may be a viable approach. Aspects that are highlighted include some recent results on state evolution for both MP-AMP algorithms, and the use of data compression to reduce communication in the MP network