2,191 research outputs found
Asymptotic Analysis of MAP Estimation via the Replica Method and Applications to Compressed Sensing
The replica method is a non-rigorous but well-known technique from
statistical physics used in the asymptotic analysis of large, random, nonlinear
problems. This paper applies the replica method, under the assumption of
replica symmetry, to study estimators that are maximum a posteriori (MAP) under
a postulated prior distribution. It is shown that with random linear
measurements and Gaussian noise, the replica-symmetric prediction of the
asymptotic behavior of the postulated MAP estimate of an n-dimensional vector
"decouples" as n scalar postulated MAP estimators. The result is based on
applying a hardening argument to the replica analysis of postulated posterior
mean estimators of Tanaka and of Guo and Verdu.
The replica-symmetric postulated MAP analysis can be readily applied to many
estimators used in compressed sensing, including basis pursuit, lasso, linear
estimation with thresholding, and zero norm-regularized estimation. In the case
of lasso estimation the scalar estimator reduces to a soft-thresholding
operator, and for zero norm-regularized estimation it reduces to a
hard-threshold. Among other benefits, the replica method provides a
computationally-tractable method for precisely predicting various performance
metrics including mean-squared error and sparsity pattern recovery probability.Comment: 22 pages; added details on the replica symmetry assumptio
Generalized Approximate Message-Passing Decoder for Universal Sparse Superposition Codes
Sparse superposition (SS) codes were originally proposed as a
capacity-achieving communication scheme over the additive white Gaussian noise
channel (AWGNC) [1]. Very recently, it was discovered that these codes are
universal, in the sense that they achieve capacity over any memoryless channel
under generalized approximate message-passing (GAMP) decoding [2], although
this decoder has never been stated for SS codes. In this contribution we
introduce the GAMP decoder for SS codes, we confirm empirically the
universality of this communication scheme through its study on various channels
and we provide the main analysis tools: state evolution and potential. We also
compare the performance of GAMP with the Bayes-optimal MMSE decoder. We
empirically illustrate that despite the presence of a phase transition
preventing GAMP to reach the optimal performance, spatial coupling allows to
boost the performance that eventually tends to capacity in a proper limit. We
also prove that, in contrast with the AWGNC case, SS codes for binary input
channels have a vanishing error floor in the limit of large codewords.
Moreover, the performance of Hadamard-based encoders is assessed for practical
implementations
Cleaning large correlation matrices: tools from random matrix theory
This review covers recent results concerning the estimation of large
covariance matrices using tools from Random Matrix Theory (RMT). We introduce
several RMT methods and analytical techniques, such as the Replica formalism
and Free Probability, with an emphasis on the Marchenko-Pastur equation that
provides information on the resolvent of multiplicatively corrupted noisy
matrices. Special care is devoted to the statistics of the eigenvectors of the
empirical correlation matrix, which turn out to be crucial for many
applications. We show in particular how these results can be used to build
consistent "Rotationally Invariant" estimators (RIE) for large correlation
matrices when there is no prior on the structure of the underlying process. The
last part of this review is dedicated to some real-world applications within
financial markets as a case in point. We establish empirically the efficacy of
the RIE framework, which is found to be superior in this case to all previously
proposed methods. The case of additively (rather than multiplicatively)
corrupted noisy matrices is also dealt with in a special Appendix. Several open
problems and interesting technical developments are discussed throughout the
paper.Comment: 165 pages, article submitted to Physics Report
Phase Transitions in Semidefinite Relaxations
Statistical inference problems arising within signal processing, data mining,
and machine learning naturally give rise to hard combinatorial optimization
problems. These problems become intractable when the dimensionality of the data
is large, as is often the case for modern datasets. A popular idea is to
construct convex relaxations of these combinatorial problems, which can be
solved efficiently for large scale datasets.
Semidefinite programming (SDP) relaxations are among the most powerful
methods in this family, and are surprisingly well-suited for a broad range of
problems where data take the form of matrices or graphs. It has been observed
several times that, when the `statistical noise' is small enough, SDP
relaxations correctly detect the underlying combinatorial structures.
In this paper we develop asymptotic predictions for several `detection
thresholds,' as well as for the estimation error above these thresholds. We
study some classical SDP relaxations for statistical problems motivated by
graph synchronization and community detection in networks. We map these
optimization problems to statistical mechanics models with vector spins, and
use non-rigorous techniques from statistical mechanics to characterize the
corresponding phase transitions. Our results clarify the effectiveness of SDP
relaxations in solving high-dimensional statistical problems.Comment: 71 pages, 24 pdf figure
Replica Symmetry Breaking in Compressive Sensing
For noisy compressive sensing systems, the asymptotic distortion with respect
to an arbitrary distortion function is determined when a general class of
least-square based reconstruction schemes is employed. The sampling matrix is
considered to belong to a large ensemble of random matrices including i.i.d.
and projector matrices, and the source vector is assumed to be i.i.d. with a
desired distribution. We take a statistical mechanical approach by representing
the asymptotic distortion as a macroscopic parameter of a spin glass and
employing the replica method for the large-system analysis. In contrast to
earlier studies, we evaluate the general replica ansatz which includes the RS
ansatz as well as RSB. The generality of the solution enables us to study the
impact of symmetry breaking. Our numerical investigations depict that for the
reconstruction scheme with the "zero-norm" penalty function, the RS fails to
predict the asymptotic distortion for relatively large compression rates;
however, the one-step RSB ansatz gives a valid prediction of the performance
within a larger regime of compression rates.Comment: 7 pages, 3 figures, presented at ITA 201
- …