40 research outputs found
The Mutual Information in Random Linear Estimation Beyond i.i.d. Matrices
There has been definite progress recently in proving the variational
single-letter formula given by the heuristic replica method for various
estimation problems. In particular, the replica formula for the mutual
information in the case of noisy linear estimation with random i.i.d. matrices,
a problem with applications ranging from compressed sensing to statistics, has
been proven rigorously. In this contribution we go beyond the restrictive
i.i.d. matrix assumption and discuss the formula proposed by Takeda, Uda,
Kabashima and later by Tulino, Verdu, Caire and Shamai who used the replica
method. Using the recently introduced adaptive interpolation method and random
matrix theory, we prove this formula for a relevant large sub-class of
rotationally invariant matrices.Comment: Presented at the 2018 IEEE International Symposium on Information
Theory (ISIT
Generalized Approximate Message-Passing Decoder for Universal Sparse Superposition Codes
Sparse superposition (SS) codes were originally proposed as a
capacity-achieving communication scheme over the additive white Gaussian noise
channel (AWGNC) [1]. Very recently, it was discovered that these codes are
universal, in the sense that they achieve capacity over any memoryless channel
under generalized approximate message-passing (GAMP) decoding [2], although
this decoder has never been stated for SS codes. In this contribution we
introduce the GAMP decoder for SS codes, we confirm empirically the
universality of this communication scheme through its study on various channels
and we provide the main analysis tools: state evolution and potential. We also
compare the performance of GAMP with the Bayes-optimal MMSE decoder. We
empirically illustrate that despite the presence of a phase transition
preventing GAMP to reach the optimal performance, spatial coupling allows to
boost the performance that eventually tends to capacity in a proper limit. We
also prove that, in contrast with the AWGNC case, SS codes for binary input
channels have a vanishing error floor in the limit of large codewords.
Moreover, the performance of Hadamard-based encoders is assessed for practical
implementations
The adaptive interpolation method for proving replica formulas. Applications to the Curie-Weiss and Wigner spike models
In this contribution we give a pedagogic introduction to the newly introduced
adaptive interpolation method to prove in a simple and unified way replica
formulas for Bayesian optimal inference problems. Many aspects of this method
can already be explained at the level of the simple Curie-Weiss spin system.
This provides a new method of solution for this model which does not appear to
be known. We then generalize this analysis to a paradigmatic inference problem,
namely rank-one matrix estimation, also refered to as the Wigner spike model in
statistics. We give many pointers to the recent literature where the method has
been succesfully applied
Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
We analyse a linear regression problem with nonconvex regularization called
smoothly clipped absolute deviation (SCAD) under an overcomplete Gaussian basis
for Gaussian random data. We propose an approximate message passing (AMP)
algorithm considering nonconvex regularization, namely SCAD-AMP, and
analytically show that the stability condition corresponds to the de
Almeida--Thouless condition in spin glass literature. Through asymptotic
analysis, we show the correspondence between the density evolution of SCAD-AMP
and the replica symmetric solution. Numerical experiments confirm that for a
sufficiently large system size, SCAD-AMP achieves the optimal performance
predicted by the replica method. Through replica analysis, a phase transition
between replica symmetric (RS) and replica symmetry breaking (RSB) region is
found in the parameter space of SCAD. The appearance of the RS region for a
nonconvex penalty is a significant advantage that indicates the region of
smooth landscape of the optimization problem. Furthermore, we analytically show
that the statistical representation performance of the SCAD penalty is better
than that of L1-based methods, and the minimum representation error under RS
assumption is obtained at the edge of the RS/RSB phase. The correspondence
between the convergence of the existing coordinate descent algorithm and RS/RSB
transition is also indicated