2,092 research outputs found
Approaching the Rate-Distortion Limit with Spatial Coupling, Belief propagation and Decimation
We investigate an encoding scheme for lossy compression of a binary symmetric
source based on simple spatially coupled Low-Density Generator-Matrix codes.
The degree of the check nodes is regular and the one of code-bits is Poisson
distributed with an average depending on the compression rate. The performance
of a low complexity Belief Propagation Guided Decimation algorithm is
excellent. The algorithmic rate-distortion curve approaches the optimal curve
of the ensemble as the width of the coupling window grows. Moreover, as the
check degree grows both curves approach the ultimate Shannon rate-distortion
limit. The Belief Propagation Guided Decimation encoder is based on the
posterior measure of a binary symmetric test-channel. This measure can be
interpreted as a random Gibbs measure at a "temperature" directly related to
the "noise level of the test-channel". We investigate the links between the
algorithmic performance of the Belief Propagation Guided Decimation encoder and
the phase diagram of this Gibbs measure. The phase diagram is investigated
thanks to the cavity method of spin glass theory which predicts a number of
phase transition thresholds. In particular the dynamical and condensation
"phase transition temperatures" (equivalently test-channel noise thresholds)
are computed. We observe that: (i) the dynamical temperature of the spatially
coupled construction saturates towards the condensation temperature; (ii) for
large degrees the condensation temperature approaches the temperature (i.e.
noise level) related to the information theoretic Shannon test-channel noise
parameter of rate-distortion theory. This provides heuristic insight into the
excellent performance of the Belief Propagation Guided Decimation algorithm.
The paper contains an introduction to the cavity method
Community detection and stochastic block models: recent developments
The stochastic block model (SBM) is a random graph model with planted
clusters. It is widely employed as a canonical model to study clustering and
community detection, and provides generally a fertile ground to study the
statistical and computational tradeoffs that arise in network and data
sciences.
This note surveys the recent developments that establish the fundamental
limits for community detection in the SBM, both with respect to
information-theoretic and computational thresholds, and for various recovery
requirements such as exact, partial and weak recovery (a.k.a., detection). The
main results discussed are the phase transitions for exact recovery at the
Chernoff-Hellinger threshold, the phase transition for weak recovery at the
Kesten-Stigum threshold, the optimal distortion-SNR tradeoff for partial
recovery, the learning of the SBM parameters and the gap between
information-theoretic and computational thresholds.
The note also covers some of the algorithms developed in the quest of
achieving the limits, in particular two-round algorithms via graph-splitting,
semi-definite programming, linearized belief propagation, classical and
nonbacktracking spectral methods. A few open problems are also discussed
Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
Factorizing low-rank matrices has many applications in machine learning and
statistics. For probabilistic models in the Bayes optimal setting, a general
expression for the mutual information has been proposed using heuristic
statistical physics computations, and proven in few specific cases. Here, we
show how to rigorously prove the conjectured formula for the symmetric rank-one
case. This allows to express the minimal mean-square-error and to characterize
the detectability phase transitions in a large set of estimation problems
ranging from community detection to sparse PCA. We also show that for a large
set of parameters, an iterative algorithm called approximate message-passing is
Bayes optimal. There exists, however, a gap between what currently known
polynomial algorithms can do and what is expected information theoretically.
Additionally, the proof technique has an interest of its own and exploits three
essential ingredients: the interpolation method introduced in statistical
physics by Guerra, the analysis of the approximate message-passing algorithm
and the theory of spatial coupling and threshold saturation in coding. Our
approach is generic and applicable to other open problems in statistical
estimation where heuristic statistical physics predictions are available
Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization
We study the problem of detecting a structured, low-rank signal matrix
corrupted with additive Gaussian noise. This includes clustering in a Gaussian
mixture model, sparse PCA, and submatrix localization. Each of these problems
is conjectured to exhibit a sharp information-theoretic threshold, below which
the signal is too weak for any algorithm to detect. We derive upper and lower
bounds on these thresholds by applying the first and second moment methods to
the likelihood ratio between these "planted models" and null models where the
signal matrix is zero. Our bounds differ by at most a factor of root two when
the rank is large (in the clustering and submatrix localization problems, when
the number of clusters or blocks is large) or the signal matrix is very sparse.
Moreover, our upper bounds show that for each of these problems there is a
significant regime where reliable detection is information- theoretically
possible but where known algorithms such as PCA fail completely, since the
spectrum of the observed matrix is uninformative. This regime is analogous to
the conjectured 'hard but detectable' regime for community detection in sparse
graphs.Comment: For sparse PCA and submatrix localization, we determine the
information-theoretic threshold exactly in the limit where the number of
blocks is large or the signal matrix is very sparse based on a conditional
second moment method, closing the factor of root two gap in the first versio
Mutual Information in Rank-One Matrix Estimation
We consider the estimation of a n-dimensional vector x from the knowledge of
noisy and possibility non-linear element-wise measurements of xxT , a very
generic problem that contains, e.g. stochastic 2-block model, submatrix
localization or the spike perturbation of random matrices. We use an
interpolation method proposed by Guerra and later refined by Korada and Macris.
We prove that the Bethe mutual information (related to the Bethe free energy
and conjectured to be exact by Lesieur et al. on the basis of the non-rigorous
cavity method) always yields an upper bound to the exact mutual information. We
also provide a lower bound using a similar technique. For concreteness, we
illustrate our findings on the sparse PCA problem, and observe that (a) our
bounds match for a large region of parameters and (b) that it exists a phase
transition in a region where the spectum remains uninformative. While we
present only the case of rank-one symmetric matrix estimation, our proof
technique is readily extendable to low-rank symmetric matrix or low-rank
symmetric tensor estimationComment: 8 pages, 1 figure
- …