3 research outputs found
Understanding Phase Transitions via Mutual Information and MMSE
The ability to understand and solve high-dimensional inference problems is
essential for modern data science. This article examines high-dimensional
inference problems through the lens of information theory and focuses on the
standard linear model as a canonical example that is both rich enough to be
practically useful and simple enough to be studied rigorously. In particular,
this model can exhibit phase transitions where an arbitrarily small change in
the model parameters can induce large changes in the quality of estimates. For
this model, the performance of optimal inference can be studied using the
replica method from statistical physics but, until recently, it was not known
if the resulting formulas were actually correct. In this chapter, we present a
tutorial description of the standard linear model and its connection to
information theory. We also describe the replica prediction for this model and
outline the authors' recent proof that it is exact
Information-theoretic limits of a multiview low-rank symmetric spiked matrix model
We consider a generalization of an important class of high-dimensional
inference problems, namely spiked symmetric matrix models, often used as
probabilistic models for principal component analysis. Such paradigmatic models
have recently attracted a lot of attention from a number of communities due to
their phenomenological richness with statistical-to-computational gaps, while
remaining tractable. We rigorously establish the information-theoretic limits
through the proof of single-letter formulas for the mutual information and
minimum mean-square error. On a technical side we improve the recently
introduced adaptive interpolation method, so that it can be used to study
low-rank models (i.e., estimation problems of "tall matrices") in full
generality, an important step towards the rigorous analysis of more complicated
inference and learning models.Comment: Presented at the 2020 International Symposium on Information Theory
(ISIT
Information-Theoretic Limits for the Matrix Tensor Product
This paper studies a high-dimensional inference problem involving the matrix
tensor product of random matrices. This problem generalizes a number of
contemporary data science problems including the spiked matrix models used in
sparse principal component analysis and covariance estimation and the
stochastic block model used in network analysis. The main results are
single-letter formulas (i.e., analytical expressions that can be approximated
numerically) for the mutual information and the minimum mean-squared error
(MMSE) in the Bayes optimal setting where the distributions of all random
quantities are known. We provide non-asymptotic bounds and show that our
formulas describe exactly the leading order terms in the mutual information and
MMSE in the high-dimensional regime where the number of rows and number of
columns scale with for some .
On the technical side, this paper introduces some new techniques for the
analysis of high-dimensional matrix-valued signals. Specific contributions
include a novel extension of the adaptive interpolation method that uses
order-preserving positive semidefinite interpolation paths, and a variance
inequality between the overlap and the free energy that is based on
continuous-time I-MMSE relations