8,763 research outputs found
Phase Transitions in Sparse PCA
We study optimal estimation for sparse principal component analysis when the
number of non-zero elements is small but on the same order as the dimension of
the data. We employ approximate message passing (AMP) algorithm and its state
evolution to analyze what is the information theoretically minimal mean-squared
error and the one achieved by AMP in the limit of large sizes. For a special
case of rank one and large enough density of non-zeros Deshpande and Montanari
[1] proved that AMP is asymptotically optimal. We show that both for low
density and for large rank the problem undergoes a series of phase transitions
suggesting existence of a region of parameters where estimation is information
theoretically possible, but AMP (and presumably every other polynomial
algorithm) fails. The analysis of the large rank limit is particularly
instructive.Comment: 6 pages, 3 figure
Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization
We study the problem of detecting a structured, low-rank signal matrix
corrupted with additive Gaussian noise. This includes clustering in a Gaussian
mixture model, sparse PCA, and submatrix localization. Each of these problems
is conjectured to exhibit a sharp information-theoretic threshold, below which
the signal is too weak for any algorithm to detect. We derive upper and lower
bounds on these thresholds by applying the first and second moment methods to
the likelihood ratio between these "planted models" and null models where the
signal matrix is zero. Our bounds differ by at most a factor of root two when
the rank is large (in the clustering and submatrix localization problems, when
the number of clusters or blocks is large) or the signal matrix is very sparse.
Moreover, our upper bounds show that for each of these problems there is a
significant regime where reliable detection is information- theoretically
possible but where known algorithms such as PCA fail completely, since the
spectrum of the observed matrix is uninformative. This regime is analogous to
the conjectured 'hard but detectable' regime for community detection in sparse
graphs.Comment: For sparse PCA and submatrix localization, we determine the
information-theoretic threshold exactly in the limit where the number of
blocks is large or the signal matrix is very sparse based on a conditional
second moment method, closing the factor of root two gap in the first versio
Statistical Physics and Representations in Real and Artificial Neural Networks
This document presents the material of two lectures on statistical physics
and neural representations, delivered by one of us (R.M.) at the Fundamental
Problems in Statistical Physics XIV summer school in July 2017. In a first
part, we consider the neural representations of space (maps) in the
hippocampus. We introduce an extension of the Hopfield model, able to store
multiple spatial maps as continuous, finite-dimensional attractors. The phase
diagram and dynamical properties of the model are analyzed. We then show how
spatial representations can be dynamically decoded using an effective Ising
model capturing the correlation structure in the neural data, and compare
applications to data obtained from hippocampal multi-electrode recordings and
by (sub)sampling our attractor model. In a second part, we focus on the problem
of learning data representations in machine learning, in particular with
artificial neural networks. We start by introducing data representations
through some illustrations. We then analyze two important algorithms, Principal
Component Analysis and Restricted Boltzmann Machines, with tools from
statistical physics
X-Ray and UV Orbital Phase Dependence in LMC X-3
The black-hole binary LMC X-3 is known to be variable on time scales of days
to years. We investigate X-ray and ultraviolet variability in the system as a
function of the 1.7 day binary phase using a 6.4 day observation with the Rossi
X-ray Timing Explorer (RXTE) from December 1998. An abrupt 14% flux decrease,
lasting nearly an entire orbit, is followed by a return to previous flux
levels. This behavior occurs twice, at nearly the same binary phase, but it is
not present in consecutive orbits. When the X-ray flux is at lower intensity, a
periodic amplitude modulation of 7% is evident in data folded modulo the
orbital period. The higher intensity data show weaker correlation with phase.
This is the first report of X-ray variability at the orbital period of LMC X-3.
Archival RXTE observations of LMC X--3 during a high flux state in December
1996 show similar phase dependence. An ultraviolet light curve obtained with
the High Speed Photometer aboard the Hubble Space Telescope shows orbital
modulation consistent with that in the optical, caused by the ellipsoidal
variation of the spatially deformed companion.
The X-ray spectrum of LMC X-3 can be acceptably represented by a
phenomenological disk-black-body plus a power law. Changes in the spectrum of
LMC X-3 during our observations are compatible with earlier observations during
which variations in the 2-10 keV flux are tracked closely by the disk geometry
spectral model parameter.Comment: 11 pages, 7 figures, ApJ in pres
Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
Factorizing low-rank matrices has many applications in machine learning and
statistics. For probabilistic models in the Bayes optimal setting, a general
expression for the mutual information has been proposed using heuristic
statistical physics computations, and proven in few specific cases. Here, we
show how to rigorously prove the conjectured formula for the symmetric rank-one
case. This allows to express the minimal mean-square-error and to characterize
the detectability phase transitions in a large set of estimation problems
ranging from community detection to sparse PCA. We also show that for a large
set of parameters, an iterative algorithm called approximate message-passing is
Bayes optimal. There exists, however, a gap between what currently known
polynomial algorithms can do and what is expected information theoretically.
Additionally, the proof technique has an interest of its own and exploits three
essential ingredients: the interpolation method introduced in statistical
physics by Guerra, the analysis of the approximate message-passing algorithm
and the theory of spatial coupling and threshold saturation in coding. Our
approach is generic and applicable to other open problems in statistical
estimation where heuristic statistical physics predictions are available
- …