8 research outputs found
Efficient inference in stochastic block models with vertex labels
We study the stochastic block model with two communities where vertices
contain side information in the form of a vertex label. These vertex labels may
have arbitrary label distributions, depending on the community memberships. We
analyze a linearized version of the popular belief propagation algorithm. We
show that this algorithm achieves the highest accuracy possible whenever a
certain function of the network parameters has a unique fixed point. Whenever
this function has multiple fixed points, the belief propagation algorithm may
not perform optimally. We show that increasing the information in the vertex
labels may reduce the number of fixed points and hence lead to optimality of
belief propagation
Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap
A simple model to study subspace clustering is the high-dimensional
-Gaussian mixture model where the cluster means are sparse vectors. Here we
provide an exact asymptotic characterization of the statistically optimal
reconstruction error in this model in the high-dimensional regime with
extensive sparsity, i.e. when the fraction of non-zero components of the
cluster means , as well as the ratio between the number of
samples and the dimension are fixed, while the dimension diverges. We identify
the information-theoretic threshold below which obtaining a positive
correlation with the true cluster means is statistically impossible.
Additionally, we investigate the performance of the approximate message passing
(AMP) algorithm analyzed via its state evolution, which is conjectured to be
optimal among polynomial algorithm for this task. We identify in particular the
existence of a statistical-to-computational gap between the algorithm that
require a signal-to-noise ratio
to perform better than random, and the information theoretic threshold at
.
Finally, we discuss the case of sub-extensive sparsity by comparing the
performance of the AMP with other sparsity-enhancing algorithms, such as
sparse-PCA and diagonal thresholding.Comment: NeurIPS camera-ready versio
Statistical Mechanics of Generalization In Graph Convolution Networks
Graph neural networks (GNN) have become the default machine learning model
for relational datasets, including protein interaction networks, biological
neural networks, and scientific collaboration graphs. We use tools from
statistical physics and random matrix theory to precisely characterize
generalization in simple graph convolution networks on the contextual
stochastic block model. The derived curves are phenomenologically rich: they
explain the distinction between learning on homophilic and heterophilic graphs
and they predict double descent whose existence in GNNs has been questioned by
recent work. Our results are the first to accurately explain the behavior not
only of a stylized graph learning model but also of complex GNNs on messy
real-world datasets. To wit, we use our analytic insights about homophily and
heterophily to improve performance of state-of-the-art graph neural networks on
several heterophilic benchmarks by a simple addition of negative self-loop
filters
Phase transitions in spiked matrix estimation: information-theoretic analysis
We study here the so-called spiked Wigner and Wishart models, where one
observes a low-rank matrix perturbed by some Gaussian noise. These models
encompass many classical statistical tasks such as sparse PCA, submatrix
localization, community detection or Gaussian mixture clustering. The goal of
these notes is to present in a unified manner recent results (as well as new
developments) on the information-theoretic limits of these spiked matrix
models. We compute the minimal mean squared error for the estimation of the
low-rank signal and compare it to the performance of spectral estimators and
message passing algorithms. Phase transition phenomena are observed: depending
on the noise level it is either impossible, easy (i.e. using polynomial-time
estimators) or hard (information-theoretically possible, but no efficient
algorithm is known to succeed) to recover the signal.Comment: These notes present in a unified manner recent results (as well as
new developments) on the information-theoretic limits in spiked matrix
estimatio