2 research outputs found
Statistical physics of linear and bilinear inference problems
The recent development of compressed sensing has led to spectacular advances
in the understanding of sparse linear estimation problems as well as in
algorithms to solve them. It has also triggered a new wave of developments in
the related fields of generalized linear and bilinear inference problems, that
have very diverse applications in signal processing and are furthermore a
building block of deep neural networks. These problems have in common that they
combine a linear mixing step and a nonlinear, probabilistic sensing step,
producing indirect measurements of a signal of interest. Such a setting arises
in problems as different as medical or astronomical imaging, clustering, matrix
completion or blind source separation. The aim of this thesis is to propose
efficient algorithms for this class of problems and to perform their
theoretical analysis. To this end, it uses belief propagation, thanks to which
high-dimensional distributions can be sampled efficiently, thus making a
Bayesian approach to inference tractable. The resulting algorithms undergo
phase transitions just as physical systems do. These phase transitions can be
analyzed using the replica method, initially developed in statistical physics
of disordered systems. The analysis reveals phases in which inference is easy,
hard or impossible. These phases correspond to different energy landscapes of
the problem. The main contributions of this thesis can be divided into three
categories. First, the application of known algorithms to concrete problems:
community detection, superposition codes and an innovative imaging system.
Second, a new, efficient message-passing algorithm for a class of problems
called blind sensor calibration. Third, a theoretical analysis of matrix
compressed sensing and of instabilities in Bayesian bilinear inference
algorithms.Comment: Phd thesi
Mean-field methods and algorithmic perspectives for high-dimensional machine learning
The main difficulty that arises in the analysis of most machine learning
algorithms is to handle, analytically and numerically, a large number of
interacting random variables. In this Ph.D manuscript, we revisit an approach
based on the tools of statistical physics of disordered systems. Developed
through a rich literature, they have been precisely designed to infer the
macroscopic behavior of a large number of particles from their microscopic
interactions. At the heart of this work, we strongly capitalize on the deep
connection between the replica method and message passing algorithms in order
to shed light on the phase diagrams of various theoretical models, with an
emphasis on the potential differences between statistical and algorithmic
thresholds. We essentially focus on synthetic tasks and data generated in the
teacher-student paradigm. In particular, we apply these mean-field methods to
the Bayes-optimal analysis of committee machines, to the worst-case analysis of
Rademacher generalization bounds for perceptrons, and to empirical risk
minimization in the context of generalized linear models. Finally, we develop a
framework to analyze estimation models with structured prior informations,
produced for instance by deep neural networks based generative models with
random weights.Comment: Ph.D manuscrip