1,146 research outputs found
Expectation Propagation for Approximate Inference: Free Probability Framework
We study asymptotic properties of expectation propagation (EP) -- a method
for approximate inference originally developed in the field of machine
learning. Applied to generalized linear models, EP iteratively computes a
multivariate Gaussian approximation to the exact posterior distribution. The
computational complexity of the repeated update of covariance matrices severely
limits the application of EP to large problem sizes. In this study, we present
a rigorous analysis by means of free probability theory that allows us to
overcome this computational bottleneck if specific data matrices in the problem
fulfill certain properties of asymptotic freeness. We demonstrate the relevance
of our approach on the gene selection problem of a microarray dataset.Comment: Both authors are co-first authors. The main body of this paper is
accepted for publication in the proceedings of the 2018 IEEE International
Symposium on Information Theory (ISIT
Statistical Mechanics of Support Vector Networks
Using methods of Statistical Physics, we investigate the generalization
performance of support vector machines (SVMs), which have been recently
introduced as a general alternative to neural networks. For nonlinear
classification rules, the generalization error saturates on a plateau, when the
number of examples is too small to properly estimate the coefficients of the
nonlinear part. When trained on simple rules, we find that SVMs overfit only
weakly. The performance of SVMs is strongly enhanced, when the distribution of
the inputs has a gap in feature space.Comment: REVTeX, 4 pages, 2 figures, accepted by Phys. Rev. Lett (typos
corrected
A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
We consider the problem of solving TAP mean field equations by iteration for
Ising model with coupling matrices that are drawn at random from general
invariant ensembles. We develop an analysis of iterative algorithms using a
dynamical functional approach that in the thermodynamic limit yields an
effective dynamics of a single variable trajectory. Our main novel contribution
is the expression for the implicit memory term of the dynamics for general
invariant ensembles. By subtracting these terms, that depend on magnetizations
at previous time steps, the implicit memory terms cancel making the iteration
dependent on a Gaussian distributed field only. The TAP magnetizations are
stable fixed points if an AT stability criterion is fulfilled. We illustrate
our method explicitly for coupling matrices drawn from the random orthogonal
ensemble.Comment: 27 pages, 6 Figures Published in Journal of Physics A: Mathematical
and Theoretical, Volume 49, Number 11, 201
- …