10,972 research outputs found
Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (or : How to Prove Kabashima's Replica Formula)
There has been a recent surge of interest in the study of asymptotic
reconstruction performance in various cases of generalized linear estimation
problems in the teacher-student setting, especially for the case of i.i.d
standard normal matrices. Here, we go beyond these matrices, and prove an
analytical formula for the reconstruction performance of convex generalized
linear models with rotationally-invariant data matrices with arbitrary bounded
spectrum, rigorously confirming a conjecture originally derived using the
replica method from statistical physics. The formula includes many problems
such as compressed sensing or sparse logistic classification. The proof is
achieved by leveraging on message passing algorithms and the statistical
properties of their iterates, allowing to characterize the asymptotic empirical
distribution of the estimator. Our proof is crucially based on the construction
of converging sequences of an oracle multi-layer vector approximate message
passing algorithm, where the convergence analysis is done by checking the
stability of an equivalent dynamical system. We illustrate our claim with
numerical examples on mainstream learning methods such as sparse logistic
regression and linear support vector classifiers, showing excellent agreement
between moderate size simulation and the asymptotic prediction.Comment: 19 pages,25 appendix,4 figure
On the convergence of gaussian belief propagation with nodes of arbitrary size
CITATION: Kamper, F., Steel, S. J. & Du Preez, J. A. 2019. On the convergence of Gaussian belief propagation with nodes of arbitrary size.
Journal of Machine Learning Research, 20(165):1-37.The original publication is available at http://jmlr.orgThis paper is concerned with a multivariate extension of Gaussian message passing applied to pairwise Markov graphs (MGs). Gaussian message passing applied to pairwise MGs is often labeled Gaussian belief propagation (GaBP) and can be used to approximate the marginal of each variable contained in the pairwise MG. We propose a multivariate extension of GaBP (we label this GaBP-m) that can be used to estimate higher-dimensional marginals. Beyond the ability to estimate higher-dimensional marginals, GaBP-m exhibits better convergence behavior than GaBP, and can also provide more accurate univariate marginals. The theoretical results of this paper are based on an extension of the computation tree analysis conducted on univariate nodes to the multivariate case. The main contribution of this paper is the development of a convergence condition for GaBP-m that moves beyond the walk-summability of the precision matrix. Based on this convergence condition, we derived an upper bound for the number of iterations required for convergence of the GaBP-m algorithm. An upper bound on the dissimilarity between the approximate and exact marginal covariance matrices was established. We argue that GaBP-m is robust towards a certain change in variables, a property not shared by iterative solvers of linear systems, such as the conjugate gradient (CG) and preconditioned conjugate gradient (PCG) methods. The advantages of using GaBP-m over GaBP are also illustrated empirically.https://www.jmlr.org/papers/v20/18-040.htmlPublisher's versio
Message-Passing Algorithms for Quadratic Minimization
Gaussian belief propagation (GaBP) is an iterative algorithm for computing
the mean of a multivariate Gaussian distribution, or equivalently, the minimum
of a multivariate positive definite quadratic function. Sufficient conditions,
such as walk-summability, that guarantee the convergence and correctness of
GaBP are known, but GaBP may fail to converge to the correct solution given an
arbitrary positive definite quadratic function. As was observed in previous
work, the GaBP algorithm fails to converge if the computation trees produced by
the algorithm are not positive definite. In this work, we will show that the
failure modes of the GaBP algorithm can be understood via graph covers, and we
prove that a parameterized generalization of the min-sum algorithm can be used
to ensure that the computation trees remain positive definite whenever the
input matrix is positive definite. We demonstrate that the resulting algorithm
is closely related to other iterative schemes for quadratic minimization such
as the Gauss-Seidel and Jacobi algorithms. Finally, we observe, empirically,
that there always exists a choice of parameters such that the above
generalization of the GaBP algorithm converges
- …