Truncating the loop series expansion for Belief Propagation

Recently, M. Chertkov and V.Y. Chernyak derived an exact expression for the partition sum (normalization constant) corresponding to a graphical model, which is an expansion around the Belief Propagation solution. By adding correction terms to the BP free energy, one for each "generalized loop" in the factor graph, the exact partition sum is obtained. However, the usually enormous number of generalized loops generally prohibits summation over all correction terms. In this article we introduce Truncated Loop Series BP (TLSBP), a particular way of truncating the loop series of M. Chertkov and V.Y. Chernyak by considering generalized loops as compositions of simple loops. We analyze the performance of TLSBP in different scenarios, including the Ising model, regular random graphs and on Promedas, a large probabilistic medical diagnostic system. We show that TLSBP often improves upon the accuracy of the BP solution, at the expense of increased computation time. We also show that the performance of TLSBP strongly depends on the degree of interaction between the variables. For weak interactions, truncating the series leads to significant improvements, whereas for strong interactions it can be ineffective, even if a high number of terms is considered.Comment: 31 pages, 12 figures, submitted to Journal of Machine Learning Researc

Belief Propagation and Loop Series on Planar Graphs

We discuss a generic model of Bayesian inference with binary variables defined on edges of a planar graph. The Loop Calculus approach of [1, 2] is used to evaluate the resulting series expansion for the partition function. We show that, for planar graphs, truncating the series at single-connected loops reduces, via a map reminiscent of the Fisher transformation [3], to evaluating the partition function of the dimer matching model on an auxiliary planar graph. Thus, the truncated series can be easily re-summed, using the Pfaffian formula of Kasteleyn [4]. This allows to identify a big class of computationally tractable planar models reducible to a dimer model via the Belief Propagation (gauge) transformation. The Pfaffian representation can also be extended to the full Loop Series, in which case the expansion becomes a sum of Pfaffian contributions, each associated with dimer matchings on an extension to a subgraph of the original graph. Algorithmic consequences of the Pfaffian representation, as well as relations to quantum and non-planar models, are discussed.Comment: Accepted for publication in Journal of Statistical Mechanics: theory and experimen

Loop Calculus for Non-Binary Alphabets using Concepts from Information Geometry

The Bethe approximation is a well-known approximation of the partition function used in statistical physics. Recently, an equality relating the partition function and its Bethe approximation was obtained for graphical models with binary variables by Chertkov and Chernyak. In this equality, the multiplicative error in the Bethe approximation is represented as a weighted sum over all generalized loops in the graphical model. In this paper, the equality is generalized to graphical models with non-binary alphabet using concepts from information geometry.Comment: 18 pages, 4 figures, submitted to IEEE Trans. Inf. Theor

Finite size corrections to disordered systems on Erd\"{o}s-R\'enyi random graphs

We study the finite size corrections to the free energy density in disorder spin systems on sparse random graphs, using both replica theory and cavity method. We derive an analytical expressions for the $O(1/N)$ corrections in the replica symmetric phase as a linear combination of the free energies of open and closed chains. We perform a numerical check of the formulae on the Random Field Ising Model at zero temperature, by computing finite size corrections to the ground state energy density.Comment: Submitted to PR

Sums over geometries and improvements on the mean field approximation

The saddle points of a Lagrangian due to Efetov are analyzed. This Lagrangian was originally proposed as a tool for calculating systematic corrections to the Bethe approximation, a mean-field approximation which is important in statistical mechanics, glasses, coding theory, and combinatorial optimization. Detailed analysis shows that the trivial saddle point generates a sum over geometries reminiscent of dynamically triangulated quantum gravity, which suggests new possibilities to design sums over geometries for the specific purpose of obtaining improved mean field approximations to $D$-dimensional theories. In the case of the Efetov theory, the dominant geometries are locally tree-like, and the sum over geometries diverges in a way that is similar to quantum gravity's divergence when all topologies are included. Expertise from the field of dynamically triangulated quantum gravity about sums over geometries may be able to remedy these defects and fulfill the Efetov theory's original promise. The other saddle points of the Efetov Lagrangian are also analyzed; the Hessian at these points is nonnormal and pseudo-Hermitian, which is unusual for bosonic theories. The standard formula for Gaussian integrals is generalized to nonnormal kernels.Comment: Accepted for publication in Physical Review D, probably in November 2007. At the reviewer's request, material was added which made the article more assertive, confident, and clear. No changes in substanc