3 research outputs found
Message-Passing Algorithms: Reparameterizations and Splittings
The max-product algorithm, a local message-passing scheme that attempts to
compute the most probable assignment (MAP) of a given probability distribution,
has been successfully employed as a method of approximate inference for
applications arising in coding theory, computer vision, and machine learning.
However, the max-product algorithm is not guaranteed to converge to the MAP
assignment, and if it does, is not guaranteed to recover the MAP assignment.
Alternative convergent message-passing schemes have been proposed to overcome
these difficulties. This work provides a systematic study of such
message-passing algorithms that extends the known results by exhibiting new
sufficient conditions for convergence to local and/or global optima, providing
a combinatorial characterization of these optima based on graph covers, and
describing a new convergent and correct message-passing algorithm whose
derivation unifies many of the known convergent message-passing algorithms.
While convergent and correct message-passing algorithms represent a step
forward in the analysis of max-product style message-passing algorithms, the
conditions needed to guarantee convergence to a global optimum can be too
restrictive in both theory and practice. This limitation of convergent and
correct message-passing schemes is characterized by graph covers and
illustrated by example.Comment: A complete rework and expansion of the previous version
Counting in Graph Covers: A Combinatorial Characterization of the Bethe Entropy Function
We present a combinatorial characterization of the Bethe entropy function of
a factor graph, such a characterization being in contrast to the original,
analytical, definition of this function. We achieve this combinatorial
characterization by counting valid configurations in finite graph covers of the
factor graph. Analogously, we give a combinatorial characterization of the
Bethe partition function, whose original definition was also of an analytical
nature. As we point out, our approach has similarities to the replica method,
but also stark differences. The above findings are a natural backdrop for
introducing a decoder for graph-based codes that we will call symbolwise
graph-cover decoding, a decoder that extends our earlier work on blockwise
graph-cover decoding. Both graph-cover decoders are theoretical tools that help
towards a better understanding of message-passing iterative decoding, namely
blockwise graph-cover decoding links max-product (min-sum) algorithm decoding
with linear programming decoding, and symbolwise graph-cover decoding links
sum-product algorithm decoding with Bethe free energy function minimization at
temperature one. In contrast to the Gibbs entropy function, which is a concave
function, the Bethe entropy function is in general not concave everywhere. In
particular, we show that every code picked from an ensemble of regular
low-density parity-check codes with minimum Hamming distance growing (with high
probability) linearly with the block length has a Bethe entropy function that
is convex in certain regions of its domain.Comment: Submitted to IEEE Trans. Inf. Theory, Nov. 20, 2010; rev. Sep. 22,
2012; current version, Oct. 9, 2012. Main changes from v1 to v2: new example
(Example 34), new lemma (Lemma 35), changed some notation, changed the domain
of the Gibbs free energy function and related functions, reordered some
sections/appendices, fixed some typos, improved the background discussion,
added some new reference
New Understanding of the Bethe Approximation and the Replica Method
In this thesis, new generalizations of the Bethe approximation and new
understanding of the replica method are proposed. The Bethe approximation is an
efficient approximation for graphical models, which gives an asymptotically
accurate estimate of the partition function for many graphical models. The
Bethe approximation explains the well-known message passing algorithm, belief
propagation, which is exact for tree graphical models. It is also known that
the cluster variational method gives the generalized Bethe approximation,
called the Kikuchi approximation, yielding the generalized belief propagation.
In the thesis, a new series of generalization of the Bethe approximation is
proposed, which is named the asymptotic Bethe approximation. The asymptotic
Bethe approximation is derived from the characterization of the Bethe free
energy using graph covers, which was recently obtained by Vontobel. The
asymptotic Bethe approximation can be expressed in terms of the edge zeta
function by using Watanabe and Fukumizu's result about the Hessian of the Bethe
entropy. The asymptotic Bethe approximation is confirmed to be better than the
conventional Bethe approximation on some conditions. For this purpose, Chertkov
and Chernyak's loop calculus formula is employed, which shows that the error of
the Bethe approximation can be expressed as a sum of weights corresponding to
generalized loops, and generalized for non-binary finite alphabets by using
concepts of information geometry.Comment: Doctoral thesi