3,831 research outputs found
Fixing Convergence of Gaussian Belief Propagation
Gaussian belief propagation (GaBP) is an iterative message-passing algorithm
for inference in Gaussian graphical models. It is known that when GaBP
converges it converges to the correct MAP estimate of the Gaussian random
vector and simple sufficient conditions for its convergence have been
established. In this paper we develop a double-loop algorithm for forcing
convergence of GaBP. Our method computes the correct MAP estimate even in cases
where standard GaBP would not have converged. We further extend this
construction to compute least-squares solutions of over-constrained linear
systems. We believe that our construction has numerous applications, since the
GaBP algorithm is linked to solution of linear systems of equations, which is a
fundamental problem in computer science and engineering. As a case study, we
discuss the linear detection problem. We show that using our new construction,
we are able to force convergence of Montanari's linear detection algorithm, in
cases where it would originally fail. As a consequence, we are able to increase
significantly the number of users that can transmit concurrently.Comment: In the IEEE International Symposium on Information Theory (ISIT)
2009, Seoul, South Korea, July 200
Polynomial Linear Programming with Gaussian Belief Propagation
Interior-point methods are state-of-the-art algorithms for solving linear
programming (LP) problems with polynomial complexity. Specifically, the
Karmarkar algorithm typically solves LP problems in time O(n^{3.5}), where
is the number of unknown variables. Karmarkar's celebrated algorithm is known
to be an instance of the log-barrier method using the Newton iteration. The
main computational overhead of this method is in inverting the Hessian matrix
of the Newton iteration. In this contribution, we propose the application of
the Gaussian belief propagation (GaBP) algorithm as part of an efficient and
distributed LP solver that exploits the sparse and symmetric structure of the
Hessian matrix and avoids the need for direct matrix inversion. This approach
shifts the computation from realm of linear algebra to that of probabilistic
inference on graphical models, thus applying GaBP as an efficient inference
engine. Our construction is general and can be used for any interior-point
algorithm which uses the Newton method, including non-linear program solvers.Comment: 7 pages, 1 figure, appeared in the 46th Annual Allerton Conference on
Communication, Control and Computing, Allerton House, Illinois, Sept. 200
A Low Density Lattice Decoder via Non-Parametric Belief Propagation
The recent work of Sommer, Feder and Shalvi presented a new family of codes
called low density lattice codes (LDLC) that can be decoded efficiently and
approach the capacity of the AWGN channel. A linear time iterative decoding
scheme which is based on a message-passing formulation on a factor graph is
given.
In the current work we report our theoretical findings regarding the relation
between the LDLC decoder and belief propagation. We show that the LDLC decoder
is an instance of non-parametric belief propagation and further connect it to
the Gaussian belief propagation algorithm. Our new results enable borrowing
knowledge from the non-parametric and Gaussian belief propagation domains into
the LDLC domain. Specifically, we give more general convergence conditions for
convergence of the LDLC decoder (under the same assumptions of the original
LDLC convergence analysis). We discuss how to extend the LDLC decoder from
Latin square to full rank, non-square matrices. We propose an efficient
construction of sparse generator matrix and its matching decoder. We report
preliminary experimental results which show our decoder has comparable symbol
to error rate compared to the original LDLC decoder.%Comment: Submitted for publicatio
Distributed Large Scale Network Utility Maximization
Recent work by Zymnis et al. proposes an efficient primal-dual interior-point
method, using a truncated Newton method, for solving the network utility
maximization (NUM) problem. This method has shown superior performance relative
to the traditional dual-decomposition approach. Other recent work by Bickson et
al. shows how to compute efficiently and distributively the Newton step, which
is the main computational bottleneck of the Newton method, utilizing the
Gaussian belief propagation algorithm.
In the current work, we combine both approaches to create an efficient
distributed algorithm for solving the NUM problem. Unlike the work of Zymnis,
which uses a centralized approach, our new algorithm is easily distributed.
Using an empirical evaluation we show that our new method outperforms previous
approaches, including the truncated Newton method and dual-decomposition
methods. As an additional contribution, this is the first work that evaluates
the performance of the Gaussian belief propagation algorithm vs. the
preconditioned conjugate gradient method, for a large scale problem.Comment: In the International Symposium on Information Theory (ISIT) 200
Message-Passing Algorithms for Quadratic Minimization
Gaussian belief propagation (GaBP) is an iterative algorithm for computing
the mean of a multivariate Gaussian distribution, or equivalently, the minimum
of a multivariate positive definite quadratic function. Sufficient conditions,
such as walk-summability, that guarantee the convergence and correctness of
GaBP are known, but GaBP may fail to converge to the correct solution given an
arbitrary positive definite quadratic function. As was observed in previous
work, the GaBP algorithm fails to converge if the computation trees produced by
the algorithm are not positive definite. In this work, we will show that the
failure modes of the GaBP algorithm can be understood via graph covers, and we
prove that a parameterized generalization of the min-sum algorithm can be used
to ensure that the computation trees remain positive definite whenever the
input matrix is positive definite. We demonstrate that the resulting algorithm
is closely related to other iterative schemes for quadratic minimization such
as the Gauss-Seidel and Jacobi algorithms. Finally, we observe, empirically,
that there always exists a choice of parameters such that the above
generalization of the GaBP algorithm converges
A New Approach to Speeding Up Topic Modeling
Latent Dirichlet allocation (LDA) is a widely-used probabilistic topic
modeling paradigm, and recently finds many applications in computer vision and
computational biology. In this paper, we propose a fast and accurate batch
algorithm, active belief propagation (ABP), for training LDA. Usually batch LDA
algorithms require repeated scanning of the entire corpus and searching the
complete topic space. To process massive corpora having a large number of
topics, the training iteration of batch LDA algorithms is often inefficient and
time-consuming. To accelerate the training speed, ABP actively scans the subset
of corpus and searches the subset of topic space for topic modeling, therefore
saves enormous training time in each iteration. To ensure accuracy, ABP selects
only those documents and topics that contribute to the largest residuals within
the residual belief propagation (RBP) framework. On four real-world corpora,
ABP performs around to times faster than state-of-the-art batch LDA
algorithms with a comparable topic modeling accuracy.Comment: 14 pages, 12 figure
- …