5,060 research outputs found
The Bregman Variational Dual-Tree Framework
Graph-based methods provide a powerful tool set for many non-parametric
frameworks in Machine Learning. In general, the memory and computational
complexity of these methods is quadratic in the number of examples in the data
which makes them quickly infeasible for moderate to large scale datasets. A
significant effort to find more efficient solutions to the problem has been
made in the literature. One of the state-of-the-art methods that has been
recently introduced is the Variational Dual-Tree (VDT) framework. Despite some
of its unique features, VDT is currently restricted only to Euclidean spaces
where the Euclidean distance quantifies the similarity. In this paper, we
extend the VDT framework beyond the Euclidean distance to more general Bregman
divergences that include the Euclidean distance as a special case. By
exploiting the properties of the general Bregman divergence, we show how the
new framework can maintain all the pivotal features of the VDT framework and
yet significantly improve its performance in non-Euclidean domains. We apply
the proposed framework to different text categorization problems and
demonstrate its benefits over the original VDT.Comment: Appears in Proceedings of the Twenty-Ninth Conference on Uncertainty
in Artificial Intelligence (UAI2013
Cycle-based Cluster Variational Method for Direct and Inverse Inference
We elaborate on the idea that loop corrections to belief propagation could be
dealt with in a systematic way on pairwise Markov random fields, by using the
elements of a cycle basis to define region in a generalized belief propagation
setting. The region graph is specified in such a way as to avoid dual loops as
much as possible, by discarding redundant Lagrange multipliers, in order to
facilitate the convergence, while avoiding instabilities associated to minimal
factor graph construction. We end up with a two-level algorithm, where a belief
propagation algorithm is run alternatively at the level of each cycle and at
the inter-region level. The inverse problem of finding the couplings of a
Markov random field from empirical covariances can be addressed region wise. It
turns out that this can be done efficiently in particular in the Ising context,
where fixed point equations can be derived along with a one-parameter log
likelihood function to minimize. Numerical experiments confirm the
effectiveness of these considerations both for the direct and inverse MRF
inference.Comment: 47 pages, 16 figure
Characterizing and Improving Generalized Belief Propagation Algorithms on the 2D Edwards-Anderson Model
We study the performance of different message passing algorithms in the two
dimensional Edwards Anderson model. We show that the standard Belief
Propagation (BP) algorithm converges only at high temperature to a paramagnetic
solution. Then, we test a Generalized Belief Propagation (GBP) algorithm,
derived from a Cluster Variational Method (CVM) at the plaquette level. We
compare its performance with BP and with other algorithms derived under the
same approximation: Double Loop (DL) and a two-ways message passing algorithm
(HAK). The plaquette-CVM approximation improves BP in at least three ways: the
quality of the paramagnetic solution at high temperatures, a better estimate
(lower) for the critical temperature, and the fact that the GBP message passing
algorithm converges also to non paramagnetic solutions. The lack of convergence
of the standard GBP message passing algorithm at low temperatures seems to be
related to the implementation details and not to the appearance of long range
order. In fact, we prove that a gauge invariance of the constrained CVM free
energy can be exploited to derive a new message passing algorithm which
converges at even lower temperatures. In all its region of convergence this new
algorithm is faster than HAK and DL by some orders of magnitude.Comment: 19 pages, 13 figure
- …