A Low Density Lattice Decoder via Non-Parametric Belief Propagation

The recent work of Sommer, Feder and Shalvi presented a new family of codes called low density lattice codes (LDLC) that can be decoded efficiently and approach the capacity of the AWGN channel. A linear time iterative decoding scheme which is based on a message-passing formulation on a factor graph is given. In the current work we report our theoretical findings regarding the relation between the LDLC decoder and belief propagation. We show that the LDLC decoder is an instance of non-parametric belief propagation and further connect it to the Gaussian belief propagation algorithm. Our new results enable borrowing knowledge from the non-parametric and Gaussian belief propagation domains into the LDLC domain. Specifically, we give more general convergence conditions for convergence of the LDLC decoder (under the same assumptions of the original LDLC convergence analysis). We discuss how to extend the LDLC decoder from Latin square to full rank, non-square matrices. We propose an efficient construction of sparse generator matrix and its matching decoder. We report preliminary experimental results which show our decoder has comparable symbol to error rate compared to the original LDLC decoder.%Comment: Submitted for publicatio

Gibbs Sampling for (Coupled) Infinite Mixture Models in the Stick Breaking Representation

Nonparametric Bayesian approaches to clustering, information retrieval, language modeling and object recognition have recently shown great promise as a new paradigm for unsupervised data analysis. Most contributions have focused on the Dirichlet process mixture models or extensions thereof for which efficient Gibbs samplers exist. In this paper we explore Gibbs samplers for infinite complexity mixture models in the stick breaking representation. The advantage of this representation is improved modeling flexibility. For instance, one can design the prior distribution over cluster sizes or couple multiple infinite mixture models (e.g. over time) at the level of their parameters (i.e. the dependent Dirichlet process model). However, Gibbs samplers for infinite mixture models (as recently introduced in the statistics literature) seem to mix poorly over cluster labels. Among others issues, this can have the adverse effect that labels for the same cluster in coupled mixture models are mixed up. We introduce additional moves in these samplers to improve mixing over cluster labels and to bring clusters into correspondence. An application to modeling of storm trajectories is used to illustrate these ideas.Comment: Appears in Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence (UAI2006

Inference in sensor networks : graphical models and particle methods

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 175-183).Sensor networks have quickly risen in importance over the last several years to become an active field of research, full of difficult problems and applications. At the same time, graphical models have shown themselves to be an extremely useful formalism for describing the underlying statistical structure of problems for sensor networks. In part, this is due to a number of efficient methods for solving inference problems defined on graphical models, but even more important is the fact that many of these methods (such as belief propagation) can be interpreted as a set of message passing operations, for which it is not difficult to describe a simple, distributed architecture in which each sensor performs local processing and fusion of information, and passes messages locally among neighboring sensors. At the same time, many of the tasks which are most important in sensor networks are characterized by such features as complex uncertainty and nonlinear observation processes. Particle filtering is one common technique for dealing with inference under these conditions in certain types of sequential problems, such as tracking of mobile objects.(cont.) However, many sensor network applications do not have the necessary structure to apply particle filtering, and even when they do there are subtleties which arise due to the nature of a distributed inference process performed on a system with limited resources (such as power, bandwidth, and so forth). This thesis explores how the ideas of graphical models and sample-based representations of uncertainty such as are used in particle filtering can be applied to problems defined for sensor networks, in which we must consider the impact of resource limitations on our algorithms. In particular, we explore three related themes. We begin by describing how sample-based representations can be applied to solve inference problems defined on general graphical models. Limited communications, the primary restriction in most practical sensor networks, means that the messages which are passed in the inference process must be approximated in some way. Our second theme explores the consequences of such message approximations, and leads to results with implications both for distributed systems and the use of belief propagation more generally.(cont.) This naturally raises a third theme, investigating the optimal cost of representing sample-based estimates of uncertainty so as to minimize the communications required. Our analysis shows several interesting differences between this problem and traditional source coding methods. We also use the metrics for message errors to define lossy or approximate4 encoders, and provide an example encoder capable of balancing communication costs with a measure on inferential error. Finally, we put all of these three themes to work to solve a difficult and important task in sensor networks. The self-localization problem for sensors networks involves the estimation of all sensor positions given a set of relative inter-sensor measurements in the network. We describe this problem as a graphical model, illustrate the complex uncertainties involved in the estimation process, and present a method of finding for both estimates of the sensor positions and their remaining uncertainty using a sample-based message passing algorithm. This method is capable of incorporating arbitrary noise distributions, including outlier processes, and by applying our lossy encoding algorithm can be used even when communications is relatively limited.(cont.) We conclude the thesis with a summary of the work and its contributions, and a description of some of the many problems which remain open within the field.y Alexander T. Ihler.Ph.D

Nonparametric Belief Propagation and Facial Appearance Estimation

In many applications of graphical models arising in computer vision, the hidden variables of interest are most naturally specified by continuous, non-Gaussian distributions. There exist inference algorithms for discrete approximations to these continuous distributions, but for the high-dimensional variables typically of interest, discrete inference becomes infeasible. Stochastic methods such as particle filters provide an appealing alternative. However, existing techniques fail to exploit the rich structure of the graphical models describing many vision problems. Drawing on ideas from regularized particle filters and belief propagation (BP), this paper develops a nonparametric belief propagation (NBP) algorithm applicable to general graphs. Each NBP iteration uses an efficient sampling procedure to update kernel-based approximations to the true, continuous likelihoods. The algorithm can accomodate an extremely broad class of potential functions, including nonparametric representations. Thus, NBP extends particle filtering methods to the more general vision problems that graphical models can describe. We apply the NBP algorithm to infer component interrelationships in a parts-based face model, allowing location and reconstruction of occluded features