Bayesian inference for inverse problems

Traditionally, the MaxEnt workshops start by a tutorial day. This paper summarizes my talk during 2001'th workshop at John Hopkins University. The main idea in this talk is to show how the Bayesian inference can naturally give us all the necessary tools we need to solve real inverse problems: starting by simple inversion where we assume to know exactly the forward model and all the input model parameters up to more realistic advanced problems of myopic or blind inversion where we may be uncertain about the forward model and we may have noisy data. Starting by an introduction to inverse problems through a few examples and explaining their ill posedness nature, I briefly presented the main classical deterministic methods such as data matching and classical regularization methods to show their limitations. I then presented the main classical probabilistic methods based on likelihood, information theory and maximum entropy and the Bayesian inference framework for such problems. I show that the Bayesian framework, not only generalizes all these methods, but also gives us natural tools, for example, for inferring the uncertainty of the computed solutions, for the estimation of the hyperparameters or for handling myopic or blind inversion problems. Finally, through a deconvolution problem example, I presented a few state of the art methods based on Bayesian inference particularly designed for some of the mass spectrometry data processing problems.Comment: Presented at MaxEnt01. To appear in Bayesian Inference and Maximum Entropy Methods, B. Fry (Ed.), AIP Proceedings. 20pages, 13 Postscript figure

Relative entropy and the multi-variable multi-dimensional moment problem

Entropy-like functionals on operator algebras have been studied since the pioneering work of von Neumann, Umegaki, Lindblad, and Lieb. The most well-known are the von Neumann entropy $trace (\rho\log \rho)$ and a generalization of the Kullback-Leibler distance $trace (\rho \log \rho - \rho \log \sigma)$, refered to as quantum relative entropy and used to quantify distance between states of a quantum system. The purpose of this paper is to explore these as regularizing functionals in seeking solutions to multi-variable and multi-dimensional moment problems. It will be shown that extrema can be effectively constructed via a suitable homotopy. The homotopy approach leads naturally to a further generalization and a description of all the solutions to such moment problems. This is accomplished by a renormalization of a Riemannian metric induced by entropy functionals. As an application we discuss the inverse problem of describing power spectra which are consistent with second-order statistics, which has been the main motivation behind the present work.Comment: 24 pages, 3 figure

A Sequential Topic Model for Mining Recurrent Activities from Long Term Video Logs

This paper introduces a novel probabilistic activity modeling approach that mines recurrent sequential patterns called motifs from documents given as word×time count matrices (e.g., videos). In this model, documents are represented as a mixture of sequential activity patterns (our motifs) where the mixing weights are defined by the motif starting time occurrences. The novelties are multi fold. First, unlike previous approaches where topics modeled only the co-occurrence of words at a given time instant, our motifs model the co-occurrence and temporal order in which the words occur within a temporal window. Second, unlike traditional Dynamic Bayesian Networks (DBN), our model accounts for the important case where activities occur concurrently in the video (but not necessarily in syn- chrony), i.e., the advent of activity motifs can overlap. The learning of the motifs in these difficult situations is made possible thanks to the introduction of latent variables representing the activity starting times, enabling us to implicitly align the occurrences of the same pattern during the joint inference of the motifs and their starting times. As a third novelty, we propose a general method that favors the recovery of sparse distributions, a highly desirable property in many topic model applications, by adding simple regularization constraints on the searched distributions to the data likelihood optimization criteria. We substantiate our claims with experiments on synthetic data to demonstrate the algorithm behavior, and on four video datasets with significant variations in their activity content obtained from static cameras. We observe that using low-level motion features from videos, our algorithm is able to capture sequential patterns that implicitly represent typical trajectories of scene objects

Deconvolution of Quantized-Input Linear Systems: An Information-Theoretic Approach

The deconvolution problem has been drawing the attention of mathematicians, physicists and engineers since the early sixties. Ubiquitous in the applications, it consists in recovering the unknown input of a convolution system from noisy measurements of the output. It is a typical instance of inverse, ill-posed problem: the existence and uniqueness of the solution are not assured and even small perturbations in the data may cause large deviations in the solution. In the last fifty years, a large amount of estimation techniques have been proposed by different research communities to tackle deconvolution, each technique being related to a peculiar engineering application or mathematical set. In many occurrences, the unknown input presents some known features, which can be exploited to develop ad hoc algorithms. For example, prior information about regularity and smoothness of the input function are often considered, as well as the knowledge of a probabilistic distribution on the input source: the estimation techniques arising in different scenarios are strongly diverse. Less effort has been dedicated to the case where the input is known to be affected by discontinuities and switches, which is becoming an important issue in modern technologies. In fact, quantized signals, that is, piecewise constant functions that can assume only a finite number of values, are nowadays widespread in the applications, given the ongoing process of digitization concerning most of information and communication systems. Moreover, hybrid systems are often encountered, which are characterized by the introduction of quantized signals into physical, analog communication channels. Motivated by such consideration, this dissertation is devoted to the study of the deconvolution of continuous systems with quantized input; in particular, our attention will be focused on linear systems. Given the discrete nature of the input, we will show that the whole problem can be interpreted as a paradigmatic digital transmission problem and we will undertake an Information-theoretic approach to tackle it. The aim of this dissertation is to develop suitable deconvolution algorithms for quantized-input linear systems, which will be derived from known decoding procedures, and to test them in different scenarios. Much consideration will be given to the theoretical analysis of these algorithms, whose performance will be rigorously described in mathematical terms

Generalized Maximum Entropy, Convexity and Machine Learning

This thesis identifies and extends techniques that can be linked to the principle of maximum entropy (maxent) and applied to parameter estimation in machine learning and statistics. Entropy functions based on deformed logarithms are used to construct Bregman divergences, and together these represent a generalization of relative entropy. The framework is analyzed using convex analysis to charac- terize generalized forms of exponential family distributions. Various connections to the existing machine learning literature are discussed and the techniques are applied to the problem of non-negative matrix factorization (NMF)

A new look at entropy for solving linear inverse problems

Abstract—Entropy-based methods are widely used for solving inverse problems, particularly when the solution is known to be positive. Here, we address linear ill-posed and noisy inverse problems of the form � � � a e � e � e � C �� � with a general convex con-straint � � � P ˆ Pˆ, P ˆ where ˆ is a convex set. Although projective methods are well adapted to this context, we study alternative methods which rely highly on some “information-based ” criteria. Our goal is to clarify the role played by entropy in this field, and to present a new point of view on entropy, using general tools and results coming from convex analysis. We present then a new and broad scheme for entropic-based inversion of linear-noisy inverse problems. This scheme was introduced by Navaza in 1985 in connection with a physical modeling for crystallographic applications, and further studied by Dacunha-Castelle and Gamboa. Important features of this paper are: i) a unified presentation of many well-known reconstruction criteria, ii) proposal of new criteria for reconstruction under various prior knowledge and with various noise statistics, iii) a description of practical inversion of data using the aforementioned criteria, and iv) a presentation of some reconstruction results