Gibbs Sampling using Anti-correlation Gaussian Data Augmentation, with Applications to L1-ball-type Models

L1-ball-type priors are a recent generalization of the spike-and-slab priors. By transforming a continuous precursor distribution to the L1-ball boundary, it induces exact zeros with positive prior and posterior probabilities. With great flexibility in choosing the precursor and threshold distributions, we can easily specify models under structured sparsity, such as those with dependent probability for zeros and smoothness among the non-zeros. Motivated to significantly accelerate the posterior computation, we propose a new data augmentation that leads to a fast block Gibbs sampling algorithm. The latent variable, named ``anti-correlation Gaussian'', cancels out the quadratic exponent term in the latent Gaussian distribution, making the parameters of interest conditionally independent so that they can be updated in a block. Compared to existing algorithms such as the No-U-Turn sampler, the new blocked Gibbs sampler has a very low computing cost per iteration and shows rapid mixing of Markov chains. We establish the geometric ergodicity guarantee of the algorithm in linear models. Further, we show useful extensions of our algorithm for posterior estimation of general latent Gaussian models, such as those involving multivariate truncated Gaussian or latent Gaussian process. Keywords: Blocked Gibbs sampler; Fast Mixing of Markov Chains; Latent Gaussian Models; Soft-thresholding

Designing EDAs by using the Elitist Convergent EDA Concept and the Boltzmann Distribution

ABSTRACT This paper presents a theoretical definition for designing EDAs called Elitist Convergent Estimation of Distribution Algorithm (ECEDA), and a practical implementation: the Boltzmann Univariate Marginal Distribution Algorithm (BUMDA). This proposal computes a Gaussian model which approximates a Boltzmann distribution via the minimization of the Kullback Leibler divergence. The resulting approach needs only one parameter: the population size. A set of problems is presented to show advantages and comparative performance of this approach with state of the art continuous EDAs

Reversible jump MCMC for nonparametric drift estimation for diffusion processes

In the context of nonparametric Bayesian estimation a Markov chain Monte Carlo algorithm is devised and implemented to sample from the posterior distribution of the drift function of a continuously or discretely observed one-dimensional diffusion. The drift is modeled by a scaled linear combination of basis functions with a Gaussian prior on the coefficients. The scaling parameter is equipped with a partially conjugate prior. The number of basis function in the drift is equipped with a prior distribution as well. For continuous data, a reversible jump Markov chain algorithm enables the exploration of the posterior over models of varying dimension. Subsequently, it is explained how data-augmentation can be used to extend the algorithm to deal with diffusions observed discretely in time. Some examples illustrate that the method can give satisfactory results. In these examples a comparison is made with another existing method as well

Towards Spatial Queries over Phenomena in Sensor Networks

Today, technology developments enable inexpensive production and deployment of tiny sensing and computing nodes. Networked through wireless radio, such senor nodes form a new platform, wireless sensor networks, which provide novel ability to monitor spatiotemporally continuous phenomena. By treating a wireless sensor network as a database system, users can pose SQL-based queries over phenomena without needing to program detailed sensor node operations. DBMS-internally, intelligent and energyefficient data collection and processing algorithms have to be implemented to support spatial query processing over sensor networks. This dissertation proposes spatial query support for two views of continuous phenomena: field-based and object-based. A field-based view of continuous phenomena depicts them as a value distribution over a geographical area. However, due to the discrete and comparatively sparse distribution of sensor nodes, estimation methods are necessary to generate a field-based query result, and it has to be computed collaboratively ‘in-the-network’ due to energy constraints. This dissertation proposes SWOP, an in-network algorithm using Gaussian Kernel estimation. The key contribution is the use of a small number of Hermite coefficients to approximate the Gaussian Kernel function for sub-clustered sensor nodes, and processes the estimation result efficiently. An object-based view of continuous phenomena is interested in aspects such as the boundary of an ‘interesting region’ (e.g. toxic plume). This dissertation presents NED, which provides object boundary detection in sensor networks. NED encodes partial event estimation results based on confidence levels into optimized, variable length messages exchanged locally among neighboring sensor nodes to save communication cost. Therefore, sensor nodes detect objects and boundaries based on moving averages to eliminate noise effects and enhance detection quality. Furthermore, the dissertation proposes the SNAKE-based approach, which uses deformable curves to track the spatiotemporal changes of such objects incrementally in sensor networks. In the proposed algorithm, only neighboring nodes exchange messages to maintain the curve structures. Based on in-network tracking of deformable curves, other types of spatial and spatiotemporal properties of objects, such as area, can be provided by the sensor network. The experimental results proved that our approaches are resource friendly within the constrained sensor networks, while providing high quality query results

Accurate PET Reconstruction from Reduced Set of Measurements based on GMM

In this paper, we provide a novel method for the estimation of unknown parameters of the Gaussian Mixture Model (GMM) in Positron Emission Tomography (PET). A vast majority of PET imaging methods are based on reconstruction model that is defined by values on some pixel/voxel grid. Instead, we propose a continuous parametric GMM model. Usually, Expectation-Maximization (EM) iterations are used to obtain the GMM model parameters from some set of point-wise measurements. The challenge of PET reconstruction is that the measurement is represented by the so called lines of response (LoR), instead of points. The goal is to estimate the unknown parameters of the Gaussian mixture directly from a relatively small set of LoR-s. Estimation of unknown parameters relies on two facts: the marginal distribution theorem of the multivariate normal distribution; and the properties of the marginal distribution of LoR-s. We propose an iterative algorithm that resembles the maximum-likelihood method to determine the unknown parameters. Results show that the estimated parameters follow the correct ones with a great accuracy. The result is promising, since the high-quality parametric reconstruction model can be obtained from lower dose measurements, and is directly suitable for further processing.Comment: 23 pages, 10 figures, submitted to "Signal Processing" by Elsevie

Capacity of Continuous Channels with Memory via Directed Information Neural Estimator

Calculating the capacity (with or without feedback) of channels with memory and continuous alphabets is a challenging task. It requires optimizing the directed information (DI) rate over all channel input distributions. The objective is a multi-letter expression, whose analytic solution is only known for a few specific cases. When no analytic solution is present or the channel model is unknown, there is no unified framework for calculating or even approximating capacity. This work proposes a novel capacity estimation algorithm that treats the channel as a `black-box', both when feedback is or is not present. The algorithm has two main ingredients: (i) a neural distribution transformer (NDT) model that shapes a noise variable into the channel input distribution, which we are able to sample, and (ii) the DI neural estimator (DINE) that estimates the communication rate of the current NDT model. These models are trained by an alternating maximization procedure to both estimate the channel capacity and obtain an NDT for the optimal input distribution. The method is demonstrated on the moving average additive Gaussian noise channel, where it is shown that both the capacity and feedback capacity are estimated without knowledge of the channel transition kernel. The proposed estimation framework opens the door to a myriad of capacity approximation results for continuous alphabet channels that were inaccessible until now

Performance in population models for count data, part II: a new SAEM algorithm.

International audienceAnalysis of count data from clinical trials using mixed effect analysis has recently become widely used. However, algorithms available for the parameter estimation, including LAPLACE and Gaussian quadrature (GQ), are associated with certain limitations, including bias in parameter estimates and the long analysis runtime. The stochastic approximation expectation maximization (SAEM) algorithm has proven to be a very efficient and powerful tool in the analysis of continuous data. The aim of this study was to implement and investigate the performance of a new SAEM algorithm for application to count data. A new SAEM algorithm was implemented in MATLAB for estimation of both, parameters and the Fisher information matrix. Stochastic Monte Carlo simulations followed by re-estimation were performed according to scenarios used in previous studies (part I) to investigate properties of alternative algorithms (Plan et al., 2008, Abstr 1372 [ http://wwwpage-meetingorg/?abstract=1372 ]). A single scenario was used to explore six probability distribution models. For parameter estimation, the relative bias was less than 0.92% and 4.13% for fixed and random effects, for all models studied including ones accounting for over- or under-dispersion. Empirical and estimated relative standard errors were similar, with distance between them being <1.7% for all explored scenarios. The longest CPU time was 95 s for parameter estimation and 56 s for SE estimation. The SAEM algorithm was extended for analysis of count data. It provides accurate estimates of both, parameters and standard errors. The estimation is significantly faster compared to LAPLACE and GQ. The algorithm is implemented in Monolix 3.1, (beta-version available in July 2009)