92,371 research outputs found
MCMC methods for functions modifying old algorithms to make\ud them faster
Many problems arising in applications result in the need\ud
to probe a probability distribution for functions. Examples include Bayesian nonparametric statistics and conditioned diffusion processes. Standard MCMC algorithms typically become arbitrarily slow under the mesh refinement dictated by nonparametric description of the unknown function. We describe an approach to modifying a whole range of MCMC methods which ensures that their speed of convergence is robust under mesh refinement. In the applications of interest the data is often sparse and the prior specification is an essential part of the overall modeling strategy. The algorithmic approach that we describe is applicable whenever the desired probability measure has density with respect to a Gaussian process or Gaussian random field prior, and to some useful non-Gaussian priors constructed through random truncation. Applications are shown in density estimation, data assimilation in fluid mechanics, subsurface geophysics and image registration. The key design principle is to formulate the MCMC method for functions. This leads to algorithms which can be implemented via minor modification of existing algorithms, yet which show enormous speed-up on a wide range of applied problems
Learning Topic Models and Latent Bayesian Networks Under Expansion Constraints
Unsupervised estimation of latent variable models is a fundamental problem
central to numerous applications of machine learning and statistics. This work
presents a principled approach for estimating broad classes of such models,
including probabilistic topic models and latent linear Bayesian networks, using
only second-order observed moments. The sufficient conditions for
identifiability of these models are primarily based on weak expansion
constraints on the topic-word matrix, for topic models, and on the directed
acyclic graph, for Bayesian networks. Because no assumptions are made on the
distribution among the latent variables, the approach can handle arbitrary
correlations among the topics or latent factors. In addition, a tractable
learning method via optimization is proposed and studied in numerical
experiments.Comment: 38 pages, 6 figures, 2 tables, applications in topic models and
Bayesian networks are studied. Simulation section is adde
Symmetrized importance samplers for stochastic differential equations
We study a class of importance sampling methods for stochastic differential
equations (SDEs). A small-noise analysis is performed, and the results suggest
that a simple symmetrization procedure can significantly improve the
performance of our importance sampling schemes when the noise is not too large.
We demonstrate that this is indeed the case for a number of linear and
nonlinear examples. Potential applications, e.g., data assimilation, are
discussed.Comment: Added brief discussion of Hamilton-Jacobi equation. Also made various
minor corrections. To appear in Communciations in Applied Mathematics and
Computational Scienc
Simulation of stochastic network dynamics via entropic matching
The simulation of complex stochastic network dynamics arising, for instance,
from models of coupled biomolecular processes remains computationally
challenging. Often, the necessity to scan a models' dynamics over a large
parameter space renders full-fledged stochastic simulations impractical,
motivating approximation schemes. Here we propose an approximation scheme which
improves upon the standard linear noise approximation while retaining similar
computational complexity. The underlying idea is to minimize, at each time
step, the Kullback-Leibler divergence between the true time evolved probability
distribution and a Gaussian approximation (entropic matching). This condition
leads to ordinary differential equations for the mean and the covariance matrix
of the Gaussian. For cases of weak nonlinearity, the method is more accurate
than the linear method when both are compared to stochastic simulations.Comment: 23 pages, 6 figures; significantly revised versio
Computing Functions of Random Variables via Reproducing Kernel Hilbert Space Representations
We describe a method to perform functional operations on probability
distributions of random variables. The method uses reproducing kernel Hilbert
space representations of probability distributions, and it is applicable to all
operations which can be applied to points drawn from the respective
distributions. We refer to our approach as {\em kernel probabilistic
programming}. We illustrate it on synthetic data, and show how it can be used
for nonparametric structural equation models, with an application to causal
inference
Asymptotic Estimates in Information Theory with Non-Vanishing Error Probabilities
This monograph presents a unified treatment of single- and multi-user
problems in Shannon's information theory where we depart from the requirement
that the error probability decays asymptotically in the blocklength. Instead,
the error probabilities for various problems are bounded above by a
non-vanishing constant and the spotlight is shone on achievable coding rates as
functions of the growing blocklengths. This represents the study of asymptotic
estimates with non-vanishing error probabilities.
In Part I, after reviewing the fundamentals of information theory, we discuss
Strassen's seminal result for binary hypothesis testing where the type-I error
probability is non-vanishing and the rate of decay of the type-II error
probability with growing number of independent observations is characterized.
In Part II, we use this basic hypothesis testing result to develop second- and
sometimes, even third-order asymptotic expansions for point-to-point
communication. Finally in Part III, we consider network information theory
problems for which the second-order asymptotics are known. These problems
include some classes of channels with random state, the multiple-encoder
distributed lossless source coding (Slepian-Wolf) problem and special cases of
the Gaussian interference and multiple-access channels. Finally, we discuss
avenues for further research.Comment: Further comments welcom
- …