6,605 research outputs found
Estimating Ratios of Normalizing Constants Using Linked Importance Sampling
Ratios of normalizing constants for two distributions are needed in both
Bayesian statistics, where they are used to compare models, and in statistical
physics, where they correspond to differences in free energy. Two approaches
have long been used to estimate ratios of normalizing constants. The `simple
importance sampling' (SIS) or `free energy perturbation' method uses a sample
drawn from just one of the two distributions. The `bridge sampling' or
`acceptance ratio' estimate can be viewed as the ratio of two SIS estimates
involving a bridge distribution. For both methods, difficult problems must be
handled by introducing a sequence of intermediate distributions linking the two
distributions of interest, with the final ratio of normalizing constants being
estimated by the product of estimates of ratios for adjacent distributions in
this sequence. Recently, work by Jarzynski, and independently by Neal, has
shown how one can view such a product of estimates, each based on simple
importance sampling using a single point, as an SIS estimate on an extended
state space. This `Annealed Importance Sampling' (AIS) method produces an
exactly unbiased estimate for the ratio of normalizing constants even when the
Markov transitions used do not reach equilibrium. In this paper, I show how a
corresponding `Linked Importance Sampling' (LIS) method can be constructed in
which the estimates for individual ratios are similar to bridge sampling
estimates. I show empirically that for some problems, LIS estimates are much
more accurate than AIS estimates found using the same computation time,
although for other problems the two methods have similar performance. Linked
sampling methods similar to LIS are useful for other purposes as well
Representing numeric data in 32 bits while preserving 64-bit precision
Data files often consist of numbers having only a few significant decimal
digits, whose information content would allow storage in only 32 bits. However,
we may require that arithmetic operations involving these numbers be done with
64-bit floating-point precision, which precludes simply representing the data
as 32-bit floating-point values. Decimal floating point gives a compact and
exact representation, but requires conversion with a slow division operation
before it can be used. Here, I show that interesting subsets of 64-bit
floating-point values can be compactly and exactly represented by the 32 bits
consisting of the sign, exponent, and high-order part of the mantissa, with the
lower-order 32 bits of the mantissa filled in by table lookup, indexed by bits
from the part of the mantissa retained, and possibly from the exponent. For
example, decimal data with 4 or fewer digits to the left of the decimal point
and 2 or fewer digits to the right of the decimal point can be represented in
this way using the lower-order 5 bits of the retained part of the mantissa as
the index. Data consisting of 6 decimal digits with the decimal point in any of
the 7 positions before or after one of the digits can also be represented this
way, and decoded using 19 bits from the mantissa and exponent as the index.
Encoding with such a scheme is a simple copy of half the 64-bit value, followed
if necessary by verification that the value can be represented, by checking
that it decodes correctly. Decoding requires only extraction of index bits and
a table lookup. Lookup in a small table will usually reference cache; even with
larger tables, decoding is still faster than conversion from decimal floating
point with a division operation. I discuss how such schemes perform on recent
computer systems, and how they might be used to automatically compress large
arrays in interpretive languages such as R
Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
Gaussian processes are a natural way of defining prior distributions over
functions of one or more input variables. In a simple nonparametric regression
problem, where such a function gives the mean of a Gaussian distribution for an
observed response, a Gaussian process model can easily be implemented using
matrix computations that are feasible for datasets of up to about a thousand
cases. Hyperparameters that define the covariance function of the Gaussian
process can be sampled using Markov chain methods. Regression models where the
noise has a t distribution and logistic or probit models for classification
applications can be implemented by sampling as well for latent values
underlying the observations. Software is now available that implements these
methods using covariance functions with hierarchical parameterizations. Models
defined in this way can discover high-level properties of the data, such as
which inputs are relevant to predicting the response
- …