9 research outputs found
Affine Invariant Interacting Langevin Dynamics for Bayesian Inference
We propose a computational method (with acronym ALDI) for sampling from a given target distribution based on first-order (overdamped) Langevin dynamics which satisfies the property of affine invariance. The central idea of ALDI is to run an ensemble of particles with their empirical covariance serving as a preconditioner for their underlying Langevin dynamics. ALDI does not require taking the inverse or square root of the empirical covariance matrix, which enables application to high-dimensional sampling problems. The theoretical properties of ALDI are studied in terms of nondegeneracy and ergodicity. Furthermore, we study its connections to diffusion on Riemannian manifolds and Wasserstein gradient flows. Bayesian inference serves as a main application area for ALDI. In case of a forward problem with additive Gaussian measurement errors, ALDI allows for a gradient-free approximation in the spirit of the ensemble Kalman filter. A computational comparison between gradient-free and gradient-based ALDI is provided for a PDE constrained Bayesian inverse problem
Affine invariant interacting Langevin dynamics for Bayesian inference
We propose a computational method (with acronym ALDI) for sampling from a
given target distribution based on first-order (overdamped) Langevin dynamics
which satisfies the property of affine invariance. The central idea of ALDI is
to run an ensemble of particles with their empirical covariance serving as a
preconditioner for their underlying Langevin dynamics. ALDI does not require
taking the inverse or square root of the empirical covariance matrix, which
enables application to high-dimensional sampling problems. The theoretical
properties of ALDI are studied in terms of non-degeneracy and ergodicity.
Furthermore, we study its connections to diffusion on Riemannian manifolds and
Wasserstein gradient flows.
Bayesian inference serves as a main application area for ALDI. In case of a
forward problem with additive Gaussian measurement errors, ALDI allows for a
gradient-free approximation in the spirit of the ensemble Kalman filter. A
computational comparison between gradient-free and gradient-based ALDI is
provided for a PDE constrained Bayesian inverse problem
Transitional annealed adaptive slice sampling for Gaussian process hyper-parameter estimation
Surrogate models have become ubiquitous in science and engineering for their capability of emulating expensive computer codes, necessary to model and investigate complex phenomena. Bayesian emulators based on Gaussian processes adequately quantify the uncertainty that results from the cost of the original simulator, and thus the inability to evaluate it on the whole input space. However, it is common in the literature that only a partial Bayesian analysis is carried out, whereby the underlying hyper-parameters are estimated via gradient-free optimization or genetic algorithms, to name a few methods. On the other hand, maximum a posteriori (MAP) estimation could discard important regions of the hyper-parameter space. In this paper, we carry out a more complete Bayesian inference, that combines Slice Sampling with some recently developed sequential Monte Carlo samplers. The resulting algorithm improves the mixing in the sampling through the delayed-rejection nature of Slice Sampling, the inclusion of an annealing scheme akin to Asymptotically Independent Markov Sampling and parallelization via transitional Markov chain Monte Carlo. Examples related to the estimation of Gaussian process hyper-parameters are presented. For the purpose of reproducibility, further development, and use in other applications, the code to generate the examples in this paper is freely available for download at http://github.com/agarbuno/ta2s2_codes
Transitional annealed adaptive slice sampling for Gaussian process hyper-parameter estimation
Surrogate models have become ubiquitous in science and engineering for their capability of emulating expensive computer codes, necessary to model and investigate complex phenomena. Bayesian emulators based on Gaussian processes adequately quantify the uncertainty that results from the cost of the original simulator, and thus the inability to evaluate it on the whole input space. However, it is common in the literature that only a partial Bayesian analysis is carried out, whereby the underlying hyper-parameters are estimated via gradient-free optimization or genetic algorithms, to name a few methods. On the other hand, maximum a posteriori (MAP) estimation could discard important regions of the hyper-parameter space. In this paper, we carry out a more complete Bayesian inference, that combines Slice Sampling with some recently developed sequential Monte Carlo samplers. The resulting algorithm improves the mixing in the sampling through the delayed-rejection nature of Slice Sampling, the inclusion of an annealing scheme akin to Asymptotically Independent Markov Sampling and parallelization via transitional Markov chain Monte Carlo. Examples related to the estimation of Gaussian process hyper-parameters are presented. For the purpose of reproducibility, further development, and use in other applications, the code to generate the examples in this paper is freely available for download at http://github.com/agarbuno/ta2s2_codes
Calibration and Uncertainty Quantification of Convective Parameters in an Idealized GCM
Parameters in climate models are usually calibrated manually, exploiting only
small subsets of the available data. This precludes both optimal calibration
and quantification of uncertainties. Traditional Bayesian calibration methods
that allow uncertainty quantification are too expensive for climate models;
they are also not robust in the presence of internal climate variability. For
example, Markov chain Monte Carlo (MCMC) methods typically require
model runs and are sensitive to internal variability noise, rendering them
infeasible for climate models. Here we demonstrate an approach to model
calibration and uncertainty quantification that requires only model
runs and can accommodate internal climate variability. The approach consists of
three stages: (i) a calibration stage uses variants of ensemble Kalman
inversion to calibrate a model by minimizing mismatches between model and data
statistics; (ii) an emulation stage emulates the parameter-to-data map with
Gaussian processes (GP), using the model runs in the calibration stage for
training; (iii) a sampling stage approximates the Bayesian posterior
distributions by sampling the GP emulator with MCMC. We demonstrate the
feasibility and computational efficiency of this calibrate-emulate-sample (CES)
approach in a perfect-model setting. Using an idealized general circulation
model, we estimate parameters in a simple convection scheme from synthetic data
generated with the model. The CES approach generates probability distributions
of the parameters that are good approximations of the Bayesian posteriors, at a
fraction of the computational cost usually required to obtain them. Sampling
from this approximate posterior allows the generation of climate predictions
with quantified parametric uncertainties
Calibrate, emulate, sample
Many parameter estimation problems arising in applications can be cast in the framework of Bayesian inversion. This allows not only for an estimate of the parameters, but also for the quantification of uncertainties in the estimates. Often in such problems the parameter-to-data map is very expensive to evaluate, and computing derivatives of the map, or derivative-adjoints, may not be feasible. Additionally, in many applications only noisy evaluations of the map may be available. We propose an approach to Bayesian inversion in such settings that builds on the derivative-free optimization capabilities of ensemble Kalman inversion methods. The overarching approach is to first use ensemble Kalman sampling (EKS) to calibrate the unknown parameters to fit the data; second, to use the output of the EKS to emulate the parameter-to-data map; third, to sample from an approximate Bayesian posterior distribution in which the parameter-to-data map is replaced by its emulator. This results in a principled approach to approximate Bayesian inference that requires only a small number of evaluations of the (possibly noisy approximation of the) parameter-to-data map. It does not require derivatives of this map, but instead leverages the documented power of ensemble Kalman methods. Furthermore, the EKS has the desirable property that it evolves the parameter ensemble towards the regions in which the bulk of the parameter posterior mass is located, thereby locating them well for the emulation phase of the methodology. In essence, the EKS methodology provides a cheap solution to the design problem of where to place points in parameter space to efficiently train an emulator of the parameter-to-data map for the purposes of Bayesian inversion
Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler
Solving inverse problems without the use of derivatives or adjoints of the forward model is highly desirable in many applications arising in science and engineering. In this paper we propose a new version of such a methodology, a framework for its analysis, and numerical evidence of the practicality of the method proposed. Our starting point is an ensemble of overdamped Langevin diffusions which interact through a single preconditioner computed as the empirical ensemble covariance. We demonstrate that the nonlinear Fokker--Planck equation arising from the mean-field limit of the associated stochastic differential equation (SDE) has a novel gradient flow structure, built on the Wasserstein metric and the covariance matrix of the noisy flow. Using this structure, we investigate large time properties of the Fokker--Planck equation, showing that its invariant measure coincides with that of a single Langevin diffusion, and demonstrating exponential convergence to the invariant measure in a number of settings. We introduce a new noisy variant on ensemble Kalman inversion (EKI) algorithms found from the original SDE by replacing exact gradients with ensemble differences; this defines the ensemble Kalman sampler (EKS). Numerical results are presented which demonstrate its efficacy as a derivative-free approximate sampler for the Bayesian posterior arising from inverse problems