MCMC for variationally sparse Gaussian processes

Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function but otherwise free-form. The result is a Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs. Code to replicate each experiment in this paper will be available shortly.JH was funded by an MRC fellowship, AM and ZG by EPSRC grant EP/I036575/1 and a Google Focussed Research award.This is the final version of the article. It first appeared from the Neural Information Processing Systems Foundation via https://papers.nips.cc/paper/5875-mcmc-for-variationally-sparse-gaussian-processe

Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations

Variational inference techniques based on inducing variables provide an elegant framework for scalable posterior estimation in Gaussian process (GP) models. Besides enabling scalability, one of their main advantages over sparse approximations using direct marginal likelihood maximization is that they provide a robust alternative for point estimation of the inducing inputs, i.e. the location of the inducing variables. In this work we challenge the common wisdom that optimizing the inducing inputs in the variational framework yields optimal performance. We show that, by revisiting old model approximations such as the fully-independent training conditionals endowed with powerful sampling-based inference methods, treating both inducing locations and GP hyper-parameters in a Bayesian way can improve performance significantly. Based on stochastic gradient Hamiltonian Monte Carlo, we develop a fully Bayesian approach to scalable GP and deep GP models, and demonstrate its state-of-the-art performance through an extensive experimental campaign across several regression and classification problems

Bayesian inference for multi-level non-stationary Gaussian processes

The complexity of most real-world phenomena requires the use of flexible models that capture intricated features present in the data. Gaussian processes (GPs) have proven valuable tools for this purpose due to their non parametric and probabilistic nature. Nevertheless, the default approach when modelling with GPs is to assume stationarity. This assumption permits easier inference but can be restrictive when the correlation of the process is not constant across the input space. This thesis investigates a class of non-stationary priors that enhance flexibility while retaining interpretability. These priors assemble GPs through input-varying parameters in the covariance. Such hierarchical constructions result in high-dimensional correlated posteriors, where Bayesian inference becomes challenging and notably expensive due to the characteristic computational constrains of GPs. Altogether, this thesis provides novel approaches for scalable Bayesian inference in 2-level GP regression models. First, we use a sparse representation of the inverse non-stationary covariance to develop and compare three different Markov chain Monte Carlo (MCMC) samplers for two hyperpriors. To maintain scalability when extending the approach to multi-dimensional problems, we propose a non-stationary additive Gaussian process (AGP) model. The efficiency and accuracy of the methodology are demonstrated in simulated experiments and a computer emulation problem. Second, we derive a hybrid variational-MCMC approach that combines low-dimensional variational distributions with MCMC to avoid further distributional and independence restrictions on the posterior of interest. The resulting approximate posterior includes an intractable likelihood that when approximated with a small-order Gauss-Hermite quadrature results in poor predictive performance. In this case, an extension to higher-dimensional settings requires specific assumptions of the non-stationary covariance. Lastly, we propose a pseudo-marginal algorithm that uses a block-Poisson estimator to circumvent numerical integration in the variationally sparse model. This strategy demonstrates an improvement in predictive performance, can be computationally more efficient, and is generally applicable to other GP-based models with intractable likelihoods

Overcoming mean-field approximations in recurrent Gaussian process models

We identify a new variational inference scheme for dynamical systems whose transition function is modelled by a Gaussian process. Inference in this setting has either employed computationally intensive MCMC methods, or relied on factorisations of the variational posterior. As we demonstrate in our experiments, the factorisation between latent system states and transition function can lead to a miscalibrated posterior and to learning unnecessarily large noise terms. We eliminate this factorisation by explicitly modelling the dependence between state trajectories and the Gaussian process posterior. Samples of the latent states can then be tractably generated by conditioning on this representation. The method we obtain (VCDT: variationally coupled dynamics and trajectories) gives better predictive performance and more calibrated estimates of the transition function, yet maintains the same time and space complexities as mean-field methods. Code is available at: g i t h u b . c o m / i a l o n g / G P t