5,574 research outputs found
Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations
Variational inference techniques based on inducing variables provide an
elegant framework for scalable posterior estimation in Gaussian process (GP)
models. Besides enabling scalability, one of their main advantages over sparse
approximations using direct marginal likelihood maximization is that they
provide a robust alternative for point estimation of the inducing inputs, i.e.
the location of the inducing variables. In this work we challenge the common
wisdom that optimizing the inducing inputs in the variational framework yields
optimal performance. We show that, by revisiting old model approximations such
as the fully-independent training conditionals endowed with powerful
sampling-based inference methods, treating both inducing locations and GP
hyper-parameters in a Bayesian way can improve performance significantly. Based
on stochastic gradient Hamiltonian Monte Carlo, we develop a fully Bayesian
approach to scalable GP and deep GP models, and demonstrate its
state-of-the-art performance through an extensive experimental campaign across
several regression and classification problems
Doubly Stochastic Variational Inference for Deep Gaussian Processes
Gaussian processes (GPs) are a good choice for function approximation as they
are flexible, robust to over-fitting, and provide well-calibrated predictive
uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of
GPs, but inference in these models has proved challenging. Existing approaches
to inference in DGP models assume approximate posteriors that force
independence between the layers, and do not work well in practice. We present a
doubly stochastic variational inference algorithm, which does not force
independence between layers. With our method of inference we demonstrate that a
DGP model can be used effectively on data ranging in size from hundreds to a
billion points. We provide strong empirical evidence that our inference scheme
for DGPs works well in practice in both classification and regression.Comment: NIPS 201
Large-scale Heteroscedastic Regression via Gaussian Process
Heteroscedastic regression considering the varying noises among observations
has many applications in the fields like machine learning and statistics. Here
we focus on the heteroscedastic Gaussian process (HGP) regression which
integrates the latent function and the noise function together in a unified
non-parametric Bayesian framework. Though showing remarkable performance, HGP
suffers from the cubic time complexity, which strictly limits its application
to big data. To improve the scalability, we first develop a variational sparse
inference algorithm, named VSHGP, to handle large-scale datasets. Furthermore,
two variants are developed to improve the scalability and capability of VSHGP.
The first is stochastic VSHGP (SVSHGP) which derives a factorized evidence
lower bound, thus enhancing efficient stochastic variational inference. The
second is distributed VSHGP (DVSHGP) which (i) follows the Bayesian committee
machine formalism to distribute computations over multiple local VSHGP experts
with many inducing points; and (ii) adopts hybrid parameters for experts to
guard against over-fitting and capture local variety. The superiority of DVSHGP
and SVSHGP as compared to existing scalable heteroscedastic/homoscedastic GPs
is then extensively verified on various datasets.Comment: 14 pages, 15 figure
- …