10,654 research outputs found
Gaussian processes for Bayesian classification via Hybrid Monte Carlo
The full Bayesian method for applying neural networks to a prediction problem is to set up the prior/hyperprior structure for the net and then perform the necessary integrals. However, these integrals are not tractable analytically, and Markov Chain Monte Carlo (MCMC) methods are slow, especially if the parameter space is high-dimensional. Using Gaussian processes we can approximate the weight space integral analytically, so that only a small number of hyperparameters need be integrated over by MCMC methods. We have applied this idea to classification problems, obtaining excellent results on the real-world problems investigated so far
Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification
Gaussian processes are a natural way of defining prior distributions over
functions of one or more input variables. In a simple nonparametric regression
problem, where such a function gives the mean of a Gaussian distribution for an
observed response, a Gaussian process model can easily be implemented using
matrix computations that are feasible for datasets of up to about a thousand
cases. Hyperparameters that define the covariance function of the Gaussian
process can be sampled using Markov chain methods. Regression models where the
noise has a t distribution and logistic or probit models for classification
applications can be implemented by sampling as well for latent values
underlying the observations. Software is now available that implements these
methods using covariance functions with hierarchical parameterizations. Models
defined in this way can discover high-level properties of the data, such as
which inputs are relevant to predicting the response
Adaptive Multiple Importance Sampling for Gaussian Processes
In applications of Gaussian processes where quantification of uncertainty is
a strict requirement, it is necessary to accurately characterize the posterior
distribution over Gaussian process covariance parameters. Normally, this is
done by means of standard Markov chain Monte Carlo (MCMC) algorithms. Motivated
by the issues related to the complexity of calculating the marginal likelihood
that can make MCMC algorithms inefficient, this paper develops an alternative
inference framework based on Adaptive Multiple Importance Sampling (AMIS). This
paper studies the application of AMIS in the case of a Gaussian likelihood, and
proposes the Pseudo-Marginal AMIS for non-Gaussian likelihoods, where the
marginal likelihood is unbiasedly estimated. The results suggest that the
proposed framework outperforms MCMC-based inference of covariance parameters in
a wide range of scenarios and remains competitive for moderately large
dimensional parameter spaces.Comment: 27 page
A New Monte Carlo Based Algorithm for the Gaussian Process Classification Problem
Gaussian process is a very promising novel technology that has been applied
to both the regression problem and the classification problem. While for the
regression problem it yields simple exact solutions, this is not the case for
the classification problem, because we encounter intractable integrals. In this
paper we develop a new derivation that transforms the problem into that of
evaluating the ratio of multivariate Gaussian orthant integrals. Moreover, we
develop a new Monte Carlo procedure that evaluates these integrals. It is based
on some aspects of bootstrap sampling and acceptancerejection. The proposed
approach has beneficial properties compared to the existing Markov Chain Monte
Carlo approach, such as simplicity, reliability, and speed
Hierarchical Gaussian process mixtures for regression
As a result of their good performance in practice and their desirable analytical properties, Gaussian process regression models are becoming increasingly of interest in statistics, engineering and other fields. However, two major problems arise when the model is applied to a large data-set with repeated measurements. One stems from the systematic heterogeneity among the different replications, and the other is the requirement to invert a covariance matrix which is involved in the implementation of the model. The dimension of this matrix equals the sample size of the training data-set. In this paper, a Gaussian process mixture model for regression is proposed for dealing with the above two problems, and a hybrid Markov chain Monte Carlo (MCMC) algorithm is used for its implementation. Application to a real data-set is reported
Pseudo-Marginal Bayesian Inference for Gaussian Processes
The main challenges that arise when adopting Gaussian Process priors in
probabilistic modeling are how to carry out exact Bayesian inference and how to
account for uncertainty on model parameters when making model-based predictions
on out-of-sample data. Using probit regression as an illustrative working
example, this paper presents a general and effective methodology based on the
pseudo-marginal approach to Markov chain Monte Carlo that efficiently addresses
both of these issues. The results presented in this paper show improvements
over existing sampling methods to simulate from the posterior distribution over
the parameters defining the covariance function of the Gaussian Process prior.
This is particularly important as it offers a powerful tool to carry out full
Bayesian inference of Gaussian Process based hierarchic statistical models in
general. The results also demonstrate that Monte Carlo based integration of all
model parameters is actually feasible in this class of models providing a
superior quantification of uncertainty in predictions. Extensive comparisons
with respect to state-of-the-art probabilistic classifiers confirm this
assertion.Comment: 14 pages double colum
- …