47 research outputs found
Doubly Stochastic Variational Inference for Deep Gaussian Processes
Gaussian processes (GPs) are a good choice for function approximation as they
are flexible, robust to over-fitting, and provide well-calibrated predictive
uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of
GPs, but inference in these models has proved challenging. Existing approaches
to inference in DGP models assume approximate posteriors that force
independence between the layers, and do not work well in practice. We present a
doubly stochastic variational inference algorithm, which does not force
independence between layers. With our method of inference we demonstrate that a
DGP model can be used effectively on data ranging in size from hundreds to a
billion points. We provide strong empirical evidence that our inference scheme
for DGPs works well in practice in both classification and regression.Comment: NIPS 201
Bayesian inference with finitely wide neural networks
The analytic inference, e.g. predictive distribution being in closed form,
may be an appealing benefit for machine learning practitioners when they treat
wide neural networks as Gaussian process in Bayesian setting. The realistic
widths, however, are finite and cause weak deviation from the Gaussianity under
which partial marginalization of random variables in a model is
straightforward. On the basis of multivariate Edgeworth expansion, we propose a
non-Gaussian distribution in differential form to model a finite set of outputs
from a random neural network, and derive the corresponding marginal and
conditional properties. Thus, we are able to derive the non-Gaussian posterior
distribution in Bayesian regression task. In addition, in the bottlenecked deep
neural networks, a weight space representation of deep Gaussian process, the
non-Gaussianity is investigated through the marginal kernel.Comment: v2: added relevant references, example of simple non-Gaussian
bivariate distribution and corresponding inferenc