9,801 research outputs found
Deep Gaussian Processes with Convolutional Kernels
Deep Gaussian processes (DGPs) provide a Bayesian non-parametric alternative
to standard parametric deep learning models. A DGP is formed by stacking
multiple GPs resulting in a well-regularized composition of functions. The
Bayesian framework that equips the model with attractive properties, such as
implicit capacity control and predictive uncertainty, makes it at the same time
challenging to combine with a convolutional structure. This has hindered the
application of DGPs in computer vision tasks, an area where deep parametric
models (i.e. CNNs) have made breakthroughs. Standard kernels used in DGPs such
as radial basis functions (RBFs) are insufficient for handling pixel
variability in raw images. In this paper, we build on the recent convolutional
GP to develop Convolutional DGP (CDGP) models which effectively capture image
level features through the use of convolution kernels, therefore opening up the
way for applying DGPs to computer vision tasks. Our model learns local spatial
influence and outperforms strong GP based baselines on multi-class image
classification. We also consider various constructions of convolution kernel
over the image patches, analyze the computational trade-offs and provide an
efficient framework for convolutional DGP models. The experimental results on
image data such as MNIST, rectangles-image, CIFAR10 and Caltech101 demonstrate
the effectiveness of the proposed approaches
On the Inductive Bias of Neural Tangent Kernels
State-of-the-art neural networks are heavily over-parameterized, making the
optimization algorithm a crucial ingredient for learning predictive models with
good generalization properties. A recent line of work has shown that in a
certain over-parameterized regime, the learning dynamics of gradient descent
are governed by a certain kernel obtained at initialization, called the neural
tangent kernel. We study the inductive bias of learning in such a regime by
analyzing this kernel and the corresponding function space (RKHS). In
particular, we study smoothness, approximation, and stability properties of
functions with finite norm, including stability to image deformations in the
case of convolutional networks, and compare to other known kernels for similar
architectures.Comment: NeurIPS 201
- …