Gaussian processes (GPs) can provide a principled approach to uncertainty quantification with easy-to-interpret kernel hyperparameters, such as the lengthscale,
which controls the correlation distance of function values. However, selecting an
appropriate kernel can be challenging. Deep GPs avoid manual kernel engineering by successively parameterizing kernels with GP layers, allowing them to learn
low-dimensional embeddings of the inputs that explain the output data. Following the architecture of deep neural networks, the most common deep GPs warp
the input space layer-by-layer but lose all the interpretability of shallow GPs. An
alternative construction is to successively parameterize the lengthscale of a kernel, improving the interpretability but ultimately giving away the notion of learning lower-dimensional embeddings. Unfortunately, both methods are susceptible
to particular pathologies which may hinder fitting and limit their interpretability.
This work proposes a novel synthesis of both previous approaches: Thin and Deep
GP (TDGP). Each TDGP layer defines locally linear transformations of the original input data maintaining the concept of latent embeddings while also retaining
the interpretation of lengthscales of a kernel. Moreover, unlike the prior methods,
TDGP induces non-pathological manifolds that admit learning lower-dimensional
representations. We show with theoretical and experimental results that i) TDGP
is, unlike previous models, tailored to specifically discover lower-dimensional
manifolds in the input data, ii) TDGP behaves well when increasing the number
of layers, and iii) TDGP performs well in standard benchmark datasets
