17 research outputs found
Bayesian Deep Net GLM and GLMM
Deep feedforward neural networks (DFNNs) are a powerful tool for functional
approximation. We describe flexible versions of generalized linear and
generalized linear mixed models incorporating basis functions formed by a DFNN.
The consideration of neural networks with random effects is not widely used in
the literature, perhaps because of the computational challenges of
incorporating subject specific parameters into already complex models.
Efficient computational methods for high-dimensional Bayesian inference are
developed using Gaussian variational approximation, with a parsimonious but
flexible factor parametrization of the covariance matrix. We implement natural
gradient methods for the optimization, exploiting the factor structure of the
variational covariance matrix in computation of the natural gradient. Our
flexible DFNN models and Bayesian inference approach lead to a regression and
classification method that has a high prediction accuracy, and is able to
quantify the prediction uncertainty in a principled and convenient way. We also
describe how to perform variable selection in our deep learning method. The
proposed methods are illustrated in a wide range of simulated and real-data
examples, and the results compare favourably to a state of the art flexible
regression and classification method in the statistical literature, the
Bayesian additive regression trees (BART) method. User-friendly software
packages in Matlab, R and Python implementing the proposed methods are
available at https://github.com/VBayesLabComment: 35 pages, 7 figure, 10 table
Variational Bayes Estimation of Discrete-Margined Copula Models with Application to Time Series
We propose a new variational Bayes estimator for high-dimensional copulas
with discrete, or a combination of discrete and continuous, margins. The method
is based on a variational approximation to a tractable augmented posterior, and
is faster than previous likelihood-based approaches. We use it to estimate
drawable vine copulas for univariate and multivariate Markov ordinal and mixed
time series. These have dimension , where is the number of observations
and is the number of series, and are difficult to estimate using previous
methods. The vine pair-copulas are carefully selected to allow for
heteroskedasticity, which is a feature of most ordinal time series data. When
combined with flexible margins, the resulting time series models also allow for
other common features of ordinal data, such as zero inflation, multiple modes
and under- or over-dispersion. Using six example series, we illustrate both the
flexibility of the time series copula models, and the efficacy of the
variational Bayes estimator for copulas of up to 792 dimensions and 60
parameters. This far exceeds the size and complexity of copula models for
discrete data that can be estimated using previous methods
Flexible Variational Bayes based on a Copula of a Mixture of Normals
Variational Bayes methods approximate the posterior density by a family of
tractable distributions and use optimisation to estimate the unknown parameters
of the approximation. Variational approximation is useful when exact inference
is intractable or very costly. Our article develops a flexible variational
approximation based on a copula of a mixture of normals, which is implemented
using the natural gradient and a variance reduction method. The efficacy of the
approach is illustrated by using simulated and real datasets to approximate
multimodal, skewed and heavy-tailed posterior distributions, including an
application to Bayesian deep feedforward neural network regression models. Each
example shows that the proposed variational approximation is much more accurate
than the corresponding Gaussian copula and a mixture of normals variational
approximations.Comment: 39 page
Random Effects Models with Deep Neural Network Basis Functions: Methodology and Computation
Deep neural networks (DNNs) are a powerful tool for functional approximation. We describe flexible versions of generalized linear and generalized linear mixed models incorporating basis functions formed by a deep neural network. The consideration of neural networks with random effects seems little used in the literature, perhaps because of the computational challenges of incorporating subject specific parameters into already complex models. Efficient computational methods for Bayesian inference are developed based on Gaussian variational approximation methods. A parsimonious but flexible factor parametrization of the covariance matrix is used in the Gaussian variational approximation. We implement natural gradient methods for the optimization, exploiting the factor structure of the variational covariance matrix to perform fast matrix vector multiplications in iterative conjugate gradient linear solvers in natural gradient computations. The method can be implemented in high dimensions, and the use of the natural gradient allows faster and more stable convergence of the variational algorithm. In the case of random effects, we compute unbiased estimates of the gradient of the lower bound in the model with the random effects integrated out by making use of Fisher's identity. The proposed methods are illustrated in several examples for DNN random effects models and high-dimensional logistic regression with sparse signal shrinkage priors