2 research outputs found
Scalable Variational Gaussian Process Regression Networks
Gaussian process regression networks (GPRN) are powerful Bayesian models for
multi-output regression, but their inference is intractable. To address this
issue, existing methods use a fully factorized structure (or a mixture of such
structures) over all the outputs and latent functions for posterior
approximation, which, however, can miss the strong posterior dependencies among
the latent variables and hurt the inference quality. In addition, the updates
of the variational parameters are inefficient and can be prohibitively
expensive for a large number of outputs. To overcome these limitations, we
propose a scalable variational inference algorithm for GPRN, which not only
captures the abundant posterior dependencies but also is much more efficient
for massive outputs. We tensorize the output space and introduce
tensor/matrix-normal variational posteriors to capture the posterior
correlations and to reduce the parameters. We jointly optimize all the
parameters and exploit the inherent Kronecker product structure in the
variational model evidence lower bound to accelerate the computation. We
demonstrate the advantages of our method in several real-world applications
Bayesian Inference in High-Dimensional Time-Serieswith the Orthogonal Stochastic Linear Mixing Model
Many modern time-series datasets contain large numbers of output response
variables sampled for prolonged periods of time. For example, in neuroscience,
the activities of 100s-1000's of neurons are recorded during behaviors and in
response to sensory stimuli. Multi-output Gaussian process models leverage the
nonparametric nature of Gaussian processes to capture structure across multiple
outputs. However, this class of models typically assumes that the correlations
between the output response variables are invariant in the input space.
Stochastic linear mixing models (SLMM) assume the mixture coefficients depend
on input, making them more flexible and effective to capture complex output
dependence. However, currently, the inference for SLMMs is intractable for
large datasets, making them inapplicable to several modern time-series
problems. In this paper, we propose a new regression framework, the orthogonal
stochastic linear mixing model (OSLMM) that introduces an orthogonal constraint
amongst the mixing coefficients. This constraint reduces the computational
burden of inference while retaining the capability to handle complex output
dependence. We provide Markov chain Monte Carlo inference procedures for both
SLMM and OSLMM and demonstrate superior model scalability and reduced
prediction error of OSLMM compared with state-of-the-art methods on several
real-world applications. In neurophysiology recordings, we use the inferred
latent functions for compact visualization of population responses to auditory
stimuli, and demonstrate superior results compared to a competing method
(GPFA). Together, these results demonstrate that OSLMM will be useful for the
analysis of diverse, large-scale time-series datasets