49,989 research outputs found
Probabilistic Recurrent State-Space Models
State-space models (SSMs) are a highly expressive model class for learning
patterns in time series data and for system identification. Deterministic
versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex
time series data. Fully probabilistic SSMs, however, are often found hard to
train, even for smaller problems. To overcome this limitation, we propose a
novel model formulation and a scalable training algorithm based on doubly
stochastic variational inference and Gaussian processes. In contrast to
existing work, the proposed variational approximation allows one to fully
capture the latent state temporal correlations. These correlations are the key
to robust training. The effectiveness of the proposed PR-SSM is evaluated on a
set of real-world benchmark datasets in comparison to state-of-the-art
probabilistic model learning methods. Scalability and robustness are
demonstrated on a high dimensional problem
Online Natural Gradient as a Kalman Filter
We cast Amari's natural gradient in statistical learning as a specific case
of Kalman filtering. Namely, applying an extended Kalman filter to estimate a
fixed unknown parameter of a probabilistic model from a series of observations,
is rigorously equivalent to estimating this parameter via an online stochastic
natural gradient descent on the log-likelihood of the observations.
In the i.i.d. case, this relation is a consequence of the "information
filter" phrasing of the extended Kalman filter. In the recurrent (state space,
non-i.i.d.) case, we prove that the joint Kalman filter over states and
parameters is a natural gradient on top of real-time recurrent learning (RTRL),
a classical algorithm to train recurrent models.
This exact algebraic correspondence provides relevant interpretations for
natural gradient hyperparameters such as learning rates or initialization and
regularization of the Fisher information matrix.Comment: 3rd version: expanded intr
Deep Probabilistic Time Series Forecasting using Augmented Recurrent Input for Dynamic Systems
The demand of probabilistic time series forecasting has been recently raised
in various dynamic system scenarios, for example, system identification and
prognostic and health management of machines. To this end, we combine the
advances in both deep generative models and state space model (SSM) to come up
with a novel, data-driven deep probabilistic sequence model. Specially, we
follow the popular encoder-decoder generative structure to build the recurrent
neural networks (RNN) assisted variational sequence model on an augmented
recurrent input space, which could induce rich stochastic sequence dependency.
Besides, in order to alleviate the issue of inconsistency between training and
predicting as well as improving the mining of dynamic patterns, we (i) propose
using a hybrid output as input at next time step, which brings training and
predicting into alignment; and (ii) further devise a generalized
auto-regressive strategy that encodes all the historical dependencies at
current time step. Thereafter, we first investigate the methodological
characteristics of the proposed deep probabilistic sequence model on toy cases,
and then comprehensively demonstrate the superiority of our model against
existing deep probabilistic SSM models through extensive numerical experiments
on eight system identification benchmarks from various dynamic systems.
Finally, we apply our sequence model to a real-world centrifugal compressor
sensor data forecasting problem, and again verify its outstanding performance
by quantifying the time series predictive distribution.Comment: 25 pages, 7 figures, 4 tables, preprint under revie
Deep Distributional Time Series Models and the Probabilistic Forecasting of Intraday Electricity Prices
Recurrent neural networks (RNNs) with rich feature vectors of past values can
provide accurate point forecasts for series that exhibit complex serial
dependence. We propose two approaches to constructing deep time series
probabilistic models based on a variant of RNN called an echo state network
(ESN). The first is where the output layer of the ESN has stochastic
disturbances and a shrinkage prior for additional regularization. The second
approach employs the implicit copula of an ESN with Gaussian disturbances,
which is a deep copula process on the feature space. Combining this copula with
a non-parametrically estimated marginal distribution produces a deep
distributional time series model. The resulting probabilistic forecasts are
deep functions of the feature vector and also marginally calibrated. In both
approaches, Bayesian Markov chain Monte Carlo methods are used to estimate the
models and compute forecasts. The proposed deep time series models are suitable
for the complex task of forecasting intraday electricity prices. Using data
from the Australian National Electricity Market, we show that our models
provide accurate probabilistic price forecasts. Moreover, the models provide a
flexible framework for incorporating probabilistic forecasts of electricity
demand as additional features. We demonstrate that doing so in the deep
distributional time series model in particular, increases price forecast
accuracy substantially
Predictive-State Decoders: Encoding the Future into Recurrent Networks
Recurrent neural networks (RNNs) are a vital modeling technique that rely on
internal states learned indirectly by optimization of a supervised,
unsupervised, or reinforcement training loss. RNNs are used to model dynamic
processes that are characterized by underlying latent states whose form is
often unknown, precluding its analytic representation inside an RNN. In the
Predictive-State Representation (PSR) literature, latent state processes are
modeled by an internal state representation that directly models the
distribution of future observations, and most recent work in this area has
relied on explicitly representing and targeting sufficient statistics of this
probability distribution. We seek to combine the advantages of RNNs and PSRs by
augmenting existing state-of-the-art recurrent neural networks with
Predictive-State Decoders (PSDs), which add supervision to the network's
internal state representation to target predicting future observations.
Predictive-State Decoders are simple to implement and easily incorporated into
existing training pipelines via additional loss regularization. We demonstrate
the effectiveness of PSDs with experimental results in three different domains:
probabilistic filtering, Imitation Learning, and Reinforcement Learning. In
each, our method improves statistical performance of state-of-the-art recurrent
baselines and does so with fewer iterations and less data.Comment: NIPS 201
- …