35 research outputs found
Probabilistic Recurrent State-Space Models
State-space models (SSMs) are a highly expressive model class for learning
patterns in time series data and for system identification. Deterministic
versions of SSMs (e.g. LSTMs) proved extremely successful in modeling complex
time series data. Fully probabilistic SSMs, however, are often found hard to
train, even for smaller problems. To overcome this limitation, we propose a
novel model formulation and a scalable training algorithm based on doubly
stochastic variational inference and Gaussian processes. In contrast to
existing work, the proposed variational approximation allows one to fully
capture the latent state temporal correlations. These correlations are the key
to robust training. The effectiveness of the proposed PR-SSM is evaluated on a
set of real-world benchmark datasets in comparison to state-of-the-art
probabilistic model learning methods. Scalability and robustness are
demonstrated on a high dimensional problem
Video Prediction Models as Rewards for Reinforcement Learning
Specifying reward signals that allow agents to learn complex behaviors is a
long-standing challenge in reinforcement learning. A promising approach is to
extract preferences for behaviors from unlabeled videos, which are widely
available on the internet. We present Video Prediction Rewards (VIPER), an
algorithm that leverages pretrained video prediction models as action-free
reward signals for reinforcement learning. Specifically, we first train an
autoregressive transformer on expert videos and then use the video prediction
likelihoods as reward signals for a reinforcement learning agent. VIPER enables
expert-level control without programmatic task rewards across a wide range of
DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction
model allows us to derive rewards for an out-of-distribution environment where
no expert data is available, enabling cross-embodiment generalization for
tabletop manipulation. We see our work as starting point for scalable reward
specification from unlabeled videos that will benefit from the rapid advances
in generative modeling. Source code and datasets are available on the project
website: https://escontrela.me/viperComment: 22 pages, 18 figures, 4 tables. under revie