36 research outputs found
Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
In reinforcement learning, policy gradient algorithms optimize the policy
directly and rely on sampling efficiently an environment. Nevertheless, while
most sampling procedures are based on direct policy sampling, self-performance
measures could be used to improve such sampling prior to each policy update.
Following this line of thought, we introduce SAUNA, a method where
non-informative transitions are rejected from the gradient update. The level of
information is estimated according to the fraction of variance explained by the
value function: a measure of the discrepancy between V and the empirical
returns. In this work, we use this metric to select samples that are useful to
learn from, and we demonstrate that this selection can significantly improve
the performance of policy gradient methods. In this paper: (a) We define
SAUNA's metric and introduce its method to filter transitions. (b) We conduct
experiments on a set of benchmark continuous control problems. SAUNA
significantly improves performance. (c) We investigate how SAUNA reliably
selects samples with the most positive impact on learning and study its
improvement on both performance and sample efficiency.Comment: Accepted at IJCAI 202
Action-Conditional Video Prediction using Deep Networks in Atari Games
Motivated by vision-based reinforcement learning (RL) problems, in particular
Atari games from the recent benchmark Aracade Learning Environment (ALE), we
consider spatio-temporal prediction problems where future (image-)frames are
dependent on control variables or actions as well as previous frames. While not
composed of natural scenes, frames in Atari games are high-dimensional in size,
can involve tens of objects with one or more objects being controlled by the
actions directly and many other objects being influenced indirectly, can
involve entry and departure of objects, and can involve deep partial
observability. We propose and evaluate two deep neural network architectures
that consist of encoding, action-conditional transformation, and decoding
layers based on convolutional neural networks and recurrent neural networks.
Experimental results show that the proposed architectures are able to generate
visually-realistic frames that are also useful for control over approximately
100-step action-conditional futures in some games. To the best of our
knowledge, this paper is the first to make and evaluate long-term predictions
on high-dimensional video conditioned by control inputs.Comment: Published at NIPS 2015 (Advances in Neural Information Processing
Systems 28