92 research outputs found
Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
In reinforcement learning, policy gradient algorithms optimize the policy
directly and rely on sampling efficiently an environment. Nevertheless, while
most sampling procedures are based on direct policy sampling, self-performance
measures could be used to improve such sampling prior to each policy update.
Following this line of thought, we introduce SAUNA, a method where
non-informative transitions are rejected from the gradient update. The level of
information is estimated according to the fraction of variance explained by the
value function: a measure of the discrepancy between V and the empirical
returns. In this work, we use this metric to select samples that are useful to
learn from, and we demonstrate that this selection can significantly improve
the performance of policy gradient methods. In this paper: (a) We define
SAUNA's metric and introduce its method to filter transitions. (b) We conduct
experiments on a set of benchmark continuous control problems. SAUNA
significantly improves performance. (c) We investigate how SAUNA reliably
selects samples with the most positive impact on learning and study its
improvement on both performance and sample efficiency.Comment: Accepted at IJCAI 202
Distral: Robust Multitask Reinforcement Learning
Most deep reinforcement learning algorithms are data inefficient in complex
and rich environments, limiting their applicability to many scenarios. One
direction for improving data efficiency is multitask learning with shared
neural network parameters, where efficiency may be improved through transfer
across related tasks. In practice, however, this is not usually observed,
because gradients from different tasks can interfere negatively, making
learning unstable and sometimes even less data efficient. Another issue is the
different reward schemes between tasks, which can easily lead to one task
dominating the learning of a shared model. We propose a new approach for joint
training of multiple tasks, which we refer to as Distral (Distill & transfer
learning). Instead of sharing parameters between the different workers, we
propose to share a "distilled" policy that captures common behaviour across
tasks. Each worker is trained to solve its own task while constrained to stay
close to the shared policy, while the shared policy is trained by distillation
to be the centroid of all task policies. Both aspects of the learning process
are derived by optimizing a joint objective function. We show that our approach
supports efficient transfer on complex 3D environments, outperforming several
related methods. Moreover, the proposed learning process is more robust and
more stable---attributes that are critical in deep reinforcement learning
- …