301 research outputs found
Towards Better Interpretability in Deep Q-Networks
Deep reinforcement learning techniques have demonstrated superior performance
in a wide variety of environments. As improvements in training algorithms
continue at a brisk pace, theoretical or empirical studies on understanding
what these networks seem to learn, are far behind. In this paper we propose an
interpretable neural network architecture for Q-learning which provides a
global explanation of the model's behavior using key-value memories, attention
and reconstructible embeddings. With a directed exploration strategy, our model
can reach training rewards comparable to the state-of-the-art deep Q-learning
models. However, results suggest that the features extracted by the neural
network are extremely shallow and subsequent testing using out-of-sample
examples shows that the agent can easily overfit to trajectories seen during
training.Comment: Accepted at AAAI-19; (16 pages, 18 figures
The primacy bias in Model-based RL
The primacy bias in deep reinforcement learning (DRL), which refers to the
agent's tendency to overfit early data and lose the ability to learn from new
data, can significantly decrease the performance of DRL algorithms. Previous
studies have shown that employing simple techniques, such as resetting the
agent's parameters, can substantially alleviate the primacy bias. However, we
observe that resetting the agent's parameters harms its performance in the
context of model-based reinforcement learning (MBRL). In fact, on further
investigation, we find that the primacy bias in MBRL differs from that in
model-free RL. In this work, we focus on investigating the primacy bias in MBRL
and propose world model resetting, which works in MBRL. We apply our method to
two different MBRL algorithms, MBPO and DreamerV2. We validate the
effectiveness of our method on multiple continuous control tasks on MuJoCo and
DeepMind Control Suite, as well as discrete control tasks on Atari 100k
benchmark. The results show that world model resetting can significantly
alleviate the primacy bias in model-based setting and improve algorithm's
performance. We also give a guide on how to perform world model resetting
effectively
- …