Exploration in sparse reward environments remains one of the key challenges
of model-free reinforcement learning. Instead of solely relying on extrinsic
rewards provided by the environment, many state-of-the-art methods use
intrinsic rewards to encourage exploration. However, we show that existing
methods fall short in procedurally-generated environments where an agent is
unlikely to visit a state more than once. We propose a novel type of intrinsic
reward which encourages the agent to take actions that lead to significant
changes in its learned state representation. We evaluate our method on multiple
challenging procedurally-generated tasks in MiniGrid, as well as on tasks with
high-dimensional observations used in prior work. Our experiments demonstrate
that this approach is more sample efficient than existing exploration methods,
particularly for procedurally-generated MiniGrid environments. Furthermore, we
analyze the learned behavior as well as the intrinsic reward received by our
agent. In contrast to previous approaches, our intrinsic reward does not
diminish during the course of training and it rewards the agent substantially
more for interacting with objects that it can control