22 research outputs found
Automatic Curriculum Learning For Deep RL: A Short Survey
Automatic Curriculum Learning (ACL) has become a cornerstone of recent
successes in Deep Reinforcement Learning (DRL).These methods shape the learning
trajectories of agents by challenging them with tasks adapted to their
capacities. In recent years, they have been used to improve sample efficiency
and asymptotic performance, to organize exploration, to encourage
generalization or to solve sparse reward problems, among others. The ambition
of this work is dual: 1) to present a compact and accessible introduction to
the Automatic Curriculum Learning literature and 2) to draw a bigger picture of
the current state of the art in ACL to encourage the cross-breeding of existing
concepts and the emergence of new ideas.Comment: Accepted at IJCAI202
Growing Action Spaces
In complex tasks, such as those with large combinatorial action spaces,
random exploration may be too inefficient to achieve meaningful learning
progress. In this work, we use a curriculum of progressively growing action
spaces to accelerate learning. We assume the environment is out of our control,
but that the agent may set an internal curriculum by initially restricting its
action space. Our approach uses off-policy reinforcement learning to estimate
optimal value functions for multiple action spaces simultaneously and
efficiently transfers data, value estimates, and state representations from
restricted action spaces to the full task. We show the efficacy of our approach
in proof-of-concept control tasks and on challenging large-scale StarCraft
micromanagement tasks with large, multi-agent action spaces