1,862 research outputs found
Curiosity Driven Exploration of Learned Disentangled Goal Spaces
International audienceIntrinsically motivated goal exploration processes enable agents to autonomously sample goals to explore efficiently complex environments with high-dimensional continuous actions. They have been applied successfully to real world robots to discover repertoires of policies producing a wide diversity of effects. Often these algorithms relied on engineered goal spaces but it was recently shown that one can use deep representation learning algorithms to learn an adequate goal space in simple environments. However, in the case of more complex environments containing multiple objects or distractors, an efficient exploration requires that the structure of the goal space reflects the one of the environment. In this paper we show that using a disentangled goal space leads to better exploration performances than an entangled goal space. We further show that when the representation is disentangled, one can leverage it by sampling goals that maximize learning progress in a modular manner. Finally, we show that the measure of learning progress, used to drive curiosity-driven exploration, can be used simultaneously to discover abstract independently controllable features of the environment. The code used in the experiments is available at https://github.com/flowersteam/ Curiosity_Driven_Goal_Exploration
Automatic Curriculum Learning For Deep RL: A Short Survey
Automatic Curriculum Learning (ACL) has become a cornerstone of recent
successes in Deep Reinforcement Learning (DRL).These methods shape the learning
trajectories of agents by challenging them with tasks adapted to their
capacities. In recent years, they have been used to improve sample efficiency
and asymptotic performance, to organize exploration, to encourage
generalization or to solve sparse reward problems, among others. The ambition
of this work is dual: 1) to present a compact and accessible introduction to
the Automatic Curriculum Learning literature and 2) to draw a bigger picture of
the current state of the art in ACL to encourage the cross-breeding of existing
concepts and the emergence of new ideas.Comment: Accepted at IJCAI202
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
In open-ended environments, autonomous learning agents must set their own
goals and build their own curriculum through an intrinsically motivated
exploration. They may consider a large diversity of goals, aiming to discover
what is controllable in their environments, and what is not. Because some goals
might prove easy and some impossible, agents must actively select which goal to
practice at any moment, to maximize their overall mastery on the set of
learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a
modular Universal Value Function Approximator with hindsight learning to
achieve a diversity of goals of different kinds within a unique policy and 2)
an automated curriculum learning mechanism that biases the attention of the
agent towards goals maximizing the absolute learning progress. Agents focus
sequentially on goals of increasing complexity, and focus back on goals that
are being forgotten. Experiments conducted in a new modular-goal robotic
environment show the resulting developmental self-organization of a learning
curriculum, and demonstrate properties of robustness to distracting goals,
forgetting and changes in body properties.Comment: Accepted at ICML 201
GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
Designing agents, capable of learning autonomously a wide range of skills is
critical in order to increase the scope of reinforcement learning. It will both
increase the diversity of learned skills and reduce the burden of manually
designing reward functions for each skill. Self-supervised agents, setting
their own goals, and trying to maximize the diversity of those goals have shown
great promise towards this end. However, a currently known limitation of agents
trying to maximize the diversity of sampled goals is that they tend to get
attracted to noise or more generally to parts of the environments that cannot
be controlled (distractors). When agents have access to predefined goal
features or expert knowledge, absolute Learning Progress (ALP) provides a way
to distinguish between regions that can be controlled and those that cannot.
However, those methods often fall short when the agents are only provided with
raw sensory inputs such as images. In this work we extend those concepts to
unsupervised image-based goal exploration. We propose a framework that allows
agents to autonomously identify and ignore noisy distracting regions while
searching for novelty in the learnable regions to both improve overall
performance and avoid catastrophic forgetting. Our framework can be combined
with any state-of-the-art novelty seeking goal exploration approaches. We
construct a rich 3D image based environment with distractors. Experiments on
this environment show that agents using our framework successfully identify
interesting regions of the environment, resulting in drastically improved
performances. The source code is available at
https://sites.google.com/view/grimgep
- …