Automatic Curriculum Learning For Deep RL: A Short Survey

Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL).These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.Comment: Accepted at IJCAI202

Computational Theories of Curiosity-Driven Learning

What are the functions of curiosity? What are the mechanisms of curiosity-driven learning? We approach these questions about the living using concepts and tools from machine learning and developmental robotics. We argue that curiosity-driven learning enables organisms to make discoveries to solve complex problems with rare or deceptive rewards. By fostering exploration and discovery of a diversity of behavioural skills, and ignoring these rewards, curiosity can be efficient to bootstrap learning when there is no information, or deceptive information, about local improvement towards these problems. We also explain the key role of curiosity for efficient learning of world models. We review both normative and heuristic computational frameworks used to understand the mechanisms of curiosity in humans, conceptualizing the child as a sense-making organism. These frameworks enable us to discuss the bi-directional causal links between curiosity and learning, and to provide new hypotheses about the fundamental role of curiosity in self-organizing developmental structures through curriculum learning. We present various developmental robotics experiments that study these mechanisms in action, both supporting these hypotheses to understand better curiosity in humans and opening new research avenues in machine learning and artificial intelligence. Finally, we discuss challenges for the design of experimental paradigms for studying curiosity in psychology and cognitive neuroscience. Keywords: Curiosity, intrinsic motivation, lifelong learning, predictions, world model, rewards, free-energy principle, learning progress, machine learning, AI, developmental robotics, development, curriculum learning, self-organization.Comment: To appear in "The New Science of Curiosity", ed. G. Gordon, Nova Science Publisher

Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Building autonomous machines that can explore open-ended environments, discover possible interactions and build repertoires of skills is a general objective of artificial intelligence. Developmental approaches argue that this can only be achieved by $autotelic$ $agents$: intrinsically motivated learning agents that can learn to represent, generate, select and solve their own problems. In recent years, the convergence of developmental approaches with deep reinforcement learning (RL) methods has been leading to the emergence of a new field: $developmental$ $reinforcement$ $learning$. Developmental RL is concerned with the use of deep RL algorithms to tackle a developmental problem -- the $intrinsically$ $motivated$ $acquisition$ $of$ $open$-$ended$ $repertoires$ $of$ $skills$. The self-generation of goals requires the learning of compact goal encodings as well as their associated goal-achievement functions. This raises new challenges compared to standard RL algorithms originally designed to tackle pre-defined sets of goals using external reward signals. The present paper introduces developmental RL and proposes a computational framework based on goal-conditioned RL to tackle the intrinsically motivated skills acquisition problem. It proceeds to present a typology of the various goal representations used in the literature, before reviewing existing methods to learn to represent and prioritize goals in autonomous systems. We finally close the paper by discussing some open challenges in the quest of intrinsically motivated skills acquisition

Life is an Adventure! An agent-based reconciliation of narrative and scientific worldviews\ud

The scientific worldview is based on laws, which are supposed to be certain, objective, and independent of time and context. The narrative worldview found in literature, myth and religion, is based on stories, which relate the events experienced by a subject in a particular context with an uncertain outcome. This paper argues that the concept of “agent”, supported by the theories of evolution, cybernetics and complex adaptive systems, allows us to reconcile scientific and narrative perspectives. An agent follows a course of action through its environment with the aim of maximizing its fitness. Navigation along that course combines the strategies of regulation, exploitation and exploration, but needs to cope with often-unforeseen diversions. These can be positive (affordances, opportunities), negative (disturbances, dangers) or neutral (surprises). The resulting sequence of encounters and actions can be conceptualized as an adventure. Thus, the agent appears to play the role of the hero in a tale of challenge and mystery that is very similar to the "monomyth", the basic storyline that underlies all myths and fairy tales according to Campbell [1949]. This narrative dynamics is driven forward in particular by the alternation between prospect (the ability to foresee diversions) and mystery (the possibility of achieving an as yet absent prospect), two aspects of the environment that are particularly attractive to agents. This dynamics generalizes the scientific notion of a deterministic trajectory by introducing a variable “horizon of knowability”: the agent is never fully certain of its further course, but can anticipate depending on its degree of prospect

The Ecology of Open-Ended Skill Acquisition: Computational framework and experiments on the interactions between environmental, adaptive, multi-agent and cultural dynamics

An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquiring new skills in order to solve them: what is called open-ended skill acquisition (OESA). Understanding the mechanisms underlying OESA is an important scientific challenge in both cognitive science (e.g. by studying infant cognitive development) and in artificial intelligence (aiming at computational architectures capable of open-ended learning). Both fields, however, mostly focus on cognitive and social mechanisms at the scale of an individual’s life. It is rarely acknowledged that OESA, an ability that is fundamentally related to the characteristics of human intelligence, has been necessarily shaped by ecological, evolutionary and cultural mechanisms interacting at multiple spatiotemporal scales. In this thesis, I present a research program aiming at understanding, modelingand simulating the dynamics of OESA in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. It relies on a conceptual framework expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics. Three main research questions are developed and I present a selection of my contributions for each of them.- What are the ecological conditions favoring the evolution of skill acquisition?- How to bootstrap the formation of a cultural repertoire in populations of adaptive agents?- What is the role of cultural evolution in the open-ended dynamics of human skill acquisition?By developing these topics, we will reveal interesting relationships between theories in human evolution and recent approaches in artificial intelligence. This will lead to the proposition of a humanist perspective on AI: using it as a family of computational tools that can help us to explore and study the mechanisms driving open-ended skill acquisition in both artificial and biological systems, as a way to better understand the dynamics of our own species within its whole ecological context. This document presents an overview of my scientific trajectory since the start of my PhD thesis in 2007, the detail of my current research program, a selection of my contributions as well as perspectives for future work

Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning

International audienceAutonomous reinforcement learning agents, like children, do not have access to predefined goals and reward functions. They must discover potential goals, learn their own reward functions and engage in their own learning trajectory. Children, however, benefit from exposure to language, helping to organize and mediate their thought. We propose LE2 (Language Enhanced Exploration), a learning algorithm leveraging intrinsic motivations and natural language (NL) interactions with a descriptive social partner (SP). Using NL descriptions from the SP, it can learn an NL-conditioned reward function to formulate goals for intrinsically motivated goal exploration and learn a goal-conditioned policy. By exploring, collecting descriptions from the SP and jointly learning the reward function and the policy, the agent grounds NL descriptions into real behavioral goals. From simple goals discovered early to more complex goals discovered by experimenting on simpler ones, our agent autonomously builds its own behavioral repertoire. This naturally occurring curriculum is supplemented by an active learning curriculum resulting from the agent's intrinsic motivations. Experiments are presented with a simulated robotic arm that interacts with several objects including tools

Computational Theories of Curiosity-Driven Learning

International audienceWhat are the functions of curiosity? What are the mechanisms of curiosity-driven learning? We approach these questions about the living using concepts and tools from machine learning and developmental robotics. We argue that curiosity-driven learning enables organisms to make discoveries to solve complex problems with rare or deceptive rewards. By fostering exploration and discovery of a diversity of behavioural skills, and ignoring these rewards, curiosity can be efficient to bootstrap learning when there is no information, or deceptive information, about local improvement towards these problems. We also explain the key role of curiosity for efficient learning of world models. We review both normative and heuristic computational frameworks used to understand the mechanisms of curiosity in humans, conceptualizing the child as a sense-making organism. These frameworks enable us to discuss the bi-directional causal links between curiosity and learning, and to provide new hypotheses about the fundamental role of curiosity in self-organizing developmental structures through curriculum learning. We present various developmental robotics experiments that study these mechanisms in action, both supporting these hypotheses to understand better curiosity in humans and opening new research avenues in machine learning and artificial intelligence. Finally, we discuss challenges for the design of experimental paradigms for studying curiosity in psychology and cognitive neuroscience

What do we learn about development from baby robots?

Understanding infant development is one of the greatest scientific challenges of contemporary science. A large source of difficulty comes from the fact that the development of skills in infants results from the interactions of multiple mechanisms at multiple spatio-temporal scales. The concepts of "innate" or "acquired" are not any more adequate tools for explanations, which call for a shift from reductionist to systemic accounts. To address this challenge, building and experimenting with robots modeling the growing infant brain and body is crucial. Systemic explanations of pattern formation in sensorimotor, cognitive and social development, viewed as a complex dynamical system, require the use of formal models based on mathematics, algorithms and robots. Formulating hypothesis about development using such models, and exploring them through experiments, allows us to consider in detail the interaction between many mechanisms and parameters. This complements traditional experimental methods in psychology and neuroscience where only a few variables can be studied at the same time. Furthermore, the use of robots is of particular importance. The laws of physics generate everywhere around us spontaneous patterns in the inorganic world. They also strongly impact the living, and in particular constrain and guide infant development through the properties of its (changing) body in interaction with the physical environment. Being able to consider the body as an experimental variable, something that can be systematically changed in order to study the impact on skill formation, has been a dream to many developmental scientists. This is today becoming possible with developmental robotics