How Evolution May Work Through Curiosity-Driven Developmental Process

International audienceInfants’ own activities create and actively select their learning experiences. Here we review recent models of embodied information seeking and curiosity-driven learning and show that these mechanisms have deep implications for development and evolution. We discuss how these mecha- nisms yield self-organized epigenesis with emergent ordered behavioral and cognitive develop- mental stages. We describe a robotic experiment that explored the hypothesis that progress in learning, in and for itself, generates intrinsic rewards: The robot learners probabilistically selected experiences according to their potential for reducing uncertainty. In these experiments, curiosity- driven learning led the robot learner to successively discover object affordances and vocal interac- tion with its peers. We explain how a learning curriculum adapted to the current constraints of the learning system automatically formed, constraining learning and shaping the developmental trajec- tory. The observed trajectories in the robot experiment share many properties with those in infant development, including a mixture of regularities and diversities in the developmental patterns. Finally, we argue that such emergent developmental structures can guide and constrain evolution, in particular with regard to the origins of language

What do we learn about development from baby robots?

Understanding infant development is one of the greatest scientific challenges of contemporary science. A large source of difficulty comes from the fact that the development of skills in infants results from the interactions of multiple mechanisms at multiple spatio-temporal scales. The concepts of "innate" or "acquired" are not any more adequate tools for explanations, which call for a shift from reductionist to systemic accounts. To address this challenge, building and experimenting with robots modeling the growing infant brain and body is crucial. Systemic explanations of pattern formation in sensorimotor, cognitive and social development, viewed as a complex dynamical system, require the use of formal models based on mathematics, algorithms and robots. Formulating hypothesis about development using such models, and exploring them through experiments, allows us to consider in detail the interaction between many mechanisms and parameters. This complements traditional experimental methods in psychology and neuroscience where only a few variables can be studied at the same time. Furthermore, the use of robots is of particular importance. The laws of physics generate everywhere around us spontaneous patterns in the inorganic world. They also strongly impact the living, and in particular constrain and guide infant development through the properties of its (changing) body in interaction with the physical environment. Being able to consider the body as an experimental variable, something that can be systematically changed in order to study the impact on skill formation, has been a dream to many developmental scientists. This is today becoming possible with developmental robotics

Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges

Computational models of emergent communication in agent populations are currently gaining interest in the machine learning community due to recent advances in Multi-Agent Reinforcement Learning (MARL). Current contributions are however still relatively disconnected from the earlier theoretical and computational literature aiming at understanding how language might have emerged from a prelinguistic substance. The goal of this paper is to position recent MARL contributions within the historical context of language evolution research, as well as to extract from this theoretical and computational background a few challenges for future research

CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to discover what is controllable in their environments, and what is not. Because some goals might prove easy and some impossible, agents must actively select which goal to practice at any moment, to maximize their overall mastery on the set of learnable goals. This paper proposes CURIOUS, an algorithm that leverages 1) a modular Universal Value Function Approximator with hindsight learning to achieve a diversity of goals of different kinds within a unique policy and 2) an automated curriculum learning mechanism that biases the attention of the agent towards goals maximizing the absolute learning progress. Agents focus sequentially on goals of increasing complexity, and focus back on goals that are being forgotten. Experiments conducted in a new modular-goal robotic environment show the resulting developmental self-organization of a learning curriculum, and demonstrate properties of robustness to distracting goals, forgetting and changes in body properties.Comment: Accepted at ICML 201

Complexity Reduction in the Negotiation of New Lexical Conventions

In the process of collectively inventing new words for new concepts in a population, conflicts can quickly become numerous, in the form of synonymy and homonymy. Remembering all of them could cost too much memory, and remembering too few may slow down the overall process. Is there an efficient behavior that could help balance the two? The Naming Game is a multi-agent computational model for the emergence of language, focusing on the negotiation of new lexical conventions, where a common lexicon self-organizes but going through a phase of high complexity. Previous work has been done on the control of complexity growth in this particular model, by allowing agents to actively choose what they talk about. However, those strategies were relying on ad hoc heuristics highly dependent on fine-tuning of parameters. We define here a new principled measure and a new strategy, based on the beliefs of each agent on the global state of the population. The measure does not rely on heavy computation, and is cognitively plausible. The new strategy yields an efficient control of complexity growth, along with a faster agreement process. Also, we show that short-term memory is enough to build relevant beliefs about the global lexicon.Comment: Published at CogSci 2018 conferenc

Scaling the impact of active topic choice in the Naming Game

International audienceHow does language emerge, evolve and gets transmitted between individuals? What mechanisms underly the formation and evolution of linguistic conventions, and what are their dynamics? Computational linguistic studies have shown that local interactions within groups of individuals (e.g. humans or robots) can lead to self-organization of lexica associating semantic categories to words [13]. However, it still doesn't scale well to complex meaning spaces and a large number of possible word-meaning associations (or lexical conventions), suggesting high competition among those conventions. In statistical machine learning and in developmental sciences, it has been argued that an active control of the complexity of learning situations can have a significant impact on the global dynamics of the learning process [2, 4, 3]. This approach has been mostly studied for single robotic agents learning sensorimotor affordances [6, 5]. However active learning might represent an evolutionary advantage for language formation at the population level as well [8, 12]. Naming Games are a computational framework, elaborated to simulate the self-organization of lexical conventions in the form of a multi-agent model [11]. Through repeated local interactions between random couples of agents (designated speaker and hearer), shared conventions emerge. Interactions consist of uttering a word – or an abstract signal – referring to a topic, and evaluating the success or failure of communication. However, in existing works processes involved in these interactions are typically random choices, especially the choice of a communication topic. The introduction of active learning algorithms in these models produces significant improvement of the convergence process towards a shared vocabulary, with the speaker [7, 9, 1] or the hearer [10] actively controlling vocabulary growth. We study here how the convergence time and the maximum level of complexity scale with population size, for three different strategies (one with random topic choice and two with active topic choice) detailed in table 1. Both active strategies use a parameter (α and n), which is each time chosen optimal in our simulations. As for the version of the Naming Game used in our work, the scenario of the interaction is described in [10]. Vocabulary is updated as described in the Minimal Naming Game, detailed in [14]. In our simulations, we choose to set N = M = W , where N is the population size, M the number of meanings, and W the number of possible words. The computed theoretical success ratio of communication is used to represent the degree of convergence toward a shared lexicon for the whole population. A value of 1 means that the population reached full convergence. Complexity level of an individual lexicon is measured as the total number of distinct associations between meanings and words in the lexicon, or in other words: memory usage. We show here (see figures 2,3) that convergence time and maximum complexity are reduced with active topic choice, a behavior that is amplified as larger populations are considered. The minimal counts strategy yields a strictly minimum complexity (equal to the complexity of a completed lexicon), while converging as fast as the success threshold strategy. Further work will deal with other variants of the Naming Game (with different vocabulary update, population replacement, and different ratio for N , M and W). For the moment only the hearer's choice scenario is studied, because of its high robustness to changes in parameter values for the different strategies [10]

Open challenges in understanding development and evolution of speech forms: The roles of embodied self-organization, motivation and active exploration

This article discusses open scientific challenges for understanding development and evolution of speech forms, as a commentary to Moulin-Frier et al. (Moulin-Frier et al., 2015). Based on the analysis of mathematical models of the origins of speech forms, with a focus on their assumptions , we study the fundamental question of how speech can be formed out of non--speech, at both developmental and evolutionary scales. In particular, we emphasize the importance of embodied self-organization , as well as the role of mechanisms of motivation and active curiosity-driven exploration in speech formation. Finally , we discuss an evolutionary-developmental perspective of the origins of speech

A new experimental setup to study the structure of curiosity-driven exploration in humans

International audienceWe are presenting an exploratory study with humans designed to analyze and measure properties of curiosity-driven exploration of a priori unknown sensorimotor spaces. More specifically, we are interested in the relation between exploration and learning progress

Autonomous development and learning in artificial intelligence and robotics: Scaling up deep learning to human–like learning

International audienceAutonomous lifelong development and learning is a fundamental capability of humans, differentiating them from current deep learning systems. However, other branches of artificial intelligence have designed crucial ingredients towards autonomous learning: curiosity and intrinsic motivation, social learning and natural interaction with peers, and embodiment. These mechanisms guide exploration and autonomous choice of goals, and integrating them with deep learning opens stimulating perspectives. Deep learning (DL) approaches made great advances in artificial intelligence, but are still far away from human learning. As argued convincingly by Lake et al., differences include human capabilities to learn causal models of the world from very little data, leveraging compositional representations and priors like intuitive physics and psychology. However, there are other fundamental differences between current DL systems and human learning, as well as technical ingredients to fill this gap, that are either superficially, or not adequately, discussed by Lake et al. These fundamental mechanisms relate to autonomous development and learning. They are bound to play a central role in artificial intelligence in the future. Current DL systems require engineers to manually specify a task-specific objective function for every new task, and learn through off-line processing of large training databases. On the contrary, humans learn autonomously open-ended repertoires of skills, deciding for themselves which goals to pursue or value, and which skills to explore, driven by intrinsic motivation/curiosity and social learning through natural interaction with peers. Such learning processes are incremental, online, and progressive. Human child development involves a progressive increase of complexity in a curriculum of learning where skills are explored, acquired, and built on each other, through particular ordering and timing. Finally, human learning happens in the physical world, and through bodily and physical experimentation, under severe constraints on energy, time, and computational resources. In the two last decades, the field of Developmental and Cognitive Robotics (Cangelosi and Schlesinger, 2015; Asada et al., 2009), in strong interaction with developmental psychology and neuroscience, has achieved significant advances in computationa