Final report key contents: main results accomplished by the EU-Funded project IM-CLeVeR - Intrinsically Motivated Cumulative Learning Versatile Robots

This document has the goal of presenting the main scientific and technological achievements of the project IM-CLeVeR. The document is organised as follows: 1. Project executive summary: a brief overview of the project vision, objectives and keywords. 2. Beneficiaries of the project and contacts: list of Teams (partners) of the project, Team Leaders and contacts. 3. Project context and objectives: the vision of the project and its overall objectives 4. Overview of work performed and main results achieved: a one page overview of the main results of the project 5. Overview of main results per partner: a bullet-point list of main results per partners 6. Main achievements in detail, per partner: a throughout explanation of the main results per partner (but including collaboration work), with also reference to the main publications supporting them

The Ecology of Open-Ended Skill Acquisition: Computational framework and experiments on the interactions between environmental, adaptive, multi-agent and cultural dynamics

An intriguing feature of the human species is our ability to continuously invent new problems and to proactively acquiring new skills in order to solve them: what is called open-ended skill acquisition (OESA). Understanding the mechanisms underlying OESA is an important scientific challenge in both cognitive science (e.g. by studying infant cognitive development) and in artificial intelligence (aiming at computational architectures capable of open-ended learning). Both fields, however, mostly focus on cognitive and social mechanisms at the scale of an individual’s life. It is rarely acknowledged that OESA, an ability that is fundamentally related to the characteristics of human intelligence, has been necessarily shaped by ecological, evolutionary and cultural mechanisms interacting at multiple spatiotemporal scales. In this thesis, I present a research program aiming at understanding, modelingand simulating the dynamics of OESA in artificial systems, grounded in theories studying its eco-evolutionary bases in the human species. It relies on a conceptual framework expressing the complex interactions between environmental, adaptive, multi-agent and cultural dynamics. Three main research questions are developed and I present a selection of my contributions for each of them.- What are the ecological conditions favoring the evolution of skill acquisition?- How to bootstrap the formation of a cultural repertoire in populations of adaptive agents?- What is the role of cultural evolution in the open-ended dynamics of human skill acquisition?By developing these topics, we will reveal interesting relationships between theories in human evolution and recent approaches in artificial intelligence. This will lead to the proposition of a humanist perspective on AI: using it as a family of computational tools that can help us to explore and study the mechanisms driving open-ended skill acquisition in both artificial and biological systems, as a way to better understand the dynamics of our own species within its whole ecological context. This document presents an overview of my scientific trajectory since the start of my PhD thesis in 2007, the detail of my current research program, a selection of my contributions as well as perspectives for future work

Curiosity-driven phonetic learning

International audienceThis article studies how developmental phonetic learning can be guided by pure curiosity-driven exploration, also called intrinsically motivated exploration. Phonetic learning refers here to learning how to control a vocal tract to reach acoustic goals. We compare three different exploration strategies for learning the auditory-motor inverse model: random motor exploration, random goal selection with reaching, and curiosity-driven active goal selection with reaching. Using a realistic vocal tract model, we show how intrinsically motivated learning driven by competence progress can generate automatically developmental structure in both articulatory and auditory modalities, displaying patterns in line with some experimental data from infants

Integrating reinforcement learning, equilibrium points and minimum variance to understand the development of reaching: a computational model

Despite the huge literature on reaching behaviour we still lack a clear idea about the motor control processes underlying its development in infants. This article contributes to overcome this gap by proposing a computational model based on three key hypotheses: (a) trial-anderror learning processes drive the progressive development of reaching; (b) the control of the movements based on equilibrium points allows the model to quickly find the initial approximate solution to the problem of gaining contact with the target objects; (c) the request of precision of the end-movement in the presence of muscular noise drives the progressive refinement of the reaching behaviour. The tests of the model, based on a two degrees of freedom simulated dynamical arm, show that it is capable of reproducing a large number of empirical findings, most deriving from longitudinal studies with children: the developmental trajectory of several dynamical and kinematic variables of reaching movements, the time evolution of submovements composing reaching, the progressive development of a bell-shaped speed profile, and the evolution of the management of redundant degrees of freedom. The model also produces testable predictions on several of these phenomena. Most of these empirical data have never been investigated by previous computational models and, more importantly, have never been accounted for by a unique model. In this respect, the analysis of the model functioning reveals that all these results are ultimately explained, sometimes in unexpected ways, by the same developmental trajectory emerging from the interplay of the three mentioned hypotheses: the model first quickly learns to perform coarse movements that assure a contact of the hand with the target (an achievement with great adaptive value), and then slowly refines the detailed control of the dynamical aspects of movement to increase accuracy

The development of numerical cognition in children and artificial systems: a review of the current knowledge and proposals for multi-disciplinary research

Numerical cognition is a distinctive component of human intelligence such that the observation of its practice provides a window into high-level brain function. The modelling of numerical abilities in artificial cognitive systems can help to confirm existing child development hypotheses and define new ones by means of computational simulations. Meanwhile, new research will help to discover innovative principles for the design of artificial agents with advanced reasoning capabilities and clarify the underlying algorithms (e.g. deep learning) that can be highly effective but difficult to understand for humans. This article promotes new investigation by providing a common resource for researchers with different backgrounds, including computer science, robotics, neuroscience, psychology, and education, who are interested in pursuing scientific collaboration on mutually stimulating research on this topic. The article emphasises the fundamental role of embodiment in the initial development of numerical cognition in children. This strong relationship with the body motivates the Cognitive Developmental Robotics (CDR) approach for new research that can (among others) help to standardise data collection and provide open databases for benchmarking computational models. Furthermore, we discuss the potential application of robots in classrooms and argue that the CDR approach can be extended to assist educators and favour mathematical education

Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcement driving both action acquisition and reward maximization: A simulated robotic study

An important issue of recent neuroscientific research is to understand the functional role of the phasic release of dopamine in the striatum, and in particular its relation to reinforcement learning. The literature is split between two alternative hypotheses: one considers phasic dopamine as a reward prediction error similar to the computational TD-error, whose function is to guide an animal to maximize future rewards; the other holds that phasic dopamine is a sensory prediction error signal that lets the animal discover and acquire novel actions. In this paper we propose an original hypothesis that integrates these two contrasting positions: according to our view phasic dopamine represents a TD-like reinforcement prediction error learning signal determined by both unexpected changes in the environment (temporary, intrinsic reinforcements) and biological rewards (permanent, extrinsic reinforcements). Accordingly, dopamine plays the functional role of driving both the discovery and acquisition of novel actions and the maximization of future rewards. To validate our hypothesis we perform a series of experiments with a simulated robotic system that has to learn different skills in order to get rewards. We compare different versions of the system in which we vary the composition of the learning signal. The results show that only the system reinforced by both extrinsic and intrinsic reinforcements is able to reach high performance in sufficiently complex conditions

Ecological active vision: four bio-inspired principles to integrate bottom-up and adaptive top-down attention tested with a simple camera-arm robot

Vision gives primates a wealth of information useful to manipulate the environment, but at the same time it can easily overwhelm their computational resources. Active vision is a key solution found by nature to solve this problem: a limited fovea actively displaced in space to collect only relevant information. Here we highlight that in ecological conditions this solution encounters four problems: 1) the agent needs to learn where to look based on its goals; 2) manipulation causes learning feedback in areas of space possibly outside the attention focus; 3) good visual actions are needed to guide manipulation actions, but only these can generate learning feedback; and 4) a limited fovea causes aliasing problems. We then propose a computational architecture ("BITPIC") to overcome the four problems, integrating four bioinspired key ingredients: 1) reinforcement-learning fovea-based top-down attention; 2) a strong vision-manipulation coupling; 3) bottom-up periphery-based attention; and 4) a novel action-oriented memory. The system is tested with a simple simulated camera-arm robot solving a class of search-and-reach tasks involving color-blob "objects." The results show that the architecture solves the problems, and hence the tasks, very ef?ciently, and highlight how the architecture principles can contribute to a full exploitation of the advantages of active vision in ecological conditions