Integration of Action and Language Knowledge: A Roadmap for Developmental Robotics

“This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder." “Copyright IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.”This position paper proposes that the study of embodied cognitive agents, such as humanoid robots, can advance our understanding of the cognitive development of complex sensorimotor, linguistic, and social learning skills. This in turn will benefit the design of cognitive robots capable of learning to handle and manipulate objects and tools autonomously, to cooperate and communicate with other robots and humans, and to adapt their abilities to changing internal, environmental, and social conditions. Four key areas of research challenges are discussed, specifically for the issues related to the understanding of: 1) how agents learn and represent compositional actions; 2) how agents learn and represent compositional lexica; 3) the dynamics of social interaction and learning; and 4) how compositional action and language representations are integrated to bootstrap the cognitive system. The review of specific issues and progress in these areas is then translated into a practical roadmap based on a series of milestones. These milestones provide a possible set of cognitive robotics goals and test scenarios, thus acting as a research roadmap for future work on cognitive developmental robotics.Peer reviewe

Self-directedness, integration and higher cognition

In this paper I discuss connections between self-directedness, integration and higher cognition. I present a model of self-directedness as a basis for approaching higher cognition from a situated cognition perspective. According to this model increases in sensorimotor complexity create pressure for integrative higher order control and learning processes for acquiring information about the context in which action occurs. This generates complex articulated abstractive information processing, which forms the major basis for higher cognition. I present evidence that indicates that the same integrative characteristics found in lower cognitive process such as motor adaptation are present in a range of higher cognitive process, including conceptual learning. This account helps explain situated cognition phenomena in humans because the integrative processes by which the brain adapts to control interaction are relatively agnostic concerning the source of the structure participating in the process. Thus, from the perspective of the motor control system using a tool is not fundamentally different to simply controlling an arm

Ecological adaptation in the context of an actor-critic

Biological beings are the result of an evolutionary and developmental process of adaptation to the environment they perceive and where they act. Animals and plants have successfully adapted to a large variety of environments, which supports the ideal of inspiring artificial agents after biology and ethology. This idea has been already suggested by previous studies and is extended throughout this thesis. However, the role of perception in the process of adaptation and its integration in an agent capable of acting for survival is not clear.Robotic architectures in AI proposed throughout the last decade have broadly addressed the problems of behaviour selection, namely deciding "what to do next", and of learning as the two main adaptive processes. Behaviour selection has been commonly related to theories of motivation, and learning has been bound to theories of reinforcement. However, the formulation of a general theory including both processes as particular cases of the same phenomenon is still an incomplete task. This thesis focuses again on behaviour selection and learning; however it proposes to integrate both processes by stressing the ecological relationship between the agent and its environment. If the selection of behaviour is an expression of the agent's motivations, the feedback of the environment due to behaviour execution can be viewed as part of the same process, since it also influences the agent's internal motivations and the learning processes via reinforcement. I relate this to an argument supporting the existence of a common neural substrate to compute motivation and reward, and therefore relating the elicitation of a behaviour to the perception of reward resulting from its executionAs in previous studies, behaviour selection is viewed as a competition among parallel pathways to gain control over the agent's actuators. Unlike for the previous cases, the computation of every motivation in this thesis is not anymore the result of an additive or multiplicative formula combining inner and outer stimuli. Instead, the ecological principle is proposed to constrain the combination of stimuli in a novel fashion that leads to adaptive behavioural patterns. This method aims at overcoming the intrinsic limitations of any formula, the use of which results in behavioural responses restricted to a set of specific patterns, and therefore to the set of ethological cases they can justify. External stimuli and internal physiology in the model introduced in this thesis are not combined a priori. Instead, these are viewed from the perspective of the agent as modulatory elements biasing the selection of one behaviour over another guided by the reward provided by the environment, being the selection performed by an actor-critic reinforcement learning algorithm aiming at the maximum cumulative reward.In this context, the agent's drives are the expression of the deficit or excess of internal resources and the reference of the agent to define its relationship with the environment. The schema to learn object affordances is integrated in an actor-critic reinforcement learning algorithm, which is the core of a motivation and reinforcement framework driving behaviour selection and learning. Its working principle is based on the capacity of perceiving changes in the environment via internal hormonal responses and of modifying the agent's behavioural patterns accordingly. To this end, the concept of reward is defined in the framework of the agent's internal physiology and is related to the condition of physiological stability introduced by Ashby, and supported by Dawkins and Meyer as a requirement for survival. In this light, the definition of the reward used for learning is defined in the physiological state, where the effect of interacting with the environment can be quantified in an ethologically consistent manner.The above ideas on motivation, behaviour selection, learning and perception have been made explicit in an architecture integrated in an simulated robotic platform. To demonstrate the reach of their validity, extensive simulation has been performed to address the affordance learning paradigm and the adaptation offered by the framework of the actor-critic. To this end, three different metrics have been proposed to measure the effect of external and internal perception on the learning and behaviour selection processes: the performance in terms of flexibility of adaptation, the physiological stability and the cycles of behaviour execution at every situation. In addition to this, the thesis has begun to frame the integration of behaviours of an appetitive and consummatory nature in a single schema. Finally, it also contributes to the arguments disambiguating the role of dopamine as a neurotransmitter in the Basal Ganglia

Should artificial agents ask for help in human-robot collaborative problem-solving?

International audienceTransferring as fast as possible the functioning of our brain to artificial intelligence is an ambitious goal that would help advance the state of the art in AI and robotics. It is in this perspective that we propose to start from hypotheses derived from an empirical study in a human-robot interaction and to verify if they are validated in the same way for children as for a basic reinforcement learning algorithm. Thus, we check whether receiving help from an expert when solving a simple close-ended task (the Towers of Hanoï) allows to accelerate or not the learning of this task, depending on whether the intervention is canonical or requested by the player. Our experiences have allowed us to conclude that, whether requested or not, a Q-learning algorithm benefits in the same way from expert help as children do

Attentional mechanisms for socially interactive robots – a survey

This review intends to provide an overview of the state of the art in the modeling and implementation of automatic attentional mechanisms for socially interactive robots. Humans assess and exhibit intentionality by resorting to multisensory processes that are deeply rooted within low-level automatic attention-related mechanisms of the brain. For robots to engage with humans properly, they should also be equipped with similar capabilities. Joint attention, the precursor of many fundamental types of social interactions, has been an important focus of research in the past decade and a half, therefore providing the perfect backdrop for assessing the current status of state-of-the-art automatic attentional-based solutions. Consequently, we propose to review the influence of these mechanisms in the context of social interaction in cutting-edge research work on joint attention. This will be achieved by summarizing the contributions already made in these matters in robotic cognitive systems research, by identifying the main scientific issues to be addressed by these contributions and analyzing how successful they have been in this respect, and by consequently drawing conclusions that may suggest a roadmap for future successful research efforts

Markerless Vision-Based Skeleton Tracking in Therapy of Gross Motor Skill Disorders in Children

This chapter presents a research towards implementation of a computer vision system for markerless skeleton tracking in therapy of gross motor skill disorders in children suffering from mild cognitive impairment. The proposed system is based on a low-cost 3D sensor and a skeleton tracking software. The envisioned architecture is scalable in the sense that the system may be used as a stand-alone assistive tool for tracking the effects of therapy or it may be integrated with an advanced autonomous conversational agent to maintain the spatial attention of the child and to increase her motivation to undergo a long-term therapy

Addressing joint action challenges in HRI: Insights from psychology and philosophy

The vast expansion of research in human-robot interactions (HRI) these last decades has been accompanied by the design of increasingly skilled robots for engaging in joint actions with humans. However, these advances have encountered significant challenges to ensure fluent interactions and sustain human motivation through the different steps of joint action. After exploring current literature on joint action in HRI, leading to a more precise definition of these challenges, the present article proposes some perspectives borrowed from psychology and philosophy showing the key role of communication in human interactions. From mutual recognition between individuals to the expression of commitment and social expectations, we argue that communicative cues can facilitate coordination, prediction, and motivation in the context of joint action. The description of several notions thus suggests that some communicative capacities can be implemented in the context of joint action for HRI, leading to an integrated perspective of robotic communication.French National Research Agency (ANR) ANR-16-CE33-0017 ANR-17-EURE-0017 FrontCog ANR-10-IDEX-0001-02 PSLJuan de la Cierva-Incorporacion grant IJC2019-040199-ISpanish Government PID2019-108870GB-I00 PID2019-109764RB-I0

The hippocampus and cerebellum in adaptively timed learning, recognition, and movement

The concepts of declarative memory and procedural memory have been used to distinguish two basic types of learning. A neural network model suggests how such memory processes work together as recognition learning, reinforcement learning, and sensory-motor learning take place during adaptive behaviors. To coordinate these processes, the hippocampal formation and cerebellum each contain circuits that learn to adaptively time their outputs. Within the model, hippocampal timing helps to maintain attention on motivationally salient goal objects during variable task-related delays, and cerebellar timing controls the release of conditioned responses. This property is part of the model's description of how cognitive-emotional interactions focus attention on motivationally valued cues, and how this process breaks down due to hippocampal ablation. The model suggests that the hippocampal mechanisms that help to rapidly draw attention to salient cues could prematurely release motor commands were not the release of these commands adaptively timed by the cerebellum. The model hippocampal system modulates cortical recognition learning without actually encoding the representational information that the cortex encodes. These properties avoid the difficulties faced by several models that propose a direct hippocampal role in recognition learning. Learning within the model hippocampal system controls adaptive timing and spatial orientation. Model properties hereby clarify how hippocampal ablations cause amnesic symptoms and difficulties with tasks which combine task delays, novelty detection, and attention towards goal objects amid distractions. When these model recognition, reinforcement, sensory-motor, and timing processes work together, they suggest how the brain can accomplish conditioning of multiple sensory events to delayed rewards, as during serial compound conditioning.Air Force Office of Scientific Research (F49620-92-J-0225, F49620-86-C-0037, 90-0128); Advanced Research Projects Agency (ONR N00014-92-J-4015); Office of Naval Research (N00014-91-J-4100, N00014-92-J-1309, N00014-92-J-1904); National Institute of Mental Health (MH-42900

SOVEREIGN: An Autonomous Neural System for Incrementally Learning Planned Action Sequences to Navigate Towards a Rewarded Goal

How do reactive and planned behaviors interact in real time? How are sequences of such behaviors released at appropriate times during autonomous navigation to realize valued goals? Controllers for both animals and mobile robots, or animats, need reactive mechanisms for exploration, and learned plans to reach goal objects once an environment becomes familiar. The SOVEREIGN (Self-Organizing, Vision, Expectation, Recognition, Emotion, Intelligent, Goaloriented Navigation) animat model embodies these capabilities, and is tested in a 3D virtual reality environment. SOVEREIGN includes several interacting subsystems which model complementary properties of cortical What and Where processing streams and which clarify similarities between mechanisms for navigation and arm movement control. As the animat explores an environment, visual inputs are processed by networks that are sensitive to visual form and motion in the What and Where streams, respectively. Position-invariant and sizeinvariant recognition categories are learned by real-time incremental learning in the What stream. Estimates of target position relative to the animat are computed in the Where stream, and can activate approach movements toward the target. Motion cues from animat locomotion can elicit head-orienting movements to bring a new target into view. Approach and orienting movements are alternately performed during animat navigation. Cumulative estimates of each movement are derived from interacting proprioceptive and visual cues. Movement sequences are stored within a motor working memory. Sequences of visual categories are stored in a sensory working memory. These working memories trigger learning of sensory and motor sequence categories, or plans, which together control planned movements. Predictively effective chunk combinations are selectively enhanced via reinforcement learning when the animat is rewarded. Selected planning chunks effect a gradual transition from variable reactive exploratory movements to efficient goal-oriented planned movement sequences. Volitional signals gate interactions between model subsystems and the release of overt behaviors. The model can control different motor sequences under different motivational states and learns more efficient sequences to rewarded goals as exploration proceeds.Riverside Reserach Institute; Defense Advanced Research Projects Agency (N00014-92-J-4015); Air Force Office of Scientific Research (F49620-92-J-0225); National Science Foundation (IRI 90-24877, SBE-0345378); Office of Naval Research (N00014-92-J-1309, N00014-91-J-4100, N00014-01-1-0624, N00014-01-1-0624); Pacific Sierra Research (PSR 91-6075-2

Layered control architectures in robots and vertebrates

We revieiv recent research in robotics, neuroscience, evolutionary neurobiology, and ethology with the aim of highlighting some points of agreement and convergence. Specifically, we com pare Brooks' (1986) subsumption architecture for robot control with research in neuroscience demonstrating layered control systems in vertebrate brains, and with research in ethology that emphasizes the decomposition of control into multiple, intertwined behavior systems. From this perspective we then describe interesting parallels between the subsumption architecture and the natural layered behavior system that determines defense reactions in the rat. We then consider the action selection problem for robots and vertebrates and argue that, in addition to subsumption- like conflict resolution mechanisms, the vertebrate nervous system employs specialized selection mechanisms located in a group of central brain structures termed the basal ganglia. We suggest that similar specialized switching mechanisms might be employed in layered robot control archi tectures to provide effective and flexible action selection