DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics

Robots are still limited to controlled conditions, that the robot designer knows with enough details to endow the robot with the appropriate models or behaviors. Learning algorithms add some flexibility with the ability to discover the appropriate behavior given either some demonstrations or a reward to guide its exploration with a reinforcement learning algorithm. Reinforcement learning algorithms rely on the definition of state and action spaces that define reachable behaviors. Their adaptation capability critically depends on the representations of these spaces: small and discrete spaces result in fast learning while large and continuous spaces are challenging and either require a long training period or prevent the robot from converging to an appropriate behavior. Beside the operational cycle of policy execution and the learning cycle, which works at a slower time scale to acquire new policies, we introduce the redescription cycle, a third cycle working at an even slower time scale to generate or adapt the required representations to the robot, its environment and the task. We introduce the challenges raised by this cycle and we present DREAM (Deferred Restructuring of Experience in Autonomous Machines), a developmental cognitive architecture to bootstrap this redescription process stage by stage, build new state representations with appropriate motivations, and transfer the acquired knowledge across domains or tasks or even across robots. We describe results obtained so far with this approach and end up with a discussion of the questions it raises in Neuroscience

Prospection in cognition: the case for joint episodic-procedural memory in cognitive robotics

Prospection lies at the core of cognition: it is the means by which an agent \u2013 a person or a cognitive robot \u2013 shifts its perspective from immediate sensory experience to anticipate future events, be they the actions of other agents or the outcome of its own actions. Prospection, accomplished by internal simulation, requires mechanisms for both perceptual imagery and motor imagery. While it is known that these two forms of imagery are tightly entwined in the mirror neuron system, we do not yet have an effective model of the mentalizing network which would provide a framework to integrate declarative episodic and procedural memory systems and to combine experiential knowledge with skillful know-how. Such a framework would be founded on joint perceptuo-motor representations. In this paper, we examine the case for this form of representation, contrasting sensory-motor theory with ideo-motor theory, and we discuss how such a framework could be realized by joint episodic-procedural memory. We argue that such a representation framework has several advantages for cognitive robots. Since episodic memory operates by recombining imperfectly recalled past experience, this allows it to simulate new or unexpected events. Furthermore, by virtue of its associative nature, joint episodic-procedural memory allows the internal simulation to be conditioned by current context, semantic memory, and the agent\u2019s value system. Context and semantics constrain the combinatorial explosion of potential perception-action associations and allow effective action selection in the pursuit of goals, while the value system provides the motives that underpin the agent\u2019s autonomy and cognitive development. This joint episodic-procedural memory framework is neutral regarding the final implementation of these episodic and procedural memories, which can be configured sub-symbolically as associative networks or symbolically as content-addressable image databases and databases of motor-control scripts

Cortex, countercurrent context, and dimensional integration of lifetime memory

The correlation between relative neocortex size and longevity in mammals encourages a search for a cortical function specifically related to the life-span. A candidate in the domain of permanent and cumulative memory storage is proposed and explored in relation to basic aspects of cortical organization. The pattern of cortico-cortical connectivity between functionally specialized areas and the laminar organization of that connectivity converges on a globally coherent representational space in which contextual embedding of information emerges as an obligatory feature of cortical function. This brings a powerful mode of inductive knowledge within reach of mammalian adaptations, a mode which combines item specificity with classificatory generality. Its neural implementation is proposed to depend on an obligatory interaction between the oppositely directed feedforward and feedback currents of cortical activity, in countercurrent fashion. Direct interaction of the two streams along their cortex-wide local interface supports a scheme of "contextual capture" for information storage responsible for the lifelong cumulative growth of a uniquely cortical form of memory termed "personal history." This approach to cortical function helps elucidate key features of cortical organization as well as cognitive aspects of mammalian life history strategies

A biologically inspired meta-control navigation system for the Psikharpax rat robot

A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e. g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment-recognized as new contexts-and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics

NEUROMORPHIC VLSI REALIZATION OF THE HIPPOCAMPAL FORMATION AND THE LATERAL SUPERIOR OLIVE

In this work, the focus is on realizing the function of the hippocampal formation (HF) and the lateral superior olive (LSO) in electronic circuits. The first major contribution of this dissertation is to realize the function of the HF in silicon. This was based on the GRIDSmap model and the Bayesian integration. For this, two novel circuits were designed and integrated with others. The first circuit was that of a Bayesian integration synapse which can perform Bayesian integration at the single neuron level. The second circuit was that of a velocity integrator which is so compact that it can enable integration of the entire system on a single chip compared to its predecessors which would have needed 27 chips! However, since the computational neuroscience models of the hippocampal place cells do not explain all the characteristics observed empirically, a novel model for the place cells, based on the sensori-motor integration of inputs is proposed. This is the second major contribution of this thesis. The third major contribution is to demonstrate a VLSI system which can perform azimuthal localization based on population response of the LSO. This system was based on the Reed and Blum's model of the LSO. For this, a novel circuit of a second order synapse and that of a conductance neuron was designed and integrated with other circuits. This synapse circuit can produce an output current whose peak is delayed and is proportional to the number of inputs it receives. The HF is thought to aid in spatial navigation and the LSO is thought to be involved in azimuthal localization of sounds both of which are useful for autonomous robotic spatial navigation. Hence, silicon realization of these two will be useful in robotics which is an area of interest for the neuromorphic engineers

Role of Anterior Cingulate Cortex in Saccade Control

Cognitive control is referred to the guidance of behavior based on internal goals rather than external stimuli. It has been postulated that prefrontal cortex is mainly involved in higher order cognitive functions. Specifically, anterior cingulate cortex (ACC), which is part of the prefrontal cortex, is suggested to be involved in performance monitoring and conflict monitoring that are considered to be cognitive control functions. Saccades are the fast eye movements that align the fovea on the objects of interest in the environment. In this thesis, I have explored the role of ACC in control of saccadic eye movements. First, I performed a resting-state fMRI study to identify areas within the ACC that are functionally connected to the frontal eye fields (FEF). It has been shown that FEF is involved in saccade generation. Therefore, the ACC areas that are functionally connected to FEF could be hypothesized to have a role in saccade control. Then, I performed simultaneous electrophysiological recordings in the ACC and FEF. Furthermore, I explored whether ACC exerts control over FEF. My results show that ACC is involved in cognitive control of saccades. Furthermore, the ACC and FEF neurons communicate through synchronized theta and beta band activity in these areas. The results of this thesis shine light on the mechanisms by which these brain areas communicate. Moreover, my findings support the notion that ACC and FEF have a unique oscillatory property, and more specifically ACC has a prominent theta band, and to a lesser extent beta band activity

Central vestibular networking for sensorimotor control, cognition, and emotion

Purpose of reviewThe aim of this study was to illuminate the extent of the bilateral central vestibular network from brainstem and cerebellum to subcortical and cortical areas and its interrelation to higher cortical functions such as spatial cognition and anxiety.Recent findingsThe conventional view that the main function of the vestibular system is the perception of self-motion and body orientation in space and the sensorimotor control of gaze and posture had to be developed further by a hierarchical organisation with bottom-up and top-down interconnections. Even the vestibulo-ocular and vestibulo-spinal reflexes are modified by perceptual cortical processes, assigned to higher vestibulo-cortical functions. A first comparative fMRI meta-analysis of vestibular stimulation and fear-conditioning studies in healthy participants disclosed widely distributed clusters of concordance, including the prefrontal cortex, anterior insula, temporal and inferior parietal lobe, thalamus, brainstem and cerebellum. In contrast, the cortical vestibular core region around the posterior insula was activated during vestibular stimulation but deactivated during fear conditioning. In recent years, there has been increasing evidence from studies in animals and humans that the central vestibular system has numerous connections related to spatial sensorimotor performance, memory, and emotion. The clinical implication of the complex interaction within various networks makes it difficult to assign some higher multisensory disorders to one particular modality, for example in spatial hemineglect or room-tilt illusion.SummaryOur understanding of higher cortical vestibular functions is still in its infancy. Different brain imaging techniques in animals and humans are one of the most promising methodological approaches for further structural and functional decoding of the vestibular and other intimately interconnected networks. The multisensory networking including cognition and emotion determines human behaviour in space