Layered control architectures in natural and artificial systems

We review recent research in robotics and neuroscience with the aim of highlighting some points of agreement and convergence. Specifically, we compare Brooks’ [9] subsumption architecture for robot control with a part of the neuroscience literature that can be interpreted as demonstrating hierarchical control systems in animal brains. We focus first on work that follows the tradition of Hughlings Jackson [23] who, in neuroscience and neuropsychology, is particularly associated with the notion of layered competence. From this perspective we further argue that recent work on the defense system of the rat can be interpreted by analogy to Brooks’ subsumption architecture. An important focus is the role of multiple learning systems in the brain, and of hierarchical learning mechanisms in the rat defense system

Layered control architectures in natural and artificial systems

We review recent research in robotics and neuroscience with the aim of highlighting some points of agreement and convergence. Specifically, we compare Brooks’ [9] subsumption architecture for robot control with a part of the neuroscience literature that can be interpreted as demonstrating hierarchical control systems in animal brains. We focus first on work that follows the tradition of Hughlings Jackson [23] who, in neuroscience and neuropsychology, is particularly associated with the notion of layered competence. From this perspective we further argue that recent work on the defense system of the rat can be interpreted by analogy to Brooks’ subsumption architecture. An important focus is the role of multiple learning systems in the brain, and of hierarchical learning mechanisms in the rat defense system

Is the short-latency dopamine response too short to signal reward error?

Unexpected stimuli that are behaviourally significant have the capacity to elicit a short-latency, short-duration burst of firing in mesencephalic dopaminergic neurones. An influential interpretation of the experimental data that characterize this response proposes that dopaminergic neurones have a crucial role in reinforcement learning because they signal error in the prediction of future reward. In this article we propose a different functional role for this ‘short-latency dopamine response’ in the mechanisms that underlie associative learning. We suggest that the initial burst of dopaminergic-neurone firing could represent an essential component in the process of switching attentional and behavioural selections to unexpected, behaviourally important stimuli. This switching response could be a crucial prerequisite for associative learning and might be part of a general short-latency response that is mediated by catecholamines and prepares the organism for an appropriate reaction to biologically significant events. Any act which in a given situation produces satisfaction becomes associated with that situation so that when the situation recurs the act is more likely than before to recur also. E.L. Thorndike (1911)

The basal ganglia: A vertebrate solution to the selection problem?

A selection problem arises whenever two or more competing systems seek simultaneous access to a restricted resource. Consideration of several selection architectures suggests there are significant advantages for systems which incorporate a central switching mechanism. We propose that the vertebrate basal ganglia have evolved as a centralized selection device, specialized to resolve conflicts over access to limited motor and cognitive resources. Analysis of basal ganglia functional architecture and its position within a wider anatomical framework suggests it can satisfy many of the requirements expected of an efficient selection mechanism

Emergence of visually-evoked reward expectation signals in dopamine neurons via the superior colliculus in V1 lesioned monkeys

Responses of midbrain dopamine (DA) neurons reflecting expected reward from sensory cues are critical for reward-based associative learning. However, critical pathways by which reward-related visual information is relayed to DA neurons remain unclear. To address this question, we investigated Pavlovian conditioning in macaque monkeys with unilateral primary visual cortex (V1) lesions (an animal model of 'blindsight'). Anticipatory licking responses to obtain juice drops were elicited in response to visual conditioned stimuli (CS) in the affected visual field. Subsequent pharmacological inactivation of the superior colliculus (SC) suppressed the anticipatory licking. Concurrent single unit recordings indicated that DA responses reflecting the reward expectation could be recorded in the absence of V1, and that these responses were also suppressed by SC inactivation. These results indicate that the subcortical visual circuit can relay reward-predicting visual information to DA neurons and integrity of the SC is necessary for visually-elicited classically conditioned responses after V1 lesion

A robot model of the basal ganglia: Behavior and intrinsic processing

The existence of multiple parallel loops connecting sensorimotor systems to the basal ganglia has given rise to proposals that these nuclei serve as a selection mechanism resolving competitions between the alternative actions available in a given context. A strong test of this hypothesis is to require a computational model of the basal ganglia to generate integrated selection sequences in an autonomous agent, we therefore describe a robot architecture into which such a model is embedded, and require it to control action selection in a robotic task inspired by animal observations. Our results demonstrate effective action selection by the embedded model under a wide range of sensory and motivational conditions. When confronted with multiple, high salience alternatives, the robot also exhibits forms of behavioral disintegration that show similarities to animal behavior in conflict situations. The model is shown to cast light on recent neurobiological findings concerning behavioral switching and sequencing

Segregated anatomical input to sub-regions of the rodent superior colliculus associated with approach and defense

The superior colliculus (SC) is responsible for sensorimotor transformations required to direct gaze toward or away from unexpected, biologically salient events. Significant changes in the external world are signaled to SC through primary multisensory afferents, spatially organized according to a retinotopic topography. For animals, where an unexpected event could indicate the presence of either predator or prey, early decisions to approach or avoid are particularly important. Rodents’ ecology dictates predators are most often detected initially as movements in upper visual field (mapped in medial SC), while appetitive stimuli are normally found in lower visual field (mapped in lateral SC). Our purpose was to exploit this functional segregation to reveal neural sites that can bias or modulate initial approach or avoidance responses. Small injections of Fluoro-Gold were made into medial or lateral sub-regions of intermediate and deep layers of SC (SCm/SCl). A remarkable segregation of input to these two functionally defined areas was found. (i) There were structures that projected only to SCm (e.g., specific cortical areas, lateral geniculate and suprageniculate thalamic nuclei, ventromedial and premammillary hypothalamic nuclei, and several brainstem areas) or SCl (e.g., primary somatosensory cortex representing upper body parts and vibrissae and parvicellular reticular nucleus in the brainstem). (ii) Other structures projected to both SCm and SCl but from topographically segregated populations of neurons (e.g., zona incerta and substantia nigra pars reticulata). (iii) There were a few brainstem areas in which retrogradely labeled neurons were spatially overlapping (e.g., pedunculopontine nucleus and locus coeruleus). These results indicate significantly more structures across the rat neuraxis are in a position to modulate defense responses evoked from SCm, and that neural mechanisms modulating SC-mediated defense or appetitive behavior are almost entirely segregated

MIRO: A robot “Mammal” with a biomimetic brain-based control system

We describe the design of a novel commercial biomimetic brain-based robot, MIRO, developed as a prototype robot companion. The MIRO robot is animal-like in several aspects of its appearance, however, it is also biomimetic in a more significant way, in that its control architecture mimics some of the key principles underlying the design of the mammalian brain as revealed by neuroscience. Specifically, MIRO builds on decades of previous work in developing robots with brain-based control systems using a layered control architecture alongside centralized mechanisms for integration and action selection. MIRO’s control system operates across three core processors, P1-P3, that mimic aspects of spinal cord, brainstem, and forebrain functionality respectively. Whilst designed as a versatile prototype for next generation companion robots, MIRO also provides developers and researchers with a new platform for investigating the potential advantages of brain-based control

Reinforcement learning or active inference?

This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain