A Neurocomputational Model of the Functional Role of Dopamine in Stimulus-Response Task Learning and Performance

Thesis (Ph.D.) - Indiana University, Psychology, 2009The neuromodulatory neurotransmitter dopamine (DA) plays a complex, but central role in the learning and performance of stimulus-response (S-R) behaviors. Studies have implicated DA's role in reward-driven learning and also its role in setting the overall level of vigor or frequency of response. Here, a neurocomputational model is developed which models DA's influence on a set of brain regions believed to be involved in the learning and execution of S-R tasks, including frontal cortex, basal ganglia, and cingulate cortex. An `actor' component of the model is trained, using `babble' (random behavior selection) and `critic' (rewarding and punishing) components of the model, to perform acceptance/rejection responses upon presentation of color stimuli in the context of recently presented auditory tones. The model behaves like an autonomous organism learning (and relearning) through `trial-and-error'. The focus of the study, the impact of hypo- and hyper-normal DA activity on this model, is investigated by three different dopaminergic pathways--two striatal and one prefrontal cortical--being manipulated independently during the learning and performance of the color response task. Hypo-DA conditions, analogous to Parkinsonism, cause slowing and reduction of frequency of learned responses, and, at extremes, degrade the learning (either initial or reversal) of the task. Hyper-DA conditions, analogous to psychostimulant effects, cause more rapid response times, but also can lead to perseveration of incorrect learning of response on the task. The presence of these effects often depends on which DA-ergic pathway is manipulated, however, which has implications for interpretation of the pharmacological experimental data. The proposed model embodies an integrative theory of dopamine function which suggests that the base rate of DA cell activity encodes the overall `activity-oriented motivation' of the organism, with hunger and/or expectation of reward driving both response vigor and tendency to generate an explorative `babble' response. This more `tonic' feature of DA functionality coexists naturally with the more extensively-studied `phasic' reward-learning features. The model may provide better insights on the role of DA system dysfunction in the cognitive and motivational symptoms of disorders such as Parkinsonism, psychostimulant abuse, ADHD, OCD, and schizophrenia, accounting for deficits in both learning and performance of tasks

A biologically inspired meta-control navigation system for the Psikharpax rat robot

A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e. g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment-recognized as new contexts-and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics

Development of a Large-Scale Integrated Neurocognitive Architecture Part 1: Conceptual Framework

The idea of creating a general purpose machine intelligence that captures many of the features of human cognition goes back at least to the earliest days of artificial intelligence and neural computation. In spite of more than a half-century of research on this issue, there is currently no existing approach to machine intelligence that comes close to providing a powerful, general-purpose human-level intelligence. However, substantial progress made during recent years in neural computation, high performance computing, neuroscience and cognitive science suggests that a renewed effort to produce a general purpose and adaptive machine intelligence is timely, likely to yield qualitatively more powerful approaches to machine intelligence than those currently existing, and certain to lead to substantial progress in cognitive science, AI and neural computation. In this report, we outline a conceptual framework for the long-term development of a large-scale machine intelligence that is based on the modular organization, dynamics and plasticity of the human brain. Some basic design principles are presented along with a review of some of the relevant existing knowledge about the neurobiological basis of cognition. Three intermediate-scale prototypes for parts of a larger system are successfully implemented, providing support for the effectiveness of several of the principles in our framework. We conclude that a human-competitive neuromorphic system for machine intelligence is a viable long- term goal, but that for the short term, substantial integration with more standard symbolic methods as well as substantial research will be needed to make this goal achievable

Executive control: balancing stability and flexibility via the duality of evolutionary neuroanatomical trends

The concept of executive functions has a rich history and remains current despite increased use of other terms, including working memory and cognitive control. Executive functions have sometimes been equated with functions subserved by the frontal cortex, but this adds little clarity, given that we so far lack a comprehensive theory of frontal function. Pending a more complete mechanistic understanding, clinically useful generalizations can help characterize both healthy cognition and multiple varieties of cognitive impairment. This article surveys several hierarchical and autoregulatory control theories, and suggests that the evolutionary cytoarchitectonic trends theory provides a valuable neuroanatomical framework to help organize research on frontal structure-function relations. The theory suggests that paleocortical/ventrolateral and archicortical/dorsomedial trends are associated with neural network flexibility and stability respectively, which comports well with multiple other conceptual distinctions that have been proposed to characterize ventral and dorsal frontal functions, including the “initiation/inhibition,” “what/where,” and “classification/expectation” hypotheses

A model of reversal learning and working memory in medicated and unmedicated patients with Parkinsons disease

Wepresent a neural network model of cognition in medicated and unmedicated patients with Parkinson’s disease (PD) in various learning and memory tasks. The model extends our prior models of the basal ganglia and PD with further modeling of the role of prefrontal cortex (PFC) dopamine in stimulus–response learning, reversal, and working memory. In our model, PD is associated with decreased dopamine levels in the basal ganglia and PFC, whereas dopamine medications increase dopamine levels in both brain structures. Simulation results suggest that dopamine medications impair stimulus–response learning in agreement with experimental data (Breitenstein et al., 2006; Gotham, Brown, & Marsden, 1988). Weshow how decreased dopamine levels in the PFC in unmedicated PD patients are associated with impaired working memory performance, as seen experimentally (Costa et al., 2003; Lange et al., 1992; Moustafa, Sherman, & Frank, 2008; Owen, Sahakian, Hodges, Summers, & Polkey, 1995). Further, our model simulations illustrate how increases in tonic dopamine levels in the PFC due to dopamine medications will enhance working memory, in accord with previous modeling and experimental results (Cohen, Braver, & Brown, 2002; Durstewitz, Seamans, & Sejnowski, 2000; Wang, Vijayraghavan, & Goldman-Rakic, 2004). The model is also consistent with data reported in Cools, Barker, Sahakian, and Robbins (2001), who showed that dopamine medications impair reversal learning. In addition, our model shows that extended training of the reversal phase leads to enhanced reversal performance in medicated PD patients, which is a new, and as yet untested, prediction of the model. Overall, our model provides a unified account for performance in various behavioral tasks using common computational principles.Research reported in this publication was supported by National Institutes of Health Award 1 P50 NS 071675-02 from the National Institute of Neurological Disorders and Stroke and by a 2013 internal UWS Research Grant Scheme award P00021210 to A.A.M

Neurobiological mechanisms for language, symbols and concepts: Clues from brain-constrained deep neural networks

Neural networks are successfully used to imitate and model cognitive processes. However, to provide clues about the neurobiological mechanisms enabling human cognition, these models need to mimic the structure and function of real brains. Brain-constrained networks differ from classic neural networks by implementing brain similarities at different scales, ranging from the micro- and mesoscopic levels of neuronal function, local neuronal links and circuit interaction to large-scale anatomical structure and between-area connectivity. This review shows how brain-constrained neural networks can be applied to study in silico the formation of mechanisms for symbol and concept processing and to work towards neurobiological explanations of specifically human cognitive abilities. These include verbal working memory and learning of large vocabularies of symbols, semantic binding carried by specific areas of cortex, attention focusing and modulation driven by symbol type, and the acquisition of concrete and abstract concepts partly influenced by symbols. Neuronal assembly activity in the networks is analyzed to deliver putative mechanistic correlates of higher cognitive processes and to develop candidate explanations founded in established neurobiological principles

Dopamine modulation of learning and memory in the prefrontal cortex: insights from studies in primates, rodents, and birds

In this review, we provide a brief overview over the current knowledge about the role of dopamine transmission in the prefrontal cortex during learning and memory. We discuss work in humans, monkeys, rats, and birds in order to provide a basis for comparison across species that might help identify crucial features and constraints of the dopaminergic system in executive function. Computational models of dopamine function are introduced to provide a framework for such a comparison. We also provide a brief evolutionary perspective showing that the dopaminergic system is highly preserved across mammals. Even birds, following a largely independent evolution of higher cognitive abilities, have evolved a comparable dopaminergic system. Finally, we discuss the unique advantages and challenges of using different animal models for advancing our understanding of dopamine function in the healthy and diseased brain