The role of prediction and outcomes in adaptive cognitive control

Humans adaptively perform actions to achieve their goals. This flexible behaviour requires two core abilities: the ability to anticipate the outcomes of candidate actions and the ability to select and implement actions in a goal-directed manner. The ability to predict outcomes has been extensively researched in reinforcement learning paradigms, but this work has often focused on simple actions that are not embedded in hierarchical and sequential structures that are characteristic of goal-directed human behaviour. On the other hand, the ability to select actions in accordance with high-level task goals, particularly in the presence of alternative responses and salient distractors, has been widely researched in cognitive control paradigms. Cognitive control research, however, has often paid less attention to the role of action outcomes. The present review attempts to bridge these accounts by proposing an outcome-guided mechanism for selection of extended actions. Our proposal builds on constructs from the hierarchical reinforcement learning literature, which emphasises the concept of reaching and evaluating informative states, i.e., states that constitute subgoals in complex actions. We develop an account of the neural mechanisms that allow outcome-guided action selection to be achieved in a network that relies on projections from cortical areas to the basal ganglia and back-projections from the basal ganglia to the cortex. These cortico-basal ganglia-thalamo-cortical ‘loops’ allow convergence – and thus integration – of information from non-adjacent cortical areas (for example between sensory and motor representations). This integration is essential in action sequences, for which achieving an anticipated sensory state signals the successful completion of an action. We further describe how projection pathways within the basal ganglia allow selection between representations, which may pertain to movements, actions, or extended action plans. The model lastly envisages a role for hierarchical projections from the striatum to dopaminergic midbrain areas that enable more rostral frontal areas to bias the selection of inputs from more posterior frontal areas via their respective representations in the basal ganglia.This work is supported by the Biotechnology and Biological Sciences Research Council (BBSRC) Grant BB/I019847/1, awarded to NY and FW

From “Oh, OK” to “Ah, yes” to “Aha!”: Hyper-systemizing and the rewards of insight\ud

Hyper-systemizers are individuals displaying an unusually strong bias toward systemizing, i.e. toward explaining events and solving problems by appeal to mechanisms that do not involve intentions or agency. Hyper-systemizing in combination with deficit mentalizing ability typically presents clinically as an autistic spectrum disorder; however, the development of hyper-systemizing in combination with normal-range mentalizing ability is not well characterized. Based on a review and synthesis of clinical, observational, experimental, and neurofunctional studies, it is hypothesized that repeated episodes of insightful problem solving by systemizing result in attentional and motivational sensitization toward further systemizing via progressive and chronic deactivation of the default network. This hypothesis is distinguished from alternatives, and its correlational and causal implications are discussed. Predictions of the default-deactivation model accessible to survey-based instruments, standard cognitive measures and neurofunctional methods are outlined, and evidence pertaining to them considered

Neurobiological Foundations Of Stability And Flexibility

In order to adapt to changing and uncertain environments, humans and other organisms must balance stability and flexibility in learning and behavior. Stability is necessary to learn environmental regularities and support ongoing behavior, while flexibility is necessary when beliefs need to be revised or behavioral strategies need to be changed. Adjusting the balance between stability and flexibility must often be based on endogenously generated decisions that are informed by information from the environment but not dictated explicitly. This dissertation examines the neurobiological bases of such endogenous flexibility, focusing in particular on the role of prefrontally-mediated cognitive control processes and the neuromodulatory actions of dopaminergic and noradrenergic systems. In the first study (Chapter 2), we examined the role of frontostriatal circuits in instructed reinforcement learning. In this paradigm, inaccurate instructions are given prior to trial-and-error learning, leading to bias in learning and choice. Abandoning the instructions thus necessitates flexibility. We utilized transcranial direct current stimulation over dorsolateral prefrontal cortex to try to establish a causal role for this area in this bias. We also assayed two genes, the COMT Val158Met genetic polymorphism and the DAT1/SLC6A3 variable number tandem repeat, which affect prefrontal and striatal dopamine, respectively. The results support the role of prefrontal cortex in biasing learning, and provide further evidence that individual differences in the balance between prefrontal and striatal dopamine may be particularly important in the tradeoff between stability and flexibility. In the second study (Chapter 3), we assess the neurobiological mechanisms of stability and flexibility in the context of exploration, utilizing fMRI to examine dynamic changes in functional brain networks associated with exploratory choices. We then relate those changes to changes in norepinephrine activity, as measured indirectly via pupil diameter. We find tentative support for the hypothesis that increased norepinephrine activity around exploration facilitates the reorganization of functional brain networks, potentially providing a substrate for flexible exploratory states. Together, this work provides further support for the framework that stability and flexibility entail both costs and benefits, and that optimizing the balance between the two involves interactions of learning and cognitive control systems under the influence of catecholamines

An interoceptive predictive coding model of conscious presence

We describe a theoretical model of the neurocognitive mechanisms underlying conscious presence and its disturbances. The model is based on interoceptive prediction error and is informed by predictive models of agency, general models of hierarchical predictive coding and dopaminergic signaling in cortex, the role of the anterior insular cortex (AIC) in interoception and emotion, and cognitive neuroscience evidence from studies of virtual reality and of psychiatric disorders of presence, specifically depersonalization/derealization disorder. The model associates presence with successful suppression by top-down predictions of informative interoceptive signals evoked by autonomic control signals and, indirectly, by visceral responses to afferent sensory signals. The model connects presence to agency by allowing that predicted interoceptive signals will depend on whether afferent sensory signals are determined, by a parallel predictive-coding mechanism, to be self-generated or externally caused. Anatomically, we identify the AIC as the likely locus of key neural comparator mechanisms. Our model integrates a broad range of previously disparate evidence, makes predictions for conjoint manipulations of agency and presence, offers a new view of emotion as interoceptive inference, and represents a step toward a mechanistic account of a fundamental phenomenological property of consciousness

The role of the anterior cingulate cortex in prediction error and signaling surprise

In the past two decades, reinforcement learning (RL) has become a popular framework for understanding brain function. A key component of RL models, prediction error, has been associated with neural signals throughout the brain, including subcortical nuclei, primary sensory cortices, and prefrontal cortex. Depending on the location in which activity is observed, the functional interpretation of prediction error may change: Prediction errors may reflect a discrepancy in the anticipated and actual value of reward, a signal indicating the salience or novelty of a stimulus, and many other interpretations. Anterior cingulate cortex (ACC) has long been recognized as a region involved in processing behavioral error, and recent computational models of the region have expanded this interpretation to include a more general role for the region in predicting likely events, broadly construed, and signaling deviations between expected and observed events. Ongoing modeling work investigating the interaction between ACC and additional regions involved in cognitive control suggests an even broader role for cingulate in computing a hierarchically structured surprise signal critical for learning models of the environment. The result is a predictive coding model of the frontal lobes, suggesting that predictive coding may be a unifying computational principle across the neocortex. This paper reviews the brain mechanisms responsible for surprise; focusing on the Anterior Cingulate Cortex (ACC), long-known to play a role in behavioral-error, with a recently-expanded role in predicting likely' events and signaling deviations between expected and observed events. It argues for ACC's role in in surprise and learning, based on recent modelling work. As such, the paper provides the neuroscience complement to the psychological and computational proposals of other papers in the volume