Neural correlates of weighted reward prediction error during reinforcement learning classify response to cognitive behavioral therapy in depression

While cognitive behavioral therapy (CBT) is an effective treatment for major depressive disorder, only up to 45% of depressed patients will respond to it. At present, there is no clinically viable neuroimaging predictor of CBT response. Notably, the lack of a mechanistic understanding of treatment response has hindered identification of predictive biomarkers. To obtain mechanistically meaningful fMRI predictors of CBT response, we capitalize on pretreatment neural activity encoding a weighted reward prediction error (RPE), which is implicated in the acquisition and processing of feedback information during probabilistic learning. Using a conventional mass-univariate fMRI analysis, we demonstrate that, at the group level, responders exhibit greater pretreatment neural activity encoding a weighted RPE in the right striatum and right amygdala. Crucially, using multivariate methods, we show that this activity offers significant out-of-sample classification of treatment response. Our findings support the feasibility and validity of neurocomputational approaches to treatment prediction in psychiatry

Brain network mechanisms in learning behavior

The study of learning has been a central focus of psychology and neuroscience since their inception. Cognitive neuroscience’s traditional approach to understanding learn-ing has been to decompose it into discrete cognitive processes with separable and localized underlying neural systems. While this focus on modular cognitive functions for individual brain areas has led to considerable progress, there is increasing evidence that much of learn-ing behavior relies on overlapping cognitive and neural systems, which may be harder to disentangle than previously envisioned. This is not surprising, as the processes underlying learning must involve widespread integration of information from sensory, affective, and motor sources. The standard tools of cognitive neuroscience limit our ability to describe processes that rely on widespread coordination of brain activity. To understand learning, it will be necessary to characterize dynamic co-activation at the circuit level. In this dissertation, I present three studies that seek to describe the roles of distrib-uted brain networks in learning. I begin by giving an overview of our current understand-ing of multiple forms of learning, describing the neural and computational mechanisms thought to underlie incremental feedback-based learning and flexible episodic memory. I will focus in particular on the difficulties in separating these processes at the cognitive level and in localizing them to individual regions at the neural level. I will then describe recent findings that have begun to characterize the brain’s large-scale network structure, emphasiz-ing the potential roles that distributed networks could play in understanding learning and cognition more generally. I will end the introduction by reviewing current attempts to char-acterize the dynamics of large-scale brain networks, which will be essential for providing a mechanistic link to learning behavior. Chapter 2 is a study demonstrating that intrinsic connectivity between the hippo-campus and the ventromedial prefrontal cortex, as well as between these regions and dis-tributed brain networks, is related to individual differences in the transfer of learning on a sensory preconditioning task. The hippocampus and ventromedial prefrontal cortex have both been shown to be involved in this type of learning, and this study represents an early attempt to link connectivity between individual regions and broader networks to learning processes. Chapter 3 is a study that takes advantage of recent developments in mathematical modeling of temporal networks to demonstrate a relationship between large-scale network dynamics and reinforcement learning within individuals. This study shows that the flexibil-ity of network connectivity in the striatum is related to learning performance over time, as well as to individual differences in parameters estimated from computational models of re-inforcement learning. Notably, connectivity between the striatum and visual as well as or-bitofrontal regions increased over the course of the task, which is consistent with an inte-grative role for the region in learning value-based associations. Network flexibility in a dis-tinct set of regions is associated with episodic memory for object images presented during the learning task. Chapter 4 examines the role of dopamine, a neurotransmitter strongly linked to val-ue updating in reinforcement learning, in the dynamic network changes occurring during learning. Patients with Parkinson’s disease, who experience a loss of dopaminergic neu-rons in the substantia nigra, performed a reversal-learning task while undergoing functional magnetic resonance imaging. Patients were scanned on and off of a dopamine precursor medication (levodopa) in a within-subject design in order to examine the impact of dopa-mine on brain network dynamics during learning. The reversal provided an experimental manipulation of dynamic connectivity, and patients on medication showed greater modula-tion of striatal-cortical connectivity. Similar results were found in a number of regions re-ceiving midbrain projections including the prefrontal cortex and medial temporal lobe. This study indicates that dopamine inputs from the midbrain modulate large-scale network dy-namics during learning, providing a direct link between reinforcement learning theories of value updating and network neuroscience accounts of dynamic connectivity. Together, these results indicate that large-scale networks play a critical role in multi-ple forms of learning behavior. Each highlights the potential importance of understanding dynamic routing and integration of information across large-scale circuits for our concep-tion of learning and other cognitive processes. Understanding the when, where, and how of this information flow in the brain may provide an alternative or compliment to traditional theories of distinct learning systems. These studies also illustrate challenges in integrating this perspective with established theories in cognitive neuroscience. Chapter 5 will situate the studies in a broader discussion of how brain activity relates to cognition in general, while pointing out current roadblocks and potential ways forward for a cognitive network neuroscience of learning

Training an Actor-Critic Reinforcement Learning Controller for Arm Movement Using Human-Generated Rewards

Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Functional Electrical Stimulation (FES) employs neuroprostheses to apply electrical current to the nerves and muscles of individuals paralyzed by spinal cord injury (SCI) to restore voluntary movement. Neuroprosthesis controllers calculate stimulation patterns to produce desired actions. To date, no existing controller is able to efficiently adapt its control strategy to the wide range of possible physiological arm characteristics, reaching movements, and user preferences that vary over time. Reinforcement learning (RL) is a control strategy that can incorporate human reward signals as inputs to allow human users to shape controller behavior. In this study, ten neurologically intact human participants assigned subjective numerical rewards to train RL controllers, evaluating animations of goal-oriented reaching tasks performed using a planar musculoskeletal human arm simulation. The RL controller learning achieved using human trainers was compared with learning accomplished using human-like rewards generated by an algorithm; metrics included success at reaching the specified target; time required to reach the target; and target overshoot. Both sets of controllers learned efficiently and with minimal differences, significantly outperforming standard controllers. Reward positivity and consistency were found to be unrelated to learning success. These results suggest that human rewards can be used effectively to train RL-based FES controllers.NIH #TRN030167Veterans Administration Rehabilitation Research & Development predoctoral fellowshipArdiem Medical Arm Control Device grant #W81XWH072004

Novelty detection and learning drives

This document presents Deliverable 5.1 of the IM-CLeVeR (Intrinsically Motivated Cumulative Learning Versatile Robots) EU FP7 project. It represents one of two deliverables from Workpackage 5 (Novelty Detection and Drives for Autonomous Learning)

Goals and information processing in human decisions

We do not make decisions in the void. Every day, we act in awareness of our context, adjusting our objectives according to the situations we find. Operating effectively under multiple goals is fundamental for appropriate learning and decision-making, and deficiencies in this capacity can be at the core of mental disorders such as anxiety, depression, or post-traumatic stress disorder. In this thesis, I present studies I conducted to investigate how goals impact different stages of the decision process, from simple perceptual choices to subjective value preferences. Previous studies have described how animals assess alternatives and integrate evidence to make decisions. Most of the time, the focus of this work has been on simplified scenarios with single goals. In this thesis, my experiments tackle the issue of how people adjust information processing in tasks that demand more than one objective. Through various manipulations of the behavioural goals, such as decision framing, I show that (i) attention and evidence accumulation, (ii) brain representations, and (iii) decision confidence were all affected by context changes. Using behavioural testing, computational models, and neuroimaging I show that goals have a crucial role in evidence integration and the allocation of visual attention. My findings indicate that brain patterns adapt to enhance goal-relevant information during learning and the valuation of alternatives. Finally, I report the presence of goal-dependent asymmetries in the generation of decision confidence, overweighting the evidence of the most-relevant option to fulfil the goal. In conclusion, I show how the entire process is highly flexible and serves the behavioural demands. These findings support the reinterpretation of some perspectives, such as reported biases and irrationalities in decisions, as attributes of adaptive processing towards goal fulfilment

Empirical experiments on intrinsic motivations and action acquisition: results, evaluation, and redefinition

This document presents Deliverable D3.2 of the EU-funded Integrated Project "IM-CLeVeR - Intrinsically Motivated Cumulative Learning Versatile Robots", contract n. FP7-ICT-IP-231722.The aims of the deliverable, as given in the original IM-CLEVER proposal were to identify new key empirical phenomena and processes, allowing the design of a second set of experiments. This report covers: (1) novelty detection and discovery of when/what/how of agency in experiments with humans ("joystick experiment") and Parkinson patients. (2) how object properties that stimulate intrinsically motivated interaction and facilitate the acquisition of adaptive knowledge and skills in monkeys and children ("board experiment")

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field

Task switching in the prefrontal cortex

The overall goal of this dissertation is to elucidate the cellular and circuit mechanisms underlying flexible behavior in the prefrontal cortex. We are often faced with situations in which the appropriate behavior in one context is inappropriate in others. If these situations are familiar, we can perform the appropriate behavior without relearning how the context relates to the behavior — an important hallmark of intelligence. Neuroimaging and lesion studies have shown that this dynamic, flexible process of remapping context to behavior (task switching) is dependent on prefrontal cortex, but the precise contributions and interactions of prefrontal subdivisions are still unknown. This dissertation investigates two prefrontal areas that are thought to be involved in distinct, but complementary executive roles in task switching — the dorsolateral prefrontal cortex (dlPFC) and the anterior cingulate cortex (ACC). Using electrophysiological recordings from macaque monkeys, I show that synchronous network oscillations in the dlPFC provide a mechanism to flexibly coordinate context representations (rules) between groups of neurons during task switching. Then, I show that, wheras the ACC neurons can represent rules at the cellular level, they do not play a significant role in switching between contexts — rather they seem to be more related to errors and motivational drive. Finally, I develop a set of web-enabled interactive visualization tools designed to provide a multi-dimensional integrated view of electrophysiological datasets. Taken together, these results contribute to our understanding of task switching by investigating new mechanisms for coordination of neurons in prefrontal cortex, clarifying the roles of prefrontal subdivisions during task switching, and providing visualization tools that enhance exploration and understanding of large, complex and multi-scale electrophysiological data

Memory-based preferential choice in large option spaces

Whether adding songs to a playlist or groceries to a shopping basket, everyday decisions often require us to choose between an innumerable set of options. Laboratory studies of preferential choice have made considerable progress in describing how people navigate fixed sets of options. Yet, questions remain about how well this generalises to more complex, everyday choices. In this thesis, I ask how people navigate large option spaces, focusing particularly on how long-term memory supports decisions. In the first project, I explore how large option spaces are structured in the mind. A topic model trained on the purchasing patterns of consumers uncovered an intuitive set of themes that centred primarily around goals (e.g., tomatoes go well in a salad), suggesting that representations are geared to support action. In the second project, I explore how such representations are queried during memory-based decisions, where options must be retrieved from memory. Using a large dataset of over 100,000 online grocery shops, results revealed that consumers query multiple systems of associative memory when determining what choose next. Attending to certain knowledge sources, as estimated by a cognitive model, predicted important retrieval errors, such as the propensity to forget or add unwanted products. In the final project, I ask how preferences could be learned and represented in large option spaces, where most options are untried. A cognitive model of sequential decision making is proposed, which learns preferences over choice attributes, allowing for the generalisation of preferences to unseen options, by virtue of their similarity to previous choices. This model explains reduced exploration patterns behaviour observed in the supermarket and preferential choices in more controlled laboratory settings. Overall, this suggests that consumers depend on associative systems in long-term memory when navigating large spaces of options, enabling inferences about the conceptual properties and subjective value of novel options