Task-specific effects of reward on task switching

Although cognitive control and reinforcement learning have been researched extensively over the last few decades, only recently have studies investigated their interrelationship. An important unanswered question concerns how the control system decides what task to execute and how vigorously to carry out the task once selected. Based on a recent theory of control formulated according to principles of hierarchical reinforcement learning, we asked whether rewards can affect top-down control over task performance at the level of task representation. Participants were rewarded for correctly performing only one of two tasks in a standard task-switching experiment. Reaction times and error rates were lower for the reinforced task compared to the non-reinforced task. Moreover, the switch cost in error rates for the non-reinforced task was significantly larger compared to the reinforced task, especially for trials in which the imperative stimulus afforded different responses for the two tasks, resulting in a "non-paradoxical" asymmetric switch cost. These findings suggest that reinforcement at the task level resulted in greater application of top-down control rather than in stronger stimulus-response pathways for the rewarded task

Electrophysiological indices of anterior cingulate cortex function reveal changing levels of cognitive effort and reward valuation that sustain task performance

Successful execution of goal-directed behaviors often requires the deployment of cognitive control, which is thought to require cognitive effort. Recent theories have proposed that anterior cingulate cortex (ACC) regulates control levels by weighing the reward-related benefits of control against its effort-related costs. However, given that the sensations of cognitive effort and reward valuation are available only to introspection, this hypothesis is difficult to investigate empirically. We have proposed that two electrophysiological indices of ACC function, frontal midline theta and the reward positivity (RewP), provide objective measures of these functions. To explore this issue, we recorded the electroencephalogram (EEG) from participants engaged in an extended, cognitively-demanding task. Participants performed a time estimation task for 2 h in which they received reward and error feedback according to their task performance. We observed that the amplitude of the RewP, a feedback-locked component of the event related brain potential associated with reward processing, decreased with time-on-task. Conversely, frontal midline theta power, which consists of 4-8 Hz EEG oscillations associated with cognitive effort, increased with time-on-task. We also explored how these phenomena changed over time by conducting within-participant multi-level modeling analyses. Our results suggest that extended execution of a cognitively-demanding task is characterized by an early phase in which high control levels foster rapid improvements in task performance, and a later phase in which high control levels were necessary to maintain stable task performance, perhaps counteracting waning reward valuation

Reward-based contextual learning supported by anterior cingulate cortex

The anterior cingulate cortex (ACC) is commonly associated with cognitive control and decision making, but its specific function is highly debated. To explore a recent theory that the ACC learns the reward values of task contexts (Holroyd & McClure in Psychological Review, 122, 54-83, 2015; Holroyd & Yeung in Trends in Cognitive Sciences, 16, 122-128, 2012), we recorded the event-related brain potentials (ERPs) from participants as they played a novel gambling task. The participants were first required to select from among three games in one "virtual casino," and subsequently they were required to select from among three different games in a different virtual casino; unbeknownst to them, the payoffs for the games were higher in one casino than in the other. Analysis of the reward positivity, an ERP component believed to reflect reward-related signals carried to the ACC by the midbrain dopamine system, revealed that the ACC is sensitive to differences in the reward values associated with both the casinos and the games inside the casinos, indicating that participants learned the values of the contexts in which rewards were delivered. These results highlight the importance of the ACC in learning the reward values of task contexts in order to guide action selection

Rightward-biased hemodynamic response of the parahippocampal system during virtual navigation

Phase reset of parahippocampal electrophysiological oscillations in the theta frequency range is said to contribute to item encoding and retrieval during spatial navigation. Although well-studied in non-human animals, this mechanism is poorly understood in humans. Previously we found that feedback stimuli presented in a virtual maze environment elicited a burst of theta power over right-posterior areas of the human scalp, and that the power and phase angle of these oscillations were greater following right turns compared to left turns in the maze. Here we investigated the source of this effect with functional magnetic resonance imaging. Consistent with our predictions, we found that 1) feedback encountered in the maze task activated right parahippocampal cortex (PHC), 2) right PHC was more activated by rewards following right turns compared to left turns in the maze, and 3) the rightward-biased activation was more pronounced in individuals who displayed good spatial abilities. These findings support our previous electrophysiological findings and highlight, in humans, a role for PHC theta oscillations in encoding salient information for the purpose of spatial navigation

Individual differences in personality associated with anterior cingulate cortex function: implication for understanding depression

We humans depend heavily on cognitive control to make decision and execute goal-directed behaviors, without which our behavior would be overpowered by automatic, stimulus-driven responses. In my dissertation, I focus on a brain region most implicated in this crucial process: the anterior cingulate cortex (ACC). The importance of this region is highlighted by lesion studies demonstrating diminished self-initiated behavior, or apathy, following ACC damage, the most severe form of which results in the near complete absence of speech production and willed actions in the presence of intact motor ability. Despite decades of research, however, its precise function is still highly debated, due particularly to ACC’s observed involvement in multiple aspects of cognition. In my dissertation I examine ACC function according to recent developments in reinforcement learning theory that posit a key role for ACC in motivating extended behavior. According to this theory, ACC is responsible for learning task values and motivating effortful control over extended behaviors based on those learned task values. The aim of my dissertation is two-fold: 1) to improve understanding of ACC function, and 2) to elucidate the contribution of ACC to depression, as revealed by individual differences in several personality traits related to motivation and reward sensitivity in a population of healthy college students. It was hypothesized that these different personality traits express, to greater or lesser degrees across individuals, ACC function, and that their abnormal expression (in particular, atypically low motivation and reward sensitivity) constitute hallmark characteristics of depression. First, this dissertation reveals that reward positivity (RewP), a key electrophysiological signature of reward processing that is believed to index the impact of reinforcement learning signals carried by the midbrain dopamine system on to ACC, is sensitive to individual differences in reward valuation, being larger for those high in reward sensitivity and smaller for those high in depression scores. Second, consistent with a previous suggestion that people high in depression or depression scores have difficulty using reward information to motivate behavior, I find these individuals to exhibit relatively poor prolonged task performance despite an apparently greater investment of cognitive control, and a reduced willingness to expend effort to obtain probable rewards, a behavior that was stable with time on task. In contrast, individuals characterized by high persistence, which is indicative of good ACC function, exhibited high self-reported task engagement and increasing effortful behaviors with time on task, particularly for trials in which reward receipt was unlikely, suggesting increased motivational control. In sum, this dissertation emphasizes the importance of understanding the basic function of ACC as assessed by individual differences in personality, which is then used to understand the impact of its dysfunction in relation to mental illnesses.Graduat

Cost-benefit analysis in physical effort expenditure: An electrophysiological registered report

Navigating through everyday life requires us to make series of choices involving effort: Is it worth the effort for what I want to accomplish? Effort-based decision making depends on evaluating the value of effort-related costs against potential rewards, and only when the rewards outweigh their effort costs do effortful behaviors tend to get carried out. Despite a surge of research on this topic, what effortful control and reward processes are involved in such decisions and whether electrophysiological measures of control and reward processes could better elucidate these processes remain unclear. Here, we will parametrically manipulate effort and reward levels to investigate their effects on different decision processes (i.e., choice evaluation, choice itself, subsequent physical effort production, reward feedback valuation). To assess these decision processes, we will examine two electrophysiological indices: frontal midline theta power and reward positivity amplitude; further, we will investigate whether these indices track cost-benefit integration, which will be reflected in subjective values derived from behavioral modelling of choices. Our goal is to understand how effort and reward affect different aspects of decision making and effort production, and how the electrophysiological and behavioral measures of these processes relate to each other

Electrophysiological measures of conflict and reward processing are associated with decisions to engage in physical effort

Anterior cingulate cortex (ACC), a key brain region involved in cognitive control and decision making, is suggested to mediate effort- and value-based decision making, but the specific role of ACC in this process remains debated. Here we used frontal midline theta (FMT) and the reward positivity (RewP) to examine ACC function in a value-based decision making task requiring physical effort. We investigated whether (1) FMT power is sensitive to the difficulty of the decision or to selecting effortful actions, and (2) RewP is sensitive to the subjective value of reward outcomes as a function of effort investment. On each trial, participants chose to execute a low-effort or a high-effort behavior (that required squeezing a hand-dynamometer) to obtain smaller or larger rewards, respectively, while their brainwaves were recorded. We replicated prior findings that tonic FMT increased over the course of the hour-long task, which suggests increased application of control in the face of growing fatigue. RewP amplitude also increased following execution of high-effort compared to low-effort behavior, consistent with increased valuation of reward outcomes by ACC. Although neither phasic nor tonic FMT were associated with decision difficulty or effort selection per se, an exploratory analysis revealed that the interaction of phasic FMT and expected value of choice predicted effort choice. This interaction suggests that phasic FMT increases specifically under situations of decision difficulty when participants ultimately select a high-effort choice. These results point to a unique role for ACC in motivating and persisting at effortful behavior when decision conflict is high