Neurocognitive basis of model-based decision making and its metacontrol in childhood

Human behavior is supported by both goal-directed (model-based) and habitual (model-free) decision-making, each differing in its flexibility, accuracy, and computational cost. The arbitration between habitual and goal-directed systems is thought to be regulated by a process known as metacontrol. However, how these systems emerge and develop remains poorly understood. Recently, we found that while children between 5 and 11 years displayed robust signatures of model-based decision-making, which increased during this developmental period, there were substantial individual differences in the display of metacontrol. Here, we inspect the neurocognitive basis of model-based decision-making and metacontrol in childhood and focus this investigation on executive functions, fluid reasoning, and brain structure. A total of 69 participants between the ages of 6-13 completed a two-step decision-making task and an extensive behavioral test battery. A subset of 44 participants also completed a structural magnetic resonance imaging scan. We find that individual differences in metacontrol are specifically associated with performance on an inhibition task and individual differences in thickness of dorsolateral prefrontal, temporal, and superior-parietal cortices. These brain regions likely reflect the involvement of cognitive processes crucial to metacontrol, such as cognitive control and contextual processing

Suboptimal human inference can invert the bias-variance trade-off for decisions with asymmetric evidence

Solutions to challenging inference problems are often subject to a fundamental trade-off between: 1) bias (being systematically wrong) that is minimized with complex inference strategies, and 2) variance (being oversensitive to uncertain observations) that is minimized with simple inference strategies. However, this trade-off is based on the assumption that the strategies being considered are optimal for their given complexity and thus has unclear relevance to forms of inference based on suboptimal strategies. We examined inference problems applied to rare, asymmetrically available evidence, which a large population of human subjects solved using a diverse set of strategies that varied in form and complexity. In general, subjects using more complex strategies tended to have lower bias and variance, but with a dependence on the form of strategy that reflected an inversion of the classic bias-variance trade-off: subjects who used more complex, but imperfect, Bayesian-like strategies tended to have lower variance but higher bias because of incorrect tuning to latent task features, whereas subjects who used simpler heuristic strategies tended to have higher variance because they operated more directly on the observed samples but lower, near-normative bias. Our results help define new principles that govern individual differences in behavior that depends on rare-event inference and, more generally, about the information-processing trade-offs that can be sensitive to not just the complexity, but also the optimality, of the inference process. &nbsp;</p

Situation-appropriate Investment of Cognitive Resources

The human brain is equipped with the ability to plan ahead, i.e. to mentally simulate the expected consequences of candidate actions to select the one with the most desirable expected long-term outcome. Insufficient planning can lead to maladaptive behaviour and may even be a contributory cause of important societal problems such as the depletion of natural resources or man-made climate change. Understanding the cognitive and neural mechanisms of forward planning and its regulation are therefore of great importance and could ultimately give us clues on how to better align our behaviour with long-term goals. Apart from its potential beneficial effects, planning is time-consuming and therefore associated with opportunity costs. It is assumed that the brain regulates the investment into planning based on a cost-benefit analysis, so that planning only takes place when the perceived benefits outweigh the costs. But how can the brain know in advance how beneficial or costly planning will be? One potential solution is that people learn from experience how valuable planning would be in a given situation. It is however largely unknown how the brain implements such learning, especially in environments with large state spaces. This dissertation tested the hypothesis that humans construct and use so-called control contexts to efficiently adjust the degree of planning to the demands of the current situation. Control contexts can be seen as abstract state representations, that conveniently cluster together situations with a similar demand for planning. Inferring context thus allows to prospectively adjust the control system to the learned demands of the global context. To test the control context hypothesis, two complex sequential decision making tasks were developed. Each of the two tasks had to fulfil two important criteria. First, the tasks should generate both situations in which planning had the potential to improve performance, as well as situations in which a simple strategy was sufficient. Second, the tasks had to feature rich state spaces requiring participants to compress their state representation for efficient regulation of planning. Participants’ planning was modelled using a parametrized dynamic programming solution to a Markov Decision Process, with parameters estimated via hierarchical Bayesian inference. The first study used a 15-step task in which participants had to make a series of decisions to achieve one or multiple goals. In this task, the computational costs of accurate forward planning increased exponentially with the length of the planning horizon. We therefore hypothesized that participants identify ‘distance from goal’ as the relevant contextual feature to guide their regulation of forward planning. As expected we found that participants predominantly relied on a simple heuristic when still far from the goal but progressively switched towards forward planning when the goal approached. In the second study participants had to sustainably invest a limited but replenishable energy resource, that was needed to accept offers, in order to accumulate a maximum number of points in the long run. The demand for planning varied across the different situations of the task, but due to the large number of possible situations (n = 448) it would be difficult for the participants to develop an expectation for each individual situation of how beneficial planning would be. We therefore hypothesized, that to regulate their forward planning participants used a compressed tasks representation, clustering together states with similar demands for planning. Consistent with this, reaction times (operationalising planning duration) increased with trial-by-trial value-conflict (operationalising approximate planning demand), but this increase was more pronounced in a context with generally high demand for planning. We further found that fMRI activity in the dorsal anterior cingulate cortex (dACC) increased with conflict, but this increase was more pronounced in a context with generally high demand for planning as well. Taken together, the results suggest that the dACC integrates representations of planning demand on different levels of abstraction to regulate prospective information sampling in an efficient and situation-appropriate way. This dissertation provides novel insights into the question how humans adapt their planning to the demands of the current situation. The results are consistent with the view that the regulation of planning is based on an integrated signal of the expected costs and benefits of planning. Furthermore, the results of this dissertation provide evidence that the regulation of planning in environments with real-world complexity critically relies on the brain’s powerful ability to construct and use abstract hierarchical representations

Learning Mechanisms to Predispose Risky Alcohol Drinking Behaviors During Young Adulthood

Alcohol use disorder (AUD) is a mental disorder that negatively affects personal health and burdens the global health system. Alcohol-attributed harms can also extend beyond the drinkers to other people in the society through increased road traffic accidents and more interpersonal violent behaviors. The effects of this disorder make it crucial to investigate predisposing mechanisms in order to identify at-risk individuals and further develop novel interventions. Although aberrant learning and dysfunctions in decision-making have been observed in individuals with AUD, it is not yet clear whether they predispose the development of risky drinking behaviors or result from repetitive alcohol use. To disentangle this, we studied the drinking behaviors of a community sample comprising participants who were 18–24, which is when the prevalence of alcohol use typically peaks. This thesis investigates whether two types of learning mechanisms—the balance between goal-directed and habitual control and the susceptibility to interference between Pavlovian cues and instrumental behaviors—are associated with the development of risky alcohol drinking behaviors. For Study 1, we assessed how goal-directed and habitual controls at 18 predispose alcohol use development over the course of 3 years. Goal-directed and habitual control, which are informed by model-based (MB) and model-free (MF) learning, were assessed with a two-step sequential decision-making task during functional magnetic resonance imaging. Three-year drinking trajectories were constructed based on the Alcohol Use Disorders Identification Test (AUDIT-C; assessed every 6 months) and a gram/drinking occasion measure (binge drinking score; assessed yearly). Latent growth curve models were applied to examine how the MB and MF controls were associated with the drinking trajectories. We found that MB control was negatively associated with the development of the binge drinking score trajectory. In contrast, MF reward prediction signals in the ventromedial prefrontal cortex and the ventral striatum (VS) were associated with a higher starting point and a steeper increase/less decrease in AUDIT-C, respectively. For Study 2, we investigated the cross-sectional association between the susceptibility to interference between Pavlovian cues and instrumental behaviors and risky (binge) drinking behaviors at age 18. During a Pavlovian-to-instrumental transfer (PIT) task, the participants were instructed to “collect good shells” and “leave bad shells” while the appetitive (monetary gain) or aversive (monetary loss) Pavlovian cues were presented in the background. The behavioral interference PIT effect was characterized by an increased error rate (ER) during incongruent trials (“collecting good shells” in the presence of an aversive Pavlovian cue or “leaving bad shells” during the presentation of an appetitive Pavlovian cue) in comparison to congruent ones. Overall, the individuals demonstrated a substantial behavioral PIT effect. Neural PIT correlates were found in the VS, dorsomedial, and lateral prefrontal cortices (dmPFC and lPFC, respectively). High-risk drinkers, in comparison to low-risk drinkers, exhibited a stronger behavioral PIT effect, decreased lPFC responses, and increased trend-level VS responses. Moreover, the effective connectivity from the VS to the lPFC during the incongruent trials was weaker for the high-risk drinkers, which indicates that the altered interplay between bottom-up and top-down neural responses may contribute to the poor interference control performance of this group. During Study 3, we further examined whether the susceptibility to Pavlovian cues during conflict trials was associated with the development of drinking behaviors over 6 years from ages 18 to 24. The drinking behaviors were again constructed based on the AUDIT-C and the binge drinking score. The PIT task was assessed at ages 18 and 21. Following Study 2, the increased ER in the incongruent condition compared with the congruent condition (along with the neural responses in the VS, lPFC, and dmPFC during the incongruent trials) were included in the latent growth curve models as predictors. A stronger VS response during a conflict at age 18 was associated with a higher starting point in both drinking trajectories but was negatively associated with the development of the binge drinking score trajectory. At age 21, high ER and enhanced neural responses in the dmPFC were associated with a risky AUDIT-C trajectory that started to emerge and develop until age 24. Through exploratory cluster analyses of the drinking trajectories, we identified two subgroups: the drinking behavior in the 'late riser' group escalated after age 21, whereas the drinking of 'early peakers' culminated at this age and then declined. The late risers displayed enhanced dmPFC responses and higher ER during conflict at age 21. Interestingly, this group also exhibited an increased ER from ages 18 to 21. Taken altogether, the unbalanced goal-directed to habitual control, informed by less MB and more MF control, appears to be a strong predisposing candidate mechanism that underlies the development of risky drinking behaviors during young adulthood. At age 18, the susceptibility to interference between Pavlovian cues and instrumental behaviors was associated with risky drinking behavior. The development of risky drinking behaviors over the 6 years was associated with the behavioral interference PIT effect at age 21 and its change from ages 18 to 21. Researchers could further explore the dynamics in PIT to predict risky drinking behaviors in the future

Dopaminergic signalling during goal-directed behaviour in a structured environment

During flexible behaviour, dopamine is thought to carry reward prediction errors (RPEs), which update values and hence modify future behaviour. However, in real-world situations where the statistical relationships in the environment can be learned, continuously adapting values is not always the most efficient way of adapting to change. Moreover, the environment is not always fully observable, and observations may provide only partial information about the current state of the world. In such partial observable structured environments, as is found in many real-world situations, it is not well understood what kind of information dopamine conveys or its causal role in shaping adaptive behaviour. To probe dopamine’s involvement in goal-directed behavioural flexibility in such environments, we measured and manipulated dopamine while mice solved a partially observable structured sequential decision task. In chapter 3 we show that mice solve such a task using state inference. In chapter 4, we recorded calcium activity from dopaminergic cell bodies in the ventral tegmental area and dopamine axonal projections in the ventral and dorsomedial striatum, as well as dopaminergic concentrations in the same striatal regions. Dopamine multiplexed a wide range of information. At different timescales dopamine signalling was consistent with carrying information about choice-specific RPEs, choice-independent reward history and lateralised movement signals. RPE computations were shaped by task structure and the inferred state of the task. Nonetheless, in chapter 5, we show that although dopamine responded strongly to rewards, optogenetically activating or inhibiting dopamine at the time of trial outcome had no effect on subsequent choice. However, in a different task context, we could show that the same stimulation had a substantial effect on animals’ choices. Therefore, we conclude that when inference guides choice, rewards have a dopamine-independent influence on policy through the information they carry about the world’s state

Decisions, decisions, decisions: the development and plasticity of reinforcement learning, social and temporal decision making in children

Human decision-making is the flexible way people respond to their environment, take actions, and plan toward long-term goals. It is commonly thought that humans rely on distinct decision-making systems, which are either more habitual and reflexive or deliberate and calculated. How we make decisions can provide insight into our social functioning, mental health and underlying psychopathology, and ability to consider the consequences of our actions. Notably, the ability to make appropriate, habitual or deliberate decisions depending on the context, here referred to as metacontrol, remains underexplored in developmental samples. This thesis aims to investigate the development of different decision-making mechanisms in middle childhood (ages 5-13) and to illuminate the potential neurocognitive mechanisms underlying value-based decision-making. Using a novel sequential decision-making task, the first experimental chapter presents robust markers of model-based decision-making in childhood (N = 85), which reflects the ability to plan through a sequential task structure, contrary to previous developmental studies. Using the same paradigm, in a new sample via both behavioral (N = 69) and MRI-based measures (N = 44), the second experimental chapter explores the neurocognitive mechanisms that may underlie model-based decision-making and its metacontrol in childhood and links individual differences in inhibition and cortical thickness to metacontrol. The third experimental chapter explores the potential plasticity of social and intertemporal decision-making in a longitudinal executive function training paradigm (N = 205) and initial relationships with executive functions. Finally, I critically discuss the results presented in this thesis and their implications and outline directions for future research in the neurocognitive underpinnings of decision-making during development

Planning complexity registers as a cost in metacontrol

Data and tasks for Kool, Gershman, &amp; Cushman (2018) in Journal of Cognitive Neuroscienc