Observational learning computations in neurons of the human anterior cingulate cortex

When learning from direct experience, neurons in the primate brain have been shown to encode a teaching signal used by algorithms in artificial intelligence: the reward prediction error (PE)—the difference between how rewarding an event is, and how rewarding it was expected to be. However, in humans and other species learning often takes place by observing other individuals. Here, we show that, when humans observe other players in a card game, neurons in their rostral anterior cingulate cortex (rACC) encode both the expected value of an observed choice, and the PE after the outcome was revealed. Notably, during the same task neurons recorded in the amygdala (AMY) and the rostromedial prefrontal cortex (rmPFC) do not exhibit this type of encoding. Our results suggest that humans learn by observing others, at least in part through the encoding of observational PEs in single neurons in the rACC

Counterfactual Choice and Learning in a Neural Network Centered on Human Lateral Frontopolar Cortex

Decision making and learning in a real-world context require organisms to track not only the choices they make and the outcomes that follow but also other untaken, or counterfactual, choices and their outcomes. Although the neural system responsible for tracking the value of choices actually taken is increasingly well understood, whether a neural system tracks counterfactual information is currently unclear. Using a three-alternative decision-making task, a Bayesian reinforcement-learning algorithm, and fMRI, we investigated the coding of counterfactual choices and prediction errors in the human brain. Rather than representing evidence favoring multiple counterfactual choices, lateral frontal polar cortex (lFPC), dorsomedial frontal cortex (DMFC), and posteromedial cortex (PMC) encode the reward-based evidence favoring the best counterfactual option at future decisions. In addition to encoding counterfactual reward expectations, the network carries a signal for learning about counterfactual options when feedback is available—a counterfactual prediction error. Unlike other brain regions that have been associated with the processing of counterfactual outcomes, counterfactual prediction errors within the identified network cannot be related to regret theory. Furthermore, individual variation in counterfactual choice-related activity and prediction error-related activity, respectively, predicts variation in the propensity to switch to profitable choices in the future and the ability to learn from hypothetical feedback. Taken together, these data provide both neural and behavioral evidence to support the existence of a previously unidentified neural system responsible for tracking both counterfactual choice options and their outcomes

Ventromedial Prefrontal and Anterior Cingulate Cortex Adopt Choice and Default Reference Frames during Sequential Multi-Alternative Choice

Although damage to the medial frontal cortex causes profound decision-making impairments, it has been difficult to pinpoint the relative contributions of key anatomical subdivisions. Here we use function magnetic resonance imaging to examine the contributions of human ventromedial prefrontal cortex (vmPFC) and dorsal anterior cingulate cortex (dACC) during sequential choices between multiple alternatives— two key features of choices made in ecological settings. By carefully constructing options whose current value at any given decision was dissociable from their longer term value, we were able to examine choices in current and long-term frames of reference. We present evidence showing that activity at choice and feedback in vmPFC and dACC was tied to the current choice and the best long-term option, respectively. vmPFC, mid-cingulate, and posterior cingulate cortex encoded the relative value between the chosen and next best option at each sequential decision, whereas dACC encoded the relative value of adapting choices from the option with the highest value in the longer term. Furthermore, at feedback we identify temporally dissociable effects that predict repetition of the current choice and adaptation away from the long-term best option in vmPFC and dACC, respectively. These functional dissociations at choice and feedback suggest that sequential choices are subject to competing cortical mechanisms

Observational learning computations in neurons of the human anterior cingulate cortex

When learning from direct experience, neurons in the primate brain have been shown to encode a teaching signal used by algorithms in artificial intelligence: the reward prediction error (PE)—the difference between how rewarding an event is, and how rewarding it was expected to be. However, in humans and other species learning often takes place by observing other individuals. Here, we show that, when humans observe other players in a card game, neurons in their rostral anterior cingulate cortex (rACC) encode both the expected value of an observed choice, and the PE after the outcome was revealed. Notably, during the same task neurons recorded in the amygdala (AMY) and the rostromedial prefrontal cortex (rmPFC) do not exhibit this type of encoding. Our results suggest that humans learn by observing others, at least in part through the encoding of observational PEs in single neurons in the rACC

The Behavioral and Neural Mechanisms Underlying the Tracking of Expertise

Evaluating the abilities of others is fundamental for successful economic and social behavior. We investigated the computational and neurobiological basis of ability tracking by designing an fMRI task that required participants to use and update estimates of both people and algorithms’ expertise through observation of their predictions. Behaviorally, we find a model-based algorithm characterized subject predictions better than several alternative models. Notably, when the agent’s prediction was concordant rather than discordant with the subject’s own likely prediction, participants credited people more than algorithms for correct predictions and penalized them less for incorrect predictions. Neurally, many components of the mentalizing network—medial prefrontal cortex, anterior cingulate gyrus, temporoparietal junction, and precuneus—represented or updated expertise beliefs about both people and algorithms. Moreover, activity in lateral orbitofrontal and medial prefrontal cortex reflected behavioral differences in learning about people and algorithms. These findings provide basic insights into the neural basis of social learning

Testing for Fictive Learning in Decision-Making Under Uncertainty

We conduct two experiments where subjects make a sequence of binary choices between risky and ambiguous binary lotteries. Risky lotteries are defined as lotteries where the relative frequencies of outcomes are known. Ambiguous lotteries are lotteries where the relative frequencies of outcomes are not known or may not exist. The trials in each experiment are divided into three phases: pre-treatment, treatment and post-treatment. The trials in the pre-treatment and post-treatment phases are the same. As such, the trials before and after the treatment phase are dependent, clustered matched-pairs, that we analyze with the alternating logistic regression (ALR) package in SAS. In both experiments, we reveal to each subject the outcomes of her actual and counterfactual choices in the treatment phase. The treatments differ in the complexity of the random process used to generate the relative frequencies of the payoffs of the ambiguous lotteries. In the first experiment, the probabilities can be inferred from the converging sample averages of the observed actual and counterfactual outcomes of the ambiguous lotteries. In the second experiment the sample averages do not converge. If we define fictive learning in an experiment as statistically significant changes in the responses of subjects before and after the treatment phase of an experiment, then we expect fictive learning in the first experiment, but no fictive learning in the second experiment. The surprising finding in this paper is the presence of fictive learning in the second experiment. We attribute this counterintuitive result to apophenia: “seeing meaningful patterns in meaningless or random data.” A refinement of this result is the inference from a subsequent Chi-squared test, that the effects of fictive learning in the first experiment are significantly different from the effects of fictive learning in the second experiment

Inferences on a multidimensional social hierarchy use a grid-like code

Generalizing experiences to guide decision-making in novel situations is a hallmark of flexible behavior. Cognitive maps of an environment or task can theoretically afford such flexibility, but direct evidence has proven elusive. In this study, we found that discretely sampled abstract relationships between entities in an unseen two-dimensional social hierarchy are reconstructed into a unitary two-dimensional cognitive map in the hippocampus and entorhinal cortex. We further show that humans use a grid-like code in entorhinal cortex and medial prefrontal cortex for inferred direct trajectories between entities in the reconstructed abstract space during discrete decisions. These grid-like representations in the entorhinal cortex are associated with decision value computations in the medial prefrontal cortex and temporoparietal junction. Collectively, these findings show that grid-like representations are used by the human brain to infer novel solutions, even in abstract and discrete problems, and suggest a general mechanism underpinning flexible decision-making and generalization

Protocol for building a cognitive map of structural knowledge in humans by integrating abstract relationships from separate experiences

Humans are adept at learning the latent structure of the relationship between abstract concepts and can build a cognitive map from limited experiences. However, examining internal representations of the cognitive map is challenging because they are unobservable and differ across individuals. Here, we introduce a behavioral training protocol designed for human participants to implicitly build a map of two-dimensional social hierarchies while making a series of binary choices and analytic tools for measuring the internal representation of this structural knowledge. For complete details on the use and execution of this protocol, please refer to Park et al. (2020a, 2020b)