A Reinforcement Learning Model of Precommitment in Decision Making

Addiction and many other disorders are linked to impulsivity, where a suboptimal choice is preferred when it is immediately available. One solution to impulsivity is precommitment: constraining one's future to avoid being offered a suboptimal choice. A form of impulsivity can be measured experimentally by offering a choice between a smaller reward delivered sooner and a larger reward delivered later. Impulsive subjects are more likely to select the smaller-sooner choice; however, when offered an option to precommit, even impulsive subjects can precommit to the larger-later choice. To precommit or not is a decision between two conditions: (A) the original choice (smaller-sooner vs. larger-later), and (B) a new condition with only larger-later available. It has been observed that precommitment appears as a consequence of the preference reversal inherent in non-exponential delay-discounting. Here we show that most models of hyperbolic discounting cannot precommit, but a distributed model of hyperbolic discounting does precommit. Using this model, we find (1) faster discounters may be more or less likely than slow discounters to precommit, depending on the precommitment delay, (2) for a constant smaller-sooner vs. larger-later preference, a higher ratio of larger reward to smaller reward increases the probability of precommitment, and (3) precommitment is highly sensitive to the shape of the discount curve. These predictions imply that manipulations that alter the discount curve, such as diet or context, may qualitatively affect precommitment

Reinventing College Physics for Biologists: Explicating an epistemological curriculum

The University of Maryland Physics Education Research Group (UMd-PERG) carried out a five-year research project to rethink, observe, and reform introductory algebra-based (college) physics. This class is one of the Maryland Physics Department's large service courses, serving primarily life-science majors. After consultation with biologists, we re-focused the class on helping the students learn to think scientifically -- to build coherence, think in terms of mechanism, and to follow the implications of assumptions. We designed the course to tap into students' productive conceptual and epistemological resources, based on a theoretical framework from research on learning. The reformed class retains its traditional structure in terms of time and instructional personnel, but we modified existing best-practices curricular materials, including Peer Instruction, Interactive Lecture Demonstrations, and Tutorials. We provided class-controlled spaces for student collaboration, which allowed us to observe and record students learning directly. We also scanned all written homework and examinations, and we administered pre-post conceptual and epistemological surveys. The reformed class enhanced the strong gains on pre-post conceptual tests produced by the best-practices materials while obtaining unprecedented pre-post gains on epistemological surveys instead of the traditional losses.Comment: 35 pages including a 15 page appendix of supplementary material

Low and High Gamma Oscillations in Rat Ventral Striatum have Distinct Relationships to Behavior, Reward, and Spiking Activity on a Learned Spatial Decision Task

Local field potential (LFP) oscillations in the brain reflect organization thought to be important for perception, attention, movement, and memory. In the basal ganglia, including dorsal striatum, dysfunctional LFP states are associated with Parkinson's disease, while in healthy subjects, dorsal striatal LFPs have been linked to decision-making processes. However, LFPs in ventral striatum have been less studied. We report that in rats running a spatial decision task, prominent gamma-50 (45–55 Hz) and gamma-80 (70–85 Hz) oscillations in ventral striatum had distinct relationships to behavior, task events, and spiking activity. Gamma-50 power increased sharply following reward delivery and before movement initiation, while in contrast, gamma-80 power ramped up gradually to reward locations. Gamma-50 power was low and contained little structure during early learning, but rapidly developed a stable pattern, while gamma-80 power was initially high before returning to a stable level within a similar timeframe. Putative fast-spiking interneurons (FSIs) showed phase, firing rate, and coherence relationships with gamma-50 and gamma-80, indicating that the observed LFP patterns are locally relevant. Furthermore, in a number of FSIs such relationships were specific to gamma-50 or gamma-80, suggesting that partially distinct FSI populations mediate the effects of gamma-50 and gamma-80

Expectancies in Decision Making, Reinforcement Learning, and Ventral Striatum

Decisions can arise in different ways, such as from a gut feeling, doing what worked last time, or planful deliberation. Different decision-making systems are dissociable behaviorally, map onto distinct brain systems, and have different computational demands. For instance, “model-free” decision strategies use prediction errors to estimate scalar action values from previous experience, while “model-based” strategies leverage internal forward models to generate and evaluate potentially rich outcome expectancies. Animal learning studies indicate that expectancies may arise from different sources, including not only forward models but also Pavlovian associations, and the flexibility with which such representations impact behavior may depend on how they are generated. In the light of these considerations, we review the results of van der Meer and Redish (2009a), who found that ventral striatal neurons that respond to reward delivery can also be activated at other points, notably at a decision point where hippocampal forward representations were also observed. These data suggest the possibility that ventral striatal reward representations contribute to model-based expectancies used in deliberative decision making

Covert Expectation-of-Reward in Rat Ventral Striatum at Decision Points

Flexible decision-making strategies (such as planning) are a key component of adaptive behavior, yet their neural mechanisms have remained resistant to experimental analysis. Theories of planning require prediction and evaluation of potential future rewards, suggesting that reward signals may covertly appear at decision points. To test this idea, we recorded ensembles of ventral striatal neurons on a spatial decision task, in which hippocampal ensembles are known to represent future possibilities at decision points. We found representations of reward which were not only activated at actual reward delivery sites, but also at a high-cost choice point and before error correction. This expectation-of-reward signal at decision points was apparent at both the single cell and the ensemble level, and vanished with behavioral automation. We conclude that ventral striatal representations of reward are more dynamic than suggested by previous reports of reward- and cue-responsive cells, and may provide the necessary signal for evaluation of internally generated possibilities considered during flexible decision-making

Cognitive Control Errors in Nonhuman Primates Resembling Those in Schizophrenia Reflect Opposing Effects of NMDA Receptor Blockade on Causal Interactions Between Cells and Circuits in Prefrontal and Parietal Cortices

Background: The causal biology underlying schizophrenia is not well understood, but it is likely to involve a malfunction in how neurons adjust synaptic connections in response to patterns of activity in networks. We examined statistical dependencies between neural signals at the cell, local circuit, and distributed network levels in prefrontal and parietal cortices of monkeys performing a variant of the AX continuous performance task paradigm. We then quantified changes in the pattern of neural interactions across levels of scale following NMDA receptor (NMDAR) blockade and related these changes to a pattern of cognitive control errors closely matching the performance of patients with schizophrenia. Methods: We recorded the spiking activity of 1762 neurons along with local field potentials at multiple electrode sites in prefrontal and parietal cortices concurrently, and we generated binary time series indicating the presence or absence of spikes in single neurons or local field potential power above or below a threshold. We then applied causal discovery analysis to the time series to detect statistical dependencies between the signals (causal interactions) and compared the pattern of these interactions before and after NMDAR blockade. Results: Global blockade of NMDAR produced distinctive and frequently opposite changes in neural interactions at the cell, local circuit, and network levels in prefrontal and parietal cortices. Cognitive control errors were associated with decreased interactions at the cell level and with opposite changes at the network level in prefrontal and parietal cortices. Conclusions: NMDAR synaptic deficits change causal interactions between neural signals at different levels of scale that correlate with schizophrenia-like deficits in cognitive control

Integrating Early Results on Ventral Striatal Gamma Oscillations in the Rat

A vast literature implicates the ventral striatum in the processing of reward-related information and in mediating the impact of such information on behavior. It is characterized by heterogeneity at the local circuit, connectivity, and functional levels. A tool for dissecting this complex structure that has received relatively little attention until recently is the analysis of ventral striatal local field potential oscillations, which are more prominent in the gamma band compared to the dorsal striatum. Here we review recent results on gamma oscillations recorded from freely moving rats. Ventral striatal gamma separates into distinct frequency bands (gamma-50 and gamma-80) with distinct behavioral correlates, relationships to different inputs, and separate populations of phase-locked putative fast-spiking interneurons. Fast switching between gamma-50 and gamma-80 occurs spontaneously but is influenced by reward delivery as well as the application of dopaminergic drugs. These results provide novel insights into ventral striatal processing and highlight the importance of considering fast-timescale dynamics of ventral striatal activity