Impairments in reinforcement learning do not explain enhanced habit formation in cocaine use disorder.

RATIONALE: Drug addiction has been suggested to develop through drug-induced changes in learning and memory processes. Whilst the initiation of drug use is typically goal-directed and hedonically motivated, over time, drug-taking may develop into a stimulus-driven habit, characterised by persistent use of the drug irrespective of the consequences. Converging lines of evidence suggest that stimulant drugs facilitate the transition of goal-directed into habitual drug-taking, but their contribution to goal-directed learning is less clear. Computational modelling may provide an elegant means for elucidating changes during instrumental learning that may explain enhanced habit formation. OBJECTIVES: We used formal reinforcement learning algorithms to deconstruct the process of appetitive instrumental learning and to explore potential associations between goal-directed and habitual actions in patients with cocaine use disorder (CUD). METHODS: We re-analysed appetitive instrumental learning data in 55 healthy control volunteers and 70 CUD patients by applying a reinforcement learning model within a hierarchical Bayesian framework. We used a regression model to determine the influence of learning parameters and variations in brain structure on subsequent habit formation. RESULTS: Poor instrumental learning performance in CUD patients was largely determined by difficulties with learning from feedback, as reflected by a significantly reduced learning rate. Subsequent formation of habitual response patterns was partly explained by group status and individual variation in reinforcement sensitivity. White matter integrity within goal-directed networks was only associated with performance parameters in controls but not in CUD patients. CONCLUSIONS: Our data indicate that impairments in reinforcement learning are insufficient to account for enhanced habitual responding in CUD.This research was funded by the Medical Research Council (MR/J012084/1) and the NIHR Cambridge Biomedical Research Centre and was conducted at the NIHR Cambridge Clinical Research Facility. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. This research was also supported in part by a Medical Research Council (MRC) Clinical Research Infrastructure award (MR/M009041/1). R.N.C. consults for Campden Instruments and receives royalties from Cambridge Enterprise, Routledge, and Cambridge University Press. RNC’s research is supported by the UK Medical Research Council (MC_PC_17213). T.W.R. discloses consultancy with Cambridge Cognition, Lundbeck, Mundipharma and Unilever; he receives royalties for CANTAB from Cambridge Cognition and editorial honoraria from Springer Verlag and Elsevier. T.V.L., G.S. P.S.J., A.A.M. and K.D.E. declare to have no potential conflict of interest

Evidence for a Long-Lasting Compulsive Alcohol Seeking Phenotype in Rats

Excessive drinking to intoxication is the major behavioral characteristic of those addicted to alcohol but it is not the only one. Indeed, individuals addicted to alcohol also crave alcoholic beverages and spend time and put much effort into compulsively seeking alcohol, before eventually drinking large amounts. Unlike this excessive drinking, for which treatments exist, compulsive alcohol seeking is therefore another key feature of the persistence of alcohol addiction since it leads to relapse and for which there are few effective treatments. Here we provide novel evidence for the existence in rats of an individual vulnerability to switch from controlled to compulsive, punishment-resistant alcohol seeking. Alcohol-preferring rats given access to alcohol under an intermittent 2-bottle choice procedure to establish their alcohol-preferring phenotype were subsequently trained instrumentally to seek and take alcohol on a chained schedule of reinforcement. When stable seeking-taking performance had been established, completion of cycles of seeking responses resulted unpredictably either in punishment (0.45 mA foot-shock) or the opportunity to make a taking response for access to alcohol. Compulsive alcohol seeking, maintained in the face of the risk of punishment, emerged in only a subset of rats with a predisposition to prefer and drink alcohol, and was maintained for almost a year. We show further that a selective and potent μ-opioid receptor antagonist (GSK1521498) reduced both alcohol seeking and alcohol intake in compulsive and non-compulsive rats, indicating its therapeutic potential to promote abstinence and prevent relapse in individuals addicted to alcohol.The present study was funded by a Medical Research Council Programme Grant (no. G1002231) and by GlaxoSmithKline (GSK), which has a commercial interest in GSK1521498. The production of the P rats was funded by the R24 Alcohol Research Resource Award grant (R24 AA015512) from NIAAA

Delay aversion but preference for large and rare rewards in two choice tasks: implications for the measurement of self-control parameters

BACKGROUND: Impulsivity is defined as intolerance/aversion to waiting for reward. In intolerance-to-delay (ID) protocols, animals must choose between small/soon (SS) versus large/late (LL) rewards. In the probabilistic discount (PD) protocols, animals are faced with choice between small/sure (SS) versus large/luck-linked (LLL) rewards. It has been suggested that PD protocols also measure impulsivity, however, a clear dissociation has been reported between delay and probability discounting. RESULTS: Wistar adolescent rats (30- to 46-day-old) were tested using either protocol in drug-free state. In the ID protocol, animals showed a marked shift from LL to SS reward when delay increased, and this despite adverse consequences on the total amount of food obtained. In the PD protocol, animals developed a stable preference for LLL reward, and maintained it even when SS and LLL options were predicted and demonstrated to become indifferent. We demonstrate a clear dissociation between these two protocols. In the ID task, the aversion to delay was anti-economical and reflected impulsivity. In the PD task, preference for large reward was maintained despite its uncertain delivery, suggesting a strong attraction for unitary rewards of great magnitude. CONCLUSION: Uncertain delivery generated no aversion, when compared to delays producing an equivalent level of large-reward rarefaction. The PD task is suggested not to reflect impulsive behavior, and to generate patterns of choice that rather resemble the features of gambling. In summary, present data do indicate the need to interpret choice behavior in ID and PD protocols differently

Perseveration and Shifting in Obsessive-Compulsive Disorder as a Function of Uncertainty, Punishment, and Serotonergic Medication

Background: The nature of cognitive flexibility deficits in obsessive-compulsive disorder (OCD), which historically have been tested with probabilistic reversal learning tasks, remains elusive. Here, a novel deterministic reversal task and inclusion of unmedicated patients in the study sample illuminated the role of fixed versus uncertain rules/contingencies and of serotonergic medication. Additionally, our understanding of probabilistic reversal was enhanced through theoretical computational modeling of cognitive flexibility in OCD. Methods: We recruited 49 patients with OCD, 21 of whom were unmedicated, and 43 healthy control participants matched for age, IQ, and gender. Participants were tested on 2 tasks: a novel visuomotor deterministic reversal learning task with 3 reversals (feedback rewarding/punishing/neutral) measuring accuracy/perseveration and a 2-choice visual probabilistic reversal learning task with uncertain feedback and a single reversal measuring win-stay and lose-shift. Bayesian computational modeling provided measures of learning rate, reinforcement sensitivity, and stimulus stickiness. Results: Unmedicated patients with OCD were impaired on the deterministic reversal task under punishment only at the first and third reversals compared with both control participants and medicated patients with OCD, who had no deficit. Perseverative errors were correlated with OCD severity. On the probabilistic reversal task, unmedicated patients were only impaired at reversal, whereas medicated patients were impaired at both the learning and reversal stages. Computational modeling showed that the overall change was reduced feedback sensitivity in both OCD groups. Conclusions: Both perseveration and increased shifting can be observed in OCD, depending on test conditions including the predictability of reinforcement. Perseveration was related to clinical severity and remediated by serotonergic medication

Serotonin Differentially Regulates Short- and Long-Term Prediction of Rewards in the Ventral and Dorsal Striatum

BACKGROUND: The ability to select an action by considering both delays and amount of reward outcome is critical for maximizing long-term benefits. Although previous animal experiments on impulsivity have suggested a role of serotonin in behaviors requiring prediction of delayed rewards, the underlying neural mechanism is unclear. METHODOLOGY/PRINCIPAL FINDINGS: To elucidate the role of serotonin in the evaluation of delayed rewards, we performed a functional brain imaging experiment in which subjects chose small-immediate or large-delayed liquid rewards under dietary regulation of tryptophan, a precursor of serotonin. A model-based analysis revealed that the activity of the ventral part of the striatum was correlated with reward prediction at shorter time scales, and this correlated activity was stronger at low serotonin levels. By contrast, the activity of the dorsal part of the striatum was correlated with reward prediction at longer time scales, and this correlated activity was stronger at high serotonin levels. CONCLUSIONS/SIGNIFICANCE: Our results suggest that serotonin controls the time scale of reward prediction by differentially regulating activities within the striatum

Bidirectional Modulation of Alcohol-Associated Memory Reconsolidation through Manipulation of Adrenergic Signaling.

Alcohol addiction is a problem of great societal concern, for which there is scope to improve current treatments. One potential new treatment for alcohol addiction is based on disrupting the reconsolidation of the maladaptive Pavlovian memories that can precipitate relapse to drug-seeking behavior. In alcohol self-administering rats, we investigated the effects of bidirectionally modulating adrenergic signaling on the strength of a Pavlovian cue-alcohol memory, using a behavioral procedure that isolates the specific contribution of one maladaptive Pavlovian memory to relapse, the acquisition of a new alcohol-seeking response for an alcohol-associated conditioned reinforcer. The β-adrenergic receptor antagonist propranolol, administered in conjunction with memory reactivation, persistently disrupted the memory that underlies the capacity of a previously alcohol-associated cue to act as a conditioned reinforcer. By contrast, enhancement of adrenergic signaling by administration of the adrenergic prodrug dipivefrin at reactivation increased the strength of the cue-alcohol memory and potentiated alcohol seeking. These data demonstrate the importance of adrenergic signaling in alcohol-associated memory reconsolidation, and suggest a pharmacological target for treatments aiming to prevent relapse through the disruption of maladaptive memories.This work was supported by a UK Medical Research Council Programme Grant (G1002231) to BJE and ALM and was conducted in the Behavioural and Clinical Neuroscience Institute (BCNI), an initiative jointly funded by the MRC and the Wellcome Trust. MJWS was supported by an MRC Doctoral Training Grant and the James Baird Fund at the Medical School of the University of Cambridge. ALM was partly supported by a BCNI lectureship and the Ferreras-Willetts Fellowship from Downing College, Cambridge.This is the author accepted manuscript. The final version is available from Nature Publishing Group via http://dx.doi.org/10.1038/npp.2015.24

Stress-induced anhedonia is associated with hypertrophy of medium spiny neurons of the nucleus accumbens

There is accumulating evidence that the nucleus accumbens (NAc) has an important role in the pathophysiology of depression. As the NAc is a key component in the neural circuitry of reward, it has been hypothesized that anhedonia, a core symptom of depression, might be related to dysfunction of this brain region. Neuronal morphology and expression of plasticity-related molecules were examined in the NAc of rats displaying anhedonic behavior (measured in the sucrose-consumption test) in response to chronic mild stress. To demonstrate the relevance of our measurements to depression, we tested whether the observed changes were sensitive to reversal with antidepressants (imipramine and fluoxetine). Data show that animals displaying anhedonic behavior display an hypertrophy of medium spiny neurons in the NAc and, in parallel, have increased expression of the genes encoding for brain-derived neurotrophic factor, neural cell adhesion molecule and synaptic protein synapsin 1. Importantly, the reversal of stress-induced anhedonia by antidepressants is linked to a restoration of gene-expression patterns and dendritic morphology in the NAc. Using an animal model of depression, we show that stress induces anhedonic behavior that is associated with specific changes in the neuronal morphology and in the gene-expression profile of the NAc that are effectively reversed after treatment with antidepressants.The present work was funded by the Portuguese Foundation for Technology (FCT), project PTDC/SAU-NEU/105180/2008. FM and PL are recipients of postdoctoral fellowships and MM is recipient of a doctoral fellowship, all from FCT, Portugal

Context-dependent preferences in starlings: linking ecology, foraging and choice

Foraging animals typically encounter opportunities that they either pursue or skip, but occasionally meet several alternatives simultaneously. Behavioural ecologists predict preferences using absolute properties of each option, while decision theorists focus on relative evaluations at the time of choice. We use European starlings (Sturnus vulgaris) to integrate ecological reasoning with decision models, linking and testing hypotheses for value acquisition and choice mechanism. We hypothesise that options' values depend jointly on absolute attributes, learning context, and subject's state. In simultaneous choices, preference could result either from comparing subjective values using deliberation time, or from processing each alternative independently, without relative comparisons. The combination of the value acquisition hypothesis and independent processing at choice time has been called the Sequential Choice Model. We test this model with options equated in absolute properties to exclude the possibility of preference being built at the time of choice. Starlings learned to obtain food by responding to four stimuli in two contexts. In context [AB], they encountered options A5 or B10 in random alternation; in context [CD], they met C10 or D20. Delay to food is denoted, in seconds, by the suffixes. Observed latency to respond (Li) to each option alone (our measure of value) ranked thus: LA≈LC<LB<<LD, consistently with value being sensitive to both delay and learning context. We then introduced simultaneous presentations of A5 vs. C10 and B10 vs. C10, using latencies in no-choice tests to predict sign and strength of preference in pairings. Starlings preferred A5 over C10 and C10 over B10. There was no detectable evaluation time, and preference magnitude was predictable from latency differentials. This implies that value reflects learning rather than choice context, that preferences are not constructed by relative judgements at the time of choice, and that mechanisms adapted for sequential decisions are effective to predict choice behaviour.This work was supported by Biotechnology and Biological Sciences Research Council (BBSRC) Grant BB/G007144/1 to AK www.bbsrc.ac.uk; TM was supported by a Doctoral Grant from the Portuguese Foundation for Science and Technology (FCT) www.fct.pt/index.phtml.en. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript