Retrospective model-based inference guides model-free credit assignment

An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions

Biases in the Explore-Exploit Tradeoff in Addictions: The Role of Avoidance of Uncertainty.

We focus on exploratory decisions across disorders of compulsivity, a potential dimensional construct for the classification of mental disorders. Behaviors associated with the pathological use of alcohol or food, in alcohol use disorders (AUD) or binge-eating disorder (BED), suggest a disturbance in explore-exploit decision-making, whereby strategic exploratory decisions in an attempt to improve long-term outcomes may diminish in favor of more repetitive or exploitatory choices. We compare exploration vs exploitation across disorders of natural (obesity with and without BED) and drug rewards (AUD). We separately acquired resting state functional MRI data using a novel multi-echo planar imaging sequence and independent components analysis from healthy individuals to assess the neural correlates underlying exploration. Participants with AUD showed reduced exploratory behavior across gain and loss environments, leading to lower-yielding exploitatory choices. Obese subjects with and without BED did not differ from healthy volunteers but when compared with each other or to AUD subjects, BED had enhanced exploratory behaviors particularly in the loss domain. All subject groups had decreased exploration or greater uncertainty avoidance to losses compared with rewards. More exploratory decisions in the context of reward were associated with frontal polar and ventral striatal connectivity. For losses, exploration was associated with frontal polar and precuneus connectivity. We further implicate the relevance and dimensionality of constructs of compulsivity across disorders of both natural and drug rewards.The study was funded by the Wellcome Trust Fellowship grant for VV (093705/Z/10/Z) and Cambridge NIHR Biomedical Research Centre. VV and NAH are Wellcome Trust (WT) intermediate Clinical Fellows. LSM is in receipt of an MRC studentship. The BCNI is supported by a WT and MRC grant. MF is funded by NIMH and NSF grants and is consultant for Hoffman LaRoche pharmaceuticals. The remaining authors declare no competing financial interests.This is the final version of the article. It first appeared from NPG via http://dx.doi.org/10.1038/npp.2015.20

Thermodynamics as a theory of decision-making with information processing costs

Perfectly rational decision-makers maximize expected utility, but crucially ignore the resource costs incurred when determining optimal actions. Here we propose an information-theoretic formalization of bounded rational decision-making where decision-makers trade off expected utility and information processing costs. Such bounded rational decision-makers can be thought of as thermodynamic machines that undergo physical state changes when they compute. Their behavior is governed by a free energy functional that trades off changes in internal energy-as a proxy for utility-and entropic changes representing computational costs induced by changing states. As a result, the bounded rational decision-making problem can be rephrased in terms of well-known concepts from statistical physics. In the limit when computational costs are ignored, the maximum expected utility principle is recovered. We discuss the relation to satisficing decision-making procedures as well as links to existing theoretical frameworks and human decision-making experiments that describe deviations from expected utility theory. Since most of the mathematical machinery can be borrowed from statistical physics, the main contribution is to axiomatically derive and interpret the thermodynamic free energy as a model of bounded rational decision-making.Comment: 26 pages, 5 figures, (under revision since February 2012

Tetracycline Inducible Gene Manipulation in Serotonergic Neurons

The serotonergic (5-HT) neuronal system has important and diverse physiological functions throughout development and adulthood. Its dysregulation during development or later in adulthood has been implicated in many neuropsychiatric disorders. Transgenic animal models designed to study the contribution of serotonergic susceptibility genes to a pathological phenotype should ideally allow to study candidate gene overexpression or gene knockout selectively in serotonergic neurons at any desired time during life. For this purpose, conditional expression systems such as the tet-system are preferable. Here, we generated a transactivator (tTA) mouse line (TPH2-tTA) that allows temporal and spatial control of tetracycline (Ptet) controlled transgene expression as well as gene deletion in 5-HT neurons. The tTA cDNA was inserted into a 196 kb PAC containing a genomic mouse Tph2 fragment (177 kb) by homologous recombination in E. coli. For functional analysis of Ptet-controlled transgene expression, TPH2-tTA mice were crossed to a Ptet-regulated lacZ reporter line (Ptet-nLacZ). In adult double-transgenic TPH2-tTA/Ptet-nLacZ mice, TPH2-tTA founder line L62-20 showed strong serotonergic β-galactosidase expression which could be completely suppressed with doxycycline (Dox). Furthermore, Ptet-regulated gene expression could be reversibly activated or inactivated when Dox was either withdrawn or added to the system. For functional analysis of Ptet-controlled, Cre-mediated gene deletion, TPH2-tTA mice (L62-20) were crossed to double transgenic Ptet-Cre/R26R reporter mice to generate TPH2-tTA/Ptet-Cre/R26R mice. Without Dox, 5-HT specific recombination started at E12.5. With permanent Dox administration, Ptet-controlled Cre-mediated recombination was absent. Dox withdrawal either postnatally or during adulthood induced efficient recombination in serotonergic neurons of all raphe nuclei, respectively. In the enteric nervous system, recombination could not be detected. We generated a transgenic mouse tTA line (TPH2-tTA) which allows both inducible and reversible transgene expression and inducible Cre-mediated gene deletion selectively in 5-HT neurons throughout life. This will allow precise delineation of serotonergic gene functions during development and adulthood

Expression proteomics of UPF1 knockdown in HeLa cells reveals autoregulation of hnRNP A2/B1 mediated by alternative splicing resulting in nonsense-mediated mRNA decay

BACKGROUND: In addition to acting as an RNA quality control pathway, nonsense-mediated mRNA decay (NMD) plays roles in regulating normal gene expression. In particular, the extent to which alternative splicing is coupled to NMD and the roles of NMD in regulating uORF containing transcripts have been a matter of debate. RESULTS: In order to achieve a greater understanding of NMD regulated gene expression we used 2D-DiGE proteomics technology to examine the changes in protein expression induced in HeLa cells by UPF1 knockdown. QPCR based validation of the corresponding mRNAs, in response to both UPF1 knockdown and cycloheximide treatment, identified 17 bona fide NMD targets. Most of these were associated with bioinformatically predicted NMD activating features, predominantly upstream open reading frames (uORFs). Strikingly, however, the majority of transcripts up-regulated by UPF1 knockdown were either insensitive to, or even down-regulated by, cycloheximide treatment. Furthermore, the mRNA abundance of several down-regulated proteins failed to change upon UPF1 knockdown, indicating that UPF1`s role in regulating mRNA and protein abundance is more complex than previously appreciated. Among the bona fide NMD targets, we identified a highly conserved AS-NMD event within the 3` UTR of the HNRNPA2B1 gene. Overexpression of GFP tagged hnRNP A2 resulted in a decrease in endogenous hnRNP A2 and B1 mRNA with a concurrent increase in the NMD sensitive isoforms. CONCLUSIONS: Despite the large number of changes in protein expression upon UPF1 knockdown, a relatively small fraction of them can be directly attributed to the action of NMD on the corresponding mRNA. From amongst these we have identified a conserved AS-NMD event within HNRNPA2B1 that appears to mediate autoregulation of HNRNPA2B1 expression levels

Gravitational Waves and Gamma-Rays from a Binary Neutron Star Merger: GW170817 and GRB 170817A

On 2017 August 17, the gravitational-wave event GW170817 was observed by the Advanced LIGO and Virgo detectors, and the gamma-ray burst (GRB) GRB 170817A was observed independently by the Fermi Gamma-ray Burst Monitor, and the Anti-Coincidence Shield for the Spectrometer for the International Gamma-Ray Astrophysics Laboratory. The probability of the near-simultaneous temporal and spatial observation of GRB 170817A and GW170817 occurring by chance is $5.0\times {10}^{-8}$. We therefore confirm binary neutron star mergers as a progenitor of short GRBs. The association of GW170817 and GRB 170817A provides new insight into fundamental physics and the origin of short GRBs. We use the observed time delay of $(+1.74\pm 0.05)\,{\rm{s}}$ between GRB 170817A and GW170817 to: (i) constrain the difference between the speed of gravity and the speed of light to be between $-3\times {10}^{-15}$ and $+7\times {10}^{-16}$ times the speed of light, (ii) place new bounds on the violation of Lorentz invariance, (iii) present a new test of the equivalence principle by constraining the Shapiro delay between gravitational and electromagnetic radiation. We also use the time delay to constrain the size and bulk Lorentz factor of the region emitting the gamma-rays. GRB 170817A is the closest short GRB with a known distance, but is between 2 and 6 orders of magnitude less energetic than other bursts with measured redshift. A new generation of gamma-ray detectors, and subthreshold searches in existing detectors, will be essential to detect similar short bursts at greater distances. Finally, we predict a joint detection rate for the Fermi Gamma-ray Burst Monitor and the Advanced LIGO and Virgo detectors of 0.1-1.4 per year during the 2018-2019 observing run and 0.3-1.7 per year at design sensitivity

Localization and broadband follow-up of the gravitational-wave transient GW150914

A gravitational-wave (GW) transient was identified in data recorded by the Advanced Laser Interferometer Gravitational-wave Observatory (LIGO) detectors on 2015 September 14. The event, initially designated G184098 and later given the name GW150914, is described in detail elsewhere. By prior arrangement, preliminary estimates of the time, significance, and sky location of the event were shared with 63 teams of observers covering radio, optical, near-infrared, X-ray, and gamma-ray wavelengths with ground- and space-based facilities. In this Letter we describe the low-latency analysis of the GW data and present the sky localization of the first observed compact binary merger. We summarize the follow-up observations reported by 25 teams via private Gamma-ray Coordinates Network circulars, giving an overview of the participating facilities, the GW sky localization coverage, the timeline, and depth of the observations. As this event turned out to be a binary black hole merger, there is little expectation of a detectable electromagnetic (EM) signature. Nevertheless, this first broadband campaign to search for a counterpart of an Advanced LIGO source represents a milestone and highlights the broad capabilities of the transient astronomy community and the observing strategies that have been developed to pursue neutron star binary merger events. Detailed investigations of the EM data and results of the EM follow-up campaign are being disseminated in papers by the individual teams

Search for gravitational waves from Scorpius X-1 in the second Advanced LIGO observing run with an improved hidden Markov model

We present results from a semicoherent search for continuous gravitational waves from the low-mass x-ray binary Scorpius X-1, using a hidden Markov model (HMM) to track spin wandering. This search improves on previous HMM-based searches of LIGO data by using an improved frequency domain matched filter, the J-statistic, and by analyzing data from Advanced LIGO's second observing run. In the frequency range searched, from 60 to 650 Hz, we find no evidence of gravitational radiation. At 194.6 Hz, the most sensitive search frequency, we report an upper limit on gravitational wave strain (at 95% confidence) of h095%=3.47×10-25 when marginalizing over source inclination angle. This is the most sensitive search for Scorpius X-1, to date, that is specifically designed to be robust in the presence of spin wandering. © 2019 American Physical Society