Stochastic Variance-Reduced Policy Gradient

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective func- tion; II) approximations in the full gradient com- putation; and III) a non-stationary sampling pro- cess. The result is SVRPG, a stochastic variance- reduced policy gradient algorithm that leverages on importance weights to preserve the unbiased- ness of the gradient estimate. Under standard as- sumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs

Distinct α subunit variations of the hypothalamic GABAA receptor triplets (αβγ) are linked to hibernating state in hamsters

<p>Abstract</p> <p>Background</p> <p>The structural arrangement of the γ-aminobutyric acid type A receptor (GABA_AR) is known to be crucial for the maintenance of cerebral-dependent homeostatic mechanisms during the promotion of highly adaptive neurophysiological events of the permissive hibernating rodent, i.e the Syrian golden hamster. In this study, <it>in vitro </it>quantitative autoradiography and <it>in situ </it>hybridization were assessed in major hypothalamic nuclei. Reverse Transcription Reaction-Polymerase chain reaction (RT-PCR) tests were performed for specific GABA_AR receptor subunit gene primers synthases of non-hibernating (NHIB) and hibernating (HIB) hamsters. Attempts were made to identify the type of αβγ subunit combinations operating during the switching ON/OFF of neuronal activities in some hypothalamic nuclei of hibernators.</p> <p>Results</p> <p>Both autoradiography and molecular analysis supplied distinct expression patterns of all α subunits considered as shown by a strong (p < 0.01) prevalence of α₁ratio (over total α subunits considered in the present study) in the medial preoptic area (MPOA) and arcuate nucleus (Arc) of NHIBs with respect to HIBs. At the same time α₂subunit levels proved to be typical of periventricular nucleus (Pe) and Arc of HIB, while strong α₄expression levels were detected during awakening state in the key circadian hypothalamic station, i.e. the suprachiasmatic nucleus (Sch; 60%). Regarding the other two subunits (β and γ), elevated β₃and γ₃mRNAs levels mostly characterized MPOA of HIBs, while prevalently elevated expression concentrations of the same subunits were also typical of Sch, even though this time during the awakening state. In the case of Arc, notably elevated levels were obtained for β₃and γ₂during hibernating conditions.</p> <p>Conclusion</p> <p>We conclude that different αβγ subunits are operating as major elements either at the onset of torpor or during induction of the arousal state in the Syrian golden hamster. The identification of a brain regional distribution pattern of distinct GABA_AR subunit combinations may prove to be very useful for highlighting GABAergic mechanisms functioning at least during the different physiological states of hibernators and this may have interesting therapeutic bearings on neurological sleeping disorders.</p

HSP90 and pCREB alterations are linked to mancozeb-dependent behavioral and neurodegenerative effects in a marine teleost

The pesticide mancozeb (mz) is recognized as a potent inducer of oxidative stress due to its ability to catalyze the production of reactive oxygen species plus inhibiting mitochondrial respiration thus becoming an environmental risk for neurodegenerative diseases. Despite numerous toxicological studies on mz have been directed to mammals, attention on marine fish is still lacking. Thus, it was our intention to evaluate neurobehavioral activities of ornate wrasses (Thalassoma pavo) exposed to 0.2mg/l of mz after a preliminary screening test (0.07-0.3mg/l). Treated fish exhibited an evident (p1000%) while exploratory attitudes (total arm entries) diminished (-50%; p<0.05) versus controls during spontaneous exploration tests. Moreover, they showed evident enhancements (+111%) of immobility in the cylinder test. Contextually, strong (-88%; p<0.01) reductions of permanence in light zone of the Light/Dark apparatus along with diminished crossings (-65%) were also detected. Conversely, wrasses displayed evident enhancements (160%) of risk assessment consisting of fast entries in the dark side of this apparatus. From a molecular point of view, a notable activation (p<0.005) of the brain transcription factor pCREB occurred during mz-exposure. Similarly, in situ hybridization supplied increased HSP90 mRNAs in most brain areas such as the lateral part of the dorsal telencephalon (Dl; +68%) and valvula of the cerebellum (VCe; +35%) that also revealed evident argyrophilic signals. Overall, these first indications suggest a possible protective role of the early biomarkers pCREB and HSP90 against fish toxicit

Catestatin and GABAAR related feeding habits rely on dopamine, ghrelin plus leptin neuroreceptor expression variations

Catestatin (CST), an endogenously small sympathoinhibitory peptide is capable of interfering with the major cerebral neuroreceptor-blocking site, i.e. γ-aminobutyric acidA receptor (GABAAR) system especially in limbic brain areas that are involved with feeding behaviors. The GABAARergic-related effects seem to derive from its interaction with other molecular neuroreceptors such as dopaminergic, ghrelin and leptinergic. In this context, the present study aimed to investigate probable feeding responses (eating and drinking) induced by treatment with CST and the GABAAR antagonist bicucullin (BIC) alone or simultaneously (CST+BIC) in the Syrian hibernating hamster (Mesocricetus auratus) model. Hamsters that received these compounds via intracerebroventricular infusions displayed notable variations of feeding and drinking bouts. In particular, an anorexigenic response was evident following treatment with CST while BIC evoked a significant increase of eating and drinking behaviors. Surprisingly when both agents were given simultaneously, a predominating anorexigenic response was detected as shown by evident CST-dependent reduction of feeding bouts. Contextually such behaviors, especially those following the combined treatment were tightly correlated with the significantly increased cerebral dopamine receptor 1 (D1) plus reduced ghrelin receptor (GhsR) and leptin receptor (LepR) transcript levels. Overall, the anorexigenic effect of CST deriving from its tight interaction with GABAARs activity plus D1 and GhsR transcripts tends to propose these neuronal elements as pivotal factors responsible for feeding disorders

Stochastic Variance-Reduced Policy Gradient

International audienceIn this paper, we propose a novel reinforcement-learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective function; II) approximations in the full gradient computation; and III) a non-stationary sampling process. The result is SVRPG, a stochastic variance-reduced policy gradient algorithm that leverages on importance weights to preserve the unbiasedness of the gradient estimate. Under standard assumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs

Stochastic Variance-Reduced Policy Gradient

In this paper, we propose a novel reinforcement-learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective function; II) approximations in the full gradient computation; and III) a non-stationary sampling process. The result is SVRPG, a stochastic variance-reduced policy gradient algorithm that leverages on importance weights to preserve the unbiasedness of the gradient estimate. Under standard assumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs