SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores

The ever-growing complexity of reinforcement learning (RL) tasks demands a distributed RL system to efficiently generate and process a massive amount of data to train intelligent agents. However, existing open-source libraries suffer from various limitations, which impede their practical use in challenging scenarios where large-scale training is necessary. While industrial systems from OpenAI and DeepMind have achieved successful large-scale RL training, their system architecture and implementation details remain undisclosed to the community. In this paper, we present a novel abstraction on the dataflows of RL training, which unifies practical RL training across diverse applications into a general framework and enables fine-grained optimizations. Following this abstraction, we develop a scalable, efficient, and extensible distributed RL system called ReaLly Scalable RL (SRL). The system architecture of SRL separates major RL computation components and allows massively parallelized training. Moreover, SRL offers user-friendly and extensible interfaces for customized algorithms. Our evaluation shows that SRL outperforms existing academic libraries in both a single machine and a medium-sized cluster. In a large-scale cluster, the novel architecture of SRL leads to up to 3.7x speedup compared to the design choices adopted by the existing libraries. We also conduct a direct benchmark comparison to OpenAI's industrial system, Rapid, in the challenging hide-and-seek environment. SRL reproduces the same solution as reported by OpenAI with up to 5x speedup in wall-clock time. Furthermore, we also examine the performance of SRL in a much harder variant of the hide-and-seek environment and achieve substantial learning speedup by scaling SRL to over 15k CPU cores and 32 A100 GPUs. Notably, SRL is the first in the academic community to perform RL experiments at such a large scale.Comment: 15 pages, 12 figures, 6 table

Effect of negative emotions evoked by light, noise and taste on trigeminal thermal sensitivity

Patients with migraine often have impaired somatosensory function and experience headache attacks triggered by exogenous stimulus, such as light, sound or taste. This study aimed to assess the influence of three controlled conditioning stimuli (visual, auditory and gustatory stimuli and combined stimuli) on affective state and thermal sensitivity in healthy human participants. All participants attended four experimental sessions with visual, auditory and gustatory conditioning stimuli and combination of all stimuli, in a randomized sequence. In each session, the somatosensory sensitivity was tested in the perioral region with use of thermal stimuli with and without the conditioning stimuli. Positive and Negative Affect States (PANAS) were assessed before and after the tests. Subject based ratings of the conditioning and test stimuli in addition to skin temperature and heart rate as indicators of arousal responses were collected in real time during the tests. The three conditioning stimuli all induced significant increases in negative PANAS scores (paired t-test, P a parts per thousand currency sign0.016). Compared with baseline, the increases were in a near dose-dependent manner during visual and auditory conditioning stimulation. No significant effects of any single conditioning stimuli were observed on trigeminal thermal sensitivity (P a parts per thousand yen0.051) or arousal parameters (P a parts per thousand yen0.057). The effects of combined conditioning stimuli on subjective ratings (P a parts per thousand currency sign0.038) and negative affect (P = 0.011) were stronger than those of single stimuli. All three conditioning stimuli provided a simple way to evoke a negative affective state without physical arousal or influence on trigeminal thermal sensitivity. Multisensory conditioning had stronger effects but also failed to modulate thermal sensitivity, suggesting that so-called exogenous trigger stimuli e.g. bright light, noise, unpleasant taste in patients with migraine may require a predisposed or sensitized nervous system.Clinical NeurologyNeurosciencesSCI(E)[email protected]

Somatosensory abnormalities in Chinese patients with painful temporomandibular disorders

Background: The somatosensory phenotype of Chinese temporomandibular disorders (TMD) patients is not sufficiently studied with the use of contemporary techniques and guidelines. Methods: A standardized quantitative sensory testing (QST) battery consisting of 13 parameters with a stringent statistical protocol developed by the German Research Network on Neuropathic Pain was performed over the most painful and corresponding contralateral sites as well as the right hand of 40 Chinese patients with TMD and pain classified according to the Diagnostic Criteria for TMD (DC/TMD). The same QST protocol was performed bilaterally over the infraorbital, mental, and hand regions of 70 age-and gender-stratified healthy Chinese controls. Z-scores and loss/gain scores were computed for each TMD patient. Results: For patients, 82.5 % had somatosensory abnormalities in the painful facial region, while 60.0 % had abnormalities confined to the right hand. The most frequent abnormalities were somatosensory gain to pinprick (35.0 %) and pressure (35.0 %) stimuli, somatosensory loss to pinprick (25.0 %), cold (22.5 %), and heat (15.0 %) nociceptive stimuli. The most frequent loss/gain score was L0G2 (no somatosensory loss combined with a gain of mechanical somatosensory function) for both the facial (40.0 %) and hand (27.5 %) regions. Involving side-to-side differences in the evaluation increased the diagnostic sensitivity by 2.5-25.0 % across different parameters. Conclusions: Somatosensory abnormalities were commonly detected in Chinese TMD pain patients both within and outside the primary painful region, strongly indicating disturbances in the central processing of somatosensory stimuli. The individual variations in somatosensory abnormalities indicate a possible need for development of individualized TMD pain management.Capital Health Research and Development [2011-4025-01]; Peking University School of Stomatology [PKUSS20150207]SCI(E)[email protected]

Atrial Natriuretic Peptide Regulates Ca2+ Channel in Early Developmental Cardiomyocytes

currents has not been investigated in developmental cardiomyocytes. by ANP was entirely abolished by erythro-9-(2-Hydroxy-3-nonyl) adenine (EHNA), a selective inhibitor of type 2 phosphodiesterase(PDE2) in most cells tested. is due to activation of particulate guanylyl cyclase (GC), cGMP production and cGMP-activation of PDE2 mediated depression of adenosine 3′, 5′–cyclic monophophate (cAMP)–cAMP-dependent protein kinase (PKA) in early cardiomyogenesis

RNA-binding protein CUGBP1 regulates insulin secretion via activation of phosphodiesterase 3B in mice

International audienceAims/hypothesis: CUG-binding protein 1 (CUGBP1) is a multifunctional RNA-binding protein that regulates RNA processing at several stages including translation, deadenylation and alternative splicing, as well as RNA stability. Recent studies indicate that CUGBP1 may play a role in metabolic disorders. Our objective was to examine its role in endocrine pancreas function through gain- and loss-of-function experiments and to further decipher the underlying molecular mechanisms.Methods: A mouse model in which type 2 diabetes was induced by a high-fat diet (HFD; 60% energy from fat) and mice on a standard chow diet (10% energy from fat) were compared. Pancreas-specific CUGBP1 overexpression and knockdown mice were generated. Different lengths of the phosphodiesterase subtype 3B (PDE3B) 3′ untranslated region (UTR) were cloned for luciferase reporter analysis. Purified CUGBP1 protein was used for gel shift experiments.Results: CUGBP1 is present in rodent islets and in beta cell lines; it is overexpressed in the islets of diabetic mice. Compared with control mice, the plasma insulin level after a glucose load was significantly lower and glucose clearance was greatly delayed in mice with pancreas-specific CUGBP1 overexpression; the opposite results were obtained upon pancreas-specific CUGBP1 knockdown. Glucose- and glucagon-like peptide1 (GLP-1)-stimulated insulin secretion was significantly attenuated in mouse islets upon CUGBP1 overexpression. This was associated with a strong decrease in intracellular cAMP levels, pointing to a potential role for cAMP PDEs. CUGBP1 overexpression had no effect on the mRNA levels of PDE1A, 1C, 2A, 3A, 4A, 4B, 4D, 7A and 8B subtypes, but resulted in increased PDE3B expression. CUGBP1 was found to directly bind to a specific ATTTGTT sequence residing in the 3′ UTR of PDE3B and stabilised PDE3B mRNA. In the presence of the PDE3 inhibitor cilostamide, glucose- and GLP-1-stimulated insulin secretion was no longer reduced by CUGBP1 overexpression. Similar to CUGBP1, PDE3B was overexpressed in the islets of diabetic mice.Conclusions/interpretation: We conclude that CUGBP1 is a critical regulator of insulin secretion via activating PDE3B. Repressing this protein might provide a potential strategy for treating type 2 diabetes

Factors related to the length of stay for major depressive disorder patients in China: A real-world retrospective study

BackgroundAs numerous patients with depression have to be hospitalized because of various reasons, the demand far exceeds the limited bed count in the psychiatry department. Controlling the length of stay (LOS) of the patient is gradually being considered an effective method to alleviate this problem. Given the lack of statistical evidence of the LOS of patients with major depressive disorder (MDD) in China and the strain on the limited psychiatric resources, the purpose of our study was to investigate the LOS of patients with MDD among in-patient samples and to analyze related factors of the LOS in China by building a regression model.MethodThe data were exported from the electronic medical record system. A total of three categories of independent variables were enrolled in our study, namely, demographic, clinical, and biochemical. Univariate analysis and binominal regression analysis were applied comprehensively to find the factors related to the LOS among MDD samples. The discrimination accuracy of the model was evaluated by the receiver operating characteristic (ROC) analysis. ROC analysis indicated that the discrimination accuracy of our model was acceptable (AUC = 0.790, 95% CI = 0.714–0.865, P < 0.001).ResultA total of 254 patients were finally brought into analysis after filtering. Regression analysis indicated that abnormal LDL was the only risk factor of long LOS (OR = 3.352, 95% CI = 1.087–10.337, P = 0.035) among all the kinds of variables. Notably, in the statistically irrelevant factors of the LOS, the category of anti-depressant drugs [serotonin–norepinephrine reuptake inhibitor (SNRI) or selective serotonin reuptake inhibitor (SSRI)] prescribed to patients with MDD was not associated statistically with the LOS, which was against our initial hypothesis that the LOS of patients with MDD treated with SNRI would vary from that of the patients treated with SSRI.ConclusionUp to our knowledge, our research is the first study to show the potential factors related to the LOS from various domains, especially biochemical indexes, and the effect of drugs, among clinical patients with MDD in China. Our results could provide a theoretical reference for efficient psychiatry hospitalization management and prioritization of allocating medical resources. Future studies are required for updating independent variables which are potentially related to the LOS and verifying existing results in a larger sample

Reassortant H9N2 Influenza Viruses Containing H5N1-Like PB1 Genes Isolated from Black-Billed Magpies in Southern China

H9N2 influenza A viruses have become endemic in different types of terrestrial poultry and wild birds in Asia, and are occasionally transmitted to humans and pigs. To evaluate the role of black-billed magpies (Pica pica) in the evolution of influenza A virus, we conducted two epidemic surveys on avian influenza viruses in wild black-billed magpies in Guangxi, China in 2005 and characterized three isolated black-billed magpie H9N2 viruses (BbM viruses). Phylogenetic analysis indicated that three BbM viruses were almost identical with 99.7 to 100% nucleotide homology in their whole genomes, and were reassortants containing BJ94-like (Ck/BJ/1/94) HA, NA, M, and NS genes, SH/F/98-like (Ck/SH/F/98) PB2, PA, and NP genes, and H5N1-like (Ck/YN/1252/03, clade 1) PB1 genes. Genetic analysis showed that BbM viruses were most likely the result of multiple reassortments between co-circulating H9N2-like and H5N1-like viruses, and were genetically different from other H9N2 viruses because of the existence of H5N1-like PB1 genes. Genotypical analysis revealed that BbM viruses evolved from diverse sources and belonged to a novel genotype (B46) discovered in our recent study. Molecular analysis suggested that BbM viruses were likely low pathogenic reassortants. However, results of our pathogenicity study demonstrated that BbM viruses replicated efficiently in chickens and a mammalian mouse model but were not lethal for infected chickens and mice. Antigenic analysis showed that BbM viruses were antigenic heterologous with the H9N2 vaccine strain. Our study is probably the first report to document and characterize H9N2 influenza viruses isolated from black-billed magpies in southern China. Our results suggest that black-billed magpies were susceptible to H9N2 influenza viruses, which raise concerns over possible transmissions of reassortant H9N2 viruses among poultry and wild birds