Foundation Reinforcement Learning: towards Embodied Generalist Agents with Foundation Prior Assistance

Recently, people have shown that large-scale pre-training from internet-scale data is the key to building generalist models, as witnessed in NLP. To build embodied generalist agents, we and many other researchers hypothesize that such foundation prior is also an indispensable component. However, it is unclear what is the proper concrete form to represent those embodied foundation priors and how they should be used in the downstream task. In this paper, we propose an intuitive and effective set of embodied priors that consist of foundation policy, value, and success reward. The proposed priors are based on the goal-conditioned MDP. To verify their effectiveness, we instantiate an actor-critic method assisted by the priors, called Foundation Actor-Critic (FAC). We name our framework as Foundation Reinforcement Learning (FRL), since it completely relies on embodied foundation priors to explore, learn and reinforce. The benefits of FRL are threefold. (1) Sample efficient. With foundation priors, FAC learns significantly faster than traditional RL. Our evaluation on the Meta-World has proved that FAC can achieve 100% success rates for 7/8 tasks under less than 200k frames, which outperforms the baseline method with careful manual-designed rewards under 1M frames. (2) Robust to noisy priors. Our method tolerates the unavoidable noise in embodied foundation models. We show that FAC works well even under heavy noise or quantization errors. (3) Minimal human intervention: FAC completely learns from the foundation priors, without the need of human-specified dense reward, or providing teleoperated demos. Thus, FAC can be easily scaled up. We believe our FRL framework could enable the future robot to autonomously explore and learn without human intervention in the physical world. In summary, our proposed FRL is a novel and powerful learning paradigm, towards achieving embodied generalist agents

Compound Danshen dripping pills prevent early diabetic retinopathy: roles of vascular protection and neuroprotection

Introduction: Diabetic retinopathy (DR) represents a major cause of adult blindness, and early discovery has led to significant increase in the number of patients with DR. The drugs currently used for treatment, such as ranibizumab, mainly focus on the middle and late periods of DR, and thus do not meet the clinical need. Here, the potential mechanisms by which compound Danshen Dripping Pills (CDDP) might protect against early DR were investigated.Methods: Db/db mice were used to establish a DR model. The initial weights and HbA1c levels of the mice were monitored, and retinal pathology was assessed by hematoxylin-eosin (HE) staining. The vascular permeability of the retina and thickness of each retinal layer were measured, and electroretinogram were performed together with fundus fluorescein angiography and optical coherence tomography. The levels of inflammatory factors were examined in retinal tissue, as well as those of intercellular adhesion molecule 1 (ICAM-1), IL-6, and monocyte chemoattractant protein 1 (MCP-1) in the serum using ELISA. Immunohistochemistry was used to evaluate levels of vascular endothelial growth factor (VEGF), B-cell lymphoma 2 (Bcl-2), and Bclassociated X protein (Bax). Retinal cell injury and apoptosis were examined by TdT-mediated dUTP Nick End Labeling (TUNEL) assays.Results: The data showed that CDDP significantly improved cellular disarrangement. Imaging data indicated that CDDP could reduce vascular permeability and the amplitude of oscillatory potentials (OPs), and restore the thickness of the ganglion cell layer. Moreover, CDDP reduced the expression levels of inflammatory factors in both the retina and serum.Conclusion: These findings strongly suggest that CDDP prevents early DR through vascular and neuroprotection

Exosomes in pathogenesis, diagnosis, and therapy of ischemic stroke

Ischemic stroke is one of the major contributors to death and disability worldwide. Thus, there is an urgent need to develop early brain tissue perfusion therapies following acute stroke and to enhance functional recovery in stroke survivors. The morbidity, therapy, and recovery processes are highly orchestrated interactions involving the brain with other tissues. Exosomes are natural and ideal mediators of intercellular information transfer and recognized as biomarkers for disease diagnosis and prognosis. Changes in exosome contents express throughout the physiological process. Accumulating evidence demonstrates the use of exosomes in exploring unknown cellular and molecular mechanisms of intercellular communication and organ homeostasis and indicates their potential role in ischemic stroke. Inspired by the unique properties of exosomes, this review focuses on the communication, diagnosis, and therapeutic role of various derived exosomes, and their development and challenges for the treatment of cerebral ischemic stroke

Abnormal Topology of the Structural Connectome in the Limbic Cortico-Basal-Ganglia Circuit and Default-Mode Network Among Primary Insomnia Patients

Purpose: Primary insomnia (PI) is the second most common mental disorder. However, the topologic alterations in structural brain connectome in patients with PI remain largely unknown.Methods: A total of 44 PI patients and 46 age-, gender-, and education level matched healthy control (HC) participants were recruited in this study. Diffusion tensor imaging (DTI) and resting state MRI were used to construct structural connectome for each participant, and the network parameters were employed by non-parametric permutations to evaluate the significant differences between the two groups. Relationships between abnormal network metrics and clinical characteristics, including the disease duration, the Pittsburgh Sleep Quality Index (PSQI), the Insomnia Severity Index (ISI), the Self-Rating Anxiety Scale (SAS), and the Self-Rating Depression Scale (SDS), were investigated with Spearman’s correlation analysis in PI patients.Results: PI patients demonstrated small-world architecture with lower global (P = 0.005) and local (P = 0.035) efficiencies compared with the HC group. The unique hub nodal properties in PI patients were mainly in the right limbic cortico-basal-ganglia circuit. Five disrupted subnetworks in PI patients were observed in the limbic cortico-basal-ganglia circuit and left default-mode networks (DMN) (P < 0.05, NBS corrected). Moreover, most unique hub nodal properties in the right limbic cortico-basal-ganglia circuit were significantly correlated with disease duration, and clinical characteristics (SAS, SDS, ISI scores) in PI processing.Conclusion: These findings suggested the abnormal anatomical network architecture may be closely linked to clinical characteristics in PI. The study provided novel insights into the neural substrates underlying symptoms and neurophysiologic mechanisms of PI

Semaphorin 3A Contributes to Secondary Blood–Brain Barrier Damage After Traumatic Brain Injury

Semaphorin 3A (SEMA3A) is a member of the Semaphorins family, a class of membrane-associated protein that participates in the construction of nerve networks. SEMA3A has been reported to affect vascular permeability previously, but its influence in traumatic brain injury (TBI) is still unknown. To investigate the effects of SEMA3A, we used a mouse TBI model with a controlled cortical impact (CCI) device and a blood–brain barrier (BBB) injury model in vitro with oxygen-glucose deprivation (OGD). We tested post-TBI changes in SEMA3A, and its related receptors (Nrp-1 and plexin-A1) expression and distribution through western blotting and double-immunofluorescence staining, respectively. Neurological outcomes were evaluated by modified neurological severity scores (mNSSs) and beam-walking test. We examined BBB damage through Evans Blue dye extravasation, brain water content, and western blotting for VE-cadherin and p-VE-cadherin in vivo, and we examined the endothelial cell barrier through hopping probe ion conductance microscopy (HPICM), transwell leakage, and western blotting for VE-cadherin and p-VE-cadherin in vitro. Changes in miR-30b-5p were assessed by RT-PCR. Finally, the neuroprotective function of miR-30b-5p is measured by brain water content, mNSSs and beam-walking test. SEMA3A expression varied following TBI and peaked on the third day which expressed approximate fourfold increase compared with sham group, with the protein concentrated at the lesion boundary. SEMA3A contributed to neurological function deficits and secondary BBB damage in vivo. Our results demonstrated that SEMA3A level following OGD injury almost doubled than control group, and the negative effects of OGD injury can be improved by blocking SEMA3A expression. Furthermore, the expression of miR-30b-5p decreased approximate 40% at the third day and 60% at the seventh day post-CCI. OGD injury also exhibited an effect to approximately decrease 50% of miR-30b-5p expression. Additionally, the expression of SEMA3A post-TBI is regulated by miR-30b-5p, and miR-30b-5p could improve neurological outcomes post-TBI efficiently. Our results demonstrate that SEMA3A is a significant factor in secondary BBB damage after TBI and can be abolished by miR-30b-5p, which represents a potential therapeutic target

Anticancer Therapy-Induced Atrial Fibrillation: Electrophysiology and Related Mechanisms

Some well-established immunotherapy, radiotherapy, postoperation, anticancer drugs such as anthracyclines, antimetabolites, human epidermal growth factor receptor 2 blockers, tyrosine kinase inhibitors, alkylating agents, checkpoint inhibitors, and angiogenesis inhibitors, are significantly linked to cardiotoxicity. Cardiotoxicity is a common complication of several cancer treatments. Some studies observed complications of cardiac arrhythmia associated with the treatment of cancer, including atrial fibrillation (AF), supraventricular arrhythmias, and cardiac repolarization abnormalities. AF increases the risk of cardiovascular morbidity and mortality; it is associated with an almost doubled risk of mortality and a nearly 5-fold increase in the risk of stroke. The occurrence of AF is also usually researched in patients with advanced cancer and those undergoing active cancer treatments. During cancer treatments, the incidence rate of AF affects the prognosis of tumor treatment and challenges the treatment strategy. The present article is mainly focused on the cardiotoxicity of cancer treatments. In our review, we discuss these anticancer therapies and how they induce AF and consequently provide information on the precaution of AF during cancer treatment

Hotspots of biogeochemical activity linked to aridity and plant traits across global drylands

14 páginas.- 4 figuras.- 67 referencias.- The online version contains supplementary material available at https://doi.org/10.1038/s41477-024-01670-7Perennial plants create productive and biodiverse hotspots, known as fertile islands, beneath their canopies. These hotspots largely determine the structure and functioning of drylands worldwide. Despite their ubiquity, the factors controlling fertile islands under conditions of contrasting grazing by livestock, the most prevalent land use in drylands, remain virtually unknown. Here we evaluated the relative importance of grazing pressure and herbivore type, climate and plant functional traits on 24 soil physical and chemical attributes that represent proxies of key ecosystem services related to decomposition, soil fertility, and soil and water conservation. To do this, we conducted a standardized global survey of 288 plots at 88 sites in 25 countries worldwide. We show that aridity and plant traits are the major factors associated with the magnitude of plant effects on fertile islands in grazed drylands worldwide. Grazing pressure had little influence on the capacity of plants to support fertile islands. Taller and wider shrubs and grasses supported stronger island effects. Stable and functional soils tended to be linked to species-rich sites with taller plants. Together, our findings dispel the notion that grazing pressure or herbivore type are linked to the formation or intensification of fertile islands in drylands. Rather, our study suggests that changes in aridity, and processes that alter island identity and therefore plant traits, will have marked effects on how perennial plants support and maintain the functioning of drylands in a more arid and grazed world.This research was supported by the European Research Council (ERC grant 647038 (BIODESERT) awarded to F.T.M.) and Generalitat Valenciana (CIDEGENT/2018/041). D.J.E. was supported by the Hermon Slade Foundation (HSF21040). J. Ding was supported by the National Natural Science Foundation of China Project (41991232) and the Fundamental Research Funds for the Central Universities of China. M.D.-B. acknowledges support from TED2021-130908B-C41/AEI/10.13039/501100011033/Unión Europea Next Generation EU/PRTR and the Spanish Ministry of Science and Innovation for the I + D + i project PID2020-115813RA-I00 funded by MCIN/AEI/10.13039/501100011033. O.S. was supported by US National Science Foundation (Grants DEB 1754106, 20-25166), and Y.L.B.-P. by a Marie Sklodowska-Curie Actions Individual Fellowship (MSCA-1018 IF) within the European Program Horizon 2020 (DRYFUN Project 656035). K.G. and N.B. acknowledge support from the German Federal Ministry of Education and Research (BMBF) SPACES projects OPTIMASS (FKZ: 01LL1302A) and ORYCS (FKZ: FKZ01LL1804A). B.B. was supported by the Taylor Family-Asia Foundation Endowed Chair in Ecology and Conservation Biology, and M. Bowker by funding from the School of Forestry, Northern Arizona University. C.B. acknowledges funding from the National Natural Science Foundation of China (41971131). D.B. acknowledges support from the Hungarian Research, Development and Innovation Office (NKFI KKP 144096), and A. Fajardo support from ANID PIA/BASAL FB 210006 and the Millennium Science Initiative Program NCN2021-050. M.F. and H.E. received funding from Ferdowsi University of Mashhad (grant 39843). A.N. and M.K. acknowledge support from FCT (CEECIND/02453/2018/CP1534/CT0001, SFRH/BD/130274/2017, PTDC/ASP-SIL/7743/2020, UIDB/00329/2020), EEA (10/CALL#5), AdaptForGrazing (PRR-C05-i03-I-000035) and LTsER Montado platform (LTER_EU_PT_001) grants. O.V. acknowledges support from the Hungarian Research, Development and Innovation Office (NKFI KKP 144096). L.W. was supported by the US National Science Foundation (EAR 1554894). Y.Z. and X.Z. were supported by the National Natural Science Foundation of China (U2003214). H.S. is supported by a María Zambrano fellowship funded by the Ministry of Universities and European Union-Next Generation plan. The use of any trade, firm or product names does not imply endorsement by any agency, institution or government. Finally, we thank the many people who assisted with field work and the landowners, corporations and national bodies that allowed us access to their land.Peer reviewe