Embrace the Change Even When You Don’t Know Where It Will Take You

Senior Project submitted to The Division of Arts of Bard College

Foundation Reinforcement Learning: towards Embodied Generalist Agents with Foundation Prior Assistance

Recently, people have shown that large-scale pre-training from internet-scale data is the key to building generalist models, as witnessed in NLP. To build embodied generalist agents, we and many other researchers hypothesize that such foundation prior is also an indispensable component. However, it is unclear what is the proper concrete form to represent those embodied foundation priors and how they should be used in the downstream task. In this paper, we propose an intuitive and effective set of embodied priors that consist of foundation policy, value, and success reward. The proposed priors are based on the goal-conditioned MDP. To verify their effectiveness, we instantiate an actor-critic method assisted by the priors, called Foundation Actor-Critic (FAC). We name our framework as Foundation Reinforcement Learning (FRL), since it completely relies on embodied foundation priors to explore, learn and reinforce. The benefits of FRL are threefold. (1) Sample efficient. With foundation priors, FAC learns significantly faster than traditional RL. Our evaluation on the Meta-World has proved that FAC can achieve 100% success rates for 7/8 tasks under less than 200k frames, which outperforms the baseline method with careful manual-designed rewards under 1M frames. (2) Robust to noisy priors. Our method tolerates the unavoidable noise in embodied foundation models. We show that FAC works well even under heavy noise or quantization errors. (3) Minimal human intervention: FAC completely learns from the foundation priors, without the need of human-specified dense reward, or providing teleoperated demos. Thus, FAC can be easily scaled up. We believe our FRL framework could enable the future robot to autonomously explore and learn without human intervention in the physical world. In summary, our proposed FRL is a novel and powerful learning paradigm, towards achieving embodied generalist agents

Compound Danshen dripping pills prevent early diabetic retinopathy: roles of vascular protection and neuroprotection

Introduction: Diabetic retinopathy (DR) represents a major cause of adult blindness, and early discovery has led to significant increase in the number of patients with DR. The drugs currently used for treatment, such as ranibizumab, mainly focus on the middle and late periods of DR, and thus do not meet the clinical need. Here, the potential mechanisms by which compound Danshen Dripping Pills (CDDP) might protect against early DR were investigated.Methods: Db/db mice were used to establish a DR model. The initial weights and HbA1c levels of the mice were monitored, and retinal pathology was assessed by hematoxylin-eosin (HE) staining. The vascular permeability of the retina and thickness of each retinal layer were measured, and electroretinogram were performed together with fundus fluorescein angiography and optical coherence tomography. The levels of inflammatory factors were examined in retinal tissue, as well as those of intercellular adhesion molecule 1 (ICAM-1), IL-6, and monocyte chemoattractant protein 1 (MCP-1) in the serum using ELISA. Immunohistochemistry was used to evaluate levels of vascular endothelial growth factor (VEGF), B-cell lymphoma 2 (Bcl-2), and Bclassociated X protein (Bax). Retinal cell injury and apoptosis were examined by TdT-mediated dUTP Nick End Labeling (TUNEL) assays.Results: The data showed that CDDP significantly improved cellular disarrangement. Imaging data indicated that CDDP could reduce vascular permeability and the amplitude of oscillatory potentials (OPs), and restore the thickness of the ganglion cell layer. Moreover, CDDP reduced the expression levels of inflammatory factors in both the retina and serum.Conclusion: These findings strongly suggest that CDDP prevents early DR through vascular and neuroprotection

Exosomes in pathogenesis, diagnosis, and therapy of ischemic stroke

Ischemic stroke is one of the major contributors to death and disability worldwide. Thus, there is an urgent need to develop early brain tissue perfusion therapies following acute stroke and to enhance functional recovery in stroke survivors. The morbidity, therapy, and recovery processes are highly orchestrated interactions involving the brain with other tissues. Exosomes are natural and ideal mediators of intercellular information transfer and recognized as biomarkers for disease diagnosis and prognosis. Changes in exosome contents express throughout the physiological process. Accumulating evidence demonstrates the use of exosomes in exploring unknown cellular and molecular mechanisms of intercellular communication and organ homeostasis and indicates their potential role in ischemic stroke. Inspired by the unique properties of exosomes, this review focuses on the communication, diagnosis, and therapeutic role of various derived exosomes, and their development and challenges for the treatment of cerebral ischemic stroke

Proteomic analysis of elite soybean Jidou17 and its parents using iTRAQ-based quantitative approaches

BACKGROUND: Derived from Hobbit as the female parent and Zao5241 as the male parent, the elite soybean cultivar Jidou17 is significantly higher yielding and shows enhanced qualities and stronger resistance to non-biological stress than its parents. The purpose of this study is to understand the difference in protein expression patterns between Jidou17 and its parental strains and to evaluate the parental contributions to its elite traits. RESULTS: Leaves (14 days old) from Jidou17 and its parental cultivars were analysed for differential expressed proteins using an iTRAQ-based (isobaric tags for relative and absolute quantitation) method. A total of 1269 proteins was detected, with 141 and 181 proteins in Jidou17 differing from its female and male parent, respectively. Functional classification and an enrichment analysis based on biological functions, biological processes, and cellular components revealed that all the differential proteins fell into many functional categories but that the number of proteins varied greatly for the different categories, with enrichment in specific categories. A pathway analysis indicated that the differentiated proteins were mainly classified into the ribosome assembly pathway. Protein expression clustering results showed that the expression profiles between Jidou17 and its female parent Hobbit were more similar than those between Jidou17 and its male parent Zao5241 and between the two parental strains. Therefore, the female parent Hobbit contributed more to the Jidou17 genotype. CONCLUSIONS: This study applied a proven technique to study proteomics in 14-day-old soybean leaves and explored the depth and breadth of soybean protein research. The results provide new data for further understanding the mechanisms of elite cultivar development