Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to find a near-optimal policy using pre-collected datasets. In real-world scenarios, data collection could be costly and risky; therefore, offline RL becomes particularly challenging when the in-domain data is limited. Given recent advances in Large Language Models (LLMs) and their few-shot learning prowess, this paper introduces $\textbf{La}$nguage Models for $\textbf{Mo}$tion Control ($\textbf{LaMo}$), a general framework based on Decision Transformers to effectively use pre-trained Language Models (LMs) for offline RL. Our framework highlights four crucial components: (1) Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively, (3) using the non-linear MLP transformation instead of linear projections, to generate embeddings, and (4) integrating an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities on languages. Empirical results indicate $\textbf{LaMo}$ achieves state-of-the-art performance in sparse-reward tasks and closes the gap between value-based offline RL methods and decision transformers in dense-reward tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples.Comment: 24 pages, 16 table

H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation

Human hands possess remarkable dexterity and have long served as a source of inspiration for robotic manipulation. In this work, we propose a human $\textbf{H}$and$\textbf{-In}$formed visual representation learning framework to solve difficult $\textbf{Dex}$terous manipulation tasks ($\textbf{H-InDex}$) with reinforcement learning. Our framework consists of three stages: (i) pre-training representations with 3D human hand pose estimation, (ii) offline adapting representations with self-supervised keypoint detection, and (iii) reinforcement learning with exponential moving average BatchNorm. The last two stages only modify $0.36\%$ parameters of the pre-trained representation in total, ensuring the knowledge from pre-training is maintained to the full extent. We empirically study 12 challenging dexterous manipulation tasks and find that H-InDex largely surpasses strong baseline methods and the recent visual foundation models for motor control. Code is available at https://yanjieze.com/H-InDex .Comment: NeurIPS 2023. Code and videos: https://yanjieze.com/H-InDe

Correlation of LC3 and the recruitment of dendritic cell and the formation of TLS in colorectal cancer and its clinical significance

Background and purpose: It has been recognized as a complex problem in tumor therapy to deal with the tumor immune escape, and over-activated autophagy can cause the increase of tumor surface antigen, which participates in anti-tumor immunity. In this study, the expressions of microtubule-associated protein light chain 3 (LC3), mature dendritic cell (mDC) and the formation of tertiary lymphoid structure (TLS), an essential autophagy factor in colorectal cancer, were detected in clinical samples. The results had important clinical implications and provided new insights for enhancing anti-tumor immunity. Methods: Immunohistochemical EnVision method was used to detect the expressions of LC3, DC-lamp and the formation of TLS in cancer tissues and normal mucosal tissues of 179 patients with T2 stage high-risk and T3 stage colorectal cancer who underwent surgical resection at Binzhou Medical University Hospital from January 2016 to June 2017. Western blot and real-time fluorescence quantitative polymerase chain reaction (RTFQ-PCR) were used to detect the expressions of LC3, NY-ESO-2, lymphotoxin-beta (LTβ), CC chemokine ligand 21 (CCL21), CXC chemokine ligand 13 (CXCL13) and interleukin-17 (IL-17) in TLS+ and TLS- colorectal cancer tissues. Then the correlation and clinical significance were analyzed. Log-rank test was used to compare the prognostic differences between groups, and COX proportional risk regression model was used for multivariate survival analysis. Results: Clinical samples showed that the expressions of LC3 and DC-lamp were higher in colorectal cancer tissues than in normal mucosa tissues (P<0.05), and the expressions of LC3 and DC-lamp were positively correlated (P<0.05). The protein and mRNA expressions of LC3, NY-ESO-2, LTβ, CXCL13 and CCL21 were higher in TLS+ group than in TLS- group. The expression of IL-17 was higher in the TLS- group than in the TLS+ group (P<0.05). The expression of LC3 was positively correlated with TLS/germinal center (GC)+ and TLS/GC- subtypes and positively correlated with the expression of NY-ESO-2, LTβ, CXCL13 and CCL21 (P<0.05). The expression of DC-lamp was higher in TLS/GC+ and TLS/GC- subtype groups than in the other two subgroups (P<0.05), and there was a positive correlation. Kaplan-Meier and COX regression models showed that LC3, DC-lamp, TLS and lymph node metastasis were closely related to the prognosis of patients with colorectal cancer, and they were independent risk factors for the prognosis of colorectal cancer. Conclusion: The abnormal expression of LC3 in colorectal cancer can activate mDC to recruit lymphocytes and promote the expression and maturation of TLS, ultimately affecting the prognosis of patients

Mitogenome characterization and diversity of the nangqian grey yak (bos grunniens)

Nangqian grey yak (Bos grunniens) is a unique yak population in Qinghai Province, China. In this study, the whole mitogenome sequences of 18 Nangqian grey yaks were sequenced based on the next-generation sequencing (NGS) technology and annotated. The total length of whole mitogenome sequence is between 16.323 bp and 16.325 bp, including a non-coding control region (D-loop region), 22 tRNA genes, 13 protein-coding genes and two rRNA genes (12S rRNA and 16S rRNA). Maternal genetic diversity based on the mitogenome variations was analyzed. A total of 12 haplotypes were identified among 18 complete mitogenome sequences, the haplotype diversity and nucleotide diversity of Nangqian grey yak were 0.948±0.033 and 0.001±0.001, respectively. Compared with the wild yak population and six other domestic yak breeds/populations in China, the haplotype diversity of Nangqian grey yak population was higher, indicating abundant maternal genetic diversity in Nangqian grey yak. The phylogenetic tree showed that Nangqian grey yak was most closely related to Tibet alpine, Xueduo, Changtai, Sibu, Zhongdian, Tianzhu white, Ashdan, Jinchuan, Jiulong, Pamir, Pali, Qinghai plateau, Huanhu, Datong, Bazhou and wild yak breeds/populations, closer to Chawula, Muli, Gannan, Niangya and Yushu yak breeds, but far away from other yak breeds (i.e. Leiwuqi and Maiwa yak)

Hydroxychloroquine Potentiates Apoptosis Induced by PPARα Antagonist in 786-O Clear Cell Renal Cell Carcinoma Cells Associated with Inhibiting Autophagy

Clear cell renal cell carcinoma (ccRCC) is the major pathological pattern of renal cell carcinoma. The ccRCC cells exhibit a certain degree of inherent drug resistance due to some genetic mutations. In recent years, peroxisome proliferator-activated receptor-α (PPARα) antagonists have been reported as a targeted therapeutic drug capable of inducing apoptosis and cell cycle arrest in the ccRCC cell line. Autophagy, which can be induced by stress in eukaryotic cells, plays a complex role in the proliferation, survival, and death of tumor cells. In our study, we found that the expression of PPARα was low in highly differentiated ccRCC tissues and 786-O cell line but high in poorly differentiated ccRCC tissues. The level of PPARα expression in ccRCC tissues is correlated to the grade of differentiation, but not to the sex or age of ccRCC patients. The findings also revealed that the PPARα antagonist GW6471 can lower cell viability and induce autophagy in the 786-O ccRCC cell line. This autophagy can be inhibited by hydroxychloroquine. When treated with a combination of hydroxychloroquine and GW6471, the viability of the 786-O cells was decreased further when compared to the treatment with GW6471 or hydroxychloroquine alone, and apoptosis was promoted. Meanwhile, when human kidney 2 cells were cotreated with hydroxychloroquine and GW6471, cell viability was only slightly influenced. Hence, our finding indicates that the combination of GW6471 and hydroxychloroquine may constitute a novel and potentially effective treatment for ccRCC. Furthermore, this approach is likely to be safe owing to its minimal effects on normal renal tissues

Investigation of Very Large Eddy Simulation Method for Applications of Supersonic Turbulent Combustion

The very large eddy simulation (VLES) method was investigated for supersonic reacting flows in the present work. The advantages and characteristics of the VLES model and the widely used improved delayed detached eddy simulation (IDDES) method were revealed through a supersonic ramped-cavity cold flow. Compared to the IDDES model, the VLES model transformed from RANS mode to LES mode faster, resulting in a smaller gray region caused by the mode transition. However, the original volume-averaging truncation length scale could lead to poor predictions of the velocity profiles and wall pressure distribution. By introducing a hybrid truncation length scale combining the maximum grid length and the shear layer adaptive (SLA) length with different coefficients, the accuracy of the VLES method was significantly improved, and the issue of the low shear layer position was solved. Moreover, owing to the resolution control function, the VLES method could adaptively model more turbulent kinetic energy and maintain a good accuracy in a coarser mesh. Finally, the modified VLES method was applied in conjunction with a hybrid combustion model constructed by the partially stirred reactor (PaSR) model and the Ingenito supersonic combustion model (ISCM) in simulations of the supersonic flame in the DLR scramjet combustor. After introducing the correction of the molecular collision frequency by the ISCM, the results obtained by the hybrid combustion model were more consistent with the experimental results, especially for the time-averaging temperature profile in the ignition zone

Periodic composites: quasi-uniform heat conduction, Janus thermal illusion, and illusion thermal diodes

Manipulating thermal conductivities at will plays a crucial role in controlling heat flow. By developing an effective medium theory including periodicity, here we experimentally show that nonuniform media can exhibit quasi-uniform heat conduction. This provides capabilities in proposing Janus thermal illusion and illusion thermal rectification. For the former, we study, via experiment and theory, a big periodic composite containing a small periodic composite with circular or elliptic particles. As a result, we reveal the Janus thermal illusion that describes the whole periodic system with both invisibility illusion along one direction and visibility illusion along the perpendicular direction, which is fundamentally different from the existing thermal illusions for misleading thermal detection. Further, the Janus illusion helps to design two different periodic systems that both work as thermal diodes but with nearly the same temperature distribution, heat fluxes and rectification ratios, thus being called illusion thermal diodes. Such thermal diodes differ from those extensively studied in the literature, and are useful for the areas that require both thermal rectification and thermal camouflage. This work not only opens a door for designing novel periodic composites in thermal camouflage and heat rectification, but also holds for achieving similar composites in other disciplines like electrostatics, magnetostatics, and particle dynamics