Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Conventional reinforcement learning (RL) needs an environment to collect fresh data, which is impractical when online interactions are costly. Offline RL provides an alternative solution by directly learning from the previously collected dataset. However, it will yield unsatisfactory performance if the quality of the offline datasets is poor. In this paper, we consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online, and propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data. Specifically, we explicitly consider the difference between the online and offline data and apply an adaptive update scheme accordingly, that is, a pessimistic update strategy for the offline dataset and an optimistic/greedy update scheme for the online dataset. Such a simple and effective method provides a way to mix the offline and online RL and achieve the best of both worlds. We further provide two detailed algorithms for implementing the framework through embedding value or policy-based RL algorithms into it. Finally, we conduct extensive experiments on popular continuous control tasks, and results show that our algorithm can learn the expert policy with high sample efficiency even when the quality of offline dataset is poor, e.g., random dataset.Comment: AAAI202

Spatio-Temporal Adaptive Embedding Makes Vanilla Transformer SOTA for Traffic Forecasting

With the rapid development of the Intelligent Transportation System (ITS), accurate traffic forecasting has emerged as a critical challenge. The key bottleneck lies in capturing the intricate spatio-temporal traffic patterns. In recent years, numerous neural networks with complicated architectures have been proposed to address this issue. However, the advancements in network architectures have encountered diminishing performance gains. In this study, we present a novel component called spatio-temporal adaptive embedding that can yield outstanding results with vanilla transformers. Our proposed Spatio-Temporal Adaptive Embedding transformer (STAEformer) achieves state-of-the-art performance on five real-world traffic forecasting datasets. Further experiments demonstrate that spatio-temporal adaptive embedding plays a crucial role in traffic forecasting by effectively capturing intrinsic spatio-temporal relations and chronological information in traffic time series.Comment: Accepted as CIKM2023 Short Pape

LBH589 Inhibits proliferation and metastasis of hepatocellular carcinoma via inhibition of gankyrin/stat3/akt pathway

Background: Gankyrin has shown to be overexpressed in human liver cancers and plays a complex role in hepatocarcinogenesis. Panobinostat (LBH589), a new hydroxamic acid-derived histone deacetylase inhibitor has shown promising anticancer effects recently. Here, we investigated the potential of LBH589 as a form of treatment for hepatocellular carcinoma (HCC). Methods: Gankyrin plasmid was transfected into HCC cells, and the cells were selected for more than 4 weeks by incubation with G418 for overexpression clones. The therapeutic effects of LBH589 were evaluated in vitro and in vivo. Cell proliferation, apoptosis, cell cycle, invasive potential, and epithelial-mesenchy-mal transition (EMT) were examined. Results: LBH589 significantly inhibited HCC growth and metastasis in vitro and in vivo. Western blotting analysis indicated that LBH589 could decrease the expression of gankyrin and subsequently reduced serine-phosphorylated Akt and tyrosine-phosphorylated STAT3 expression although the total Akt and STAT3 were unaffected. LBH589 inhibited metastasis in vitro via down-regulation of N-cadherin, vimentin, TWIST1, VEGF and up-regulation of E-cadherin. LBH589 also induced apoptosis and G1 phase arrest in HCC cell lines. Ectopic expression of gankyrin attenuated the effects of LBH589, which indicates that gankyrin might play an important role in LBH589 mediated anticancer effects. Lastly, in vivo study indicated that LBH589 inhibited tumor growth and metastasis, without discernable adverse effects comparing to control group, with abrogating gankyrin/STAT3/Akt pathway. Conclusions: Our results suggested that LBH589 could inhibit HCC growth and metastasis through down-regulating gankyrin/STAT3/Akt pathway. LBH589 may present itself as a novel therapeutic strategy for HCC

The ectonucleotidases CD39 and CD73 on T cells: The new pillar of hematological malignancy

Hematological malignancy develops and applies various mechanisms to induce immune escape, in part through an immunosuppressive microenvironment. Adenosine is an immunosuppressive metabolite produced at high levels within the tumor microenvironment (TME). Adenosine signaling through the A2A receptor expressed on immune cells, such as T cells, potently dampens immune responses. Extracellular adenosine generated by ectonucleoside triphosphate diphosphohydrolase-1 (CD39) and ecto-5’-nucleotidase (CD73) molecules is a newly recognized ‘immune checkpoint mediator’ and leads to the identification of immunosuppressive adenosine as an essential regulator in hematological malignancies. In this Review, we provide an overview of the detailed distribution and function of CD39 and CD73 ectoenzymes in the TME and the effects of CD39 and CD73 inhibition on preclinical hematological malignancy data, which provides insights into the potential clinical applications for immunotherapy

Role of intestinal flora and 5-HT in depression- and anxiety-like behaviors in mice exposed to PM2.5

BackgroundSome studies have shown that PM2.5 exposure is closely related to central nervous system diseases that lead to cognitive dysfunction and change the composition of intestinal flora. However, there are few studies on the role of intestinal flora in PM2.5-induced depression- and anxiety-like behaviors in mice. ObjectiveTo observe the effects of PM2.5 exposure on depression- and anxiety-like behaviors and the composition of intestinal flora in mice, and to explore the role of intestinal flora in regulating 5-hydroxytryptamine (5-HT) in depression- and anxiety-like behaviors in mice exposed to PM2.5. MethodsEight-week-old male SPF C57BL/6J mice were randomly divided into control group (NS group), probiotic group (LGG group), PM2.5 group (PM group), and combined exposure group (PML group), 6 mice in each group. Mice in the PM group and the PML group were exposed to PM2.5 in a dynamic exposure cabinet for 6 h per day, 6 d a week for 7 consecutive weeks, and the PM2.5 concentrations were approximately 8 times higher than the outdoor concentration. The LGG group and the PML group were orally administered with Lactobacillus rhamnosus while the NS group and the PM group were orally administered with the same amount of saline. Elevated plus maze test and open field test were used to detect depression and anxiety in mice. Fecal samples of mice were collected to evaluate intestinal flora abundance, diversity, and structure between groups using high-throughput sequencing of 16S rRNA. ELISA was employed to detect the levels of 5-HT in serum and hippocampus. Spearman correlation was used to analyze the correlations of differential intestinal flora with 5-HT level in hippocampus and depression- and anxiety-like behavior indicators in mice. ResultsThe percentage of open-arm entry [M(P25, P75)] in the PM group was 0.0% (0.0%, 33.3%), lower than those in the NS group [47.7% (25.0%, 50.8%) ] and the PML group [46.9% (40.0%, 50.0%)], and the differences were statistically significant (P<0.05). The total travelled distance and the time spent in central area (\begin{document}$\bar x \pm s$\end{document}) in the PM group were (2.01±0.90) m and (10.31±1.99) s respectively, shorter than those of the NS group [(3.80±0.89) m, (14.47±3.07) s], the total travelled distance in the PML group [(2.73±1.12) m] was shorter than those of the NS group and the LGG group [(4.21±1.08) m], and the differences were statistically significant (P<0.05). Compared to the NS group, the Simpson index of the PM group significantly increased (P<0.05). Compared to the LGG group, the Simpson index of the PML group significantly decreased (P<0.05). The results of Beta diversity analysis showed that there were differences in the composition of intestinal flora among the four groups of mice. Compared with the NS group and the LGG group, the abundances of Erysipelotrichaceae and Dubosiella in the PM group and the PML group increased, while the abundances of Prevotellaceae_UCG-001 decreased, and the differences were statistically significant (P<0.05). In hippocampus, the level of 5-HT in the PM group [(135.02±10.31) μg·g−1] was lower than those in the NS group [(178.77±43.15) μg·g−1] and the LGG group [(224.85±22.98) μg·g−1], and the level of 5-HT in the PML group [(161.27±15.81) μg·g−1] was lower than that in the LGG group (P<0.05). 5-HT level in hippocampus was significantly positively correlated with the relative abundance of Prevotellaceae_UCG-001 (r=0.6090, P=0.012). The percentage of open-arm entry was significantly negatively correlated with the relative abundance of Dubosiella (r=−0.4630, P=0.023). ConclusionAtmospheric PM2.5 exposure may cause depression- and anxiety-like behaviors in mice. The observed behavior dysfunction may be associated with the changes in diversity and relative abundance of intestinal flora as well as the decrease of 5-HT level. Such depression- and anxiety-like behaviors are alleviated after adding probiotics

Plasmoid ejection and secondary current sheet generation from magnetic reconnection in laser-plasma interaction

Reconnection of the self-generated magnetic fields in laser-plasma interaction was first investigated experimentally by Nilson {\it et al.} [Phys. Rev. Lett. 97, 255001 (2006)] by shining two laser pulses a distance apart on a solid target layer. An elongated current sheet (CS) was observed in the plasma between the two laser spots. In order to more closely model magnetotail reconnection, here two side-by-side thin target layers, instead of a single one, are used. It is found that at one end of the elongated CS a fan-like electron outflow region including three well-collimated electron jets appears. The ($>1$ MeV) tail of the jet energy distribution exhibits a power-law scaling. The enhanced electron acceleration is attributed to the intense inductive electric field in the narrow electron dominated reconnection region, as well as additional acceleration as they are trapped inside the rapidly moving plasmoid formed in and ejected from the CS. The ejection also induces a secondary CS

The role of autophagy in cardiovascular disease: Cross-interference of signaling pathways and underlying therapeutic targets

Autophagy is a conserved lysosomal pathway for the degradation of cytoplasmic proteins and organelles, which realizes the metabolic needs of cells and the renewal of organelles. Autophagy-related genes (ATGs) are the main molecular mechanisms controlling autophagy, and their functions can coordinate the whole autophagic process. Autophagy can also play a role in cardiovascular disease through several key signaling pathways, including PI3K/Akt/mTOR, IGF/EGF, AMPK/mTOR, MAPKs, p53, Nrf2/p62, Wnt/β-catenin and NF-κB pathways. In this paper, we reviewed the signaling pathway of cross-interference between autophagy and cardiovascular diseases, and analyzed the development status of novel cardiovascular disease treatment by targeting the core molecular mechanism of autophagy as well as the critical signaling pathway. Induction or inhibition of autophagy through molecular mechanisms and signaling pathways can provide therapeutic benefits for patients. Meanwhile, we hope to provide a unique insight into cardiovascular treatment strategies by understanding the molecular mechanism and signaling pathway of crosstalk between autophagy and cardiovascular diseases