Efficient Diffusion Policies for Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to learn optimal policies from offline datasets, where the parameterization of policies is crucial but often overlooked. Recently, Diffsuion-QL significantly boosts the performance of offline RL by representing a policy with a diffusion model, whose success relies on a parametrized Markov Chain with hundreds of steps for sampling. However, Diffusion-QL suffers from two critical limitations. 1) It is computationally inefficient to forward and backward through the whole Markov chain during training. 2) It is incompatible with maximum likelihood-based RL algorithms (e.g., policy gradient methods) as the likelihood of diffusion models is intractable. Therefore, we propose efficient diffusion policy (EDP) to overcome these two challenges. EDP approximately constructs actions from corrupted ones at training to avoid running the sampling chain. We conduct extensive experiments on the D4RL benchmark. The results show that EDP can reduce the diffusion policy training time from 5 days to 5 hours on gym-locomotion tasks. Moreover, we show that EDP is compatible with various offline RL algorithms (TD3, CRR, and IQL) and achieves new state-of-the-art on D4RL by large margins over previous methods. Our code is available at https://github.com/sail-sg/edp.Comment: preprin

Offline Prioritized Experience Replay

Offline reinforcement learning (RL) is challenged by the distributional shift problem. To address this problem, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy. However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy. To alleviate this issue, we propose Offline Prioritized Experience Replay (OPER), featuring a class of priority functions designed to prioritize highly-rewarding transitions, making them more frequently visited during training. Through theoretical analysis, we show that this class of priority functions induce an improved behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution. We develop two practical strategies to obtain priority weights by estimating advantages based on a fitted value network (OPER-A) or utilizing trajectory returns (OPER-R) for quick computation. OPER is a plug-and-play component for offline RL algorithms. As case studies, we evaluate OPER on five different algorithms, including BC, TD3+BC, Onestep RL, CQL, and IQL. Extensive experiments demonstrate that both OPER-A and OPER-R significantly improve the performance for all baseline methods. Codes and priority weights are availiable at https://github.com/sail-sg/OPER.Comment: preprin

Disturbance Rejection Control for Autonomous Trolley Collection Robots with Prescribed Performance

Trajectory tracking control of autonomous trolley collection robots (ATCR) is an ambitious work due to the complex environment, serious noise and external disturbances. This work investigates a control scheme for ATCR subjecting to severe environmental interference. A kinematics model based adaptive sliding mode disturbance observer with fast convergence is first proposed to estimate the lumped disturbances. On this basis, a robust controller with prescribed performance is proposed using a backstepping technique, which improves the transient performance and guarantees fast convergence. Simulation outcomes have been provided to illustrate the effectiveness of the proposed control scheme

Epstein-Barr Virus Nuclear Antigen 3C Stabilizes Gemin3 to Block p53-mediated Apoptosis

The Epstein-Barr nuclear antigen 3C (EBNA3C), one of the essential latent antigens for Epstein-Barr virus (EBV)-induced immortalization of primary human B lymphocytes in vitro, has been implicated in regulating cell proliferation and anti-apoptosis via interaction with several cellular and viral factors. Gemin3 (also named DDX20 or DP103) is a member of DEAD RNA helicase family which exhibits diverse cellular functions including DNA transcription, recombination and repair, and RNA metabolism. Gemin3 was initially identified as a binding partner to EBNA2 and EBNA3C. However, the mechanism by which EBNA3C regulates Gemin3 function remains unclear. Here, we report that EBNA3C directly interacts with Gemin3 through its C-terminal domains. This interaction results in increased stability of Gemin3 and its accumulation in both B lymphoma cells and EBV transformed lymphoblastoid cell lines (LCLs). Moreover, EBNA3C promotes formation of a complex with p53 and Gemin3 which blocks the DNA-binding affinity of p53. Small hairpin RNA based knockdown of Gemin3 in B lymphoma or LCL cells remarkably attenuates the ability of EBNA3C to inhibit the transcription activity of p53 on its downstream genes p21 and Bax, as well as apoptosis. These findings provide the first evidence that Gemin3 may be a common target of oncogenic viruses for driving cell proliferation and anti-apoptotic activities

Protection of Pentoxifylline against Testis Injury Induced by Intermittent Hypobaric Hypoxia

To investigate the effect of pentoxifylline (PTX) on spermatogenesis dysfunction induced by intermittent hypobaric hypoxia (IHH) and unveil the underlying mechanism, experimental animals were assigned to Control, IHH+Vehicle, and IHH+PTX groups and exposed to 4 cycles of 96 h of hypobaric hypoxia followed by 96 h of normobaric normoxia for 32 days. PTX was administered for 32 days. Blood and tissue samples were collected 7 days thereafter. Serum malondialdehyde levels were used to assess lipid peroxidation; ferric-reducing antioxidant power (FRAP), superoxide dismutase, and catalase and glutathione peroxidase enzyme activities were assessed to determine antioxidant capacity in various samples. Testis histopathology was assessed after hematoxylin-eosin staining by Johnsen’s testicular scoring system. Meanwhile, testosterone synthase and vimentin amounts were assessed by immunohistochemistry. Sperm count, motility, and density were assessed to determine epididymal sperm quality. IHH treatment induced significant pathological changes in testicular tissue and enhanced serum lipid peroxide levels, while reducing serum FRAP, antioxidant enzyme activities, and testosterone synthase expression. Moreover, IHH impaired epididymal sperm quality and vimentin structure in Sertoli cells. Oral administration of PTX improved the pathological changes in the testis. IHH may impair spermatogenesis function of testicular tissues by inducing oxidative stress, but this impairment could be attenuated by administration of PTX

Collaborative Trolley Transportation System with Autonomous Nonholonomic Robots

Cooperative object transportation using multiple robots has been intensively studied in the control and robotics literature, but most approaches are either only applicable to omnidirectional robots or lack a complete navigation and decision-making framework that operates in real time. This paper presents an autonomous nonholonomic multi-robot system and an end-to-end hierarchical autonomy framework for collaborative luggage trolley transportation. This framework finds kinematic-feasible paths, computes online motion plans, and provides feedback that enables the multi-robot system to handle long lines of luggage trolleys and navigate obstacles and pedestrians while dealing with multiple inherently complex and coupled constraints. We demonstrate the designed collaborative trolley transportation system through practical transportation tasks, and the experiment results reveal their effectiveness and reliability in complex and dynamic environments

Pandemic fatigue impedes mitigation of COVID-19 in Hong Kong

Hong Kong has implemented stringent public health and social measures (PHSMs) to curb each of the four COVID-19 epidemic waves since January 2020. The third wave between July and September 2020 was brought under control within 2 m, while the fourth wave starting from the end of October 2020 has taken longer to bring under control and lasted at least 5 mo. Here, we report the pandemic fatigue as one of the potential reasons for the reduced impact of PHSMs on transmission in the fourth wave. We contacted either 500 or 1,000 local residents through weekly random-digit dialing of landlines and mobile telephones from May 2020 to February 2021. We analyze the epidemiological impact of pandemic fatigue by using the large and detailed cross-sectional telephone surveys to quantify risk perception and self-reported protective behaviors and mathematical models to incorporate population protective behaviors. Our retrospective prediction suggests that an increase of 100 daily new reported cases would lead to 6.60% (95% CI: 4.03, 9.17) more people worrying about being infected, increase 3.77% (95% CI: 2.46, 5.09) more people to avoid social gatherings, and reduce the weekly mean reproduction number by 0.32 (95% CI: 0.20, 0.44). Accordingly, the fourth wave would have been 14% (95% CI%: −53%, 81%) smaller if not for pandemic fatigue. This indicates the important role of mitigating pandemic fatigue in maintaining population protective behaviors for controlling COVID-19

Kaposi's Sarcoma Herpesvirus Upregulates Aurora A Expression to Promote p53 Phosphorylation and Ubiquitylation

Aberrant expression of Aurora A kinase has been frequently implicated in many cancers and contributes to chromosome instability and phosphorylation-mediated ubiquitylation and degradation of p53 for tumorigenesis. Previous studies showed that p53 is degraded by Kaposi's sarcoma herpesvirus (KSHV) encoded latency-associated nuclear antigen (LANA) through its SOCS-box (suppressor of cytokine signaling, LANASOCS) motif-mediated recruitment of the EC5S ubiquitin complex. Here we demonstrate that Aurora A transcriptional expression is upregulated by LANA and markedly elevated in both Kaposi's sarcoma tissue and human primary cells infected with KSHV. Moreover, reintroduction of Aurora A dramatically enhances the binding affinity of p53 with LANA and LANASOCS-mediated ubiquitylation of p53 which requires phosphorylation on Ser215 and Ser315. Small hairpin RNA or a dominant negative mutant of Aurora A kinase efficiently disrupts LANA-induced p53 ubiquitylation and degradation, and leads to induction of p53 transcriptional and apoptotic activities. These studies provide new insights into the mechanisms by which LANA can upregulate expression of a cellular oncogene and simultaneously destabilize the activities of the p53 tumor suppressor in KSHV-associated human cancers

Solidification/Stabilization of Textile Sludge as Subgrade: Usage of Binders and Skeleton Material

This study investigates the disposal of textile sludge via laboratory and field tests while protecting the eco-environment. Solidification/stabilization (S/S) technology and skeleton construction method are introduced to investigate the application of S/S sludge for subgrade material. S/S is to enhance the sludge strength and stabilize the metal(loid)s and hazardous organics in the textile sludge. Skeleton construction method aims to decrease the liquid-solid ratio in mixture to reduce the binder dosage and save binder cost. In the laboratory, binders and skeleton material are implemented to investigate the differences in unconfined compressive strength (UCS) to explore the optimal mixture. Results illustrate that UCS of binder-sludge is below 100 kPa and enhanced more than 400 kPa after adding gypsum and skeleton material. Skeleton soil material with high plasticity index and low moisture content improves UCS significantly. Scanning electron microscopy test shows the physical microstructure of sludge is greatly improved for the particular space grid structure formed by the particles and cementitious products. The leaching test shows the metal(loid)s and organics in leachate are decreased after S/S treatment and below the standard value. Finally, the textile sludge was disposed for subgrade via the technology. The strength and leaching results of field tests are in good agreement with the laboratory results. The bearing capacity of the practical subgrade meets the design requirements