Offline Prioritized Experience Replay

Offline reinforcement learning (RL) is challenged by the distributional shift problem. To address this problem, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy. However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy. To alleviate this issue, we propose Offline Prioritized Experience Replay (OPER), featuring a class of priority functions designed to prioritize highly-rewarding transitions, making them more frequently visited during training. Through theoretical analysis, we show that this class of priority functions induce an improved behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution. We develop two practical strategies to obtain priority weights by estimating advantages based on a fitted value network (OPER-A) or utilizing trajectory returns (OPER-R) for quick computation. OPER is a plug-and-play component for offline RL algorithms. As case studies, we evaluate OPER on five different algorithms, including BC, TD3+BC, Onestep RL, CQL, and IQL. Extensive experiments demonstrate that both OPER-A and OPER-R significantly improve the performance for all baseline methods. Codes and priority weights are availiable at https://github.com/sail-sg/OPER.Comment: preprin

Exvivo Experiments of Human Ovarian Cancer Ascites-Derived Exosomes Presented by Dendritic Cells Derived from Umbilical Cord Blood for Immunotherapy Treatment

Objectives Exosomes, a type of membrane vesicles, released from tumor cells have been shown to be capable of transferring tumor antigens to dendritic cells and activating specific cytotoxic T-lymphocytes. Recent work has demonstrated the presence of high numbers of exosomes in malignant effusions. Umbilical cord blood (UCB) is a rich source of hematopoietic stem cells and from which a significant number of dendritic cells can be produced. We hypothesized that the exosomes released from metastatic ovarian carcinoma were able to present tumor specific antigen to dendritic cells derived from unrelated umbilical cord blood, then could stimulate resting T cells to differentiate and induce effective cytotoxicity. Study Design Exosomes were isolated by ultracentrifugation of malignant ascites from ovarian cancer patients (n = 10). Purified exosomes were further characterized by Western blot analyses and immunoelectronic microscopy. Dendritic cells were collected from unrelated umbilical cord blood and cultured in the presence of GM-CSF, IL-4 and TNF-α. Resting T cells were mixed with dentritic cells previously primed with exosomes and the cytotoxicity were measured by MTT method. T cells were activated by DCs presented with exosomes. Results 1) the exosomes isolated from the ascites were membrane vesicles of about 30-90nm in diameter; 2) the exosomes expressed MHC class I molecules, HSP70, HSP90, Her2/Neu, and Mart1; and 3)umbilical cord blood-derived DCs previously exosome-primed stimulated resting T cells to differentiate and produce effective cytotoxicity. Conclusions These results suggested that tumor-specific antigens present on exosomes can be presented by DCs derived from unrelated umbilical cord blood to induce tumor specific cytotoxicity and this may represent as a novel immunotherapy for ovarian cancer

Characterization of a new feather-degrading bacterium from Calotes versicolor feces

A total of 842 spore-forming strains were isolated from 221 animal feces samples, in which a new feather-degrading bacterium identified as Bacillus sp. 50-3 based on morphological, biochemical and 16S rDNA tests was isolated from Calotes versicolor (an agamid lizard) feces. The bacterium can degrade native feather completely at mild conditions and in much shorter time (36 h) when using chicken feather as the sole carbon and nitrogen source. It presented optimum growth and maximum keratinase activity (680±25 U/ml, 36 h) at 37°C and pH 7.0 in feather meal medium. The keratinase of Bacillus sp. 50-3 was active on feather keratin as substrate and presented optimum additive quantity of 1.0% (w/v). So the high efficiency of Bacillus sp. 50-3 on feather-degradation suggested its potential use in biotechnological processes, especially in decreasing the environment pollution.Keywords: Animal feces, Bacillus, chicken feathers, keratinase, optimum conditions