Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration

Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration and restrains the overall performance. In many realistic tasks, e.g. autonomous driving, large-scale expert demonstration data are available. We argue that extracting expert policy from offline data to guide online exploration is a promising solution to mitigate the conserveness issue. Large-capacity models, e.g. decision transformers (DT), have been proven to be competent in offline policy learning. However, data collected in real-world scenarios rarely contain dangerous cases (e.g., collisions), which makes it prohibitive for the policies to learn safety concepts. Besides, these bulk policy networks cannot meet the computation speed requirements at inference time on real-world tasks such as autonomous driving. To this end, we propose Guided Online Distillation (GOLD), an offline-to-online safe RL framework. GOLD distills an offline DT policy into a lightweight policy network through guided online safe RL training, which outperforms both the offline DT policy and online safe RL algorithms. Experiments in both benchmark safe RL tasks and real-world driving tasks based on the Waymo Open Motion Dataset (WOMD) demonstrate that GOLD can successfully distill lightweight policies and solve decision-making problems in challenging safety-critical scenarios

Hydrocarbon generation and expulsion modeling of different lithological combination source rocks from the Funing Formation in the Subei Basin

The oil expulsion efficiency and retention efficiency of shale affect the enrichment and preservation of shale oil. Two series of semi-closed hydrous pyrolysis experiments were performed under in situ geological conditions on a Paleogene shale sample as a comparable analog to evaluate the generation and preservation potential of shale oil in the Funing Formation shale in the Subei Basin. The results show that 1) the oil-generation capacity evolution of different lithological combination source rocks in the Funing Formation of the Subei Basin can be roughly divided into four stages: a) relatively slow oil-generating and slow gas-generating, b) relatively fast oil-generating and slow gas-generating, c) oil cracking into gas, and d) kerogen cracking into gas; 2) different lithological combinations have different hydrocarbon generation, expulsion, and retention efficiencies. The total oil generation rate and gas generation rate of pure shale are higher than those of shale with a silty interlayer, and the exchange point between the oil expulsion rate and retention rate of pure shale is earlier than that of shale with the silty interlayer, which indicates that the pure shale experienced the expulsion and retention process earlier. Oil retention mainly occurs at an EqVRo of 0.84%–1.12%, while oil is mainly discharged to the adjacent siltstone at an EqVRo of 1.12%–1.28%. Based on the simulation under geological conditions, it is recognized that for shale oil exploration in the Subei Basin, the favorable thermal maturity is at an EqVRo of 0.84%–1.12%, and the favorable lithology is the shale with the silty interlayer. On one hand, the siltstone interlayer can provide pore space for the early generated oil, and the concentration difference of hydrocarbons between the shale and the interlayer can be formed so that the generated shale can continuously enter the interlayer. On the other hand, the shale above the interlayer can be used as a cap rock to preserve shale oil. The favorable area for shale oil exploration in the Subei Basin is the area with relatively high maturity (at a VR value of about 1.1%

The Bibliometric Analysis of Microplastics in Soil Environments: Hotspots of Research and Trends of Development

Microplastics are persistent and complex contaminants and have been recognized as a global concern. Recently, increasing efforts have been devoted to studying the influence of microplastics on soils. However, the complexity of microplastics and the diversity of extraction methods result in a lack of systematic analysis and comprehensive review in this field. In this paper, we used CiteSpace software to summarize the development of this field. Then, we visualized and analyzed the knowledge structure, research hotspots, and trend directions of this field. We found that the number of publications escalated dramatically, and 281 institutions in 69 countries have published articles in this field. Among them, China was the most productive contributor. However, according to the scientific collaboration analysis, we found that more than 90% of the authors who contributed to the field had no close connection. In co-occurrence analysis for subject categories, we found that the research in this field covered environmental science, engineering, ecology, and agriculture. Additionally, the effect of soil microplastics on agriculture was the most important problem in scientific research. The keyword co-occurrence cluster analysis revealed a total of 6 clusters, including “Identification” (#0), “Microbial community” (#1), “Oxidative stress” (#2), “Adsorption” (#3), “Porous media” (#4), and “Abundance” (#5). We discussed several aspects in detail, including detection methods, characteristics, environmental effects, adsorption capacity, removal and degradation, and toxicity. According to these results, we summarized the current research hotspots and evaluated future research trends in soil microplastics. This study is the first to specifically visualize the research field, and these results provide a reference for future research in the field of soil microplastics

Data from: High invasion potential of Hydrilla verticillata in the Americas predicted using ecological niche modeling combined with genetic data

Ecological niche modeling is an effective tool to characterize the spatial distribution of suitable areas for species, and it is especially useful for predicting the potential distribution of invasive species. The widespread submerged plant Hydrilla verticillata (hydrilla) has an obvious phylogeographical pattern: Four genetic lineages occupy distinct regions in native range, and only one lineage invades the Americas. Here, we aimed to evaluate climatic niche conservatism of hydrilla in North America at the intraspecific level and explore its invasion potential in the Americas by comparing climatic niches in a phylogenetic context. Niche shift was found in the invasion process of hydrilla in North America, which is probably mainly attributed to high levels of somatic mutation. Dramatic changes in range expansion in the Americas were predicted in the situation of all four genetic lineages invading the Americas or future climatic changes, especially in South America; this suggests that there is a high invasion potential of hydrilla in the Americas. Our findings provide useful information for the management of hydrilla in the Americas and give an example of exploring intraspecific climatic niche to better understand species invasion

Additional file 4: of Genetic and geographical structure of boreal plants in their southern range: phylogeography of Hippuris vulgaris in China

Modeling of the numbers of genetic clusters in Hippuris vulgaris for (a) all 91 populations, (b) the 18 populations in lineage A, and (c) the 63 populations in lineage B, respectively, using STRUCTURE. (DOC 140 kb

Integrated Analysis of the Transcriptome and Microbial Diversity in the Intestine of Miniature Pig Obesity Model

Obesity, a key contributor to metabolic disorders, necessitates an in-depth understanding of its pathogenesis and prerequisites for prevention. Guangxi Bama miniature pig (GBM) offers an apt model for obesity-related studies. In this research, we used transcriptomics and 16S rRNA gene sequencing to discern the differentially expressed genes (DEGs) within intestinal (jejunum, ileum, and colon) tissues and variations in microbial communities in intestinal contents of GBM subjected to normal diets (ND) and high-fat, high-carbohydrate diets (HFHCD). After a feeding duration of 26 weeks, the HFHCD-fed experimental group demonstrated notable increases in backfat thickness, BMI, abnormal blood glucose metabolism, and blood lipid levels alongside the escalated serum expression of pro-inflammatory factors and a marked decline in intestinal health status when compared to the ND group. Transcriptomic analysis revealed a total of 1669 DEGs, of which 27 had similar differences in three intestinal segments across different groups, including five immune related genes: COL6A6, CYP1A1, EIF2AK2, NMI, and LGALS3B. Further, we found significant changes in the microbiota composition, with a significant decrease in beneficial bacterial populations within the HFHCD group. Finally, the results of integrated analysis of microbial diversity with transcriptomics show a positive link between certain microbial abundance (Solibacillus, norank_f__Saccharimonadaceae, Candidatus_Saccharimonas, and unclassified_f__Butyricicoccaceae) and changes in gene expression (COL6A6 and NMI). Overall, HFHCD appears to co-contribute to the initiation and progression of obesity in GBM by aggravating inflammatory responses, disrupting immune homeostasis, and creating imbalances in intestinal flora