In-Sample Policy Iteration for Offline Reinforcement Learning

Offline reinforcement learning (RL) seeks to derive an effective control policy from previously collected data. To circumvent errors due to inadequate data coverage, behavior-regularized methods optimize the control policy while concurrently minimizing deviation from the data collection policy. Nevertheless, these methods often exhibit subpar practical performance, particularly when the offline dataset is collected by sub-optimal policies. In this paper, we propose a novel algorithm employing in-sample policy iteration that substantially enhances behavior-regularized methods in offline RL. The core insight is that by continuously refining the policy used for behavior regularization, in-sample policy iteration gradually improves itself while implicitly avoids querying out-of-sample actions to avert catastrophic learning failures. Our theoretical analysis verifies its ability to learn the in-sample optimal policy, exclusively utilizing actions well-covered by the dataset. Moreover, we propose competitive policy improvement, a technique applying two competitive policies, both of which are trained by iteratively improving over the best competitor. We show that this simple yet potent technique significantly enhances learning efficiency when function approximation is applied. Lastly, experimental results on the D4RL benchmark indicate that our algorithm outperforms previous state-of-the-art methods in most tasks

Research progress and prospects of utilizing carbon-based nanomaterials in enhanced oil recovery

Carbon-based nanomaterials have received heightened global interest by petroleum researchers because of their abundant stocks of necessary raw materials, ease of size control, readiness for modification, and high stability. In light of the practical demand for oil development, this study reviews the recent progress in the research of enhancing oil recovery using carbon-based nanomaterials of various dimensions, including carbon dots, carbon nanotubes, carbon nanofibers, and graphene and its derivatives. Moreover, the study elaborates on the application of these materials in high-efficiency oil displacement, profile control and water shutoff, as well as the fracturing process. The related challenges and solutions in practical oil exploration and development are analyzed, and the application prospects of these materials in future oil reservoirs and oilfields are predicted. This review provides valuable theoretical and experimental references for the large-scale application of carbon-based nanomaterials.Document Type: Invited reviewCited as: Shen, M., Zhang, C., Yan, X., Wang, L., Wu, Y., Jin, X. Research progress and prospects of utilizing carbon-based nanomaterials in enhanced oil recovery. Advances in Geo-Energy Research, 2024, 14(3): 201-214. https://doi.org/10.46690/ager.2024.12.0

Diagnosis of glioma molecular markers by terahertz technologies

This review considers glioma molecular markers in brain tissues and body fluids, shows the pathways of their formation, and describes traditional methods of analysis. The most important optical properties of glioma markers in the terahertz (THz) frequency range are also presented. New metamaterial-based technologies for molecular marker detection at THz frequencies are discussed. A variety of machine learning methods, which allow the marker detection sensitivity and differentiation of healthy and tumor tissues to be improved with the aid of THz tools, are considered. The actual results on the application of THz techniques in the intraoperative diagnosis of brain gliomas are shown. THz technologies’ potential in molecular marker detection and defining the boundaries of the glioma’s tissue is discussed

A versatile multimodal optical modality based on Brillouin light scattering and photoacoustic effect

Multimodal optical imaging techniques are useful for various applications, including imaging biological samples for providing comprehensive material properties. In this work, we developed a new modality that can measure a set of mechanical, optical, and acoustical properties of a sample at microscopic resolution, which is based on the integration of Brillouin (Br) and photoacoustic (PA) microscopy. The proposed multimodal imaging technique not only can acquire co-registered Br and PA signals but also allows us to utilize the sound speed measured by PA to quantify the sample’s refractive index, which is a fundamental property of the material and cannot be measured by either technique individually. We demonstrated the colocalization of Br and time-resolved PA signals in a synthetic phantom made of kerosene and CuSO4aqueous solution. In addition, we measured the refractive index of saline solutions and validated the result against published data with a relative error of 0.3 %. This multimodal Br-PA modality could open a new way for characterizing biological samples in physiological and pathological conditions.</jats:p

POSS-based meso-/macroporous covalent networks: supporting and stabilizing Pd for Suzuki–Miyaura reaction at room temperature

Porous covalent organic networks synthesized by Schiff base chemistry reaction of POSS and terephthalic aldehyde could serve as both supports and stabilizers for Pd catalyst, which exhibited excellent performances for Suzuki-Miyaura reactions.</p