In-Sample Policy Iteration for Offline Reinforcement Learning

Offline reinforcement learning (RL) seeks to derive an effective control policy from previously collected data. To circumvent errors due to inadequate data coverage, behavior-regularized methods optimize the control policy while concurrently minimizing deviation from the data collection policy. Nevertheless, these methods often exhibit subpar practical performance, particularly when the offline dataset is collected by sub-optimal policies. In this paper, we propose a novel algorithm employing in-sample policy iteration that substantially enhances behavior-regularized methods in offline RL. The core insight is that by continuously refining the policy used for behavior regularization, in-sample policy iteration gradually improves itself while implicitly avoids querying out-of-sample actions to avert catastrophic learning failures. Our theoretical analysis verifies its ability to learn the in-sample optimal policy, exclusively utilizing actions well-covered by the dataset. Moreover, we propose competitive policy improvement, a technique applying two competitive policies, both of which are trained by iteratively improving over the best competitor. We show that this simple yet potent technique significantly enhances learning efficiency when function approximation is applied. Lastly, experimental results on the D4RL benchmark indicate that our algorithm outperforms previous state-of-the-art methods in most tasks

Ethyl Pyruvate Attenuates CaCl2-Induced Tubular Epithelial Cell Injury by Inhibiting Autophagy and Inflammatory Responses

Background/Aims: Nephrolithiasis is one of the most prevalent diseases of the urinary system. Approximately 80% of human kidney stones are composed of calcium oxalate (CaOx), and hypercalciuria is one of the most common metabolic disorders. Emerging evidence indicates that autophagy and inflammatory responses are related to the formation of CaOx nephrolithiasis. However, the roles of autophagy and inflammation in patients with hypercalciuria remain unclear. Ethyl pyruvate (EP) displays protective effects in experimental models of many illnesses. In this study, we investigated the protective effects of EP in vitro through its inhibition of autophagy and inflammatory responses after CaCl2-induced tubular epithelial cell injury. Methods: First, we cultured human tubular epithelial (HK-2) cells in the presence of various concentrations of CaCl2 (0, 0.1, 0.25, 0.5, 1.0, 1.5, and 2.0 mg/ml) for 12 h and EP (0, 1.0, 2.5, 5.0, and 10.0 mM) for 2 h to select the optimum concentration using the Cell Counting Kit-8 assay and lactate dehydrogenase (LDH) assay. Cells in culture were stimulated with CaCl2 (1.0 mg/ml, 12 h) with or without EP pretreatment (2.5 mM, 2 h). After the exposure, we detected the expression of inflammation-related proteins using an enzyme-linked immunosorbent assay and Western blot analysis. Finally, the levels of autophagy-related proteins were determined through Western blot analysis, and the number of GFP-LC3 dots and autophagic vacuoles was detected under confocal microscopy. Results: With the use of the Cell Counting Kit-8 assay and the LDH assay, we identified the optimum concentration for CaCl2 (1.0 mg/ml) treatment and EP pretreatment (2.5 mM). Our research indicated that CaCl2 can induce autophagy and inflammatory responses in HK-2 cells. Furthermore, treatment with EP prior to CaCl2 stimulation attenuated HK-2 cell injury by inhibiting autophagy and inflammation. Conclusion: Our results provide evidence that EP attenuates CaCl2-induced injury of HK-2 cells by downregulating the expression of inflammation and autophagy proteins that may be associated with the inhibition of the high-mobility group box-1 (HMGB1)/toll-like receptor 4 (TLR4)/NF-κB pathway and the competitive interaction with Beclin-1 of HMGB1

Short-Term Power Prediction of a Wind Farm Based on Empirical Mode Decomposition and Mayfly Algorithm–Back Propagation Neural Network

With the improvement of energy consumption structure, the installed capacity of wind power increases gradually. However, the inherent intermittency and instability of wind energy bring severe challenges to the dispatching operation. Wind power forecasting is one of the main solutions. In this work, a new combined wind power prediction model is proposed. First, a quartile method is used for data cleaning, namely, identifying and eliminating the abnormal data. Then, the wind power data sequence is decomposed by empirical mode decomposition to eliminate non-stationary characteristics. Finally, the wind generator data are trained by the MA-BP network to establish the wind power prediction model. Also, the simulation tests verify the prediction effect of the proposed method. Specifically speaking, the average MAPE is decreased to 12.4979% by the proposed method. Also, the average RMSE and MAE are 107.1728 and 71.604 kW, respectively

Optimization of pre-swirl stators based on CFD for a chemical product carrier

The viscous self-propulsion flow fields of a model-scaled 55k DWT chemical product carrier fitted with a rudder-bulb-fin and a pre-swirl stator are numerically simulated based on the CFD general code FLUENT. The energy saving effects of stators are evaluated through the increase of propulsive efficiency. It is found that the computed changing tendencies of almost all self-propulsion factors after being equipped with a stator are the same as in the experiments, such as a decreased revolution rate, increased thrust deduction and mean wake. A wake energy analysis is also conducted to verify the energy-saving effects of stators, and it shows that the stator decreases the flow of kinetic energy behind the propeller through its contra-propeller pre-swirl. Next, an optimization of pre-swirl stators is conducted by CFD. Aside from the prototype stator, three modified stators are designed and the self-propulsion characteristics with these stators are also numerically simulated. The increase order of the evaluated energy-saving effects of these modified stators is seen to be the same as in the design idea. The case with the highest propulsive efficiency shows the largest increase of Ktotal before the propeller and the largest decrease of Ktotal behind the propeller relative to cases without stators

Collagen/Chitosan Complexes: Preparation, Antioxidant Activity, Tyrosinase Inhibition Activity, and Melanin Synthesis

Bioactive collagen/chitosan complexes were prepared by an ion crosslinking method using fish skin collagen and chitosan solution as raw materials. Scanning electron microscopy observation confirmed that the collagen/chitosan complexes were of a uniform spherical shape and uniform particle size. The complexes were stable at different pH values for a certain period of time through swelling experiments. Differential scanning calorimetry (DSC) showed the collagen/ chitosan complexes were more stable than collagen. X-ray diffraction (XRD) showed that the complexes had a strong crystal structure, and Fourier transform infrared spectroscopy (FTIR) data revealed the changes in the secondary structure of the protein due to chitosan and TPP crosslinking. The content of malondialdehyde (MDA) in the complex treatment group was considerably lower, but the content of SOD was significantly higher than that of the collagen group or chitosan group. In addition, the collagen/chitosan complexes could considerably reduce melanin content, inhibit tyrosinase activity, and down-regulate tyrosinase mRNA expression. In conclusion, the collagen/chitosan complexes were potential oral protein preparation for antioxidant enhancement and inhibiting melanin synthesis

Predicting the Microstructure of a Valve Head during the Hot Forging of Steel 21-4N

Valve microstructure is important during hot forging. Austenitic 21-4N steel is often used in exhaust valves. In this study, the microstructure evolution of the forging valve process was predicted using the internal state variables (i.e., average grain size, recrystallized fraction, and dislocation density) modus for 21-4N. First, 21-4N was subjected to hot compression tests on a Gleeble-1500D and static grain growth tests in a heating furnace. A set of uniform viscoplastic constitutive equations was established based on experimental data. Next, the determined unified constitutive equations were conducted in DEFORM-3D, and the microstructure evolution of 21-4N during forging was calculated. Finally, the simulation results of grain size evolution were validated via experiments. Results showed good consistency between the simulations and experiments. Thus, the models adequately predicted the microstructure evolution

Ecological efficiency index system.

As China’s urbanization accelerates, ecological environmental issues have become increasingly prominent, and how to achieve the synergistic development of urbanization and ecological environment is worth exploring. The paper uses the Super-SBM model and the improved entropy method to calculate the ecological efficiency and the new urbanization in 63 counties in Zhejiang Province from 2000 to 2019. Furthermore, the coupling coordination degree between new urbanization and ecological efficiency is discussed with the coupling degree model, Markov chain, and spatial correlation methods, and its influencing factors are explored by the geographic detector. The results show that: (1) The development trends of new urbanization and ecological efficiency in Zhejiang Province counties both present a "U" shape. Their inflection points appeared in 2005 and 2006, respectively. The gap between counties is gradually narrowing. (2) The coupling coordination degree between new urbanization and ecological efficiency in Zhejiang Province counties also develops in a "U" shape with the minimum value appearing in 2006. Its temporal evolution is dominated by advancement towards a higher level and maintenance of the original type, with most countries advancing from General Disorder to Preliminary Coordination. There is a good positive correlation in the spatial distribution, showing significant High-High and Low-Low agglomeration. (3) In detecting the driving factors, the explanatory power of economic development, natural conditions and social conditions diminishes sequentially. The interaction groups mostly are nonlinear enhancements, and the rest are all two-factor enhancements. Social factors are the main interaction objects. (4) The empirical analysis verified the efficacy of the "Two Mountains" theory and the importance of government investment in the regional coordinated development.</div