Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiency

Deep reinforcement learning (DRL) algorithms require substantial samples and computational resources to achieve higher performance, which restricts their practical application and poses challenges for further development. Given the constraint of limited resources, it is essential to leverage existing computational work (e.g., learned policies, samples) to enhance sample efficiency and reduce the computational resource consumption of DRL algorithms. Previous works to leverage existing computational work require intrusive modifications to existing algorithms and models, designed specifically for specific algorithms, lacking flexibility and universality. In this paper, we present the Snapshot Reinforcement Learning (SnapshotRL) framework, which enhances sample efficiency by simply altering environments, without making any modifications to algorithms and models. By allowing student agents to choose states in teacher trajectories as the initial state to sample, SnapshotRL can effectively utilize teacher trajectories to assist student agents in training, allowing student agents to explore a larger state space at the early training phase. We propose a simple and effective SnapshotRL baseline algorithm, S3RL, which integrates well with existing DRL algorithms. Our experiments demonstrate that integrating S3RL with TD3, SAC, and PPO algorithms on the MuJoCo benchmark significantly improves sample efficiency and average return, without extra samples and additional computational resources.Comment: Under revie

The Load Distribution of the Main Shaft Bearing Considering Combined Load and Misalignment in a Floating Direct-Drive Wind Turbine

The main shaft tapered double-inner ring bearing (TDIRB) of floating direct-drive wind turbine system (FDDWT) is one of the most critical components in FDDWT, and its failure accounts for a large proportion of wind turbine malfunctions and faults. Over the past decades, a significant number of methods have been proposed to calculate the contact load distribution along the roller length in TDIRB, since the contact load distribution of roller is the key factor of fatigue life of TDIRB. Most of methods, however, neglected the misalignment of inner ring with respect to outer ring and friction force. In this paper, with the help of comprehensive and accurate quasi-static mathematical method, the contact load distribution of internal loads in TDIRB are analysed by considering the effects of combined loads, angular misalignment and friction force at different wind speeds for FDDWT. The simulation results show that the amount of combined load has an apparent effect on the contact load distribution along the TDIRB raceways and flanges in both rows. Furthermore, the slight change of tilted misalignment has a great influence on the contact load distribution. In addition, the slight angular misalignment easily produces noncontact zone for the bearing raceway in both rows, which is significantly disadvantage for the external load uniform transmitting to each roller

The Load Distribution of the Main Shaft Bearing Considering Combined Load and Misalignment in a Floating Direct-Drive Wind Turbine

The main shaft tapered double-inner ring bearing (TDIRB) of floating direct-drive wind turbine system (FDDWT) is one of the most critical components in FDDWT, and its failure accounts for a large proportion of wind turbine malfunctions and faults. Over the past decades, a significant number of methods have been proposed to calculate the contact load distribution along the roller length in TDIRB, since the contact load distribution of roller is the key factor of fatigue life of TDIRB. Most of methods, however, neglected the misalignment of inner ring with respect to outer ring and friction force. In this paper, with the help of comprehensive and accurate quasi-static mathematical method, the contact load distribution of internal loads in TDIRB are analysed by considering the effects of combined loads, angular misalignment and friction force at different wind speeds for FDDWT. The simulation results show that the amount of combined load has an apparent effect on the contact load distribution along the TDIRB raceways and flanges in both rows. Furthermore, the slight change of tilted misalignment has a great influence on the contact load distribution. In addition, the slight angular misalignment easily produces noncontact zone for the bearing raceway in both rows, which is significantly disadvantage for the external load uniform transmitting to each roller

Dehydroepiandrosterone Antagonizes Pain Stress-Induced Suppression of Testosterone Production in Male Rats

Background: Leydig cells secrete the steroid hormone, testosterone, which is essential for male fertility and reproductive health. Stress increases the secretion of glucocorticoid [corticosterone, (CORT) in rats] that decreases circulating testosterone levels in part through a direct action on its receptors in Leydig cells. Intratesticular CORT level is dependent on oxidative inactivation of CORT by 11β-hydroxysteroid dehydrogenase 1 (HSD11B1) in rat Leydig cells. Pain may cause the stress, thus affecting testosterone production in Leydig cells.Methods: Adult male Sprague–Dawley rats orally received vehicle control or 5 or 10 mg/kg dehydroepiandrosterone (DHEA) 0.5 h before being subjected to pain stimulation for 1, 3, and 6 h. In the present study, we investigated the time-course changes of steroidogenic gene expression levels after acute pain-induced stress in rats and the possible mechanism of DHEA that prevented it. Plasma CORT, luteinizing hormone (LH), and testosterone (T) levels were measured, and Leydig cell gene expression levels were determined. The direct regulation of HSD11B1 catalytic direction by DHEA was detected in purified rat Leydig, liver, and rat Hsd11b1-transfected COS1 cells.Results: Plasma CORT levels were significantly increased at hour 1, 3, and 6 during the pain stimulation, while plasma T levels were significantly decreased starting at hour 3 and 6. Pain-induced stress also decreased Star, Hsd3b1, and Cyp17a1 expression levels at hour 3. When 5 and 10 mg/kg DHEA were orally administered to rats 0.5 h before starting pain stimulation, DHEA prevented pain-mediated decrease in plasma T levels and the expression of Star, Hsd3b1, and Cyp17a1 without affecting plasma CORT levels. DHEA was found to modulate HSD11B1 activities by increasing its oxidative activity and decreasing its reductive activity, thus decreasing the intracellular CORT levels in Leydig cells.Conclusion: Stress induced by acute pain can inhibit Leydig cell T production by upregulation of corticosterone. DHEA can prevent the negative effects of excessive corticosterone by modulating HSD11B1 activity

Table_1_Dehydroepiandrosterone Antagonizes Pain Stress-Induced Suppression of Testosterone Production in Male Rats.DOCX

<p>Background: Leydig cells secrete the steroid hormone, testosterone, which is essential for male fertility and reproductive health. Stress increases the secretion of glucocorticoid [corticosterone, (CORT) in rats] that decreases circulating testosterone levels in part through a direct action on its receptors in Leydig cells. Intratesticular CORT level is dependent on oxidative inactivation of CORT by 11β-hydroxysteroid dehydrogenase 1 (HSD11B1) in rat Leydig cells. Pain may cause the stress, thus affecting testosterone production in Leydig cells.</p><p>Methods: Adult male Sprague–Dawley rats orally received vehicle control or 5 or 10 mg/kg dehydroepiandrosterone (DHEA) 0.5 h before being subjected to pain stimulation for 1, 3, and 6 h. In the present study, we investigated the time-course changes of steroidogenic gene expression levels after acute pain-induced stress in rats and the possible mechanism of DHEA that prevented it. Plasma CORT, luteinizing hormone (LH), and testosterone (T) levels were measured, and Leydig cell gene expression levels were determined. The direct regulation of HSD11B1 catalytic direction by DHEA was detected in purified rat Leydig, liver, and rat Hsd11b1-transfected COS1 cells.</p><p>Results: Plasma CORT levels were significantly increased at hour 1, 3, and 6 during the pain stimulation, while plasma T levels were significantly decreased starting at hour 3 and 6. Pain-induced stress also decreased Star, Hsd3b1, and Cyp17a1 expression levels at hour 3. When 5 and 10 mg/kg DHEA were orally administered to rats 0.5 h before starting pain stimulation, DHEA prevented pain-mediated decrease in plasma T levels and the expression of Star, Hsd3b1, and Cyp17a1 without affecting plasma CORT levels. DHEA was found to modulate HSD11B1 activities by increasing its oxidative activity and decreasing its reductive activity, thus decreasing the intracellular CORT levels in Leydig cells.</p><p>Conclusion: Stress induced by acute pain can inhibit Leydig cell T production by upregulation of corticosterone. DHEA can prevent the negative effects of excessive corticosterone by modulating HSD11B1 activity.</p

PLGA-PEG-fucoxanthin nanoparticles protect against ischemic stroke in vivo

Fucoxanthin is the most abundant marine carotenoid derived from edible brown seaweeds and has been used as a functional ingredient for the treatment of obesity. However, poor brain bioavailability of fucoxanthin has limited its application in combating neurological diseases. In this study, we demonstrated that 40 % fucoxanthin was released from polylactic-co-glycolic acid-polyethylene glycol (PLGA-PEG) nanoparticles at one day after incubation. Fucoxanthin nanoparticles (20–40 mg/kg, i.v.) significantly prevented neurological and behavioral deficits in middle cerebral artery occlusion/reperfusion (MCAO)-treated rats via the reduction of M1 microglial activation, decrease of pro-inflammatory factor production, inactivation of the nuclear factor-κB (NF-κB) pathway, and elevation of the nuclear factor-E2-related factor 2 (Nrf2) pathway in ischemic penumbra. Moreover, fucoxanthin nanoparticles (5–15 μg/mL) inhibited hypoxia-stimulated neurotoxicity in a mouse hippocampal neuron cell line (HT22 cells). These results suggest that fucoxanthin nanoparticles enhanced brain bioavailability and produced neuroprotection against ischemic stroke in vivo with much lower effective concentrations than fucoxanthin itself, providing further support for the development of fucoxanthin as a functional ingredient for the prevention of ischemic stroke with proper nanoparticle formulations