Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk

Though deep reinforcement learning (DRL) has obtained substantial success, it may encounter catastrophic failures due to the intrinsic uncertainty of both transition and observation. Most of the existing methods for safe reinforcement learning can only handle transition disturbance or observation disturbance since these two kinds of disturbance affect different parts of the agent; besides, the popular worst-case return may lead to overly pessimistic policies. To address these issues, we first theoretically prove that the performance degradation under transition disturbance and observation disturbance depends on a novel metric of Value Function Range (VFR), which corresponds to the gap in the value function between the best state and the worst state. Based on the analysis, we adopt conditional value-at-risk (CVaR) as an assessment of risk and propose a novel reinforcement learning algorithm of CVaR-Proximal-Policy-Optimization (CPPO) which formalizes the risk-sensitive constrained optimization problem by keeping its CVaR under a given threshold. Experimental results show that CPPO achieves a higher cumulative reward and is more robust against both observation and transition disturbances on a series of continuous control tasks in MuJoCo

Task Aware Dreamer for Task Generalization in Reinforcement Learning

A long-standing goal of reinforcement learning is to acquire agents that can learn on training tasks and generalize well on unseen tasks that may share a similar dynamic but with different reward functions. A general challenge is to quantitatively measure the similarities between these different tasks, which is vital for analyzing the task distribution and further designing algorithms with stronger generalization. To address this, we present a novel metric named Task Distribution Relevance (TDR) via optimal Q functions of different tasks to capture the relevance of the task distribution quantitatively. In the case of tasks with a high TDR, i.e., the tasks differ significantly, we show that the Markovian policies cannot differentiate them, leading to poor performance. Based on this insight, we encode all historical information into policies for distinguishing different tasks and propose Task Aware Dreamer (TAD), which extends world models into our reward-informed world models to capture invariant latent features over different tasks. In TAD, we calculate the corresponding variational lower bound of the data log-likelihood, including a novel term to distinguish different tasks via states, to optimize reward-informed world models. Extensive experiments in both image-based control tasks and state-based control tasks demonstrate that TAD can significantly improve the performance of handling different tasks simultaneously, especially for those with high TDR, and demonstrate a strong generalization ability to unseen tasks

Bridging photochemistry and photomechanics with NMR crystallography: the molecular basis for the macroscopic expansion of an anthracene ester nanorod

Crystals composed of photoreactive molecules represent a new class of photomechanical materials with the potential to generate large forces on fast timescales. An example is the photodimerization of 9-tert-butyl-anthracene ester (9TBAE) in molecular crystal nanorods that leads to an average elongation of 8%. Previous work showed that this expansion results from the formation of a metastable crystalline product. In this article, it is shown how a novel combination of ensemble oriented-crystal solid-state NMR, X-ray diffraction, and first principles computational modeling can be used to establish the absolute unit cell orientations relative to the shape change, revealing the atomic-resolution mechanism for the photomechanical response and enabling the construction of a model that predicts an elongation of 7.4%, in good agreement with the experimental value. According to this model, the nanorod expansion does not result from an overall change in the volume of the unit cell, but rather from an anisotropic rearrangement of the molecular contents. The ability to understand quantitatively how molecular-level photochemistry generates mechanical displacements allows us to predict that the expansion could be tuned from +9% to −9.5% by controlling the initial orientation of the unit cell with respect to the nanorod axis. This application of NMR-assisted crystallography provides a new tool capable of tying the atomic-level structural rearrangement of the reacting molecular species to the mechanical response of a nanostructured sample

A Search for Light Fermionic Dark Matter Absorption on Electrons in PandaX-4T

We report a search on a sub-MeV fermionic dark matter absorbed by electrons with an outgoing active neutrino using the 0.63 tonne-year exposure collected by PandaX-4T liquid xenon experiment. No significant signals are observed over the expected background. The data are interpreted into limits to the effective couplings between such dark matter and electrons. For axial-vector or vector interactions, our sensitivity is competitive in comparison to existing astrophysical bounds on the decay of such dark matter into photon final states. In particular, we present the first direct detection limits for an axial-vector (vector) interaction which are the strongest in the mass range from 25 to 45 (35 to 50) keV/c$^2$

Hybrid Organic–Inorganic Photomechanical Actuators Based on Organized Molecular Crystals

Photomechanical actuators are smart materials that transform photon energy into mechanical energy without any wired connections. Photonmechanical molecular crystals have shown superior performance on nanoscopic and microscopic scales due to their high elastic moduli, high-energy density and fast response time compared with polymer actuators. However, the development of macroscopic crystal actuators that can be used in the real world has yet to be demonstrated.The work in this dissertation solves the problem by organizing photomechanical nanocrystals in porous inorganic templates Al2O3. It combines the best of both worlds where it is processable like polymers but it also attains high performance of organic crystals. 1,2-bis(2-methyl-5-phenyl-3-thienyl)perfluorocyclopentene, a diarylethene (DAE) derivative that undergoes a unimolecular photoisomerization between its ring-open and ring-closed isomers was used as the active organic element. The isomers are thermally stable but can be switched back and forth using UV and visible light for hundreds of cycles. Very small (sub-mg) amounts of material can move masses that are 105 times larger without being physically connected to an external power source. A light-to-work conversion efficiency for a molecular crystal photomechanical actuator has been for the first time quantitatively estimated, providing a starting point for future optimization.Additional studies investigated how both the organic and inorganic components could affect the photomechancial response. Two alumina templates with different morphology and elastic modulus were compared. The template with a lower average elastic modulus generate about one order of magnitude more photomechanical work. Two diarylethene derivatives with different crystal deformations were studied to determine the dependence of photomechanical response on chemical and crystal structure. The hybrid approach was then extended to porous SiO2 membranes with variable thicknesses in order to improve optical penetration and reproducibility. The photon-induced bending could be repeated for 150 cycles without loss of performance which suggests good fatigue resistance. The thickness dependence of the performance was studied, revealing thinner templates gave larger displacements. This work demonstrates that the hybrid method for making photomechanical actuators on macroscopic scales can be extended to other host materials and template geometries without sacrificing performance

Research on changes and circulations of sea-ice in Eurasian in recent 50 years

Using the monthly 1° x 1° sea-ice concentration data of Hadley center and the monthly NCEP geopotential height data from January 1953 to February 2003, temporal and spatial changing characters of sea-ice are examined. The results show almost all of the sea-ice of eight regions was decreasing, especially all seasons in Europe. But in Asia part, those display some increasing trends in spring and winter. Abrupt times of sea-ice in Europe were at end of 1970's and in Asia the times in summer/fall (spring/winter) were at end of 1980's

Performance of Composite Glass–Diarylethene Crystal Photomechanical Actuator Membranes

Hybrid organic–inorganic composites based on organic photochromic crystals embedded in inorganic templates provide a new approach to photomechanical materials. Diarylethene (DAE) nanowire crystals grown in Al2O3 membranes have exhibited reversible photoinduced bending and lifting [Dong, X., Chem. Mater. 2019, 31, 1016−1022]. In this paper, the hybrid approach is extended to porous SiO2 membranes. Despite the different template material (SiO2 instead of Al2O3) and much larger channels (5 μm diameter instead of 0.2 μm diameter), similar photomechanical behavior is observed for this new class of organic–inorganic hybrid actuators. The ability to reuse individual glass templates across different DAE filling cycles allows us to show that the DAE filling step is crucial for determining the mechanical work done by the bending template. The bending curvature also depends quadratically on the template thickness, in good agreement with theory. The light-induced bending can be repeated for up to 150 cycles without loss of performance, suggesting good fatigue resistance. The results in this paper demonstrate that the hybrid organic–inorganic approach can be extended to other host materials and template geometries. They also suggest that optimizing the organic filling and template thickness could improve the work output by an order of magnitude

Using light intensity to control reaction kinetics and reversibility in photomechanical crystals.

4-Fluoro-9-anthracenecarboxylic acid (4F-9AC) is a thermally reversible (T-type) photomechanical molecular crystal. The photomechanical response is driven by a [4 + 4] photodimerization reaction, while the photodimer dissociation determines the reset time. In this paper, both the chemical kinetics of dimer dissociation (using a microscopic fluorescence-recovery-after-photobleaching experiment) and mechanical reset dynamics (by imaging bending microneedles) for single 4F-9AC crystals are measured. The dissociation kinetics depend strongly on the initial concentration of photodimer, slowing down and becoming nonexponential at high dimer concentrations. This dose-dependent behavior is also observed in the mechanical response of bending microneedles. A new feature in the photomechanical behavior is identified: the ability of a very weak control beam to suppress dimer dissociation after large initial dimer conversions. This phenomenon provides a way to optically control the mechanical response of this photomechanical crystal. To gain physical insight into the origin of the nonexponential recovery curves, the experimental results are analyzed in terms of a standard first-order kinetic model and a nonlinear Finke-Watzky (FW) model. The FW model can qualitatively reproduce the transition from exponential to sigmoidal recovery with larger initial conversions, but neither model can reproduce the suppression of the recovery in the presence of a weak holding beam. These results highlight the need for more sophisticated theories to describe cooperative phenomena in solid-state crystalline reactions, as well as demonstrating how this behavior could lead to new properties and/or improved performance in photomechanical materials