Risk-Sensitive Reinforcement Learning: A Constrained Optimization Viewpoint

The classic objective in a reinforcement learning (RL) problem is to find a policy that minimizes, in expectation, a long-run objective such as the infinite-horizon discounted or long-run average cost. In many practical applications, optimizing the expected value alone is not sufficient, and it may be necessary to include a risk measure in the optimization process, either as the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., mean-variance tradeoff, exponential utility, the percentile performance, value at risk, conditional value at risk, prospect theory and its later enhancement, cumulative prospect theory. In this article, we focus on the combination of risk criteria and reinforcement learning in a constrained optimization framework, i.e., a setting where the goal to find a policy that optimizes the usual objective of infinite-horizon discounted/average cost, while ensuring that an explicit risk constraint is satisfied. We introduce the risk-constrained RL framework, cover popular risk measures based on variance, conditional value-at-risk and cumulative prospect theory, and present a template for a risk-sensitive RL algorithm. We survey some of our recent work on this topic, covering problems encompassing discounted cost, average cost, and stochastic shortest path settings, together with the aforementioned risk measures in a constrained framework. This non-exhaustive survey is aimed at giving a flavor of the challenges involved in solving a risk-sensitive RL problem, and outlining some potential future research directions

Partial alignment and measurement of residual dipolar couplings of proteins under high hydrostatic pressure

High-pressure NMR spectroscopy has emerged as a complementary approach for investigating various structural and thermodynamic properties of macromolecules. Noticeably absent from the array of experimental restraints that have been employed to characterize protein structures at high hydrostatic pressure is the residual dipolar coupling, which requires the partial alignment of the macromolecule of interest. Here we examine five alignment media that are commonly used at ambient pressure for this purpose. We find that the spontaneous alignment of Pf1 phage, d(GpG) and a C12E5/n-hexnanol mixture in a magnetic field is preserved under high hydrostatic pressure. However, DMPC/ DHPC bicelles and collagen gel are found to be unsuitable. Evidence is presented to demonstrate that pressure-induced structural changes can be identified using the residual dipolar coupling

One-step error correction for multipartite polarization entanglement

We present two economical one-step error-correction protocols for multipartite polarization-entangled systems in a Greenberger-Horne-Zeilinger state. One uses spatial entanglement to correct errors in the polarization entanglement of an N-photon system, resorting to linear optical elements. The other uses frequency entanglement to correct errors in the polarization entanglement of an N-photon system. The parties in quantum communication can obtain a maximally entangled state from each N-photon system transmitted with one step in these two protocols, and both of their success probabilities are 100%, in principle. That is, they both work in a deterministic way, and they do not largely consume the less-entangled photon systems, which is far different from conventional multipartite entanglement purification schemes. These features may make these two protocols more useful for practical applications in long-distance quantum communication.Comment: 8 pages, 2 figure

Analytical Mechanics of Chemical Reactions. V. Application to the Linear Reactive H +H_2 Systems

Natural collision coordinates and a zeroth‐order vibrational–adiabatic approximation are used to treat linear reactive collisions. Nonadiabatic effects on barrier transmission and on vibrational state of products are calculated. The present results are classical and are compared with exact classical numerical results for the H+H_2 reaction in the range 7–20 kcal/mol of initial relative translational energy. The agreement is encouraging and the results support the concepts introduced earlier of statistical adiabaticity and of nonadiabatic leak. At low energies the reaction is adiabatic on the average (initial vibrational phase average), thus justifying activated complex theory for this system. The relative importance of reaction path curvature and of vibrational frequency variation along the reaction path in inducing nonadiabatic effects is described. Implications for a quantum treatment, activated complex theory, and highly nonadiabatic systems are noted

The flavour asymmetry of polarized anti-quarks in the nucleon

We present a study of the flavour asymmetry of polarized anti-quarks in the nucleon using the meson cloud model. We include contributions both from the vector mesons and the interference terms of pseudoscalar and vector mesons. Employing the bag model, we first give the polarized valence quark distribution of the $\rho$ meson and the interference distributions. Our calculations show that the interference effect mildly increases the prediction for \Delta \dbar(x)-\Delta \ubar(x) at intermediate $x$ region. We also discuss the contribution of `Pauli blocking' to the asymmetry.Comment: 22 pages, LaTex, 5 PS figures. Version to appear in Eur. Phys. J. C. An appendix is added for expressions for the helicity dependent fluctuation functions. An error in the programme for fluctuation function f_{(\pi\rho)\Delta /N} is corrected, which increases numerical results by about 10%. Unchanged conclusion

Analytical Mechanics of Chemical Reactions. VI. Rotational and Vibrational Distributions of the H + H_2 Reaction in a Plane

Rotational and vibrational distributions for the exchange reaction H+H_(2)→H_(2)+H are obtained numerically for reaction in a plane and compared with the vibrationally adiabatic solutions. Evidence is obtained regarding the two adiabatic solutions for the final rotational state predicted earlier and for the disappearance of one of these at higher initial relative velocity. Good agreement between calculations based on natural collision coordinates and these based on Cartesian coordinates was found, where tested. The vibrational motion for reaction in a plane is fairly adiabatic on the average at energies of thermal interest. Connections with other properties are noted. The results support the idea of statistical adiabaticity suggested earlier in this series and thus support a derivation of activated complex theory based on this concept

Primordial perturbation with a modified dispersion relation

In this paper we study the generation of primordial perturbation with a modified dispersion relation in various cosmological evolutions. We stress that the formation of the power spectrum is strongly dependent on the background. Working in a bounce model with a matter-like contracting phase, we obtain a red tilt spectrum due to the modified dispersion relation.Comment: 6 pages, 2 figures, 1 tabl

Thermal-stress analysis for a wood composite blade

A thermal-stress analysis of a wind turbine blade made of wood composite material is reported. First, the governing partial differential equation on heat conduction is derived, then, a finite element procedure using variational approach is developed for the solution of the governing equation. Thus, the temperature distribution throughout the blade is determined. Next, based on the temperature distribution, a finite element procedure using potential energy approach is applied to determine the thermal-stress distribution. A set of results is obtained through the use of a computer, which is considered to be satisfactory. All computer programs are contained in the report

Single-Site Vanadyl Species Isolated within Molybdenum Oxide Monolayers in Propane Oxidation

The cooperation of metal oxide subunits in complex mixed metal oxide catalysts for selective oxidation of alkanes still needs deeper understanding to allow for a rational tuning of catalyst performance. Herein we analyze the interaction between vanadium and molybdenum oxide species in a monolayer supported on mesoporous silica SBA-15. Catalysts with variable Mo/V ratio between 10 and 1 were studied in the oxidation of propane and characterized by FTIR, Raman, and EPR spectroscopies, temperature-programmed reduction, UV/vis spectroscopy in combination with DFT calculations, and time-resolved experiments to analyze the redox properties of the catalysts. Molybdenum oxide (sub)monolayers on silica contain mainly dioxo (Si–O−)2Mo(═O)2 species. Dilution of silica-supported vanadium oxide species by (Si–O−)2Mo(═O)2 prevents the formation of V–O–V bonds, which are abundant in the pure vanadium oxide catalyst that predominantly contains two-dimensional vanadium oxide oligomers. Existing single vanadyl (Si–O−)3V(═O) sites and neighboring (Si–O−)2Mo(═O)2 sites do not strongly interact. The rates of reduction in propane and of oxidation in oxygen are lower for single metal oxide sites compared to those for oligomers. The rate of propane oxidation correlates with the overall redox rates of the catalysts but not linearly with the chemical composition. Retarded redox behavior facilitates selectivity toward acrolein on single-site catalysts. The abundance of M–O–M bonds is more important in terms of activity and selectivity compared to the nature of the central atom (molybdenum versus vanadium)