Composing Efficient, Robust Tests for Policy Selection

Modern reinforcement learning systems produce many high-quality policies throughout the learning process. However, to choose which policy to actually deploy in the real world, they must be tested under an intractable number of environmental conditions. We introduce RPOSST, an algorithm to select a small set of test cases from a larger pool based on a relatively small number of sample evaluations. RPOSST treats the test case selection problem as a two-player game and optimizes a solution with provable $k$-of-$N$ robustness, bounding the error relative to a test that used all the test cases in the pool. Empirical results demonstrate that RPOSST finds a small set of test cases that identify high quality policies in a toy one-shot game, poker datasets, and a high-fidelity racing simulator.Comment: 26 pages, 13 figures. To appear in Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI 2023

Electron Transport through Disordered Domain Walls: Coherent and Incoherent Regimes

We study electron transport through a domain wall in a ferromagnetic nanowire subject to spin-dependent scattering. A scattering matrix formalism is developed to address both coherent and incoherent transport properties. The coherent case corresponds to elastic scattering by static defects, which is dominant at low temperatures, while the incoherent case provides a phenomenological description of the inelastic scattering present in real physical systems at room temperature. It is found that disorder scattering increases the amount of spin-mixing of transmitted electrons, reducing the adiabaticity. This leads, in the incoherent case, to a reduction of conductance through the domain wall as compared to a uniformly magnetized region which is similar to the giant magnetoresistance effect. In the coherent case, a reduction of weak localization, together with a suppression of spin-reversing scattering amplitudes, leads to an enhancement of conductance due to the domain wall in the regime of strong disorder. The total effect of a domain wall on the conductance of a nanowire is studied by incorporating the disordered regions on either side of the wall. It is found that spin-dependent scattering in these regions increases the domain wall magnetoconductance as compared to the effect found by considering only the scattering inside the wall. This increase is most dramatic in the narrow wall limit, but remains significant for wide walls.Comment: 23 pages, 12 figure

Relative Roles of Climate Sensitivity and Forcing in Defining the Ocean Circulation Response to Climate Change

Abstract in HTML and technical report in PDF available on the Massachusetts Institute of Technology Joint Program on the Science and Policy of Global Change website (http://mit.edu/globalchange/www/).The response of the ocean’s meridional overturning circulation (MOC) to increased greenhouse gas forcing is examined using a coupled model of intermediate complexity, including a dynamic 3D ocean subcomponent. Parameters are the increase in CO2 forcing (with stabilization after a specified time interval) and the model’s climate sensitivity. In this model, the cessation of deep sinking in the north “Atlantic” (hereinafter, a “collapse”), as indicated by changes in the MOC, behaves like a simple bifurcation. The final surface air temperature (SAT) change, which is closely predicted by the product of the radiative forcing and the climate sensitivity, determines whether a collapse occurs. The initial transient response in SAT is largely a function of the forcing increase, with higher sensitivity runs exhibiting delayed behavior; accordingly, high CO2-low sensitivity scenarios can be assessed as a recovering or collapsing circulation shortly after stabilization, whereas low CO2-high sensitivity scenarios require several hundred additional years to make such a determination. We also systemically examine how the rate of forcing, for a given CO2 stabilization, affects the ocean response. In contrast with previous studies based on results using simpler ocean models, we find that except for a narrow range of marginally stable to marginally unstable scenarios, the forcing rate has little impact on whether the run collapses or recovers. In this narrow range, however, forcing increases on a time scale of slow ocean advective processes results in weaker declines in overturning strength and can permit a run to recover that would otherwise collapse.This research was supported in part by the Methods and Models for Integrated Assessments Program of the National Science Foundation, Grant ATM-9909139, by the Office of Science (BER), U.S. Department of Energy, Grant No. DE-FG02-93ER61677, and by the MIT Joint Program on the Science and Policy of Global Change (JPSPGC)

Fracture toughness of the cancellous bone of FNF femoral heads in relation to its microarchitecture

This study considers the relationship between microarchitecture and mechanical properties for cancellous bone specimens collected from a cohort of patients who had suffered fractured necks of femur. OP is an acute skeletal condition with huge socioeconomic impact [1] and it is associated with changes in both bone quantity and quality [2], which affect greatly the strength and toughness of the tissue [3].Support was provided by the EPSRC (EP/K020196: Point-ofCare High Accuracy Fracture Risk Prediction), the UK Department of Transport under the BOSCOS (Bone Scanning for Occupant Safety) project, and approved by Gloucester and Cheltenham NHS Trust hospitals under ethical consent (BOSCOS – Mr. Curwen CI REC ref 01/179G)

The Putative Role of Resveratrol in SIRT-1 Mediated Modulation of the Vitamin D Pathway

The nuclear vitamin D receptor (VDR) modulates gene transcription in 1,25-dihydroxyvitamin D (1,25D) target tissues such as kidney, colon, and bone. The 1,25D hormone is derived from vitamin D in the skin or from the diet, and binds to and activates the VDR. We have previously shown that resveratrol, an antioxidant found in the skin of red grapes, has the ability to activate the VDR signaling pathway. Moreover, cells treated with both resveratrol and 1,25D resulted in an additive or even synergistic stimulation of VDR-mediated transcription compared to cells treated with 1,25D alone. Based on these initial results, experiments were designed to test the significance of mutations in the hormone-binding domain of VDR. Identical hormone treatments were applied to “wild-type” (non-mutated) and single point VDR mutations. 1,25D displayed a significant drop in activity caused by these mutations, while the ability of resveratrol to activate VDR was only modestly attenuated. One possible interpretation of these results is that resveratrol may affect VDR activity indirectly, perhaps via the ability of resveratrol to activate SIRT1, an enzyme which has been shown to deacetylate (and thereby activate) other nuclear receptors such as the liver X receptor (LXR). In support of this hypothesis, radiolabeled 1,25D displacement assays revealed an increase in bound radiolabeled 1,25D only in the presence of resveratrol, suggesting that direct binding of resveratrol to VDR is unlikely. Additionally, we observed increased transcriptional activity in response to resveratrol in a subset of other nuclear receptors, including the liver X receptor (LXR), which is closely related to VDR and is known to be deacetylated by SIRT1. Finally, we tested receptor-mediated transcriptional activity in a system containing VDR in the absence and presence of overexpressed SIRT1. Transcriptional activity was higher in cells expressing SIRT1, and synergistic activity of 1,25D combined with resveratrol was observed. We are currently conducting additional experiments employing the VDR/SIRT1 assay in multiple cellular contexts. In conclusion, this study elucidates, for the first time, a potential novel pathway for crosstalk between two nutritionally derived lipids, vitamin D and resveratrol

Agent-Based Markov Modeling for Improved COVID-19 Mitigation Policies

You are currently viewing a research paper that was included in the August 2021 Good Systems Network Digest.Office of the VP for Researc

Empowerment for Continuous Agent-Environment Systems

This paper develops generalizations of empowerment to continuous states. Empowerment is a recently introduced information-theoretic quantity motivated by hypotheses about the efficiency of the sensorimotor loop in biological organisms, but also from considerations stemming from curiosity-driven learning. Empowemerment measures, for agent-environment systems with stochastic transitions, how much influence an agent has on its environment, but only that influence that can be sensed by the agent sensors. It is an information-theoretic generalization of joint controllability (influence on environment) and observability (measurement by sensors) of the environment by the agent, both controllability and observability being usually defined in control theory as the dimensionality of the control/observation spaces. Earlier work has shown that empowerment has various interesting and relevant properties, e.g., it allows us to identify salient states using only the dynamics, and it can act as intrinsic reward without requiring an external reward. However, in this previous work empowerment was limited to the case of small-scale and discrete domains and furthermore state transition probabilities were assumed to be known. The goal of this paper is to extend empowerment to the significantly more important and relevant case of continuous vector-valued state spaces and initially unknown state transition probabilities. The continuous state space is addressed by Monte-Carlo approximation; the unknown transitions are addressed by model learning and prediction for which we apply Gaussian processes regression with iterated forecasting. In a number of well-known continuous control tasks we examine the dynamics induced by empowerment and include an application to exploration and online model learning

Intervention to Lower Household Wood Smoke Exposure in Guatemala Reduces ST-Segment Depression on Electrocardiograms

Background: A large body of evidence suggests that fine particulate matter (PM) air pollution is a cause of cardiovascular disease, but little is known in particular about the cardiovascular effects of indoor air pollution from household use of solid fuels in developing countries. RESPIRE (Randomized Exposure Study of Pollution Indoors and Respiratory Effects) was a randomized trial of a chimney woodstove that reduces wood smoke exposure. Objectives: We tested the hypotheses that the stove intervention, compared with open fire use, would reduce ST-segment depression and increase heart rate variability (HRV). Methods: We used two complementary study designs: a) between-groups comparisons based on randomized stove assignment, and b) before-and-after comparisons within control subjects who used open fires during the trial and received chimney stoves after the trial. Electrocardiogram sessions that lasted 20 hr were repeated up to three times among 49 intervention and 70 control women 38–84 years of age, and 55 control subjects were also assessed after receiving stoves. HRV and ST-segment values were assessed for each 30-min period. ST-segment depression was defined as an average value below –1.00 mm. Personal fine PM [aerodynamic diameter ≤ 2.5 μm (PM$_{2.5}$)] exposures were measured for 24 hr before each electrocardiogram. Results: PM$_{2.5}$ exposure means were 266 and 102 μg/m$^3$ during the trial period in the control and intervention groups, respectively. During the trial, the stove intervention was associated with an odds ratio of 0.26 (95% confidence interval, 0.08–0.90) for ST-segment depression. We found similar associations with the before-and-after comparison. The intervention was not significantly associated with HRV. Conclusions: The stove intervention was associated with reduced occurrence of nonspecific ST-segment depression, suggesting that household wood smoke exposures affect ventricular repolarization and potentially cardiovascular health