Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi.Comment: Accepted at the European Conference on Machine Learning 2019 (ECML

\u3cem\u3eRhizobium japonicum\u3c/em\u3e Mutants Defective in Symbiotic Nitrogen Fixation

Rhizobium japonicum strains 3I1b110 and 61A76 were mutagenized to obtain 25 independently derived mutants that produced soybean nodules defective in nitrogen fixation, as assayed by acetylene reduction. The proteins of both the bacterial and the plant portions of the nodules were analyzed by two-dimensional polyacrylamide gel electrophoresis. All of the mutants had lower-than-normal levels of the nitrogenase components, and all but four contained a prominent bacteroid protein not observed in wild-type bacteroids. Experiments with bacteria grown ex planta suggested that this protein was derepressed by the absence of ammonia. Nitrogenase component II of one mutant was altered in isoelectric point. The soluble plant fraction of the nodules of seven mutants had very low levels of heme, yet the nodules of five of these seven mutants contained the polypeptide of leghemoglobin. Thus, the synthesis of the globin may not be coupled to the content of available heme in soybean nodules. The nodules of the other two of these seven mutants lacked not only leghemoglobin but most of the other normal plant and bacteroid proteins. Ultrastructural examination of nodules formed by these two mutants indicated normal ramification of infection threads but suggested a problem in subsequent survival of the bacteria and their release from the infection threads

Fe XVII X-ray Line Ratios for Accurate Astrophysical Plasma Diagnostics

New laboratory measurements using an Electron Beam Ion Trap (EBIT) and an x-ray microcalorimeter are presented for the n=3 to n=2 Fe XVII emission lines in the 15 {\AA} to 17 {\AA} range, along with new theoretical predictions for a variety of electron energy distributions. This work improves upon our earlier work on these lines by providing measurements at more electron impact energies (seven values from 846 to 1185 eV), performing an in situ determination of the x-ray window transmission, taking steps to minimize the ion impurity concentrations, correcting the electron energies for space charge shifts, and estimating the residual electron energy uncertainties. The results for the 3C/3D and 3s/3C line ratios are generally in agreement with the closest theory to within 10%, and in agreement with previous measurements from an independent group to within 20%. Better consistency between the two experimental groups is obtained at the lowest electron energies by using theory to interpolate, taking into account the significantly different electron energy distributions. Evidence for resonance collision effects in the spectra is discussed. Renormalized values for the absolute cross sections of the 3C and 3D lines are obtained by combining previously published results, and shown to be in agreement with the predictions of converged R-matrix theory. This work establishes consistency between results from independent laboratories and improves the reliability of these lines for astrophysical diagnostics. Factors that should be taken into account for accurate diagnostics are discussed, including electron energy distribution, polarization, absorption/scattering, and line blends.Comment: 29 pages, including 7 figure

Redox Fluctuations Control the Coupled Cycling of Iron and Carbon in Tropical Forest Soils.

Oscillating redox conditions are a common feature of humid tropical forest soils, driven by an ample supply and dynamics of reductants, high moisture, microbial oxygen consumption, and finely textured clays that limit diffusion. However, the net result of variable soil redox regimes on iron (Fe) mineral dynamics and associated carbon (C) forms and fluxes is poorly understood in tropical soils. Using a 44-day redox incubation experiment with humid tropical forest soils from Puerto Rico, we examined patterns in Fe and C transformations under four redox regimes: static anoxic, "flux 4-day" (4d oxic, 4d anoxic), "flux 8-day" (8d oxic, 4d anoxic) and static oxic. Prolonged anoxia promoted reductive dissolution of Fe-oxides, and led to an increase in soluble Fe(II) and amorphous Fe oxide pools. Preferential dissolution of the less-crystalline Fe pool was evident immediately following a shift in bulk redox status (oxic to anoxic), and coincided with increased dissolved organic C, presumably due to acidification or direct release of organic matter (OM) from dissolving Fe(III) mineral phases. The average nominal oxidation state of water-soluble C was lowest under persistent anoxic conditions, suggesting that more reduced organic compounds were metabolically unavailable for microbial consumption under reducing conditions. Anoxic soil compounds had high H/C values (and were similar to lignin-like compounds) whereas oxic soil compounds had higher O/C values, akin to tannin- and cellulose-like components. Cumulative respiration derived from native soil organic C was highest in static oxic soils. These results show how Fe minerals and Fe-OM interactions in tropical soils are highly sensitive to variable redox effects. Shifting soil oxygen availability rapidly impacted exchanges between mineral-sorbed and aqueous C pools, increased the dissolved organic C pool under anoxic conditions implying that the periodicity of low-redox events may control the fate of C in wet tropical soils

Problems of Proof for the Ban on Female Athletes with Endogenously High Testosterone Levels

At the time of this writing, a new International Association of Athletics Federations regulation preventing women with naturally high testosterone from competing in certain international athletics events has reignited the controversy over the male-female distinction in sports and its implications on individuals’ right to compete. A recent case filed by runner Caster Semenya and Athletics South Africa challenging this regulation before the Court of Arbitration for Sport, an arbitral tribunal that adjudicates disputes in international sports, sought to have the regulation overturned as discriminatory against women with a genetic intersex condition. Drawing on established international arbitration law, international norms in arbitrations, and relevant precedent, this Comment explores the evidentiary issues before the Court of Arbitration for Sport in Semenya’s challenge. In particular, this Comment argues that, given the high stakes of the case as well as the inequity in resources between the parties, the Court of Arbitration for Sport should have adopted unconventional rules with respect to the allocation of the burden of proof, the requisite standard of proof, and the evaluation of scientific evidence to ensure a fair hearing on the matter. The Comment ultimately concludes that the suggested changes are well within the discretion and ability of the Court of Arbitration for Sport to implement, slight challenges to the adoption of each proposed measure notwithstanding

Influence of next-nearest-neighbor electron hopping on the static and dynamical properties of the 2D Hubbard model

Comparing experimental data for high temperature cuprate superconductors with numerical results for electronic models, it is becoming apparent that a hopping along the plaquette diagonals has to be included to obtain a quantitative agreement. According to recent estimations the value of the diagonal hopping $t'$ appears to be material dependent. However, the values for $t'$ discussed in the literature were obtained comparing theoretical results in the weak coupling limit with experimental photoemission data and band structure calculations. The goal of this paper is to study how $t'$ gets renormalized as the interaction between electrons, $U$, increases. For this purpose, the effect of adding a bare diagonal hopping $t'$ to the fully interacting two dimensional Hubbard model Hamiltonian is investigated using numerical techniques. Positive and negative values of $t'$ are analyzed. Spin-spin correlations, $n(\bf{k})$, $\langle n\rangle$ vs $\mu$, and local magnetic moments are studied for values of $U/t$ ranging from 0 to 6, and as a function of the electronic density. The influence of the diagonal hopping in the spectral function $A(\bf{k},\omega)$ is also discussed, and the changes in the gap present in the density of states at half-filling are studied. We introduce a new criterion to determine probable locations of Fermi surfaces at zero temperature from $n(\bf{k})$ data obtained at finite temperature. It appears that hole pockets at ${\bf{k}}=(\pi/2,\pi/2)$ may be induced for negative $t'$ while a positive $t'$ produces similar features at ${\bf{k}}=(\pi,0)$ and $(0,\pi)$. Comparisons with the standard 2D Hubbard ($t'=0$) model indicate that a negative $t'$ hopping amplitude appears to be dynamically generated. In general, we conclude that it is very dangerous to extract a bare parameter of the Hamiltonian $(t')$ from PES data whereComment: 9 pages (RevTex 3.0), 12 figures (postscript), files packed with uufile

Toward a first-principles integrated simulation of tokamak edge plasmas

Performance of the ITER is anticipated to be highly sensitive to the edge plasma condition. The edge pedestal in ITER needs to be predicted from an integrated simulation of the necessary first-principles, multi-scale physics codes. The mission of the SciDAC Fusion Simulation Project (FSP) Prototype Center for Plasma Edge Simulation (CPES) is to deliver such a code integration framework by (1) building new kinetic codes XGC0 and XGC1, which can simulate the edge pedestal buildup; (2) using and improving the existing MHD codes ELITE, M3D-OMP, M3D-MPP and NIMROD, for study of large-scale edge instabilities called Edge Localized Modes (ELMs); and (3) integrating the codes into a framework using cutting-edge computer science technology. Collaborative effort among physics, computer science, and applied mathematics within CPES has created the first working version of the End-to-end Framework for Fusion Integrated Simulation (EFFIS), which can be used to study the pedestal-ELM cycles

Parsec-scale Properties of Brightest Cluster Galaxies

We present new VLBI observations at 5 GHz of a complete sample of Brightest Cluster Galaxies (BCGs) in nearby Abell Clusters (distance class <3). Combined with data from the literature, this provides parsec-scale information for 34 BCGs. Our analysis of their parsec scale radio emission and cluster X-ray properties shows a possible dichotomy between BCGs in cool core clusters and those in non cool core clusters. Among resolved sources, those in cool core clusters tend to have two-sided parsec-scale jets, while those in less relaxed clusters have predominantly one-sided parsec-scale jets. We suggest that this difference could be the result of interplay between the jets and the surrounding medium. The one-sided structure in non cool core clusters could be due to Doppler boosting effects in relativistic, intrinsically symmetric jets; two-sided morphology in cool core clusters is likely related to the presence of heavy and mildly relativistic jets slowed down on the parsec-scale. Evidence of recurrent activity are also found in BCGs in cool core clusters.Comment: 20 pages, 10 figures, accepted for publication in A&

K-shell dielectronic resonances in photoabsorption: differential oscillator strengths for Li-like C IV, O VI, and Fe XXIV

Recently X-ray photoabsorption in KLL resonances of O VI was predicted [Pradhan, Astrophys.J. Lett. 545, L165 (2000)], and detected by the Chandra X-ray Observatory [Lee et al, Astrophys. J. {\it Lett.}, submitted]. The required resonance oscillator strengths f_r, are evaluated in terms of the differential oscillator strength df/de that relates bound and continuum absorption. We present the f_r values from radiatively damped and undamped photoionization cross sections for Li-like C,O, and Fe calculated using relativistic close coupling Breit-Pauli R-matrix method. The KLL resonances of interest here are: 1s2p (^3P^o) 2s [^4P^o_{1/2,3/2}, ^2P^o_{1/2,3/2}] and 1s2p (^1P^o) 2s [^2P^o_{1/2,3/2}]. The KLL photoabsorption resonances in Fe XXIV are fully resolved up to natural autoionization profiles for the first time. It is demonstrated that the undamped f_r independently yield the resonance radiative decay rates, and thereby provide a precise check on the resolution of photoionization calculations in general. The predicted photoabsorption features should be detectable by the X-ray space observatories and enable column densities in highly ionized astrophysical plasmas to be determined from the calculated f_r. The dielectronic satellites may appear as redward broadening of resonances lines in emission and absorption.Comment: 9 pages, 2 figurs, Phys. Rev. A, Rapid Communication (submitted