Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?

Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during training if most of the training data contain such spurious correlations. This phenomenon is particularly pronounced in domains such as robotics, with potentially large gaps between the open- and closed-loop performance of an agent. In such settings, causally confused models may appear to perform well according to open-loop metrics during training but fail catastrophically when deployed in the real world. In this paper, we study causal confusion in offline reinforcement learning. We investigate whether selectively sampling appropriate points from a dataset of demonstrations may enable offline reinforcement learning agents to disambiguate the underlying causal mechanisms of the environment, alleviate causal confusion in offline reinforcement learning, and produce a safer model for deployment. To answer this question, we consider a set of tailored offline reinforcement learning datasets that exhibit causal ambiguity and assess the ability of active sampling techniques to reduce causal confusion at evaluation. We provide empirical evidence that uniform and active sampling techniques are able to consistently reduce causal confusion as training progresses and that active sampling is able to do so significantly more efficiently than uniform sampling.Comment: Published in Proceedings of the 2nd Conference on Causal Learning and Reasoning (CLeaR 2021

Adding 6 months of androgen deprivation therapy to postoperative radiotherapy for prostate cancer: a comparison of short-course versus no androgen deprivation therapy in the RADICALS-HD randomised controlled trial

Background Previous evidence indicates that adjuvant, short-course androgen deprivation therapy (ADT) improves metastasis-free survival when given with primary radiotherapy for intermediate-risk and high-risk localised prostate cancer. However, the value of ADT with postoperative radiotherapy after radical prostatectomy is unclear. Methods RADICALS-HD was an international randomised controlled trial to test the efficacy of ADT used in combination with postoperative radiotherapy for prostate cancer. Key eligibility criteria were indication for radiotherapy after radical prostatectomy for prostate cancer, prostate-specific antigen less than 5 ng/mL, absence of metastatic disease, and written consent. Participants were randomly assigned (1:1) to radiotherapy alone (no ADT) or radiotherapy with 6 months of ADT (short-course ADT), using monthly subcutaneous gonadotropin-releasing hormone analogue injections, daily oral bicalutamide monotherapy 150 mg, or monthly subcutaneous degarelix. Randomisation was done centrally through minimisation with a random element, stratified by Gleason score, positive margins, radiotherapy timing, planned radiotherapy schedule, and planned type of ADT, in a computerised system. The allocated treatment was not masked. The primary outcome measure was metastasis-free survival, defined as distant metastasis arising from prostate cancer or death from any cause. Standard survival analysis methods were used, accounting for randomisation stratification factors. The trial had 80% power with two-sided α of 5% to detect an absolute increase in 10-year metastasis-free survival from 80% to 86% (hazard ratio [HR] 0·67). Analyses followed the intention-to-treat principle. The trial is registered with the ISRCTN registry, ISRCTN40814031, and ClinicalTrials.gov, NCT00541047. Findings Between Nov 22, 2007, and June 29, 2015, 1480 patients (median age 66 years [IQR 61–69]) were randomly assigned to receive no ADT (n=737) or short-course ADT (n=743) in addition to postoperative radiotherapy at 121 centres in Canada, Denmark, Ireland, and the UK. With a median follow-up of 9·0 years (IQR 7·1–10·1), metastasis-free survival events were reported for 268 participants (142 in the no ADT group and 126 in the short-course ADT group; HR 0·886 [95% CI 0·688–1·140], p=0·35). 10-year metastasis-free survival was 79·2% (95% CI 75·4–82·5) in the no ADT group and 80·4% (76·6–83·6) in the short-course ADT group. Toxicity of grade 3 or higher was reported for 121 (17%) of 737 participants in the no ADT group and 100 (14%) of 743 in the short-course ADT group (p=0·15), with no treatment-related deaths. Interpretation Metastatic disease is uncommon following postoperative bed radiotherapy after radical prostatectomy. Adding 6 months of ADT to this radiotherapy did not improve metastasis-free survival compared with no ADT. These findings do not support the use of short-course ADT with postoperative radiotherapy in this patient population

Duration of androgen deprivation therapy with postoperative radiotherapy for prostate cancer: a comparison of long-course versus short-course androgen deprivation therapy in the RADICALS-HD randomised trial

Background Previous evidence supports androgen deprivation therapy (ADT) with primary radiotherapy as initial treatment for intermediate-risk and high-risk localised prostate cancer. However, the use and optimal duration of ADT with postoperative radiotherapy after radical prostatectomy remains uncertain. Methods RADICALS-HD was a randomised controlled trial of ADT duration within the RADICALS protocol. Here, we report on the comparison of short-course versus long-course ADT. Key eligibility criteria were indication for radiotherapy after previous radical prostatectomy for prostate cancer, prostate-specific antigen less than 5 ng/mL, absence of metastatic disease, and written consent. Participants were randomly assigned (1:1) to add 6 months of ADT (short-course ADT) or 24 months of ADT (long-course ADT) to radiotherapy, using subcutaneous gonadotrophin-releasing hormone analogue (monthly in the short-course ADT group and 3-monthly in the long-course ADT group), daily oral bicalutamide monotherapy 150 mg, or monthly subcutaneous degarelix. Randomisation was done centrally through minimisation with a random element, stratified by Gleason score, positive margins, radiotherapy timing, planned radiotherapy schedule, and planned type of ADT, in a computerised system. The allocated treatment was not masked. The primary outcome measure was metastasis-free survival, defined as metastasis arising from prostate cancer or death from any cause. The comparison had more than 80% power with two-sided α of 5% to detect an absolute increase in 10-year metastasis-free survival from 75% to 81% (hazard ratio [HR] 0·72). Standard time-to-event analyses were used. Analyses followed intention-to-treat principle. The trial is registered with the ISRCTN registry, ISRCTN40814031, and ClinicalTrials.gov , NCT00541047 . Findings Between Jan 30, 2008, and July 7, 2015, 1523 patients (median age 65 years, IQR 60–69) were randomly assigned to receive short-course ADT (n=761) or long-course ADT (n=762) in addition to postoperative radiotherapy at 138 centres in Canada, Denmark, Ireland, and the UK. With a median follow-up of 8·9 years (7·0–10·0), 313 metastasis-free survival events were reported overall (174 in the short-course ADT group and 139 in the long-course ADT group; HR 0·773 [95% CI 0·612–0·975]; p=0·029). 10-year metastasis-free survival was 71·9% (95% CI 67·6–75·7) in the short-course ADT group and 78·1% (74·2–81·5) in the long-course ADT group. Toxicity of grade 3 or higher was reported for 105 (14%) of 753 participants in the short-course ADT group and 142 (19%) of 757 participants in the long-course ADT group (p=0·025), with no treatment-related deaths. Interpretation Compared with adding 6 months of ADT, adding 24 months of ADT improved metastasis-free survival in people receiving postoperative radiotherapy. For individuals who can accept the additional duration of adverse effects, long-course ADT should be offered with postoperative radiotherapy. Funding Cancer Research UK, UK Research and Innovation (formerly Medical Research Council), and Canadian Cancer Society

Population Properties of Compact Objects from the Second LIGO–Virgo Gravitational-Wave Transient Catalog

Abstract: We report on the population of 47 compact binary mergers detected with a false-alarm rate of < in the second LIGO–Virgo Gravitational-Wave Transient Catalog. We observe several characteristics of the merging binary black hole (BBH) population not discernible until now. First, the primary mass spectrum contains structure beyond a power law with a sharp high-mass cutoff; it is more consistent with a broken power law with a break at or a power law with a Gaussian feature peaking at (90% credible interval). While the primary mass distribution must extend to or beyond, only of systems have primary masses greater than . Second, we find that a fraction of BBH systems have component spins misaligned with the orbital angular momentum, giving rise to precession of the orbital plane. Moreover, %– % of BBH systems have spins tilted by more than 90°, giving rise to a negative effective inspiral spin parameter, . Under the assumption that such systems can only be formed by dynamical interactions, we infer that between 25% and 93% of BBHs with nonvanishing are dynamically assembled. Third, we estimate merger rates, finding for BBHs and for binary neutron stars. We find that the BBH rate likely increases with redshift ( credibility) but not faster than the star formation rate ( credibility). Additionally, we examine recent exceptional events in the context of our population models, finding that the asymmetric masses of GW190412 and the high component masses of GW190521 are consistent with our models, but the low secondary mass of GW190814 makes it an outlier

Tests of general relativity with binary black holes from the second LIGO-Virgo gravitational-wave transient catalog

Gravitational waves enable tests of general relativity in the highly dynamical and strong-field regime. Using events detected by LIGO-Virgo up to 1 October 2019, we evaluate the consistency of the data with predictions from the theory. We first establish that residuals from the best-fit waveform are consistent with detector noise, and that the low- and high-frequency parts of the signals are in agreement. We then consider parametrized modifications to the waveform by varying post-Newtonian and phenomenological coefficients, improving past constraints by factors of similar to 2; we also find consistency with Kerr black holes when we specifically target signatures of the spin-induced quadrupole moment. Looking for gravitational-wave dispersion, we tighten constraints on Lorentz-violating coefficients by a factor of similar to 2.6 and bound the mass of the graviton to m(g) &lt;= 1.76 x 10(-23) eV/c(2) with 90% credibility. We also analyze the properties of the merger remnants by measuring ringdown frequencies and damping times, constraining fractional deviations away from the Kerr frequency to delta(f) over cap (220) = 0.03(-0.35)(+0.38) for the fundamental quadrupolar mode, and delta(f) over cap (221) = 0.04(-0.32)(+0.27) for the first overtone; additionally, we find no evidence for postmerger echoes. Finally, we determine that our data are consistent with tensorial polarizations through a template-independent method. When possible, we assess the validity of general relativity based on collections of events analyzed jointly. We find no evidence for new physics beyond general relativity, for black hole mimickers, or for any unaccounted systematics

GWTC-2: Compact Binary Coalescences Observed by LIGO and Virgo during the First Half of the Third Observing Run

We report on gravitational-wave discoveries from compact binary coalescences detected by Advanced LIGO and Advanced Virgo in the first half of the third observing run (O3a) between 1 April 2019 15: 00 UTC and 1 October 2019 15: 00 UTC. By imposing a false-alarm-rate threshold of two per year in each of the four search pipelines that constitute our search, we present 39 candidate gravitational-wave events. At this threshold, we expect a contamination fraction of less than 10%. Of these, 26 candidate events were reported previously in near-real time through gamma-ray coordinates network notices and circulars; 13 are reported here for the first time. The catalog contains events whose sources are black hole binary mergers up to a redshift of approximately 0.8, as well as events whose components cannot be unambiguously identified as black holes or neutron stars. For the latter group, we are unable to determine the nature based on estimates of the component masses and spins from gravitational-wave data alone. The range of candidate event masses which are unambiguously identified as binary black holes (both objects &gt;= 3 M-circle dot) is increased compared to GWTC-1, with total masses from approximately 14 M-circle dot for GW190924_021846 to approximately 150 M-circle dot for GW190521. For the first time, this catalog includes binary systems with significantly asymmetric mass ratios, which had not been observed in data taken before April 2019. We also find that 11 of the 39 events detected since April 2019 have positive effective inspiral spins under our default prior (at 90% credibility), while none exhibit negative effective inspiral spin. Given the increased sensitivity of Advanced LIGO and Advanced Virgo, the detection of 39 candidate events in approximately 26 weeks of data (approximately 1.5 per week) is consistent with GWTC-1

Search for intermediate mass black hole binaries in the first and second observing runs of the Advanced LIGO and Virgo network

International audienceGravitational-wave astronomy has been firmly established with the detection of gravitational waves from the merger of ten stellar-mass binary black holes and a neutron star binary. This paper reports on the all-sky search for gravitational waves from intermediate mass black hole binaries in the first and second observing runs of the Advanced LIGO and Virgo network. The search uses three independent algorithms: two based on matched filtering of the data with waveform templates of gravitational-wave signals from compact binaries, and a third, model-independent algorithm that employs no signal model for the incoming signal. No intermediate mass black hole binary event is detected in this search. Consequently, we place upper limits on the merger rate density for a family of intermediate mass black hole binaries. In particular, we choose sources with total masses M=m1+m2∈[120,800]  M⊙ and mass ratios q=m2/m1∈[0.1,1.0]. For the first time, this calculation is done using numerical relativity waveforms (which include higher modes) as models of the real emitted signal. We place a most stringent upper limit of 0.20  Gpc-3 yr-1 (in comoving units at the 90% confidence level) for equal-mass binaries with individual masses m1,2=100  M⊙ and dimensionless spins χ1,2=0.8 aligned with the orbital angular momentum of the binary. This improves by a factor of ∼5 that reported after Advanced LIGO’s first observing run

Gravitational-wave Constraints on the Equatorial Ellipticity of Millisecond Pulsars

We present a search for continuous gravitational waves from five radio pulsars, comprising three recycled pulsars (PSR J0437-4715, PSR J0711-6830, and PSR J0737-3039A) and two young pulsars: the Crab pulsar (J0534+2200) and the Vela pulsar (J0835-4510). We use data from the third observing run of Advanced LIGO and Virgo combined with data from their first and second observing runs. For the first time, we are able to match (for PSR J0437-4715) or surpass (for PSR J0711-6830) the indirect limits on gravitational-wave emission from recycled pulsars inferred from their observed spin-downs, and constrain their equatorial ellipticities to be less than 10(-8). For each of the five pulsars, we perform targeted searches that assume a tight coupling between the gravitational-wave and electromagnetic signal phase evolution. We also present constraints on PSR J0711-6830, the Crab pulsar, and the Vela pulsar from a search that relaxes this assumption, allowing the gravitational-wave signal to vary from the electromagnetic expectation within a narrow band of frequencies and frequency derivatives