Reinforcement learning in continuous state- and action-space

Reinforcement learning in the continuous state-space poses the problem of the inability to store the values of all state-action pairs in a lookup table, due to both storage limitations and the inability to visit all states sufficiently often to learn the correct values. This can be overcome with the use of function approximation techniques with generalisation capability, such as artificial neural networks, to store the value function. When this is applied we can select the optimal action by comparing the values of each possible action; however, when the action-space is continuous this is not possible. In this thesis we investigate methods to select the optimal action when artificial neural networks are used to approximate the value function, through the application of numerical optimization techniques. Although it has been stated in the literature that gradient-ascent methods can be applied to the action selection [47], it is also stated that solving this problem would be infeasible, and therefore, is claimed that it is necessary to utilise a second artificial neural network to approximate the policy function [21, 55]. The major contributions of this thesis include the investigation of the applicability of action selection by numerical optimization methods, including gradient-ascent along with other derivative-based and derivative-free numerical optimization methods,and the proposal of two novel algorithms which are based on the application of two alternative action selection methods: NM-SARSA [40] and NelderMead-SARSA. We empirically compare the proposed methods to state-of-the-art methods from the literature on three continuous state- and action-space control benchmark problems from the literature: minimum-time full swing-up of the Acrobot; Cart-Pole balancing problem; and a double pole variant. We also present novel results from the application of the existing direct policy search method genetic programming to the Acrobot benchmark problem [12, 14]

Erratum: "A Gravitational-wave Measurement of the Hubble Constant Following the Second Observing Run of Advanced LIGO and Virgo" (2021, ApJ, 909, 218)

[no abstract available

Search for gravitational waves from Scorpius X-1 in the second Advanced LIGO observing run with an improved hidden Markov model

We present results from a semicoherent search for continuous gravitational waves from the low-mass x-ray binary Scorpius X-1, using a hidden Markov model (HMM) to track spin wandering. This search improves on previous HMM-based searches of LIGO data by using an improved frequency domain matched filter, the J-statistic, and by analyzing data from Advanced LIGO's second observing run. In the frequency range searched, from 60 to 650 Hz, we find no evidence of gravitational radiation. At 194.6 Hz, the most sensitive search frequency, we report an upper limit on gravitational wave strain (at 95% confidence) of h095%=3.47×10-25 when marginalizing over source inclination angle. This is the most sensitive search for Scorpius X-1, to date, that is specifically designed to be robust in the presence of spin wandering. © 2019 American Physical Society

Search for Tensor, Vector, and Scalar Polarizations in the Stochastic Gravitational-Wave Background

The detection of gravitational waves with Advanced LIGO and Advanced Virgo has enabled novel tests of general relativity, including direct study of the polarization of gravitational waves. While general relativity allows for only two tensor gravitational-wave polarizations, general metric theories can additionally predict two vector and two scalar polarizations. The polarization of gravitational waves is encoded in the spectral shape of the stochastic gravitational-wave background, formed by the superposition of cosmological and individually unresolved astrophysical sources. Using data recorded by Advanced LIGO during its first observing run, we search for a stochastic background of generically polarized gravitational waves. We find no evidence for a background of any polarization, and place the first direct bounds on the contributions of vector and scalar polarizations to the stochastic background. Under log-uniform priors for the energy in each polarization, we limit the energy densities of tensor, vector, and scalar modes at 95% credibility to Ω0T<5.58×10-8, Ω0V<6.35×10-8, and Ω0S<1.08×10-7 at a reference frequency f0=25 Hz. © 2018 American Physical Society

Search for Gravitational Waves Associated with Gamma-Ray Bursts Detected by Fermi and Swift during the LIGO-Virgo Run O3b

We search for gravitational-wave signals associated with gamma-ray bursts (GRBs) detected by the Fermi and Swift satellites during the second half of the third observing run of Advanced LIGO and Advanced Virgo (2019 November 1 15:00 UTC-2020 March 27 17:00 UTC). We conduct two independent searches: A generic gravitational-wave transients search to analyze 86 GRBs and an analysis to target binary mergers with at least one neutron star as short GRB progenitors for 17 events. We find no significant evidence for gravitational-wave signals associated with any of these GRBs. A weighted binomial test of the combined results finds no evidence for subthreshold gravitational-wave signals associated with this GRB ensemble either. We use several source types and signal morphologies during the searches, resulting in lower bounds on the estimated distance to each GRB. Finally, we constrain the population of low-luminosity short GRBs using results from the first to the third observing runs of Advanced LIGO and Advanced Virgo. The resulting population is in accordance with the local binary neutron star merger rate. © 2022. The Author(s). Published by the American Astronomical Society

The current state of the use of large wood in river restoration and management

Trees fall naturally into rivers generating flow heterogeneity, inducing geomorphological features, and creating habitats for biota. Wood is increasingly used in restoration projects and the potential of wood acting as leaky barriers to deliver natural flood management by “slowing the flow” is recognised. However, wood in rivers can pose a risk to infrastructure and locally increase flood hazards. The aim of this paper is to provide an up-to-date summary of the benefits and risks associated with using wood to promote geomorphological processes to restore and manage rivers. This summary was developed through a workshop that brought together academics, river managers, restoration practitioners and consultants in the UK to share science and best-practice on wood in rivers. A consensus was developed on four key issues: (i) hydro-geomorphological effects, (ii) current use in restoration and management, (iii) uncertainties and risks, and (iv) tools and guidance required to inform process-based restoration and management

Narrowband Searches for Continuous and Long-duration Transient Gravitational Waves from Known Pulsars in the LIGO-Virgo Third Observing Run

Isolated neutron stars that are asymmetric with respect to their spin axis are possible sources of detectable continuous gravitational waves. This paper presents a fully coherent search for such signals from eighteen pulsars in data from LIGO and Virgo's third observing run (O3). For known pulsars, efficient and sensitive matched-filter searches can be carried out if one assumes the gravitational radiation is phase-locked to the electromagnetic emission. In the search presented here, we relax this assumption and allow both the frequency and the time derivative of the frequency of the gravitational waves to vary in a small range around those inferred from electromagnetic observations. We find no evidence for continuous gravitational waves, and set upper limits on the strain amplitude for each target. These limits are more constraining for seven of the targets than the spin-down limit defined by ascribing all rotational energy loss to gravitational radiation. In an additional search, we look in O3 data for long-duration (hours-months) transient gravitational waves in the aftermath of pulsar glitches for six targets with a total of nine glitches. We report two marginal outliers from this search, but find no clear evidence for such emission either. The resulting duration-dependent strain upper limits do not surpass indirect energy constraints for any of these targets. © 2022. The Author(s). Published by the American Astronomical Society

Dexamethasone intravitreal implant in previously treated patients with diabetic macular edema : Subgroup analysis of the MEAD study

Background: Dexamethasone intravitreal implant 0.7 mg (DEX 0.7) was approved for treatment of diabetic macular edema (DME) after demonstration of its efficacy and safety in the MEAD registration trials. We performed subgroup analysis of MEAD study results to evaluate the efficacy and safety of DEX 0.7 treatment in patients with previously treated DME. Methods: Three-year, randomized, sham-controlled phase 3 study in patients with DME, best-corrected visual acuity (BCVA) of 34.68 Early Treatment Diabetic Retinopathy Study letters (20/200.20/50 Snellen equivalent), and central retinal thickness (CRT) 65300 \u3bcm measured by time-domain optical coherence tomography. Patients were randomized to 1 of 2 doses of DEX (0.7 mg or 0.35 mg), or to sham procedure, with retreatment no more than every 6 months. The primary endpoint was 6515-letter gain in BCVA at study end. Average change in BCVA and CRT from baseline during the study (area-under-the-curve approach) and adverse events were also evaluated. The present subgroup analysis evaluated outcomes in patients randomized to DEX 0.7 (marketed dose) or sham based on prior treatment for DME at study entry. Results: Baseline characteristics of previously treated DEX 0.7 (n = 247) and sham (n=261) patients were similar. In the previously treated subgroup, mean number of treatments over 3 years was 4.1 for DEX 0.7 and 3.2 for sham, 21.5 % of DEX 0.7 patients versus 11.1 % of sham had 6515-letter BCVA gain from baseline at study end (P = 0.002), mean average BCVA change from baseline was +3.2 letters with DEX 0.7 versus +1.5 letters with sham (P = 0.024), and mean average CRT change from baseline was -126.1 \u3bcm with DEX 0.7 versus -39.0 \u3bcm with sham(P < 0.001). Cataract-related adverse events were reported in 70.3 % of baseline phakic patients in the previously treated DEX 0.7 subgroup; vision gains were restored following cataract surgery. Conclusions: DEX 0.7 significantly improved visual and anatomic outcomes in patients with DME previously treated with laser, intravitreal anti-vascular endothelial growth factor, intravitreal triamcinolone acetonide, or a combination of these therapies. The safety profile of DEX 0.7 in previously treated patients was similar to its safety profile in the total study population

GW190814: gravitational waves from the coalescence of a 23 solar mass black hole with a 2.6 solar mass compact object

We report the observation of a compact binary coalescence involving a 22.2–24.3 Me black hole and a compact object with a mass of 2.50–2.67 Me (all measurements quoted at the 90% credible level). The gravitational-wave signal, GW190814, was observed during LIGO’s and Virgo’s third observing run on 2019 August 14 at 21:10:39 UTC and has a signal-to-noise ratio of 25 in the three-detector network. The source was localized to 18.5 deg2 at a distance of - + 241 45 41 Mpc; no electromagnetic counterpart has been confirmed to date. The source has the most unequal mass ratio yet measured with gravitational waves, - + 0.112 0.009 0.008, and its secondary component is either the lightest black hole or the heaviest neutron star ever discovered in a double compact-object system. The dimensionless spin of the primary black hole is tightly constrained to �0.07. Tests of general relativity reveal no measurable deviations from the theory, and its prediction of higher-multipole emission is confirmed at high confidence. We estimate a merger rate density of 1–23 Gpc−3 yr−1 for the new class of binary coalescence sources that GW190814 represents. Astrophysical models predict that binaries with mass ratios similar to this event can form through several channels, but are unlikely to have formed in globular clusters. However, the combination of mass ratio, component masses, and the inferred merger rate for this event challenges all current models of the formation and mass distribution of compact-object binaries

On the progenitor of binary neutron star merger GW170817

On 2017 August 17 the merger of two compact objects with masses consistent with two neutron stars was discovered through gravitational-wave (GW170817), gamma-ray (GRB 170817A), and optical (SSS17a/AT 2017gfo) observations. The optical source was associated with the early-type galaxy NGC 4993 at a distance of just ∼40 Mpc, consistent with the gravitational-wave measurement, and the merger was localized to be at a projected distance of ∼2 kpc away from the galaxy's center. We use this minimal set of facts and the mass posteriors of the two neutron stars to derive the first constraints on the progenitor of GW170817 at the time of the second supernova (SN). We generate simulated progenitor populations and follow the three-dimensional kinematic evolution from binary neutron star (BNS) birth to the merger time, accounting for pre-SN galactic motion, for considerably different input distributions of the progenitor mass, pre-SN semimajor axis, and SN-kick velocity. Though not considerably tight, we find these constraints to be comparable to those for Galactic BNS progenitors. The derived constraints are very strongly influenced by the requirement of keeping the binary bound after the second SN and having the merger occur relatively close to the center of the galaxy. These constraints are insensitive to the galaxy's star formation history, provided the stellar populations are older than 1 Gyr