Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed

Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial

Background: In this study, we aimed to evaluate the effects of tocilizumab in adult patients admitted to hospital with COVID-19 with both hypoxia and systemic inflammation. Methods: This randomised, controlled, open-label, platform trial (Randomised Evaluation of COVID-19 Therapy [RECOVERY]), is assessing several possible treatments in patients hospitalised with COVID-19 in the UK. Those trial participants with hypoxia (oxygen saturation <92% on air or requiring oxygen therapy) and evidence of systemic inflammation (C-reactive protein ≥75 mg/L) were eligible for random assignment in a 1:1 ratio to usual standard of care alone versus usual standard of care plus tocilizumab at a dose of 400 mg–800 mg (depending on weight) given intravenously. A second dose could be given 12–24 h later if the patient's condition had not improved. The primary outcome was 28-day mortality, assessed in the intention-to-treat population. The trial is registered with ISRCTN (50189673) and ClinicalTrials.gov (NCT04381936). Findings: Between April 23, 2020, and Jan 24, 2021, 4116 adults of 21 550 patients enrolled into the RECOVERY trial were included in the assessment of tocilizumab, including 3385 (82%) patients receiving systemic corticosteroids. Overall, 621 (31%) of the 2022 patients allocated tocilizumab and 729 (35%) of the 2094 patients allocated to usual care died within 28 days (rate ratio 0·85; 95% CI 0·76–0·94; p=0·0028). Consistent results were seen in all prespecified subgroups of patients, including those receiving systemic corticosteroids. Patients allocated to tocilizumab were more likely to be discharged from hospital within 28 days (57% vs 50%; rate ratio 1·22; 1·12–1·33; p<0·0001). Among those not receiving invasive mechanical ventilation at baseline, patients allocated tocilizumab were less likely to reach the composite endpoint of invasive mechanical ventilation or death (35% vs 42%; risk ratio 0·84; 95% CI 0·77–0·92; p<0·0001). Interpretation: In hospitalised COVID-19 patients with hypoxia and systemic inflammation, tocilizumab improved survival and other clinical outcomes. These benefits were seen regardless of the amount of respiratory support and were additional to the benefits of systemic corticosteroids. Funding: UK Research and Innovation (Medical Research Council) and National Institute of Health Research

Convalescent plasma in patients admitted to hospital with COVID-19 (RECOVERY): a randomised controlled, open-label, platform trial

Background: Many patients with COVID-19 have been treated with plasma containing anti-SARS-CoV-2 antibodies. We aimed to evaluate the safety and efficacy of convalescent plasma therapy in patients admitted to hospital with COVID-19. Methods: This randomised, controlled, open-label, platform trial (Randomised Evaluation of COVID-19 Therapy [RECOVERY]) is assessing several possible treatments in patients hospitalised with COVID-19 in the UK. The trial is underway at 177 NHS hospitals from across the UK. Eligible and consenting patients were randomly assigned (1:1) to receive either usual care alone (usual care group) or usual care plus high-titre convalescent plasma (convalescent plasma group). The primary outcome was 28-day mortality, analysed on an intention-to-treat basis. The trial is registered with ISRCTN, 50189673, and ClinicalTrials.gov, NCT04381936. Findings: Between May 28, 2020, and Jan 15, 2021, 11558 (71%) of 16287 patients enrolled in RECOVERY were eligible to receive convalescent plasma and were assigned to either the convalescent plasma group or the usual care group. There was no significant difference in 28-day mortality between the two groups: 1399 (24%) of 5795 patients in the convalescent plasma group and 1408 (24%) of 5763 patients in the usual care group died within 28 days (rate ratio 1·00, 95% CI 0·93–1·07; p=0·95). The 28-day mortality rate ratio was similar in all prespecified subgroups of patients, including in those patients without detectable SARS-CoV-2 antibodies at randomisation. Allocation to convalescent plasma had no significant effect on the proportion of patients discharged from hospital within 28 days (3832 [66%] patients in the convalescent plasma group vs 3822 [66%] patients in the usual care group; rate ratio 0·99, 95% CI 0·94–1·03; p=0·57). Among those not on invasive mechanical ventilation at randomisation, there was no significant difference in the proportion of patients meeting the composite endpoint of progression to invasive mechanical ventilation or death (1568 [29%] of 5493 patients in the convalescent plasma group vs 1568 [29%] of 5448 patients in the usual care group; rate ratio 0·99, 95% CI 0·93–1·05; p=0·79). Interpretation: In patients hospitalised with COVID-19, high-titre convalescent plasma did not improve survival or other prespecified clinical outcomes. Funding: UK Research and Innovation (Medical Research Council) and National Institute of Health Research

A spatial decision support system to allow the investigation of the impact of emissions from major point sources under different operating policies

A Spatial Decision Support System (SDSS) is being developed for Her Majesty's Inspectorate of Pollution (HMIP). It enables the investigation of the spatial implications of different operating procedures from large point sources of pollution. The environmental effect of emissions is assessed using the critical loads methodology developed at ITE and modelled deposition footprints. This approach allows an effect per unit emission or pollution potential to be determined for each source. Individual sources are modelled and included within the SDSS if their current emissions are above a given threshold. The SDSS provides a graphical user interface (GUI) which facilitates a fast, efficient and effective means to specify and to examine the effect of different operating policies. Mapping, statistical and optimization facilities are provided to help describe the effect of any specified strategy. Maps may be produced as deposition rates or exceedance values. Statistics may be visualised as histograms and scatter plots. The optimization facility uses linear programming to minimise the total environmental impact (estimated from emissions and critical loads) or maximise power produced within environmental limits. The SDSS has been written in the Arc/Info Macro Language (AML) and provides an invisible interface with standard GIS facilities and programmes written in C