Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe

Convalescent plasma in patients admitted to hospital with COVID-19 (RECOVERY): a randomised controlled, open-label, platform trial

Background: Many patients with COVID-19 have been treated with plasma containing anti-SARS-CoV-2 antibodies. We aimed to evaluate the safety and efficacy of convalescent plasma therapy in patients admitted to hospital with COVID-19. Methods: This randomised, controlled, open-label, platform trial (Randomised Evaluation of COVID-19 Therapy [RECOVERY]) is assessing several possible treatments in patients hospitalised with COVID-19 in the UK. The trial is underway at 177 NHS hospitals from across the UK. Eligible and consenting patients were randomly assigned (1:1) to receive either usual care alone (usual care group) or usual care plus high-titre convalescent plasma (convalescent plasma group). The primary outcome was 28-day mortality, analysed on an intention-to-treat basis. The trial is registered with ISRCTN, 50189673, and ClinicalTrials.gov, NCT04381936. Findings: Between May 28, 2020, and Jan 15, 2021, 11558 (71%) of 16287 patients enrolled in RECOVERY were eligible to receive convalescent plasma and were assigned to either the convalescent plasma group or the usual care group. There was no significant difference in 28-day mortality between the two groups: 1399 (24%) of 5795 patients in the convalescent plasma group and 1408 (24%) of 5763 patients in the usual care group died within 28 days (rate ratio 1·00, 95% CI 0·93–1·07; p=0·95). The 28-day mortality rate ratio was similar in all prespecified subgroups of patients, including in those patients without detectable SARS-CoV-2 antibodies at randomisation. Allocation to convalescent plasma had no significant effect on the proportion of patients discharged from hospital within 28 days (3832 [66%] patients in the convalescent plasma group vs 3822 [66%] patients in the usual care group; rate ratio 0·99, 95% CI 0·94–1·03; p=0·57). Among those not on invasive mechanical ventilation at randomisation, there was no significant difference in the proportion of patients meeting the composite endpoint of progression to invasive mechanical ventilation or death (1568 [29%] of 5493 patients in the convalescent plasma group vs 1568 [29%] of 5448 patients in the usual care group; rate ratio 0·99, 95% CI 0·93–1·05; p=0·79). Interpretation: In patients hospitalised with COVID-19, high-titre convalescent plasma did not improve survival or other prespecified clinical outcomes. Funding: UK Research and Innovation (Medical Research Council) and National Institute of Health Research