Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe

Ticagrelor in patients with diabetes and stable coronary artery disease with a history of previous percutaneous coronary intervention (THEMIS-PCI) : a phase 3, placebo-controlled, randomised trial

Background: Patients with stable coronary artery disease and diabetes with previous percutaneous coronary intervention (PCI), particularly those with previous stenting, are at high risk of ischaemic events. These patients are generally treated with aspirin. In this trial, we aimed to investigate if these patients would benefit from treatment with aspirin plus ticagrelor. Methods: The Effect of Ticagrelor on Health Outcomes in diabEtes Mellitus patients Intervention Study (THEMIS) was a phase 3 randomised, double-blinded, placebo-controlled trial, done in 1315 sites in 42 countries. Patients were eligible if 50 years or older, with type 2 diabetes, receiving anti-hyperglycaemic drugs for at least 6 months, with stable coronary artery disease, and one of three other mutually non-exclusive criteria: a history of previous PCI or of coronary artery bypass grafting, or documentation of angiographic stenosis of 50% or more in at least one coronary artery. Eligible patients were randomly assigned (1:1) to either ticagrelor or placebo, by use of an interactive voice-response or web-response system. The THEMIS-PCI trial comprised a prespecified subgroup of patients with previous PCI. The primary efficacy outcome was a composite of cardiovascular death, myocardial infarction, or stroke (measured in the intention-to-treat population). Findings: Between Feb 17, 2014, and May 24, 2016, 11 154 patients (58% of the overall THEMIS trial) with a history of previous PCI were enrolled in the THEMIS-PCI trial. Median follow-up was 3·3 years (IQR 2·8–3·8). In the previous PCI group, fewer patients receiving ticagrelor had a primary efficacy outcome event than in the placebo group (404 [7·3%] of 5558 vs 480 [8·6%] of 5596; HR 0·85 [95% CI 0·74–0·97], p=0·013). The same effect was not observed in patients without PCI (p=0·76, p interaction=0·16). The proportion of patients with cardiovascular death was similar in both treatment groups (174 [3·1%] with ticagrelor vs 183 (3·3%) with placebo; HR 0·96 [95% CI 0·78–1·18], p=0·68), as well as all-cause death (282 [5·1%] vs 323 [5·8%]; 0·88 [0·75–1·03], p=0·11). TIMI major bleeding occurred in 111 (2·0%) of 5536 patients receiving ticagrelor and 62 (1·1%) of 5564 patients receiving placebo (HR 2·03 [95% CI 1·48–2·76], p<0·0001), and fatal bleeding in 6 (0·1%) of 5536 patients with ticagrelor and 6 (0·1%) of 5564 with placebo (1·13 [0·36–3·50], p=0·83). Intracranial haemorrhage occurred in 33 (0·6%) and 31 (0·6%) patients (1·21 [0·74–1·97], p=0·45). Ticagrelor improved net clinical benefit: 519/5558 (9·3%) versus 617/5596 (11·0%), HR=0·85, 95% CI 0·75–0·95, p=0·005, in contrast to patients without PCI where it did not, p interaction=0·012. Benefit was present irrespective of time from most recent PCI. Interpretation: In patients with diabetes, stable coronary artery disease, and previous PCI, ticagrelor added to aspirin reduced cardiovascular death, myocardial infarction, and stroke, although with increased major bleeding. In that large, easily identified population, ticagrelor provided a favourable net clinical benefit (more than in patients without history of PCI). This effect shows that long-term therapy with ticagrelor in addition to aspirin should be considered in patients with diabetes and a history of PCI who have tolerated antiplatelet therapy, have high ischaemic risk, and low bleeding risk

Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical science. © The Author(s) 2019. Published by Oxford University Press