The Accuracy and Reliability of Crowdsource Annotations of Digital Retinal Images

Purpose: Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification. Methods: We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy. Results: In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%-74%) and 87% (95% CI, 86%-88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91-0.96). For image annotation, a maximal Dice coefficient (~0.6) was achieved with a consensus threshold of 0.25. Conclusions: This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group. Translational Relevance: The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis

Allopurinol versus usual care in UK patients with ischaemic heart disease (ALL-HEART): a multicentre, prospective, randomised, open-label, blinded-endpoint trial

BACKGROUND: Allopurinol is a urate-lowering therapy used to treat patients with gout. Previous studies have shown that allopurinol has positive effects on several cardiovascular parameters. The ALL-HEART study aimed to determine whether allopurinol therapy improves major cardiovascular outcomes in patients with ischaemic heart disease. METHODS: ALL-HEART was a multicentre, prospective, randomised, open-label, blinded-endpoint trial done in 18 regional centres in England and Scotland, with patients recruited from 424 primary care practices. Eligible patients were aged 60 years or older, with ischaemic heart disease but no history of gout. Participants were randomly assigned (1:1), using a central web-based randomisation system accessed via a web-based application or an interactive voice response system, to receive oral allopurinol up-titrated to a dose of 600 mg daily (300 mg daily in participants with moderate renal impairment at baseline) or to continue usual care. The primary outcome was the composite cardiovascular endpoint of non-fatal myocardial infarction, non-fatal stroke, or cardiovascular death. The hazard ratio (allopurinol vs usual care) in a Cox proportional hazards model was assessed for superiority in a modified intention-to-treat analysis (excluding randomly assigned patients later found to have met one of the exclusion criteria). The safety analysis population included all patients in the modified intention-to-treat usual care group and those who took at least one dose of randomised medication in the allopurinol group. This study is registered with the EU Clinical Trials Register, EudraCT 2013-003559-39, and ISRCTN, ISRCTN32017426. FINDINGS: Between Feb 7, 2014, and Oct 2, 2017, 5937 participants were enrolled and then randomly assigned to receive allopurinol or usual care. After exclusion of 216 patients after randomisation, 5721 participants (mean age 72·0 years [SD 6·8], 4321 [75·5%] males, and 5676 [99·2%] white) were included in the modified intention-to-treat population, with 2853 in the allopurinol group and 2868 in the usual care group. Mean follow-up time in the study was 4·8 years (1·5). There was no evidence of a difference between the randomised treatment groups in the rates of the primary endpoint. 314 (11·0%) participants in the allopurinol group (2·47 events per 100 patient-years) and 325 (11·3%) in the usual care group (2·37 events per 100 patient-years) had a primary endpoint (hazard ratio [HR] 1·04 [95% CI 0·89–1·21], p=0·65). 288 (10·1%) participants in the allopurinol group and 303 (10·6%) participants in the usual care group died from any cause (HR 1·02 [95% CI 0·87–1·20], p=0·77). INTERPRETATION: In this large, randomised clinical trial in patients aged 60 years or older with ischaemic heart disease but no history of gout, there was no difference in the primary outcome of non-fatal myocardial infarction, non-fatal stroke, or cardiovascular death between participants randomised to allopurinol therapy and those randomised to usual care. FUNDING: UK National Institute for Health and Care Research