Crowdsourcing as a novel technique for retinal fundus photography classification: analysis of images in the EPIC Norfolk cohort on behalf of the UK Biobank Eye and Vision Consortium.

AIM: Crowdsourcing is the process of outsourcing numerous tasks to many untrained individuals. Our aim was to assess the performance and repeatability of crowdsourcing for the classification of retinal fundus photography. METHODS: One hundred retinal fundus photograph images with pre-determined disease criteria were selected by experts from a large cohort study. After reading brief instructions and an example classification, we requested that knowledge workers (KWs) from a crowdsourcing platform classified each image as normal or abnormal with grades of severity. Each image was classified 20 times by different KWs. Four study designs were examined to assess the effect of varying incentive and KW experience in classification accuracy. All study designs were conducted twice to examine repeatability. Performance was assessed by comparing the sensitivity, specificity and area under the receiver operating characteristic curve (AUC). RESULTS: Without restriction on eligible participants, two thousand classifications of 100 images were received in under 24 hours at minimal cost. In trial 1 all study designs had an AUC (95%CI) of 0.701(0.680-0.721) or greater for classification of normal/abnormal. In trial 1, the highest AUC (95%CI) for normal/abnormal classification was 0.757 (0.738-0.776) for KWs with moderate experience. Comparable results were observed in trial 2. In trial 1, between 64-86% of any abnormal image was correctly classified by over half of all KWs. In trial 2, this ranged between 74-97%. Sensitivity was ≥ 96% for normal versus severely abnormal detections across all trials. Sensitivity for normal versus mildly abnormal varied between 61-79% across trials. CONCLUSIONS: With minimal training, crowdsourcing represents an accurate, rapid and cost-effective method of retinal image analysis which demonstrates good repeatability. Larger studies with more comprehensive participant training are needed to explore the utility of this compelling technique in large scale medical image analysis

Colorectal neuroendocrine carcinomas and adenocarcinomas share oncogenic pathways. A clinico-pathologic study of 12 cases

OBJECTIVE: Neuroendocrine carcinomas (NECs) are rare neoplasms with an increasing incidence. Oncogenetic pathways of colorectal NEC are still poorly understood, and no treatment standards are available for these rare tumors. METHODS: We analyzed retrospectively the clinical records and histology of 12 patients with colorectal NEC. KRAS and BRAF mutations were investigated after the dissection of exoendocrine and neuroendocrine components. ALK alterations and EML4-ALK transcripts were detected by in-situ hybridization and determination of fusion transcripts, respectively. RESULTS: At the time of diagnosis, the mean age of the patients was 60 years (40-79) and 10 patients had synchronous metastases. A transient response occurred in two patients and one patient treated with cisplatin-etoposide or fluoropyrimidine-oxaliplatin, respectively. Tumor progression-related death occurred in 11 of 12 patients. Ten tumors contained an exocrine component, accounting for 5-70% of the tumor, and the other two contained an amphicrine component. BRAF/KRAS mutations were found in six of 10 tumors, corresponding to BRAF(V600E) (n=2) or KRAS(G12D) (n=2), KRAS(G12V) or KRAS(G13D). DNA was obtained from both exocrine and endocrine components in seven cases, and the BRAF/KRAS status was identical in all cases. Split of the ALK locus was detected in a minority of tumor cells in two of eight cases, but EML4-ALK transcripts were absent. CONCLUSION: The association of an exocrine component in all cases and the similar profile of BRAF/KRAS mutations indicate that colorectal NEC may correspond to a high-grade transformation of colorectal carcinoma. New chemotherapy regimens using targeted therapies should be assessed in these tumors

The Accuracy and Reliability of Crowdsource Annotations of Digital Retinal Images

PURPOSE: Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification. METHODS: We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy. RESULTS: In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%-74%) and 87% (95% CI, 86%-88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91-0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25. CONCLUSIONS: This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group. TRANSLATIONAL RELEVANCE: The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis