Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study

Background Dermoscopy is commonly used for the evaluation of pigmented lesions, but agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labeling of medical data is a bottleneck in the development of machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images. Objective The aim of this study is to demonstrate that crowdsourcing can be used to label basic dermoscopic structures from images of pigmented lesions with similar reliability to a group of experts. Methods First, we obtained labels of 248 images of melanocytic lesions with 31 dermoscopic “subfeatures” labeled by 20 dermoscopy experts. These were then collapsed into 6 dermoscopic “superfeatures” based on structural similarity, due to low interrater reliability (IRR): dots, globules, lines, network structures, regression structures, and vessels. These images were then used as the gold standard for the crowd study. The commercial platform DiagnosUs was used to obtain annotations from a nonexpert crowd for the presence or absence of the 6 superfeatures in each of the 248 images. We replicated this methodology with a group of 7 dermatologists to allow direct comparison with the nonexpert crowd. The Cohen κ value was used to measure agreement across raters. Results In total, we obtained 139,731 ratings of the 6 dermoscopic superfeatures from the crowd. There was relatively lower agreement for the identification of dots and globules (the median κ values were 0.526 and 0.395, respectively), whereas network structures and vessels showed the highest agreement (the median κ values were 0.581 and 0.798, respectively). This pattern was also seen among the expert raters, who had median κ values of 0.483 and 0.517 for dots and globules, respectively, and 0.758 and 0.790 for network structures and vessels. The median κ values between nonexperts and thresholded average–expert readers were 0.709 for dots, 0.719 for globules, 0.714 for lines, 0.838 for network structures, 0.818 for regression structures, and 0.728 for vessels. Conclusions This study confirmed that IRR for different dermoscopic features varied among a group of experts; a similar pattern was observed in a nonexpert crowd. There was good or excellent agreement for each of the 6 superfeatures between the crowd and the experts, highlighting the similar reliability of the crowd for labeling dermoscopic images. This confirms the feasibility and dependability of using crowdsourcing as a scalable solution to annotate large sets of dermoscopic images, with several potential clinical and educational applications, including the development of novel, explainable ML tools

Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study

BackgroundDermoscopy is commonly used for the evaluation of pigmented lesions, but agreement between experts for identification of dermoscopic structures is known to be relatively poor. Expert labeling of medical data is a bottleneck in the development of machine learning (ML) tools, and crowdsourcing has been demonstrated as a cost- and time-efficient method for the annotation of medical images. ObjectiveThe aim of this study is to demonstrate that crowdsourcing can be used to label basic dermoscopic structures from images of pigmented lesions with similar reliability to a group of experts. MethodsFirst, we obtained labels of 248 images of melanocytic lesions with 31 dermoscopic “subfeatures” labeled by 20 dermoscopy experts. These were then collapsed into 6 dermoscopic “superfeatures” based on structural similarity, due to low interrater reliability (IRR): dots, globules, lines, network structures, regression structures, and vessels. These images were then used as the gold standard for the crowd study. The commercial platform DiagnosUs was used to obtain annotations from a nonexpert crowd for the presence or absence of the 6 superfeatures in each of the 248 images. We replicated this methodology with a group of 7 dermatologists to allow direct comparison with the nonexpert crowd. The Cohen κ value was used to measure agreement across raters. ResultsIn total, we obtained 139,731 ratings of the 6 dermoscopic superfeatures from the crowd. There was relatively lower agreement for the identification of dots and globules (the median κ values were 0.526 and 0.395, respectively), whereas network structures and vessels showed the highest agreement (the median κ values were 0.581 and 0.798, respectively). This pattern was also seen among the expert raters, who had median κ values of 0.483 and 0.517 for dots and globules, respectively, and 0.758 and 0.790 for network structures and vessels. The median κ values between nonexperts and thresholded average–expert readers were 0.709 for dots, 0.719 for globules, 0.714 for lines, 0.838 for network structures, 0.818 for regression structures, and 0.728 for vessels. ConclusionsThis study confirmed that IRR for different dermoscopic features varied among a group of experts; a similar pattern was observed in a nonexpert crowd. There was good or excellent agreement for each of the 6 superfeatures between the crowd and the experts, highlighting the similar reliability of the crowd for labeling dermoscopic images. This confirms the feasibility and dependability of using crowdsourcing as a scalable solution to annotate large sets of dermoscopic images, with several potential clinical and educational applications, including the development of novel, explainable ML tools

Topical treatment of actinic keratoses in organ transplant recipients: a feasibility study for SPOT (Squamous cell carcinoma Prevention in Organ transplant recipients using Topical treatments).

BACKGROUND: The risk of cutaneous squamous cell carcinoma (cSCC) is significantly increased in organ transplant recipients (OTRs). Clearance of actinic keratoses (AKs) is generally regarded as a surrogate biomarker for cSCC prevention. OTR-cSCC chemoprevention with topical AK treatments has not been investigated in randomized controlled trials (RCTs), although there is evidence that 5% 5-fluorouracil (5-FU) may be chemoprotective in immunocompetent patients. OBJECTIVES: To assess the feasibility, activity and evaluation outcomes relevant to the design of a future phase III RCT of topical cSCC chemoprevention in OTRs. METHODS: OTRs with 10 or more AKs in predefined areas were randomized 1 : 1 : 1 to topical 5-FU, 5% imiquimod (IMIQ) or sunscreen (sun-protective factor 30+) in a phase II, open-label RCT over 15 months. Feasibility outcomes included proportions of eligible OTRs randomized, completing treatment and willing to be re-treated. AK activity [AK clearance, new AK development, patient-centred outcomes (toxicity, health-related quality of life, HRQoL)] and evaluation methodology (clinical vs. photographic) were assessed. RESULTS: Forty OTRs with 903 AKs were randomized. All feasibility outcomes were met (56% of eligible OTRs were randomized; 89% completed treatment; 81% were willing to be re-treated). AK activity analyses found 5-FU and IMIQ were superior to sunscreen for AK clearance and prevention of new AKs. 5-FU was more effective than IMIQ in AK clearance and prevention in exploratory analyses. Although toxicity was greater with 5-FU, HRQoL outcomes were similar. CONCLUSIONS: Trials of topical AK treatments in OTRs for cSCC chemoprevention are feasible and AK activity results support further investigation of 5-FU-based treatments in future phase III trials. What is already known about this topic? Cutaneous squamous cell carcinoma (cSCC) is significantly more common in immunocompromised individuals including organ transplant recipients (OTRs) compared with immunocompetent populations. cSCC chemoprevention activity of sunscreen and 5-fluorouracil-based (5-FU) actinic keratosis (AK) treatments has been demonstrated in randomized controlled trials (RCTs) in immunocompetent populations but not in OTRs. AKs are cSCC precursors and their clearance and prevention are generally regarded as surrogate endpoint biomarkers for potential cSCC chemoprevention activity. What does this study add? SPOT (SCC Prevention in OTRs using Topical treatments) has confirmed that RCTs of OTR-cSCC chemoprevention with topical AK treatments are feasible. It also suggests that topical 5-FU may be superior to 5% imiquimod and sunscreen in AK clearance and prevention. Together with recent evidence from several RCTs in the general population, these data provide a compelling rationale for further studies of intervention with 5-FU-based topical chemoprevention approaches in OTR-cSCC prevention