Simple and Efficient Confidence Score for Grading Whole Slide Images

Grading precancerous lesions on whole slide images is a challenging task: the continuous space of morphological phenotypes makes clear-cut decisions between different grades often difficult, leading to low inter- and intra-rater agreements. More and more Artificial Intelligence (AI) algorithms are developed to help pathologists perform and standardize their diagnosis. However, those models can render their prediction without consideration of the ambiguity of the classes and can fail without notice which prevent their wider acceptance in a clinical context. In this paper, we propose a new score to measure the confidence of AI models in grading tasks. Our confidence score is specifically adapted to ordinal output variables, is versatile and does not require extra training or additional inferences nor particular architecture changes. Comparison to other popular techniques such as Monte Carlo Dropout and deep ensembles shows that our method provides state-of-the art results, while being simpler, more versatile and less computationally intensive. The score is also easily interpretable and consistent with real life hesitations of pathologists. We show that the score is capable of accurately identifying mispredicted slides and that accuracy for high confidence decisions is significantly higher than for low-confidence decisions (gap in AUC of 17.1% on the test set). We believe that the proposed confidence score could be leveraged by pathologists directly in their workflow and assist them on difficult tasks such as grading precancerous lesions

Diagnosis with Confidence: Deep Learning for Reliable Classification of Squamous Lesions of the Upper Aerodigestive Tract

Abstract Diagnosis of head and neck squamous dysplasia and carcinomas is critical for patient care, cure and follow-up. It can be challenging, especially for intraepithelial lesions. Even though the last WHO classification simplified the grading of dysplasia with only two grades (except for oral or oropharyngeal lesions), the inter and intra-observer variability remains substantial, especially for non-specialized pathologists. In this study we investigated the potential of deep learning to assist the pathologist with automatic and reliable classification of head and neck squamous lesions following the 2022 WHO classification system for the hypopharynx, larynx, trachea and parapharyngeal space. We created, for the first time, a large scale database of histological samples intended for developing an automatic diagnostic tool. We developed and trained a weakly supervised model performing classification from whole slides images. A dual blind review was carried out to define a gold standard test set on which our model was able to classify lesions with high accuracy on every class (average AUC: 0.878 (95% CI: [0.834-0.918])). Finally, we defined a confidence score for the model predictions, which can be used to identify ambiguous or difficult cases. When the algorithm is applied as a screening tool, such cases can then be submitted to pathologists in priority. Our results demonstrate that the model, associated with confidence measurements, can help in the difficult task of classifying head and neck squamous lesions

Diagnosis with confidence: deep learning for reliable classification of laryngeal dysplasia

International audienceBackground Diagnosis of head and neck (HN) squamous dysplasias and carcinomas is critical for patient care, cure, and follow‐up. It can be challenging, especially for grading intraepithelial lesions. Despite recent simplification in the last WHO grading system, the inter‐ and intraobserver variability remains substantial, particularly for nonspecialized pathologists, exhibiting the need for new tools to support pathologists. Methods In this study we investigated the potential of deep learning to assist the pathologist with automatic and reliable classification of HN lesions following the 2022 WHO classification system. We created, for the first time, a large‐scale database of histological samples (>2000 slides) intended for developing an automatic diagnostic tool. We developed and trained a weakly supervised model performing classification from whole‐slide images (WSI). We evaluated our model on both internal and external test sets and we defined and validated a new confidence score to assess the predictions that can be used to identify difficult cases. Results Our model demonstrated high classification accuracy across all lesion types on both internal and external test sets (respectively average area under the curve [AUC]: 0.878 (95% confidence interval [CI]: [0.834–0.918]) and 0.886 (95% CI: [0.813–0.947])) and the confidence score allowed for accurate differentiation between reliable and uncertain predictions. Conclusion Our results demonstrate that the model, associated with confidence measurements, can help in the difficult task of classifying HN squamous lesions by limiting variability and detecting ambiguous cases, taking us one step closer to a wider adoption of AI‐based assistive tools

Automatic grading of cervical biopsies by combining full and self-supervision

Abstract In computational pathology, the application of Deep Learning to the analysis of Whole Slide Images (WSI) has provided results of unprecedented quality. Due to their enormous size, WSIs have to be split into small images (tiles) which are first encoded and whose representations are then agglomerated in order to solve prediction tasks, such as prognosis or treatment response. The choice of the encoding strategy plays a key role in such algorithms. Current approaches include the use of encodings trained on unrelated data sources, full supervision or self-supervision. In particular, self-supervised learning (SSL) offers a great opportunity to exploit all the unlabelled data available. However, it often requires large computational resources and can be challenging to train. On the other end of the spectrum, fully-supervised methods make use of valuable prior knowledge about the data but involve a costly amount of expert time. This paper proposes a framework to reconcile SSL and full supervision and measures the trade-off between long SSL training and annotation effort, showing that a combination of both has the potential to substantially increase performance. On a recently organized challenge on grading Cervical Biopsies, we show that our mixed supervision scheme reaches high performance (weighted accuracy (WA): 0.945), outperforming both SSL (WA: 0.927) and transfer learning from ImageNet (WA: 0.877). We further provide insights and guidelines to train a clinically impactful classifier with a limited expert and/or computational workload budget. We expect that the combination of full and self-supervision is an interesting strategy for many tasks in computational pathology and will be widely adopted by the field

AI-Augmented Pathology for Head and Neck Squamous Lesions Improves Non-HN Pathologist Agreement to Expert Level

Abstract Importance Diagnosis of head and neck squamous dysplasias and carcinomas is challenging, with a moderate inter-rater agreement. Nowadays, new artificial intelligence (AI) models are developed to automatically detect and grade lesions, but their contribution to the performance of pathologists hasn’t been assessed. Objective To evaluate the contribution of our AI tool in assisting pathologists in diagnosing squamous dysplasia and carcinoma in the head and neck region. Design, Setting, and Participants We evaluated the effectiveness of our previously described AI model, which combines an automatic classification of laryngeal and pharyngeal squamous lesions with a confidence score, on a panel of eight pathologists coming from different backgrounds and with different levels of experience on a subset of 115 slides. Main Outcomes and Measures The main outcome was the inter-rater agreement, measured by the weighted linear kappa. Other outcomes on diagnostic efficiency were assessed using paired t tests. Results AI-Assistance significantly improved the inter-rater agreement (linear kappa 0.73, 95%CI [0.711-0.748] with assistance versus 0.675, 95%CI [0.579-0.765] without assistance, p < 0.001). The agreement was even better on high confidence predictions (mean linear kappa 0.809, 95%CI [0.784-0.834] for assisted review, versus 0.731, 95%CI [0.681-0.781] non-assisted, p = 0.018). These improvements were particularly strong for non-specialized and younger pathologists. Hence, the AI-Assistance enabled the panel to perform on par with the expert panel described in the literature. Conclusions and Relevance Our AI-Assistance is of great value for helping pathologists in the difficult task of diagnosing squamous dysplasias and carcinomas, improving for the first time the inter-rater agreement. It demonstrates the possibility of a truly Augmented Pathology in complex tasks such as the classification of head and neck squamous lesions