A comprehensive multi-domain dataset for mitotic figure detection

The prognostic value of mitotic figures in tumor tissue is well-established for many tumor types and automating this task is of high research interest. However, especially deep learning-based methods face performance deterioration in the presence of domain shifts, which may arise from different tumor types, slide preparation and digitization devices. We introduce the MIDOG++ dataset, an extension of the MIDOG 2021 and 2022 challenge datasets. We provide region of interest images from 503 histological specimens of seven different tumor types with variable morphology with in total labels for 11,937 mitotic figures: breast carcinoma, lung carcinoma, lymphosarcoma, neuroendocrine tumor, cutaneous mast cell tumor, cutaneous melanoma, and (sub)cutaneous soft tissue sarcoma. The specimens were processed in several laboratories utilizing diverse scanners. We evaluated the extent of the domain shift by using state-of-the-art approaches, observing notable differences in single-domain training. In a leave-one-domain-out setting, generalizability improved considerably. This mitotic figure dataset is the first that incorporates a wide domain shift based on different tumor types, laboratories, whole slide image scanners, and species

Domain generalization across tumor types, laboratories, and species — Insights from the 2022 edition of the Mitosis Domain Generalization Challenge

Recognition of mitotic figures in histologic tumor specimens is highly relevant to patient outcome assessment. This task is challenging for algorithms and human experts alike, with deterioration of algorithmic performance under shifts in image representations. Considerable covariate shifts occur when assessment is performed on different tumor types, images are acquired using different digitization devices, or specimens are produced in different laboratories. This observation motivated the inception of the 2022 challenge on MItosis Domain Generalization (MIDOG 2022). The challenge provided annotated histologic tumor images from six different domains and evaluated the algorithmic approaches for mitotic figure detection provided by nine challenge participants on ten independent domains. Ground truth for mitotic figure detection was established in two ways: a three-expert majority vote and an independent, immunohistochemistry-assisted set of labels. This work represents an overview of the challenge tasks, the algorithmic strategies employed by the participants, and potential factors contributing to their success. With an score of 0.764 for the top-performing team, we summarize that domain generalization across various tumor domains is possible with today’s deep learning-based recognition pipelines. However, we also found that domain characteristics not present in the training set (feline as new species, spindle cell shape as new morphology and a new scanner) led to small but significant decreases in performance. When assessed against the immunohistochemistry-assisted reference standard, all methods resulted in reduced recall scores, with only minor changes in the order of participants in the ranking

Digital Pathology for the Primary Diagnosis of Breast Histopathological Specimens: An Innovative Validation and Concordance Study

Aim: To train and individually validate a group of breast pathologists in specialty specific digital primary diagnosis using a novel protocol endorsed by the Royal College of Pathologists’ new guideline for digital pathology. The protocol allows early exposure to live digital reporting, in a risk mitigated environment, and focusses on patient safety and professional development. Methods and Results: 3 specialty breast pathologist completed training in use of a digital microscopy system, and were exposed to a training set of 20 challenging cases, designed to help them identify personal digital diagnostic pitfalls. Following this, the 3 pathologists viewed a total of 694 live, entire breast cases. All primary diagnoses were made on digital slides, with immediate glass review and reconciliation before final case sign out. There was complete clinical concordance between the glass and digital impression of the case in 98.8% of cases. Only 1.2% of cases had a clinically significant difference in diagnosis/prognosis on glass and digital slide reads. All pathologists elected to continue using the digital microscope as standard for breast histopathology specimens, with deferral to glass for a limited number of clinical/histological scenarios as a safety net. Conclusion: Individual training and validation for digital primary diagnosis allows pathologists to develop competence and confidence in their digital diagnostic skills, and aids safe and responsible transition from the light microscope to the digital microscope

Deep Learning-Based Grading of Ductal Carcinoma In Situ in Breast Histopathology Images

Ductal carcinoma in situ (DCIS) is a non-invasive breast cancer that can progress into invasive ductal carcinoma (IDC). Studies suggest DCIS is often overtreated since a considerable part of DCIS lesions may never progress into IDC. Lower grade lesions have a lower progression speed and risk, possibly allowing treatment de-escalation. However, studies show significant inter-observer variation in DCIS grading. Automated image analysis may provide an objective solution to address high subjectivity of DCIS grading by pathologists. In this study, we developed a deep learning-based DCIS grading system. It was developed using the consensus DCIS grade of three expert observers on a dataset of 1186 DCIS lesions from 59 patients. The inter-observer agreement, measured by quadratic weighted Cohen's kappa, was used to evaluate the system and compare its performance to that of expert observers. We present an analysis of the lesion-level and patient-level inter-observer agreement on an independent test set of 1001 lesions from 50 patients. The deep learning system (dl) achieved on average slightly higher inter-observer agreement to the observers (o1, o2 and o3) ($\kappa_{o1,dl}=0.81, \kappa_{o2,dl}=0.53, \kappa_{o3,dl}=0.40$) than the observers amongst each other ($\kappa_{o1,o2}=0.58, \kappa_{o1,o3}=0.50, \kappa_{o2,o3}=0.42$) at the lesion-level. At the patient-level, the deep learning system achieved similar agreement to the observers ($\kappa_{o1,dl}=0.77, \kappa_{o2,dl}=0.75, \kappa_{o3,dl}=0.70$) as the observers amongst each other ($\kappa_{o1,o2}=0.77, \kappa_{o1,o3}=0.75, \kappa_{o2,o3}=0.72$). In conclusion, we developed a deep learning-based DCIS grading system that achieved a performance similar to expert observers. We believe this is the first automated system that could assist pathologists by providing robust and reproducible second opinions on DCIS grade

Assessment of algorithms for mitosis detection in breast cancer histopathology images

The proliferative activity of breast tumors, which is routinely estimated by counting of mitotic figures in hematoxylin and eosin stained histology sections, is considered to be one of the most important prognostic markers. However, mitosis counting is laborious, subjective and may suffer from low inter-observer agreement. With the wider acceptance of whole slide images in pathology labs, automatic image analysis has been proposed as a potential solution for these issues. In this paper, the results from the Assessment of Mitosis Detection Algorithms 2013 (AMIDA13) challenge are described. The challenge was based on a data set consisting of 12 training and 11 testing subjects, with more than one thousand annotated mitotic figures by multiple observers. Short descriptions and results from the evaluation of eleven methods are presented. The top performing method has an error rate that is comparable to the inter-observer agreement among pathologists

CONFIDENT-trial protocol: A pragmatic template for clinical implementation of artificial intelligence assistance in pathology

Introduction Artificial intelligence (AI) has been on the rise in the field of pathology. Despite promising results in retrospective studies, and several CE-IVD certified algorithms on the market, prospective clinical implementation studies of AI have yet to be performed, to the best of our knowledge. In this trial, we will explore the benefits of an AI-assisted pathology workflow, while maintaining diagnostic safety standards. Methods and analysis This is a Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence compliant single-centre, controlled clinical trial, in a fully digital academic pathology laboratory. We will prospectively include prostate cancer patients who undergo prostate needle biopsies (CONFIDENT-P) and breast cancer patients who undergo a sentinel node procedure (CONFIDENT-B) in the University Medical Centre Utrecht. For both the CONFIDENT-B and CONFIDENT-P trials, the specific pathology specimens will be pseudo-randomised to be assessed by a pathologist with or without AI assistance in a pragmatic (bi-)weekly sequential design. In the intervention group, pathologists will assess whole slide images (WSI) of the standard hematoxylin and eosin (H&E)-stained sections assisted by the output of the algorithm. In the control group, pathologists will assess H&E WSI according to the current clinical workflow. If no tumour cells are identified or when the pathologist is in doubt, immunohistochemistry (IHC) staining will be performed. At least 80 patients in the CONFIDENT-P and 180 patients in the CONFIDENT-B trial will need to be enrolled to detect superiority, allocated as 1:1. Primary endpoint for both trials is the number of saved resources of IHC staining procedures for detecting tumour cells, since this will clarify tangible cost savings that will support the business case for AI. Ethics and dissemination The ethics committee (MREC NedMec) waived the need of official ethical approval, since participants are not subjected to procedures nor are they required to follow rules. Results of both trials (CONFIDENT-B and CONFIDENT-P) will be published in scientific peer-reviewed journals

Deep learning supported mitoses counting on whole slide images: A pilot study for validating breast cancer grading in the clinical workflow

INTRODUCTION: Breast cancer (BC) prognosis is largely influenced by histopathological grade, assessed according to the Nottingham modification of Bloom-Richardson (BR). Mitotic count (MC) is a component of histopathological grading but is prone to subjectivity. This study investigated whether mitoses counting in BC using digital whole slide images (WSI) compares better to light microscopy (LM) when assisted by artificial intelligence (AI), and to which extent differences in digital MC (AI assisted or not) result in BR grade variations. METHODS: Fifty BC patients with paired core biopsies and resections were randomly selected. Component scores for BR grade were extracted from pathology reports. MC was assessed using LM, WSI, and AI. Different modalities (LM-MC, WSI-MC, and AI-MC) were analyzed for correlation with scatterplots and linear regression, and for agreement in final BR with Cohen's κ. RESULTS: MC modalities strongly correlated in both biopsies and resections: LM-MC and WSI-MC (R 2 0.85 and 0.83, respectively), LM-MC and AI-MC (R 2 0.85 and 0.95), and WSI-MC and AI-MC (R 2 0.77 and 0.83). Agreement in BR between modalities was high in both biopsies and resections: LM-MC and WSI-MC (κ 0.93 and 0.83, respectively), LM-MC and AI-MC (κ 0.89 and 0.83), and WSI-MC and AI-MC (κ 0.96 and 0.73). CONCLUSION: This first validation study shows that WSI-MC may compare better to LM-MC when using AI. Agreement between BR grade based on the different mitoses counting modalities was high. These results suggest that mitoses counting on WSI can well be done, and validate the presented AI algorithm for pathologist supervised use in daily practice. Further research is required to advance our knowledge of AI-MC, but it appears at least non-inferior to LM-MC

External validation and clinical utility assessment of PREDICT breast cancer prognostic model in young, systemic treatment-naïve women with node-negative breast cancer

Background: The validity of the PREDICT breast cancer prognostic model is unclear for young patients without adjuvant systemic treatment. This study aimed to validate PREDICT and assess its clinical utility in young women with node-negative breast cancer who did not receive systemic treatment. Methods: We selected all women from the Netherlands Cancer Registry who were diagnosed with node-negative breast cancer under age 40 between 1989 and 2000, a period when adjuvant systemic treatment was not standard practice for women with node-negative disease. We evaluated the calibration and discrimination of PREDICT using the observed/expected (O/E) mortality ratio, and the area under the receiver operating characteristic curve (AUC), respectively. Additionally, we compared the potential clinical utility of PREDICT for selectively administering chemotherapy to the chemotherapy-to-all strategy using decision curve analysis at predefined thresholds. Results: A total of 2264 women with a median age at diagnosis of 36 years were included. Of them, 71.2% had estrogen receptor (ER)-positive tumors and 44.0% had grade 3 tumors. Median tumor size was 16 mm. PREDICT v2.2 underestimated 10-year all-cause mortality by 33% in all women (O/E ratio:1.33, 95%CI:1.22–1.43). Model discrimination was moderate overall (AUC10-year:0.65, 95%CI:0.62–0.68), and poor for women with ER-negative tumors (AUC10-year:0.56, 95%CI:0.51–0.62). Compared to the chemotherapy-to-all strategy, PREDICT only showed a slightly higher net benefit in women with ER-positive tumors, but not in women with ER-negative tumors. Conclusions: PREDICT yields unreliable predictions for young women with node-negative breast cancer. Further model updates are needed before PREDICT can be routinely used in this patient subset.</p

Pros and cons of artificial intelligence implementation in diagnostic pathology

The rapid introduction of digital pathology has greatly facilitated development of artificial intelligence (AI) models in pathology that have shown great promise in assisting morphological diagnostics and quantitation of therapeutic targets. We are now at a tipping point where companies have started to bring algorithms to the market, and questions arise whether the pathology community is ready to implement AI in routine workflow. However, concerns also arise about the use of AI in pathology. This article reviews the pros and cons of introducing AI in diagnostic pathology