Impact of Different Mammography Systems on Artificial Intelligence Performance in Breast Cancer Screening

Peer reviewedPostprin

Service and clinical impacts of reader bias in breast cancer screening : a retrospective study

Acknowledgements iCAIRD Radiology Collaboration: Corri Black, Alison D. Murray and Katie Wilde, University of Aberdeen. James D. Blackwood, NHS Greater Glasgow and Clyde. Claire Butterly and John Zurowski, University of Glasgow. Jon Eilbeck and Colin McSkimming, NHS Grampian. Canon Medical Research Europe Ltd. – SHAIP platform. We would like to acknowledge the support of the Grampian Data Safe Haven (DaSH) facility within the Aberdeen Centre for Health Data Science and the associated financial support of the University of Aberdeen, and NHS Research Scotland (through NHS Grampian investment in DaSH). Funding This work is supported by the Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD). iCAIRD was funded by Innovate UK on behalf of UK Research and Innovation (UKRI) [project number: 104690]. The funders were not involved in the study’s design; the collection, analysis or interpretation of the data; or the decision to submit the manuscript for publication.Peer reviewedPublisher PD

Use of “Hidden in Plain Sight” de-identification methodology in electronic healthcare data provides minimal risk of misidentification: Results from the iCAIRD Safe Haven Artificial Intelligence Platform.

Objectives To determine the risk of misidentification when using a “Hidden In Plain Sight (HIPS)” Named Entity Recognition (NER) de-identification methodology applied to Scottish healthcare data within The Industrial Centre for Artificial Intelligence Research in Digital Diagnostics (iCAIRD) Safe Haven Artificial Intelligence Platform (SHAIP). Approach Rather than the traditional redaction of potential identifiable information in routinely collected healthcare data, our HIPS methodology utilises an NER “find and replace” approach to de-identification that keeps the structure of text intact. This ensures that context is maintained, key to the interpretation of free text information and potential Artificial Intelligence applications. To our knowledge these methods have been previously untested on Scottish healthcare data. We therefore performed assessment of this approach in terms of potential risk of misidentification using HIPS on structured Scottish data deployed in SHAIP as part of the iCAIRD programme. Results Five individual cohorts, with a total of 169,964 patients were included. For each cohort the HIPS approach was applied, and then compared to actual patient information from within the same region, in order to determine the risk of misidentification. The following fields were included: Forename, Surname, Previous Name, Gender, Date of Birth (DOB), and Postcode. Across the five cohorts and varying combinations of identifiable data fields there were a total of 94 instances of potential misidentification (0.06%). 85/94 (90.4%) of these were for the combination of Gender, Date of Birth and Postcode. Across the five cohorts there were only 3 instances (0.002%) of Forename/Surname/DOB, and 5 instances (0.003%) of Forename/Surname/Postcode potential misidentification amongst the 169,964 patients. Conclusions The iCAIRD NER HIPS Methodology provides an acceptably low misidentification rate. Further work is now required to determine the recall and precision rates. Benefits of this approach include retaining the structure of free text, as well as reducing the ability to detect any potential leaked identifiable data

Impact of different mammography systems on artificial intelligence performance in breast cancer screening

Artificial intelligence (AI) tools may assist breast screening mammography programs, but limited evidence supports their generalizability to new settings. This retrospective study used a 3-year dataset (April 1, 2016–March 31, 2019) from a U.K. regional screening program. The performance of a commercially available breast screening AI algorithm was assessed with a prespecified and site-specific decision threshold to evaluate whether its performance was transferable to a new clinical site. The dataset consisted of women (aged approximately 50–70 years) who attended routine screening, excluding self-referrals, those with complex physical requirements, those who had undergone a previous mastectomy, and those who underwent screening that had technical recalls or did not have the four standard image views. In total, 55 916 screening attendees (mean age, 60 years ± 6 [SD]) met the inclusion criteria. The prespecified threshold resulted in high recall rates (48.3%, 21 929 of 45 444), which reduced to 13.0% (5896 of 45 444) following threshold calibration, closer to the observed service level (5.0%, 2774 of 55 916). Recall rates also increased approximately threefold following a software upgrade on the mammography equipment, requiring per–software version thresholds. Using software-specific thresholds, the AI algorithm would have recalled 277 of 303 (91.4%) screen-detected cancers and 47 of 138 (34.1%) interval cancers. AI performance and thresholds should be validated for new clinical settings before deployment, while quality assurance systems should monitor AI performance for consistency