Can Machine Learning Be Better than Biased Readers?

Background: Training machine learning (ML) models in medical imaging requires large amounts of labeled data. To minimize labeling workload, it is common to divide training data among multiple readers for separate annotation without consensus and then combine the labeled data for training a ML model. This can lead to a biased training dataset and poor ML algorithm prediction performance. The purpose of this study is to determine if ML algorithms can overcome biases caused by multiple readers’ labeling without consensus. Methods: This study used a publicly available chest X-ray dataset of pediatric pneumonia. As an analogy to a practical dataset without labeling consensus among multiple readers, random and systematic errors were artificially added to the dataset to generate biased data for a binary-class classification task. The Resnet18-based convolutional neural network (CNN) was used as a baseline model. A Resnet18 model with a regularization term added as a loss function was utilized to examine for improvement in the baseline model. Results: The effects of false positive labels, false negative labels, and random errors (5–25%) resulted in a loss of AUC (0–14%) when training a binary CNN classifier. The model with a regularized loss function improved the AUC (75–84%) over that of the baseline model (65–79%). Conclusion: This study indicated that it is possible for ML algorithms to overcome individual readers’ biases when consensus is not available. It is recommended to use regularized loss functions when allocating annotation tasks to multiple readers as they are easy to implement and effective in mitigating biased labels

Error Consistency for Machine Learning Evaluation and Validation with Application to Biomedical Diagnostics

Supervised machine learning classification is the most common example of artificial intelligence (AI) in industry and in academic research. These technologies predict whether a series of measurements belong to one of multiple groups of examples on which the machine was previously trained. Prior to real-world deployment, all implementations need to be carefully evaluated with hold-out validation, where the algorithm is tested on different samples than it was provided for training, in order to ensure the generalizability and reliability of AI models. However, established methods for performing hold-out validation do not assess the consistency of the mistakes that the AI model makes during hold-out validation. Here, we show that in addition to standard methods, an enhanced technique for performing hold-out validation—that also assesses the consistency of the sample-wise mistakes made by the learning algorithm—can assist in the evaluation and design of reliable and predictable AI models. The technique can be applied to the validation of any supervised learning classification application, and we demonstrate the use of the technique on a variety of example biomedical diagnostic applications, which help illustrate the importance of producing reliable AI models. The validation software created is made publicly available, assisting anyone developing AI models for any supervised classification application in the creation of more reliable and predictable technologies

Analyzing Barriers and Enablers for the Acceptance of Artificial Intelligence Innovations into Radiology Practice: A Scoping Review

Objectives: This scoping review was conducted to determine the barriers and enablers associated with the acceptance of artificial intelligence/machine learning (AI/ML)-enabled innovations into radiology practice from a physician’s perspective. Methods: A systematic search was performed using Ovid Medline and Embase. Keywords were used to generate refined queries with the inclusion of computer-aided diagnosis, artificial intelligence, and barriers and enablers. Three reviewers assessed the articles, with a fourth reviewer used for disagreements. The risk of bias was mitigated by including both quantitative and qualitative studies. Results: An electronic search from January 2000 to 2023 identified 513 studies. Twelve articles were found to fulfill the inclusion criteria: qualitative studies (n = 4), survey studies (n = 7), and randomized controlled trials (RCT) (n = 1). Among the most common barriers to AI implementation into radiology practice were radiologists’ lack of acceptance and trust in AI innovations; a lack of awareness, knowledge, and familiarity with the technology; and perceived threat to the professional autonomy of radiologists. The most important identified AI implementation enablers were high expectations of AI’s potential added value; the potential to decrease errors in diagnosis; the potential to increase efficiency when reaching a diagnosis; and the potential to improve the quality of patient care. Conclusions: This scoping review found that few studies have been designed specifically to identify barriers and enablers to the acceptance of AI in radiology practice. The majority of studies have assessed the perception of AI replacing radiologists, rather than other barriers or enablers in the adoption of AI. To comprehensively evaluate the potential advantages and disadvantages of integrating AI innovations into radiology practice, gathering more robust research evidence on stakeholder perspectives and attitudes is essential

Missed Breast Cancers on MRI in High-Risk Patients: A Retrospective Case–Control Study

Purpose: To determine if MRI features and molecular subtype influence the detectability of breast cancers on MRI in high-risk patients. Methods and Materials: Breast cancers in a high-risk population of 104 patients were diagnosed following MRI describing a BI-RADS 4–5 lesion. MRI characteristics at the time of diagnosis were compared with previous MRI, where a BI-RADS 1–2–3 lesion was described. Results: There were 77 false-negative MRIs. A total of 51 cancers were overlooked and 26 were misinterpreted. There was no association found between MRI characteristics, the receptor type and the frequency of missed cancers. The main factors for misinterpreted lesions were multiple breast lesions, prior biopsy/surgery and long-term stability. Lesions were mostly overlooked because of their small size and high background parenchymal enhancement. Among missed lesions, 50% of those with plateau kinetics on initial MRI changed for washout kinetics, and 65% of initially progressively enhancing lesions then showed plateau or washout kinetics. There were more basal-like tumours in BRCA1 carriers (50%) than in non-carriers (13%), p = 0.0001, OR = 6.714, 95% CI = [2.058–21.910]. The proportion of missed cancers was lower in BRCA carriers (59%) versus non-carriers (79%), p < 0.05, OR = 2.621, 95% CI = [1.02–6.74]. Conclusions: MRI characteristics or molecular subtype do not influence breast cancer detectability. Lesions in a post-surgical breast should be assessed with caution. Long-term stability does not rule out malignancy and multimodality evaluation is of added value. Lowering the biopsy threshold for lesions with an interval change in kinetics for a type 2 or 3 curve should be considered. There was a higher rate of interval cancers in BRCA 1 patients attributed to lesions more aggressive in nature

Early detection of sensorineural hearing loss in Muckle-Wells-syndrome

Abstract Background Muckle-Wells-syndrome (MWS) is an autoinflammatory disease characterized by systemic and organ-specific inflammation due to excessive interleukin (IL)-1 release. Inner ear inflammation results in irreversible sensorineural hearing loss, if untreated. Early recognition and therapy may prevent deafness. The aims of the study were to characterize the spectrum of hearing loss, optimize the otologic assessment for early disease and determine responsiveness to anti-IL-1-therapy regarding hearing. Methods A single center prospective cohort study of children and adults with MWS was performed. Standardized clinical, laboratory and otologic assessments including standard pure tone audiometry, additional high tone thresholds, vestibular organ testing, tinnitus evaluation and functional disability classes were determined serially. Pure-tone-average models were developed and evaluated. Risk factors for hearing loss and the impact of anti-IL-1 treatment were determined. Results A total of 23 patients with genetically confirmed MWS were included, of whom 63 % were females; 52 % were children. At baseline all patients had active MWS; 91 % reported clinically impaired hearing with 74 % having an abnormal standard assessment (0.5–4 kHz). In contrast, high frequency pure tone averages (HF-PTA) were abnormal in all symptomatic patients including those with early hearing loss (sensitivity 100 %). Females were at highest risk for hearing loss even after adjustment for age (p = 0.008). Treatment with IL-1 blockade resulted in improved or stable hearing in 91 % of patients. Conclusions Early inner ear inflammation in MWS primarily affects the high frequencies, beyond the range of standard otologic assessment tools. The HF-PTA is a sensitive tool to detect imminent hearing loss and monitor treatment response