Radiology AI Deployment and Assessment Rubric (RADAR) to bring value-based AI into radiological practice

Objective: To provide a comprehensive framework for value assessment of artificial intelligence (AI) in radiology. Methods: This paper presents the RADAR framework, which has been adapted from Fryback and Thornbury’s imaging efficacy framework to facilitate the valuation of radiology AI from conception to local implementation. Local efficacy has been newly introduced to underscore the importance of appraising an AI technology within its local environment. Furthermore, the RADAR framework is illustrated through a myriad of study designs that help assess value. Results: RADAR presents a seven-level hierarchy, providing radiologists, researchers, and policymakers with a structured approach to the comprehensive assessment of value in radiology AI. RADAR is designed to be dynamic and meet the different valuation needs throughout the AI’s lifecycle. Initial phases like technical and diagnostic efficacy (RADAR-1 and RADAR-2) are assessed pre-clinical deployment via in silico clinical trials and cross-sectional studies. Subsequent stages, spanning from diagnostic thinking to patient outcome efficacy (RADAR-3 to RADAR-5), require clinical integration and are explored via randomized controlled trials and cohort studies. Cost-effectiveness efficacy (RADAR-6) takes a societal perspective on financial feasibility, addressed via health-economic evaluations. The final level, RADAR-7, determines how prior valuations translate locally, evaluated through budget impact analysis, multi-criteria decision analyses, and prospective monitoring.Conclusion: The RADAR framework offers a comprehensive framework for valuing radiology AI. Its layered, hierarchical structure, combined with a focus on local relevance, aligns RADAR seamlessly with the principles of value-based radiology. Critical relevance statement: The RADAR framework advances artificial intelligence in radiology by delineating a much-needed framework for comprehensive valuation. </p

Towards clinical implementation of an AI-algorithm for detection of cervical spine fractures on computed tomography

BackgroundArtificial intelligence (AI) applications can facilitate detection of cervical spine fractures on CT and reduce time to diagnosis by prioritizing suspected cases.PurposeTo assess the effect on time to diagnose cervical spine fractures on CT and diagnostic accuracy of a commercially available AI application.Materials and methodsIn this study (June 2020 - March 2022) with historic controls and prospective evaluation, we evaluated regulatory-cleared AI-software to prioritize cervical spine fractures on CT. All patients underwent non-contrast CT of the cervical spine. The time between CT acquisition and the moment the scan was first opened (DNT) was compared between the retrospective and prospective cohorts. The reference standard for determining diagnostic accuracy was the radiology report created in routine clinical workflow and adjusted by a senior radiologist. Discrepant cases were reviewed and clinical relevance of missed fractures was determined.Results2973 (mean age, 55.4 ± 19.7 [standard deviation]; 1857 men) patients were analyzed by AI, including 2036 retrospective and 938 prospective cases. Overall prevalence of cervical spine fractures was 7.6 %. The DNT was 18 % (5 min) shorter in the prospective cohort. In scans positive for cervical spine fracture according to the reference standard, DNT was 46 % (16 min) shorter in the prospective cohort. Overall sensitivity of the AI application was 89.8 % (95 % CI: 84.2–94.0 %), specificity was 95.3 % (95 % CI: 94.2–96.2 %), and diagnostic accuracy was 94.8 % (95 % CI: 93.8–95.8 %). Negative predictive value was 99.1 % (95 % CI: 98.5–99.4 %) and positive predictive value was 63.0 % (95 % CI: 58.0–67.8 %). 22 fractures were missed by AI of which 5 required stabilizing therapy.ConclusionA time gain of 16 min to diagnosis for fractured cases was observed after introducing AI. Although AI-assisted workflow prioritization of cervical spine fractures on CT shows high diagnostic accuracy, clinically relevant cases were missed

Towards clinical implementation of an AI-algorithm for detection of cervical spine fractures on computed tomography

BackgroundArtificial intelligence (AI) applications can facilitate detection of cervical spine fractures on CT and reduce time to diagnosis by prioritizing suspected cases.PurposeTo assess the effect on time to diagnose cervical spine fractures on CT and diagnostic accuracy of a commercially available AI application.Materials and methodsIn this study (June 2020 - March 2022) with historic controls and prospective evaluation, we evaluated regulatory-cleared AI-software to prioritize cervical spine fractures on CT. All patients underwent non-contrast CT of the cervical spine. The time between CT acquisition and the moment the scan was first opened (DNT) was compared between the retrospective and prospective cohorts. The reference standard for determining diagnostic accuracy was the radiology report created in routine clinical workflow and adjusted by a senior radiologist. Discrepant cases were reviewed and clinical relevance of missed fractures was determined.Results2973 (mean age, 55.4 ± 19.7 [standard deviation]; 1857 men) patients were analyzed by AI, including 2036 retrospective and 938 prospective cases. Overall prevalence of cervical spine fractures was 7.6 %. The DNT was 18 % (5 min) shorter in the prospective cohort. In scans positive for cervical spine fracture according to the reference standard, DNT was 46 % (16 min) shorter in the prospective cohort. Overall sensitivity of the AI application was 89.8 % (95 % CI: 84.2–94.0 %), specificity was 95.3 % (95 % CI: 94.2–96.2 %), and diagnostic accuracy was 94.8 % (95 % CI: 93.8–95.8 %). Negative predictive value was 99.1 % (95 % CI: 98.5–99.4 %) and positive predictive value was 63.0 % (95 % CI: 58.0–67.8 %). 22 fractures were missed by AI of which 5 required stabilizing therapy.ConclusionA time gain of 16 min to diagnosis for fractured cases was observed after introducing AI. Although AI-assisted workflow prioritization of cervical spine fractures on CT shows high diagnostic accuracy, clinically relevant cases were missed

Broadening the HTA of medical AI:A review of the literature to inform a tailored approach

Objectives: As current health technology assessment (HTA) frameworks do not provide specific guidance on the assessment of medical artificial intelligence (AI), this study aimed to propose a conceptual framework for a broad HTA of medical AI. Methods: A systematic literature review and a targeted search of policy documents was conducted to distill the relevant medical AI assessment elements. Three exemplary cases were selected to illustrate various elements: (1) An application supporting radiologists in stroke-care (2) A natural language processing application for clinical data abstraction (3) An ICU-discharge decision-making application. Results: A total of 31 policy documents and 9 academic publications were selected, from which a list of 29 issues was distilled. The issues were grouped by four focus areas: (1) Technology &amp; Performance, (2) Human &amp; Organizational, (3) Legal &amp; Ethical and (4) Transparency &amp; Usability. Each assessment element was extensively discussed in the test, and the elements clinical effectiveness, clinical workflow, workforce, interoperability, fairness and explainability were further highlighted through the exemplary cases. Conclusion: The current methodology of HTA requires extension to make it suitable for a broad evaluation of medical AI technologies. The 29-item assessment list that we propose needs a tailored approach for distinct types of medical AI, since the conceptualisation of the issues differs across applications.</p

Broadening the HTA of medical AI:A review of the literature to inform a tailored approach

Objectives: As current health technology assessment (HTA) frameworks do not provide specific guidance on the assessment of medical artificial intelligence (AI), this study aimed to propose a conceptual framework for a broad HTA of medical AI. Methods: A systematic literature review and a targeted search of policy documents was conducted to distill the relevant medical AI assessment elements. Three exemplary cases were selected to illustrate various elements: (1) An application supporting radiologists in stroke-care (2) A natural language processing application for clinical data abstraction (3) An ICU-discharge decision-making application. Results: A total of 31 policy documents and 9 academic publications were selected, from which a list of 29 issues was distilled. The issues were grouped by four focus areas: (1) Technology &amp; Performance, (2) Human &amp; Organizational, (3) Legal &amp; Ethical and (4) Transparency &amp; Usability. Each assessment element was extensively discussed in the test, and the elements clinical effectiveness, clinical workflow, workforce, interoperability, fairness and explainability were further highlighted through the exemplary cases. Conclusion: The current methodology of HTA requires extension to make it suitable for a broad evaluation of medical AI technologies. The 29-item assessment list that we propose needs a tailored approach for distinct types of medical AI, since the conceptualisation of the issues differs across applications.</p

Preoperative Classification of Peripheral Nerve Sheath Tumors on MRI Using Radiomics

Malignant peripheral nerve sheath tumors (MPNSTs) are aggressive soft-tissue tumors prevalent in neurofibromatosis type 1 (NF1) patients, posing a significant risk of metastasis and recurrence. Current magnetic resonance imaging (MRI) imaging lacks decisiveness in distinguishing benign peripheral nerve sheath tumors (BPNSTs) and MPNSTs, necessitating invasive biopsies. This study aims to develop a radiomics model using quantitative imaging features and machine learning to distinguish MPNSTs from BPNSTs. Clinical data and MRIs from MPNST and BPNST patients (2000–2019) were collected at a tertiary sarcoma referral center. Lesions were manually and semi-automatically segmented on MRI scans, and radiomics features were extracted using the Workflow for Optimal Radiomics Classification (WORC) algorithm, employing automated machine learning. The evaluation was conducted using a 100× random-split cross-validation. A total of 35 MPNSTs and 74 BPNSTs were included. The T1-weighted (T1w) MRI radiomics model outperformed others with an area under the curve (AUC) of 0.71. The incorporation of additional MRI scans did not enhance performance. Combining T1w MRI with clinical features achieved an AUC of 0.74. Experienced radiologists achieved AUCs of 0.75 and 0.66, respectively. Radiomics based on T1w MRI scans and clinical features show some ability to distinguish MPNSTs from BPNSTs, potentially aiding in the management of these tumors.</p

Endovascular repair versus open surgery in patients with ruptured abdominal aortic aneurysms: Clinical outcomes with 1-year follow-up

ObjectiveTo compare the clinical outcomes of treatment after endovascular repair and open surgery in patients with ruptured infrarenal abdominal aortic aneurysms (AAAs), including 1-year follow-up.MethodsAll consecutive conscious patients with ruptured infrarenal AAAs who presented to our tertiary care teaching hospital between January 1, 2001, and December 31, 2005, were included in this study (n = 55). Twenty-six patients underwent endovascular repair, and 29 patients underwent open surgery. Patients who were hemodynamically too unstable to undergo a computed tomography angiography scan were excluded. Outcomes evaluated were intraoperative mortality, 30-day mortality, systemic complications, complications necessitating surgical intervention, and mortality and complications during 1-year follow-up. The statistical tests we used were the Student t test, χ2 test, Fisher exact test, and Mann-Whitney U test (two sided; α = .05).ResultsThirty-day mortality was 8 (31%) of 26 patients who underwent endovascular repair and 9 (31%) of 29 patients who underwent open surgery (P = .98). Systemic complications and complications necessitating surgical intervention during the initial hospital stay were similar in both treatment groups (8/26 [31%] and 5/26 [19%] for endovascular repair, respectively, and 9/29 [31%] and 8/29 [28%] for open surgery, respectively; P > .40). During 1-year follow-up, two patients initially treated with endovascular repair died as a result of non–aneurysm-related causes; no death occurred in the open surgery group. Complications during 1-year follow-up were 1 (5%) of 20 for endovascular repair and 4 (16%) of 25 for open surgery (P = .36).ConclusionsOn the basis of our study with a highly selected population, the mortality and complication rates after endovascular repair may be similar compared with those after open surgery in patients treated for ruptured infrarenal AAAs

Preoperative Classification of Peripheral Nerve Sheath Tumors on MRI Using Radiomics

Malignant peripheral nerve sheath tumors (MPNSTs) are aggressive soft-tissue tumors prevalent in neurofibromatosis type 1 (NF1) patients, posing a significant risk of metastasis and recurrence. Current magnetic resonance imaging (MRI) imaging lacks decisiveness in distinguishing benign peripheral nerve sheath tumors (BPNSTs) and MPNSTs, necessitating invasive biopsies. This study aims to develop a radiomics model using quantitative imaging features and machine learning to distinguish MPNSTs from BPNSTs. Clinical data and MRIs from MPNST and BPNST patients (2000–2019) were collected at a tertiary sarcoma referral center. Lesions were manually and semi-automatically segmented on MRI scans, and radiomics features were extracted using the Workflow for Optimal Radiomics Classification (WORC) algorithm, employing automated machine learning. The evaluation was conducted using a 100× random-split cross-validation. A total of 35 MPNSTs and 74 BPNSTs were included. The T1-weighted (T1w) MRI radiomics model outperformed others with an area under the curve (AUC) of 0.71. The incorporation of additional MRI scans did not enhance performance. Combining T1w MRI with clinical features achieved an AUC of 0.74. Experienced radiologists achieved AUCs of 0.75 and 0.66, respectively. Radiomics based on T1w MRI scans and clinical features show some ability to distinguish MPNSTs from BPNSTs, potentially aiding in the management of these tumors.</p

A novel method to perform morphological measurements on three-dimensional (3D) models of the calcaneus based on computed tomography (CT)-imaging

Background: While current preoperative and postoperative assessment of the fractured and surgically reconstructed calcaneus relies on computed tomography (CT)-imaging, there are no established methods to quantify calcaneus morphology on CT-images. This study aims to develop a semi-automated method for morphological measurements of the calcaneus on three-dimensional (3D) models derived from CT-imaging. Methods: Using CT data, 3D models were created from healthy, fractured, and surgically reconstructed calcanei. Böhler's angle (BA) and Critical angle of Gissane (CAG) were measured on conventional lateral radiographs and corresponding 3D CT reconstructions using a novel point-based method with semiautomatic landmark placement by three observers. Intraobserver and interobserver reliability scores were calculated using intra-class correlation coefficient (ICC). In addition, consensus among observers was calculated for a maximal allowable discrepancy of 5 and 10 degrees for both methods. Results: Imaging data from 119 feet were obtained (40 healthy, 39 fractured, 40 reconstructed). Semiautomated measurements on 3D models of BA and CAG showed excellent reliability (ICC: 0.87-1.00). The manual measurements on conventional radiographs had a poor-to-excellent reliability (ICC: 0.22-0.96). In addition, the percentage of consensus among observers was much higher for the 3D method when compared to conventional two-dimensional (2D) measurements. Conclusions: The proposed method enables reliable and reproducible quantification of calcaneus morphology in 3D models of healthy, fractured and reconstructed calcanei.</p