Inter- and intra-observer agreement of BI-RADS-based subjective visual estimation of amount of fibroglandular breast tissue with magnetic resonance imaging: comparison to automated quantitative assessment.

PURPOSE: To evaluate the inter-/intra-observer agreement of BI-RADS-based subjective visual estimation of the amount of fibroglandular tissue (FGT) with magnetic resonance imaging (MRI), and to investigate whether FGT assessment benefits from an automated, observer-independent, quantitative MRI measurement by comparing both approaches. MATERIALS AND METHODS: Eighty women with no imaging abnormalities (BI-RADS 1 and 2) were included in this institutional review board (IRB)-approved prospective study. All women underwent un-enhanced breast MRI. Four radiologists independently assessed FGT with MRI by subjective visual estimation according to BI-RADS. Automated observer-independent quantitative measurement of FGT with MRI was performed using a previously described measurement system. Inter-/intra-observer agreements of qualitative and quantitative FGT measurements were assessed using Cohen's kappa (k). RESULTS: Inexperienced readers achieved moderate inter-/intra-observer agreement and experienced readers a substantial inter- and perfect intra-observer agreement for subjective visual estimation of FGT. Practice and experience reduced observer-dependency. Automated observer-independent quantitative measurement of FGT was successfully performed and revealed only fair to moderate agreement (k = 0.209-0.497) with subjective visual estimations of FGT. CONCLUSION: Subjective visual estimation of FGT with MRI shows moderate intra-/inter-observer agreement, which can be improved by practice and experience. Automated observer-independent quantitative measurements of FGT are necessary to allow a standardized risk evaluation. KEY POINTS: • Subjective FGT estimation with MRI shows moderate intra-/inter-observer agreement in inexperienced readers. • Inter-observer agreement can be improved by practice and experience. • Automated observer-independent quantitative measurements can provide reliable and standardized assessment of FGT with MRI

Development and evaluation of machine learning in whole-body magnetic resonance imaging for detecting metastases in patients with lung or colon cancer: a diagnostic test accuracy study.

OBJECTIVES: Whole-body magnetic resonance imaging (WB-MRI) has been demonstrated to be efficient and cost-effective for cancer staging. The study aim was to develop a machine learning (ML) algorithm to improve radiologists' sensitivity and specificity for metastasis detection and reduce reading times. MATERIALS AND METHODS: A retrospective analysis of 438 prospectively collected WB-MRI scans from multicenter Streamline studies (February 2013-September 2016) was undertaken. Disease sites were manually labeled using Streamline reference standard. Whole-body MRI scans were randomly allocated to training and testing sets. A model for malignant lesion detection was developed based on convolutional neural networks and a 2-stage training strategy. The final algorithm generated lesion probability heat maps. Using a concurrent reader paradigm, 25 radiologists (18 experienced, 7 inexperienced in WB-/MRI) were randomly allocated WB-MRI scans with or without ML support to detect malignant lesions over 2 or 3 reading rounds. Reads were undertaken in the setting of a diagnostic radiology reading room between November 2019 and March 2020. Reading times were recorded by a scribe. Prespecified analysis included sensitivity, specificity, interobserver agreement, and reading time of radiology readers to detect metastases with or without ML support. Reader performance for detection of the primary tumor was also evaluated. RESULTS: Four hundred thirty-three evaluable WB-MRI scans were allocated to algorithm training (245) or radiology testing (50 patients with metastases, from primary 117 colon [n = 117] or lung [n = 71] cancer). Among a total 562 reads by experienced radiologists over 2 reading rounds, per-patient specificity was 86.2% (ML) and 87.7% (non-ML) (-1.5% difference; 95% confidence interval [CI], -6.4%, 3.5%; P = 0.39). Sensitivity was 66.0% (ML) and 70.0% (non-ML) (-4.0% difference; 95% CI, -13.5%, 5.5%; P = 0.344). Among 161 reads by inexperienced readers, per-patient specificity in both groups was 76.3% (0% difference; 95% CI, -15.0%, 15.0%; P = 0.613), with sensitivity of 73.3% (ML) and 60.0% (non-ML) (13.3% difference; 95% CI, -7.9%, 34.5%; P = 0.313). Per-site specificity was high (>90%) for all metastatic sites and experience levels. There was high sensitivity for the detection of primary tumors (lung cancer detection rate of 98.6% with and without ML [0.0% difference; 95% CI, -2.0%, 2.0%; P = 1.00], colon cancer detection rate of 89.0% with and 90.6% without ML [-1.7% difference; 95% CI, -5.6%, 2.2%; P = 0.65]). When combining all reads from rounds 1 and 2, reading times fell by 6.2% (95% CI, -22.8%, 10.0%) when using ML. Round 2 read-times fell by 32% (95% CI, 20.8%, 42.8%) compared with round 1. Within round 2, there was a significant decrease in read-time when using ML support, estimated as 286 seconds (or 11%) quicker (P = 0.0281), using regression analysis to account for reader experience, read round, and tumor type. Interobserver variance suggests moderate agreement, Cohen κ = 0.64; 95% CI, 0.47, 0.81 (with ML), and Cohen κ = 0.66; 95% CI, 0.47, 0.81 (without ML). CONCLUSIONS: There was no evidence of a significant difference in per-patient sensitivity and specificity for detecting metastases or the primary tumor using concurrent ML compared with standard WB-MRI. Radiology read-times with or without ML support fell for round 2 reads compared with round 1, suggesting that readers familiarized themselves with the study reading method. During the second reading round, there was a significant reduction in reading time when using ML support

O-RADS MRI classification of indeterminate adnexal lesions: time-intensity curve analysis is better than visual assessment.

Background The MRI Ovarian-Adnexal Reporting and Data System (O-RADS) enables risk stratification of sonographically indeterminate adnexal lesions, partly based on time-intensity curve (TIC) analysis, which may not be universally available. Purpose To compare the diagnostic accuracy of visual assessment with that of TIC assessment of dynamic contrast-enhanced MRI scans to categorize adnexal lesions as benign or malignant and to evaluate the influence on the O-RADS MRI score. Materials and Methods The European Adnex MR Study Group, or EURAD, database, a prospective multicenter study of women undergoing MRI for indeterminate adnexal lesions between March 2013 and March 2018, was queried retrospectively. Women undergoing surgery for an adnexal lesion with solid tissue were included. Solid tissue enhancement relative to outer myometrium was assessed visually and with TIC. Contrast material washout was recorded. Lesions were categorized according to the O-RADS MRI score with visual and TIC assessment. Per-lesion diagnostic accuracy was calculated. Results A total of 320 lesions (207 malignant, 113 benign) in 244 women (mean age, 55.3 years ± 15.8 [standard deviation]) were analyzed. Sensitivity for malignancy was 96% (198 of 207) and 76% (157 of 207) for TIC and visual assessment, respectively. TIC was more accurate than visual assessment (86% [95% CI: 81, 90] vs 78% [95% CI: 73, 82]; P < .001) for benign lesions, predominantly because of higher specificity (95% [95% CI: 92, 98] vs 76% [95% CI: 68, 81]). A total of 21% (38 of 177) of invasive lesions were rated as low risk visually. Contrast material washout and high-risk enhancement (defined as earlier enhancement than in the myometrium) were highly specific for malignancy for both TIC (97% [95% CI: 91, 99] and 94% [95% CI: 90, 97], respectively) and visual assessment (97% [95% CI: 92, 99] and 93% [95% CI: 88, 97], respectively). O-RADS MRI score was more accurate with TIC than with visual assessment (area under the receiver operating characteristic curve, 0.87 [95% CI: 0.83, 0.90] vs 0.73 [95% CI: 0.68, 0.78]; P < .001). Conclusion Time-intensity curve analysis was more accurate than visual assessment for achieving optimal diagnostic accuracy with the Ovarian-Adnexal Reporting and Data System MRI score. Clinical trial registration no. NCT01738789 © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Vargas and Woo in this issue