A multicenter assessment of interreader reliability of LI-RADS version 2018 for MRI and CT

Background: Various limitations have impacted research evaluating reader agreement for Liver Imaging-Reporting and Data System (LI-RADS). Purpose: To assess reader agreement of LI-RADS in an international multi-center, multireader setting using scrollable images. Materials and Methods: This retrospective study used de-identified clinical multiphase CT and MRI examinations and reports with at least one untreated observation from six institutions and three countries; only qualifying examinations were submitted. Examination dates were October 2017 – August 2018 at the coordinating center. One untreated observation per examination was randomly selected using observation identifiers, and its clinically assigned features were extracted from the report. The corresponding LI-RADS v2018 category was computed as a re-scored clinical read. Each examination was randomly assigned to two of 43 research readers who independently scored the observation. Agreement for an ordinal modified four-category LI-RADS scale (LR-1/2, LR-3, LR-4, LR-5/M/tumor in vein) was computed using intra-class correlation coefficients (ICC). Agreement was also computed for dichotomized malignancy (LR-4/LR5/LR-M/LR-tumor in vein), LR-5, and LR-M. Agreement was compared between researchversus-research reads and research-versus-clinical reads. Results: 484 patients (mean age, 62 years ±10 [SD]; 156 women; 93 CT, 391 MRI) were included. ICCs for ordinal LI-RADS, dichotomized malignancy, LR-5, and LR-M were 0.68 (95% CI: 0.62, 0.74), 0.63 (95% CI: 0.56, 0.71), 0.58 (95% CI: 0.50, 0.66), and 0.46 (95% CI: 0.31, 0.61) respectively. Research-versus-research reader agreement was higher than research-versus-clinical agreement for modified four-category LI-RADS (ICC, 0.68 vs. 0.62, P = .03) and for dichotomized malignancy (ICC, 0.63 vs. 0.53, P = .005), but not for LR-5 (P = .14) or LR-M (P = .94). Conclusion: There was moderate agreement for Liver Imaging-Reporting and Data System v2018 overall. For some comparisons, research-versus-research reader agreement was higher than research-versus-clinical reader agreement, indicating differences between the clinical and research environments that warrant further study