A head-to-head comparison of the inter- and intraobserver agreement of COVID-RADS and CO-RADS grading systems in a population with high estimated prevalence of COVID-19

Purpose To evaluate the inter- and intraobserver agreement of COVID-RADS and CO-RADS reporting systems among differently experienced radiologists in a population with high estimated prevalence of COVID-19. Materials and Methods Chest CT scans of patients with clinically-epidemiologically diagnosed COVID-19 were retrieved from an open-source MosMedData dataset, randomised, and independently assigned COVID-RADS and CO-RADS grades by an abdominal radiology fellow, thoracic imaging fellow and a consultant cardiothoracic radiologist. The inter- and intraobserver agreement of the two systems were assessed using the Fleiss’ and Cohen’s kappa coefficients, respectively. Results A total of 200 studies were included in the analysis. Both systems demonstrated moderate interobserver agreement, with kappa values of 0.51 (95% CI: 0.46-0.56) and 0.55 (95% CI: 0.50-0.59) for COVID-RADS and CO-RADS, respectively. When COVID-RADS and CO-RADS grades were dichotomised at cut-off values of 2B and 4 to evaluate the agreement between grades representing different levels of clinical suspicion for COVID-19, the interobserver agreement became substantial with kappa values of 0.74 (95% CI: 0.66-0.82) for COVID-RADS and 0.73 (95% CI: 0.65-0.81) for CO-RADS. The median intraobserver agreement was considerably higher for CO-RADS reaching 0.81 (95% CI: 0.43-0.76) compared with 0.60 (95% CI: 0.43-0.76) of COVID-RADS. Conclusions COVID-RADS and CO-RADS showed comparable interobserver agreement, which was moderate when grades were compared head-to-head and substantial when grades were dichotomised to better reflect the underlying levels of suspicion for COVID-19. The median intraobserver agreement of CO-RADS was, however, considerably higher compared with COVID-RADS. Advances in knowledge This paper provides a comprehensive review of the newly introduced COVID-19 chest CT reporting systems, which will help radiologists of all sub-specialties and experience levels make an informed decision on which system to use in their own practice.Acknowledgments: The authors acknowledge support from Cancer Research UK, National Institute of Health Research Cambridge Biomedical Research Centre, Cancer Research UK and the Engineering and Physical Sciences Research Council Imaging Centre in Cambridge and Manchester and the Cambridge Experimental Cancer Medicine Centre