Fehr, J

Grant, A

Gunda, R

Hanekom, W

Lippert, C

Ndung'u, T

Siedner, MJ

Wong, EB

English

UCL Discovery

INT J TUBERC LUNG DIS 27(2):157–160Q 2023 The Unionhttp://dx.doi.org/10.5588/ijtld.22.0437LETTERCAD4TB software updates: different triaging thresholds requirecaution by users and regulation by authoritiesDear Editor,The recent recommendations by the WHO forsystematic screening for TB with digital chest X-ray(CXR) and automated imaging interpretation1 hasled to an explosion in the use of computer-assisteddiagnostic (CAD) algorithms. We previously foundthat the performance of CAD4TB (QDelft Imaging,Hertogenbosch, The Netherlands) is comparable to ahuman radiologist during community-based TBscreening in rural South Africa.2 CAD4TB quantifieslung field abnormalities suggestive of active TB,assigning a score between 0 and 100. Using CAD4TBrequires screening programmes to select a triagingthreshold above which participants receive sputumtesting. Triaging thresholds are not universal andrequire adjustment based on demographic character-istics, laboratory capacities, budget, healthcare set-tings and programmatic goals.2–7 CAD4TB isupdated annually, and the 7th version has beenreleased recently. Screening programmes might beeager to use new versions because studies reportimproved performance,8,9 but no recommendationson adopting software updates currently exist. Here,we have evaluated the triaging performance charac-teristics and optimal thresholds of the latest version ofCAD4TB (v7) compared to the two most recentversions (v5 and v6).During the first year of a community-based multi-morbidity study in a rural district in KwaZulu-Natal,South Africa, 9,912 local residents above 15 years ofage received free TB screening at a mobile camp (asdescribed previously).2,10 Briefly, TB screening in-cluded digital posterior-anterior CXR imaging andassessment of symptoms. Following WHO guidelinesfor TB prevalence surveys,11 participants were triagedfor sputum collection for any TB-related symptom(fever, weight loss, cough or night sweats) or for anyCXR lung field abnormality. CXRs were analysedusing CAD4TB v5 and scored between 0 and 100 toindicate the likelihood of TB-related lung fieldabnormality. As described previously,2 those withCAD4TB v5 .25 (a triaging threshold with asensitivity of 85% for lung field abnormality)2 weretriaged for sputum examination using Xpertw MTB/RIF Ultra (Cepheid, Sunnyvale, CA, USA) andMGITe (BD, Franklin Lakes, NJ, USA) liquidculture, and defined as microbiologically confirmedTB if either test was positive. Among the 9,912participants who underwent CXR, 5,594 (56.4%)were referred for sputum testing, 4,976 (89.0%) ofwhom were able to produce sputum. A total of 99(1.0%) participants had microbiologically positivesputum. A senior radiologist (blinded to CAD4TBscores and patient information) interpreted CXRs ashaving normal or abnormal lung fields. CXRs wereretrospectively analysed using CAD4TB v6 and v7.The distribution of CAD4TB scores (v5–v7) andpercentage of participants required to test werecompared among all CXRs (n¼ 9,912). Performancecharacteristics and triage threshold that most closelymatched the radiologist’s performance were com-pared (v5–v7) among individuals with sputum testresults (n ¼ 4,976). Participants provided writteninformed consent to participate in the study. Ethicsapproval was obtained from the University ofKwaZulu-Natal Biomedical Research Ethics Com-mittee (BE560/17), KwaZulu-Natal, South Africa;the London School of Hygiene & Tropical MedicineEthics Committee (14722), London, UK; and theMass General Brigham Institutional Review Board,Boston, MA, USA (2018P001802).The overall performance between CAD4TB v5, v6and v7 (area under the curve [AUC] v5: 0.78, 95% CI0.73–0.83; v6: 0.79, 95% CI 0.73–0.84; v7: 0.80,95% CI 0.75–0.85; P . 0.1; Figure Panel A) wassimilar, but the distribution of scores across the 100-point scale varied greatly across the three versions(median scores were v5: 28, interquartile range [IQR]22–41; v6: 35, IQR 16–46; and v7: 11, IQR 5.2–27; P, 0.001; Figure Panel B). Between the three versions,each numerical threshold had strikingly differentperformance. For example, triaging with a CAD4TBthreshold of 40 would result in a range of screeningsensitivities (v5: 79.8%, v6: 88.9%, v7: 66.7%) andspecificities (v5: 57.4%, v6: 33.3%, v7: 84.6%). Asno threshold from any version met the WHO targetproduct profile of 90% sensitivity and 70%specificity,12 we identified one threshold for eachCAD4TB version that most closely matched theradiologist sensitivity at 80.8% (95% CI 71.7–88.0). The matching thresholds were v5: 40(79.8%, 95% CI 70.5–87.2); v6: 47 (82.8%, 95%CI 73.9–89.7); and v7: 20 (79.8%, 95% CI 70.5–87.2). At these thresholds, the three CAD4TBversions had lower specificity than the radiologist(radiologist: 66.9%, 95% CI 65.6–68.2; v5: 40,Article submitted 9 August 2022. Final version accepted 13 October 2022.57.4%, 95% CI 56.0–58.8; v6: 47, 62.6%, 95% CI61.2–64.0; v7: 20, 56.6%, 95% CI 55.2–58.0),leading to a higher percentage of participants whowould require microbiological sputum testing relativeto all participants (n¼9,912) (radiologist: 20.2%; v5:40, 27.0%; v6: 47, 23.7%; v7: 20, 33.5%; FigurePanel C and Supplementary Data S1). Substantialvariations were also observed in the number of casesof microbiologically positive sputum that would be‘missed’ using potential triaging thresholds for thedifferent CAD4TB versions (Figure Panel D). Forexample, triaging with CAD4TB threshold 40, wouldresult in sputum testing for 27.0% (v5), 45.9% (v6)and 10.3% (v7) of participants. At the samethreshold, the percentage of microbiologically con-firmed TB cases missed would be 20.2% (v5), 11.1%(v6) and 33.3% (v7). To note, despite previousreports that showed improved performance withnewer versions,8,9 in these real-world data v7 didnot outperform v6, as measured by AUC andspecificity matched at the radiologist sensitivity.Despite similar AUC, v7 performed at higherspecificity but lower sensitivity at each triagingthreshold compared to v5 and v6 (SupplementaryTable S1).The change in scales and resulting wide variationsin triaging thresholds between different CAD4TBversions poses a risk to end-users in TB screeningFigure Performance of CAD4TB v5, v6 and v7 to identify microbiologically confirmed TB. TB was defined if sputum was found to bepositive on either Xpert Ultra or microbiological culture. A) For individuals with sputum results (n¼ 4,976), performance is shown interms of sensitivity and specificity and AUC. Annotations show thresholds that closest matched the radiologist’s sensitivity; B)distributions and most frequent CAD4TB scores of all three versions obtained for all chest X-rays (n ¼ 9,912); C) percentage ofparticipants triaged for sputum testing among all participants at each CAD4TB threshold (n¼9,912); D) percentage of missed positivesputum among TB-positive individuals (n¼99) at each CAD4TB threshold. The performance of the senior radiologist is marked with across (A) and dashed lines (C, D). CAD4TB thresholds that matched the radiologist’s performance (v5: 40, v6: 47, v7: 20) are markedwith numbers (A) and grey vertical lines (C, D). AUC¼ area under the receiver-operating curve.158 The International Journal of Tuberculosis and Lung Diseaseprogrammes who may unintentionally introducesystematic screening errors by adopting softwareupdates without adjusting the selected triagingthresholds. Using incorrect triaging thresholds mayhave severe consequences and result in missing peoplewith TB (triage threshold inadvertently too high) orutilising microbiological testing excessively (triagingthreshold inadvertently too low). To accommodateintra-version variation, screening programmes needto select new triaging thresholds for each newsoftware update. Previous work2,13,14 and the devel-oper15 suggest that it is necessary to conduct pilotstudies to finding triage thresholds that optimallyserve the goals of each screening exercise. It is nowunclear whether each software update requires newpiloting for re-adjustment or whether this can beachieved through retrospective analysis of the newestversion’s performance against population specificCXR collections. It is unknown whether our findingsof significant variation between CAD4TB versions isapplicable to other image interpretation algorithmsused for TB screening – this information needs to beestablished urgently.15 For anyone designing TBscreening programmes, decisions about programmat-ic adjustments to new versions are especially difficultbecause the underlying algorithmic or data changesbetween software versions are not communicated bymanufacturers. Changes to the underlying referencestandard for algorithm training may require re-adjustment of triaging thresholds, whereas smallchanges for faster radiograph interpretation, mightnot. However, information about the changes be-tween versions is not transparently shared with thecommunity because it has been considered proprie-tary by developers.15Based on the results presented here, we call forregulation to require CAD-developing companies tocommunicate changes between software versions andgive guidance for medical or public health end-usersto effectively adopt software version updates in TBscreening programmes. Continued vigilance andperformance auditing of successive CAD softwareversions should be an integral requirement forauthorisation by the WHO and regulatory agencies.These findings also contribute to ongoing scientificdebates on how to successfully adopt artificialintelligence-based tools for healthcare.J. FEHR,1,2,3 R. GUNDA,1,4,5 M. J. SIEDNER,1,6,7,8W. HANEKOM,1 T. NDUNG’U,1,5,9,10 A. GRANT,1,4,11,12C. LIPPERT,2,3,13 E. B. WONG1,141Africa Health Research Institute, KwaZulu-Natal,South Africa; 2Digital Health & Machine Learning,Hasso Plattner Institute for Digital Engineering,Potsdam, 3Digital Engineering Faculty, University ofPotsdam, Potsdam, Germany; 4School of Nursingand Public Health, College of Health Sciences,University of KwaZulu-Natal, KwaZulu-Natal,South Africa; 5Division of Infection and Immunity,University College London, London, UK; 6School ofClinical Medicine, College of Health Sciences,University of KwaZulu-Natal, KwaZulu-Natal,South Africa; 7Harvard Medical School, Boston,MA, 8Division of Infectious Diseases, MassachusettsGeneral Hospital, Boston, MA, USA; 9HIVPathogenesis Programme, The Doris Duke MedicalResearch Institute, University of KwaZulu-Natal,Durban, South Africa; 10Ragon Institute of MGH,MIT and Harvard University, Cambridge, MA, USA,11London School of Hygiene & Tropical Medicine,London, UK, 12School of Clinical Medicine, Collegeof Health Sciences, University of KwaZulu-Natal,KwaZulu-Natal, South Africa; 13Hasso PlattnerInstitute for Digital Health at Mount Sinai, IcahnSchool of Medicine at Mount Sinai, New York, NY,14Division of Infectious Diseases, University ofAlabama at Birmingham, AL, USACorrespondence to: Emily B Wong, K-RITH Tower,Nelson R Mandela School of Medicine, 719 UmbiloRd, Durban, Kwa-Zulu Natal 4001, South Africa.E-Mail: emily.wong@ahri.orgAcknowledgmentsThis research was funded in part, by the Wellcome Trust, London,UK (Grant number 201433/Z/16/A).This is an Open Access article distributed under the terms of theCreative Commons Attribution License, which permits unrestricteduse, distribution, and reproduction in any medium, provided theoriginal author and source are credited.References1 World Health Organization. WHO consolidated guidelines ontuberculosis. Module 2: Screening Systematic screening fortuberculosis disease. Geneva, Switzerland: WHO, 2021.2 Fehr J, et al. Computer-aided interpretation of chest radiogra-phy reveals the spectrum of tuberculosis in rural South Africa.Npj Digit Med 2021;4(1):20.3 Zaidi SMA, et al. Evaluation of the diagnostic accuracy ofComputer-Aided Detection of tuberculosis on chest radiogra-phy among private sector patients in Pakistan. Sci Rep2018;8(1):1–9.4 Koesoemadinata RC, et al. Computer-assisted chest radiogra-phy reading for tuberculosis screening in people living withdiabetes mellitus. Int J Tuberc Lung Dis 2018;22(9):1088–1094.5 Qin ZZ, et al. Tuberculosis detection from chest x-rays fortriaging in a high tuberculosis-burden setting: an evaluation offive artificial intelligence algorithms. Lancet Digit Health2021;3(9):e543–554.6 Khan FA, et al. Articles Chest x-ray analysis with deep learning-based software as a triage test for pulmonary tuberculosis : aprospective study of diagnostic accuracy for culture-confirmeddisease. Lancet Digit Health 2020;2(11):e573–581.7 Tavaziva G, et al. Chest X-ray analysis with deep learning-basedsoftware as a triage test for pulmonary tuberculosis: anindividual patient data meta-analysis of diagnostic accuracy.Clin Infect Dis 2022;74(8):1390–1400.8 Qin ZZ, et al. Comparing different versions of computer-aideddetection products when reading chest X-rays for tuberculosis.PLoS Digit Health 2022;1(6):e0000067.9 Murphy K, et al. Computer aided detection of tuberculosis onchest radiographs: an evaluation of the CAD4TB v6 system. SciRep 2020;10(1):1–11.Letter 15910 Wong EB, et al. Convergence of infectious and non-communi-cable disease epidemics in rural South Africa: a cross-sectional,population-based multimorbidity study. Lancet Glob Health2021;9(7):e967–976.11 World Health Organization. Tuberculosis prevalence surveys: ahandbook. The Lime Book. 2nd ed. Geneva, Switzerland: WHO,2011.12 World Health Organization. High-priority target productprofiles for new tuberculosis diagnostics: report of a consensusmeeting. Geneva, Switzerland: WHO, 2014.13 Qin ZZ, et al. Using artificial intelligence to read chestradiographs for tuberculosis detection: a multi-site evaluationof the diagnostic accuracy of three deep learning systems. SciRep 2019;9:15000.14 World Health Organization, UNICEF/UNDP/World Bank/WHO Special Programme for Research and Training inTropical Diseases. Determining the local calibration of com-puter-assisted detection (CAD) thresholds and other parame-ters: a toolkit to support the effective use of CAD for TBscreening. Geneva, Switzerland: WHO, 2021.15 Qin ZZ, et al. A new resource on artificial intelligence poweredcomputer automated detection software products for tubercu-losis programmes and implementers. Tuberculosis2021;127:102049.160 The International Journal of Tuberculosis and Lung Disease

CAD4TB software updates: different triaging thresholds require caution by users and regulation by authorities

https://discovery.ucl.ac.uk/10180924/1/s16.pdf

CAD4TB software updates: different triaging thresholds require caution by users and regulation by authorities

Abstract

Similar works

Full text

Available Versions

UCL Discovery