Accurate, reproducible diagnoses can be difficult to make in haemato-oncology due to multi-parameter clinical data, complex diagnostic criteria and time-pressured environments. We have designed a decision tree application (DTA) that reflects WHO diagnostic criteria to support accurate diagnoses of myeloid malignancies. The DTA returned the correct diagnoses in 94% of clinical cases tested. The DTA maintained a high level of accuracy in a second validation using artificially generated clinical cases. Optimisations have been made to the DTA based on the validations, and the revised version is now publicly available for use at http://bit.do/ADAtool.The article is available via Open Access. Click on the 'Additional link' above to access the full-text.Unknow

Bean, Daniel

Coats, Thomas

Coppell, Jason

Dobson, Richard

El-Bashir, Razan

Furtado, Michelle

Ireland, Robin

Kulasakeraraj, Austin

Medd, Patrick

Panopoulou, Aikaterini

Pawade, Joya

Tucker, David

Vatopoulou, Theodora

Vijayavalli, Dhanapal

Wimalachandra, Manujasri

Wood, Henry

eJHaem

PubMed

Abstract Accurate, reproducible diagnoses can be difficult to make in haemato‐oncology due to multi‐parameter clinical data, complex diagnostic criteria and time‐pressured environments. We have designed a decision tree application (DTA) that reflects WHO diagnostic criteria to support accurate diagnoses of myeloid malignancies. The DTA returned the correct diagnoses in 94% of clinical cases tested. The DTA maintained a high level of accuracy in a second validation using artificially generated clinical cases. Optimisations have been made to the DTA based on the validations, and the revised version is now publicly available for use at http://bit.do/ADAtool

Thomas Coats

Daniel Bean

Theodora Vatopoulou

Dhanapal Vijayavalli

Razan El‐Bashir

Aikaterini Panopoulou

Henry Wood

Manujasri Wimalachandra

Jason Coppell

Patrick Medd

Michelle Furtado

David Tucker

Austin Kulasakeraraj

Joya Pawade

Richard Dobson

Robin Ireland

Directory of Open Access Journals

An open‐source, expert‐designed decision tree application to support accurate diagnosis of myeloid malignancies

Lavelle J

Coats T

Crossref

Accurate, reproducible diagnoses can be difficult to make in haemato-oncology due to multi-parameter clinical data, complex diagnostic criteria and time-pressured environments. We have designed a decision tree application (DTA) that reflects WHO diagnostic criteria to support accurate diagnoses of myeloid malignancies. The DTA returned the correct diagnoses in 94% of clinical cases tested. The DTA maintained a high level of accuracy in a second validation using artificially generated clinical cases. Optimisations have been made to the DTA based on the validations, and the revised version is now publicly available for use at http://bit.do/ADAtool.UnknownThe article is available via Open Access. Click on the 'Additional link' above to access the full-text

RD&E Research Repository

An open-source, expert-designed decision tree application to support accurate diagnosis of myeloid malignancies

Accurate, reproducible diagnoses can be difficult to make in haemato‐oncology due to multi‐parameter clinical data, complex diagnostic criteria and time‐pressured environments. We have designed a decision tree application (DTA) that reflects WHO diagnostic criteria to support accurate diagnoses of myeloid malignancies. The DTA returned the correct diagnoses in 94% of clinical cases tested. The DTA maintained a high level of accuracy in a second validation using artificially generated clinical cases. Optimisations have been made to the DTA based on the validations, and the revised version is now publicly available for use at http://bit.do/ADAtool

El‐Bashir, Razan

PubMed Central

Accurate, reproducible diagnoses can be difficult to make in haemato-oncology due to multi-parameter clinical data, complex diagnostic criteria and time-pressured environments. We have designed a decision tree application (DTA) that reflects WHO diagnostic criteria to support accurate diagnoses of myeloid malignancies. The DTA returned the correct diagnoses in 94% of clinical cases tested. The DTA maintained a high level of accuracy in a second validation using artificially generated clinical cases. Optimisations have been made to the DTA based on the validations, and the revised version is now publicly available for use at http://bit.do/ADAtool

UCL Discovery

Received: 16 February 2021 Accepted: 1March 2021DOI: 10.1002/jha2.182S HORT R E PORTAn open-source, expert-designed decision tree application tosupport accurate diagnosis of myeloidmalignanciesThomas Coats1 Daniel Bean2,3 Theodora Vatopoulou4 Dhanapal Vijayavalli5,6Razan El-Bashir5 Aikaterini Panopoulou7,8 HenryWood6ManujasriWimalachandra6 Jason Coppell1 PatrickMedd9 Michelle Furtado10David Tucker10 Austin Kulasakeraraj6 Joya Pawade11 Richard Dobson2,3Robin Ireland61 Department of Haematology, Royal Devon and Exeter NHS Foundation Trust, Exeter, UK2 Biostatistics andHealth Informatics, King’s College London, London, UK3 Health Data Research UK London, University College London, London, UK4 Department of Haematology, St George’s University Hospitals NHS Foundation Trust, London, UK5 Department of Haematology, MedwayNHS Foundation Trust, Kent, UK6 Department of Haematological Medicine, King’s College Hospital NHS Foundation Trust, London, UK7 Department of Haematology, Darent Valley Hospital, Kent, UK8 Department of Haematology, RoyalMarsdenNHS Foundation Trust, London, UK9 Department of Haematology, Derriford Hospital, Plymouth, UK10 Department of Haematology, Royal Cornwall NHS Trust, Truro, UK11 Department of Pathology, North Bristol NHS Trust, Bristol, UKCorrespondenceDanielBean, SocialGenetic andDevelopmen-tal PsychiatryCentre,MemoryLane, London,SE58AF,UK.Email:Daniel.bean@kcl.ac.ukFunding informationEuropeanUnion’sHorizon2020,Grant/AwardNumber: 116074;UKResearchand Innova-tion,Grant/AwardNumber:MR/S00310X/1;National Institute forHealthResearch (NIHR)MaudsleyBiomedicalResearchCentre atSouthLondonandMaudsleyNHSFoundationTrust andKing’sCollegeLondonAbstractAccurate, reproducible diagnoses can be difficult to make in haemato-oncology due tomulti-parameter clinical data, complex diagnostic criteria and time-pressured environ-ments.Wehave designed a decision tree application (DTA) that reflectsWHOdiagnos-tic criteria to support accurate diagnoses of myeloid malignancies. The DTA returnedthe correct diagnoses in 94% of clinical cases tested. The DTA maintained a high levelof accuracy in a second validation using artificially generated clinical cases. Optimisa-tions have been made to the DTA based on the validations, and the revised version isnow publicly available for use at http://bit.do/ADAtool.KEYWORDSclassifications, clinical haematology, diagnostic haematology, myeloid leukaemiaThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, providedthe original work is properly cited.© 2021 The Authors. eJHaem published by British Society for Haematology and JohnWiley & Sons Ltd.eJHaem. 2021;2:261–265. wileyonlinelibrary.com/journal/jha2 261262 COATS ET AL.1 INTRODUCTIONNational institute for health and care excellence (NICE) guidance rec-ommends multi-disciplinary meetings (MDMs) as best practice in thediagnostic work-up of suspected haemato-oncology patients. Com-pared to some areas of medicine, haematological malignancies arealready a very well-studied discipline. A sizeable proportion of treat-ments are based on randomised controlled trial data, and with regardto the diagnostic criteria, the commonly used WHO Classification ofTumours is supported by 4500 references [1, 2].Despite the wealth of information to support clinicians, makingaccurate diagnoses in a busy MDM can be difficult—this same abun-dance of information can be difficult to navigate, is subject to periodicupdates and requires the integration ofmultiple sources of data acrossdifferent platforms (clinical, morphological, genetic, radiological), all ina time-pressured environment [3]. Poor documentation may lead onto inefficiencies of clinical service, incorrect treatment strategies andpoor quality of data for local and national audit [4].Technology is widely recognised as offering an opportunity toimprove the accuracy and efficiency of diagnostics in medicine [5, 6].These include big data analyses of hospital records, artificial intelli-gence or automated algorithms to transform howwe care for patients.The use of algorithms is not new in diagnostic healthcare; they helpto standardise how a diagnosis is reached and arewidespread in every-day clinical practice [7]. Typically, these algorithms are described andshared as text documents or flowcharts and are designed for humanuse, not for automated or computational use [8,9]. These formats arealso not amenable to automated testing or verification. However,manyof the diagnostic criteria in theWHO are clearly set out and thereforegive the possibility of designing a digitised algorithm to support clini-cians with the complexity of the diagnostics.We have previously shown that digital solutions can improvethe accuracy of MDS-subtyping in a single-centre retrospectivestudy [10]. We have now developed a comprehensive decision treeapplication (DTA) that recapitulates the 2016 WHO diagnostic cri-teria for adult myelodysplastic syndrome (MDS), acute myeloidleukaemia (AML), myeloproliferative neoplasm (MPN) and myelodys-plastic/myeloproliferative syndrome (MDS/MPN) disease categories.This multi-centre study set out to validate the DTA by independenttesting. It shows our DTA is accurate and can improve precision in thediagnosis of commonly occurringmyeloid malignancies.2 METHODSTheDTAwas designed using the open source platformesyNand is pub-licly available as a web application [11]. The DTA clinical algorithmswere derived using the WHO criteria transformed into esyN using agraphical interface which does not require writing code. Users enterclinical data within the following categories: full blood count readings,bone marrow morphology/immunophenotyping, cytogenetics, molec-ular and clinical information. Clinical data are fed into the decisiontree, and a WHO diagnosis is displayed. Thirty-five different myeloiddiagnoses, as defined by WHO criteria, are available within the DTA,and a ‘non diagnostic’ outcome. Acute leukaemias with ambiguous lin-eage, myeloid neoplasms with germline predisposition, myeloid sar-coma, myeloid proliferations associated with Down syndrome, masto-cytosis, blastic plasmacytoid dendritic cell neoplasam and JMML arenot included.A clinician at each centre received training on using the DTA. Caseswith a diagnostic bonemarrow, documented localMDMdiscussion andaWHO diagnosis were selected for analysis. Clinical data were loggedusing the DTA, and the suggested diagnosis was recorded. No clinicaldata were uploaded to or recorded in esyn.org. All analyses were runlocally in a web browser. In cases of discrepancy between the DTA andthe MDM outcome, a second independent clinical opinion was soughtto identify whether (i) the DTA had suggested an incorrect diagnosis or(ii) theMDMdiagnosis needed revision.A second validation was performed by creating 10,000 in silico casereports with random values for the different clinical variables. Eachcase that was selected for subsequent validation covered a uniqueroute through the decision tree—this ensured the whole tree wastested, with additional robustness from testing unpredictable combi-nations of variables. Each case report was reviewed by two haematol-ogists working independently to establish the most appropriate diag-nosis for the artificial cases (a team of eight FRCPath-certified haema-tologists performed this step). Any case where there was discrepancybetween the DTA and either doctor was reviewed by a team of threeindependent haematologists to adjudicate on the most likely diagno-sis. Any of the randomly generated cases that were considered highlyimprobable by the independent review were removed from furtheranalysis.Accuracy and F1 scores (the harmonic mean of precision and recall)were calculated using scikit-learn 0.20.3 in Python 3.7.0.3 RESULTS AND DISCUSSIONSixty-two cases of myeloid malignancy were logged in the DTA fromfive centres, covering 24 of the possible 35 diagnoses in the DTA(Figure 1). Fifty-five cases (89%) were concordant between the DTAand the MDM diagnosis. On independent review of the seven discor-dant cases, three caseswere recommended to revise theMDMdiagno-sis to the DTA-generated diagnosis. Four cases were discordant due toan error in the logic coded in the DTA. Therefore, overall the DTA wasaccurate in 58 of 62 cases (94%). An interesting observation from onemyeloid case was that greater clarity was required from the trephinereport to be able to run the DTA. When the biopsy was reviewed andthe report updated, this led to a recommended revision of the MDMdiagnosis.Subsequently, minor adjustments were made to improve the DTAaccuracy (DTA version 1.1). For example, three of the DTA errorswere due to one missing logic step for cases of myelodysplas-tic/myeloproliferative syndrome, unclassifiable (MDS-MPN-U). Theclinical data from the 62 cases were re-run through the updated DTA,COATS ET AL. 263F IGURE 1 Schema summarising workflow of cases run through the decision tree application (DTA) and the DTA accuracyand all 62 cases gave an output that matched the amended MDMdiagnosis. To further validate the updated DTA, three additional caseswith a known diagnosis of MDS-MPN-U that would have been mis-diagnosed by the original version, were tested; the correct diagnosiswas returned in all three.To further test the DTA, 108 randomly generated clinical cases thatcovered all of the major branches and diagnoses within the DTA wereselected for validation by pairs of clinicians working independently.Eight cases were subsequently removed for being implausible. Afterindependent review, the DTA was found to be incorrect in 15 of 100cases, giving an overall accuracy of the DTA of 85%. The average accu-racy for the individual doctors was 53% (range 43%–71%).Adjustments to the DTA were made to reflect the feedback fromthe panel of independent experts (DTA version 1.2) (Figure 1). The100 plausible randomly generated cases were re-run through theDTA. Ninety-four cases had agreement between the diagnosis and theupdatedDTA outcome giving an overall accuracy of 94% andmisclassi-fication rate of 6%. TheDTA ismost accurate in correctly categorising acase where a myeloid malignancy is confirmed, as opposed to discrim-inating a non-diagnostic case from a malignant diagnosis, with high F1scores (>0.80) and accuracy (>0.90) returned for all WHO categories(Figure 2). Sixty of the original 65 clinical cases had sufficient clinicalinformation to run on the updated DTA. For 59 of the 60 cases, thecorrect diagnosis was returned. These two validations show that analgorithmic approach is amenable to sequential optimisations leadingto high levels of accuracy.The revised version of the DTA is available for use at http://bit.do/ADAtool.The accuracy of the DTAwas lower when processing randomly gen-erated cases compared to genuine clinical cases. The case selectionwasdesigned to test the decision boundaries between diagnoses; so manycases only had small differences between them, or nearly fit the crite-ria for analternativediagnosis. Feedback fromclinicians assigningdiag-noses to these cases was that there was more heterogeneity of clinicalfeatures than normally found in routine clinical practice. These factorsmayexplain the lower thanexpectedaccuracyof the individual doctors.The ability of the DTA to performwell in this context suggests that thealgorithm is durable evenwith artificially generated, atypical presenta-tions.The review process of the artificially generated cases demonstratedsome clinical scenarios where consensus was difficult to reach. Forexample, establishing clear criteria for MPN-U was difficult. Apply-ing an algorithmic approach also highlights the areas in the WHOdiagnostic criteria that are more open to interpretation, for exam-264 COATS ET AL.F IGURE 2 Performance of 100 artificially generated cases run through DTA v1.2ple, whether the presence of a non-reactive thrombocytosis wouldre-classify an otherwise typical chronic myelomonocytic leukaemia(CMML) or chronic eosinophilic leukaemia, not otherwise specified(CEL-NOS), case into anMDS-MPN-U. In these circumstances, a differ-ential diagnosis of valid WHO diagnoses is likely to be more appropri-ate, and future iterations of the DTAwill look to address this.4 CONCLUSIONThese DTA provide an example of how technology can be used to sup-port clinicians’ assessment of multi-parameter, complex data, to pro-duce improved diagnostic accuracy and facilitate MDM recording ofoutcomes. The DTA can highlight additional tests that are required toadvance the diagnostic pathway or indicate relevant data points thatare missing. Its use encourages a consistent reporting approach andcould also be useful in education. As illustrated in this study, adjust-ments can bemade to theDTA and therefore incorporate newdiagnos-tic criteria that can be applied to both recent and historical patients foraudit, research or direct clinical care. Further work is now required toexpand the validation to a larger cohort of real clinical cases, establish-ing safety for use in clinical practice.DTA was sequentially optimised from version v1.0 to v1.1 and v1.2.(A) DTA v1.0 was tested on cohort of 62 real clinical cases. DTA v1.1was run on original 62 cases plus three additional MDS-MPN-U cases.One hundred and eight artificially generated cases were created andrun throughDTAv1.1. (B)DTAv1.2was re-runon100artificially gener-ated cases and on 60 of the original 65 clinical cases (five clinical caseslacked sufficient information to run cases through DTA v1.2).Confusionmatrix demonstrating the performance of thewholeDTAtree assessed by DTA v1.2 analysing 100 artificially generated cases.A correct diagnosis is where the WHO diagnosis returned by theDTA (Predicted label) matches the actual diagnosis (True label). Thenumbers show the percentage of the 100 cases with correspondingCOATS ET AL. 265predicted-true label combinations.Matching prediction and true labelsare found along the diagonal. Colour shading represents the percent-age recall (0%–100%) for each diagnosis tested.ACKNOWLEDGEMENTSDaniel Bean holds a UK Research and Innovation (UKRI) Fellowship aspart of Health Data Research UK (HDRUK) MR/S00310X/1. RichardDobson is supportedbyHealthDataResearchUK;TheBigData@HeartConsortium, European Union’s Horizon 2020 grant agreement num-ber: 116074; The National Institute for Health Research (NIHR) Uni-versity College London Hospitals Biomedical Research Centre and theNIHRBiomedical ResearchCentre at South LondonandMaudsleyNHSFoundation Trust and King’s College London; and the UKResearch andInnovation London Medical Imaging and Artificial Intelligence Centrefor Value BasedHealthcare.DATA AVAILABILITY STATEMENTSource results frompatient records used at all sites in the studywill notbe available due to inability to safely fully anonymise up to the Infor-mationCommissionerOffice (ICO) standards given the highly sensitiveand granular nature of the data (e.g. blood results, genetic tests, diag-noses etc.). A subset of the data that support the findings of this studyis available on reasonable request to the corresponding author.CONFLICT OF INTERESTThe authors declare that there is no conflict of interest that could beperceived as prejudicing the impartiality of the research reported.AUTHOR CONTRIBUTIONSResearch design: Thomas Coats, Daniel Bean, Richard Dobsonand Robin Ireland. Collected patient data: Theodora Vatopoulou,Jason Coppell, Aikaterini Panopoulou, Manujasri Wimalachandraand Razan El-Bashir. Assessment of artificially generated cases:Dhanapal Vijayavalli, Aikaterini Panopoulou, Jason Coppell, HenryWood, Patrick Medd, Michelle Furtado, David Tucker and AustinKulasakeraraj. Reviewed the discrepant cases: Theodora Vatopoulou,Joya Pawade and Robin Ireland. Statistical analysis: Thomas Coats andDaniel Bean. Analysis and interpretation of data: ThomasCoats, DanielBean, Richard Dobson and Robin Ireland. Drafting of the manuscript:Thomas Coats, Daniel Bean, Richard Dobson and Robin Ireland.ORCIDThomasCoats https://orcid.org/0000-0003-4341-4361HenryWood https://orcid.org/0000-0002-3503-5399David Tucker https://orcid.org/0000-0002-0481-4111REFERENCES1. Djulbegovic B, KumarA, SoaresH,Hozo I, BeplerG. Treatment successin cancer: new cancer treatment successes identified in phase 3 ran-domized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. Arch InternMed. 2008;168:632–42.2. Swerdlow SH, World Health Organization, International Agency forResearch on Cancer. WHO classification of tumours of haematopoi-etic and lymphoid tissues. 2018. http://publications.iarc.fr/Book-And-Report-Series/Who-Iarc-Classification-Of-Tumours/Who-Classification-Of-Tumours-Of-Haematopoietic-And-Lymphoid-Tissues-2017. Accessed 1 July 2018.3. Soukup T, Lamb BW, Arora S, Darzi A, Sevdalis N, Green JSA. Success-ful strategies in implementing a multidisciplinary team working in thecare of patientswith cancer: an overviewand synthesis of the availableliterature. JMultidiscip Healthc. 2018;11:49–61.4. Singh H, Meyer AND, Thomas EJ. The frequency of diagnostic errorsin outpatient care: estimations from three large observational stud-ies involving US adult populations. BMJ Qual Saf. 2014;23(9):727–31.Available from: https://pubmed.ncbi.nlm.nih.gov/24742777/5. NHSHealth Education England. Topol review . 2020. https://www.hee.nhs.uk/our-work/topol-review. Accessed 3 September 2020.6. The King’s Fund. Adoption and spread of innovation in the NHS. 2020.https://www.kingsfund.org.uk/publications/innovation-nhs. Accessed3 September 2020.7. Lavelle J, Schast A, Keren R. Standardizing care processes and improv-ing quality using pathways and continuous quality improvement. CurrTreat Options Pediatr. 2015;1(4):347–58.8. HochhausA, BaccaraniM, SilverRT, SchifferC, Apperley JF, CervantesF, et al. European LeukemiaNet 2020 recommendations for treatingchronic myeloid leukemia. Leukemia. 2020;34:966–84.9. McMullin MF, Harrison CN, Ali S, Cargo C, Chen F, Ewing J, et al.A guideline for the diagnosis and management of polycythaemiavera. A British Society for Haematology Guideline. Br J Haematol.2019;184(2):176–91.10. Coats T, Bean D, Ireland R. Automated data-extraction and decision-tree program to improve diagnostic accuracy in MDS. Br J Haematol.2019;185(S1):3–202.11. Bean DM, Heimbach J, Ficorella L, Micklem G, Oliver SG, FavrinG. esyN: Network building, sharing and publishing. PLoS One.2014;9(9):e106035.SUPPORTING INFORMATIONAdditional supporting informationmay be found online in the Support-ing Information section at the end of the article.How to cite this article: Coats T, BeanD, Vatopoulou T, et al.An open-source, expert-designed decision tree application tosupport accurate diagnosis of myeloid malignancies. eJHaem.2021;2:261–265. https://doi.org/10.1002/jha2.182

English

https://discovery.ucl.ac.uk/10145154/1/Bean_An%20open%20source%2C%20expert%20designed%20decision%20tree%20application%20to%20support%20accurate%20diagnosis%20of%20myeloid%20malignancies_VoR.pdf

An open-source, expert-designed decision tree application to support accurate diagnosis of myeloid malignancies

Abstract

Similar works

Full text

Available Versions

Directory of Open Access Journals

Crossref

RD&E Research Repository

PubMed Central

UCL Discovery