Beyer FR

Inskip A

Johnson EE

Moradi N

Robinson T

Sagoo GS

Thomson K

Uteh CO

English

Newcastle University E-Prints

1  Uncertainty in matching-adjusted indirect comparisons using real-world evidence: 1 Evidence Assessment Group Perspective 2  3 Cyril Onwuelazu Uteh, Research Assistant, Newcastle University, 0000-0003-2005-2679* 4 Eugenie Evelynne Johnson, Research Assistant, Newcastle University, 0000-0003-3324-5 7141* 6 Tomos Robinson, Senior Research Associate, Newcastle University, 0000-0001-8695-9738  7 Najmeh Moradi, Research Associate, Newcastle University, 0000-0003-4172-4411 8 Alex Inskip, Research Assistant, Newcastle University, 0009-0006-9946-667X 9 Fiona R Beyer, Principal Research Associate, Newcastle University, 0000-0002-6396-3467 10 Katie Thomson, Senior Research Associate, Newcastle University, 0000-0002-9614-728X 11 Gurdeep S Sagoo, Senior Lecturer, Newcastle University, 0000-0003-1427-1437 12 * These authors should be considered joint first author 13 Corresponding author:  14 Dr Gurdeep S Sagoo, gurdeep.sagoo@ncl.ac.uk 15  16 Funding: The independent EAG report was commissioned by the NIHR Evidence Synthesis 17 Programme as project number NIHR135664. See the Health Technology Assessment (HTA) 18 programme website for further project information (https://www.nihr.ac.uk/explore-19 nihr/funding-programmes/health-technology-assessment.htm). This summary of the EAG 20 report was compiled after NICE issued the Final Appraisal Determination. The views and 21 opinions expressed herein are those of the authors and do not necessarily reflect those of 22 NICE or the Department of Health. 23 Ethics approval: Not applicable 24 Informed consent: Not applicable 25 Data availability: Not applicable 26 Consent to participate: Not applicable 27 Consent for publication: Not applicable 28 Code availability: Not applicable 29 2  Author contributions: All authors contributed to the study conception and design and were 30 involved with the work of the EAG. 31 Acknowledgements: Expert advice and support was received by Dr Nick Meader 32 (Newcastle University) during the development of the EAG report. 33   34 3  1. Introduction 35 The National Institute for Health and Care Excellence (NICE) have adopted a real-world 36 evidence (RWE) framework to identify when RWE can be used to reduce uncertainties and 37 improve guidance; and to clearly describe best practices for planning, conducting and 38 reporting RWE studies.(1) As noted by the framework, although randomised controlled trials 39 (RCTs) are the preferred source of evidence for intervention effectiveness, RCTs are 40 sometimes not available or appropriate to base decisions on. In these circumstances, RWE 41 can improve understanding of health and social care delivery and the effects of 42 interventions.(1)  43 Non-randomised studies are increasingly being used in the initial assessment of medicines, 44 with more medicines being granted regulatory approval based on single-arm trials.(1) This is 45 particularly true for rare diseases, where adequately powering RCTs to detect an effect is 46 challenging.(2) Presenting single-arm studies as the evidence base for clinical evidence has 47 occurred within NICE’s Highly Specialised Technology Appraisals (TAs),(3, 4) which focus on 48 very rare diseases.(5) A review of NICE TAs from 2022 noted that 13 of 73 submissions 49 were based on single-arm trials; interestingly, 11 of these were in oncology.(6) 50 The same review suggests that a form of indirect treatment comparison called matching-51 adjusted indirect comparisons (MAICs) were used in 6 of the 13 TAs to compare the results 52 of the single-arm study with an external control (such as data from a registry or database).(6) 53 However, when using external controls, there needs to be careful consideration of how well 54 these external controls compare with the trial population to ensure reliability in results.(7) 55 This commentary highlights the observations and learnings of the Evidence Assessment 56 Group (EAG) on the use of real-world evidence in a MAIC within the TA submitted to NICE 57 as TA1001: Zanubrutinib for treating marginal zone lymphoma after anti-CD20-based 58 treatment.(8)  59 2. Considerations for matching-adjusted indirect comparisons using real-world 60 evidence 61 MAICs are a type of indirect treatment comparison (ITC) using individual patient data (IPD) 62 to adjust for population differences, often when there are doubts about the validity of the 63 transitivity assumption (consistency between direct and indirect evidence) foundational to 64 standard ITCs. Commonly, companies have access to IPD from their own clinical trials, 65 which is used to match to the population used for another treatment where only aggregate 66 data may be available.(9) MAICs are stratified into two types: anchored and unanchored. 67 Anchored MAICs compare two interventions with a common comparator (e.g. two 68 randomised controlled trials for the same population both using a placebo control). 69 4  Conversely, unanchored MAICs compare interventions that do not have a common 70 comparator (e.g. comparing single-arm studies). 71 As described by NICE Technical Support Document 18, unanchored MAICs are inherently 72 open to uncertainty as they assume that all effect modifiers and prognostic factors are 73 accounted for; in reality, this is an assumption that is considered almost impossible to 74 achieve.(10) As such, Phillippo et al. make the recommendation that the propensity score 75 weighting and outcome regression methods used within the MAIC adjust for all effect 76 modifiers and prognostic variables.(11) However, in practice, this is likely unfeasible, 77 particularly when using an external control such as registry data. This is because patient 78 registry data can be subject to a lack of detail in the data collected and the potential for 79 variable quality in data.(12) Such limitations make conducting an unanchored MAIC with 80 reduced uncertainty more challenging, as was the case with TA1001. 81 3. Case study: zanubrutinib for marginal zone lymphoma after anti-CD20-based 82 treatment 83 NICE invited the manufacturer of zanubrutinib (Brukinsa®), BeiGene, to submit evidence on 84 the clinical- and cost-effectiveness of the product for treating adults with marginal zone 85 lymphoma (MZL) who had already been previously treated with an anti-CD20-based therapy. 86 MZL is a rare type of slow-growing Non-Hodgkin Lymphoma, which can broadly be 87 categorised into three different subtypes: mucosa-associated lymphoid tissue (MALT) 88 lymphoma; modal MZL; and splenic MZL.(13)  89 The major source of clinical evidence within the company’s submission was derived from two 90 international, multi-centre, single-arm trials, AU-003 (Phase I-2) and MAGNOLIA (Phase 2); 91 only MAGNOLIA included participants from the UK.(14, 15) The results of these two trials 92 were combined into a single unit, known as MAGNOLIA-003. The results of MAGNOLIA-003 93 were then compared with IPD taken from the UK-based Haematological Malignancy 94 Research Network (HMRN) registry using an unanchored MAIC.(16) The company stated 95 that patient numbers for individual treatments within the HMRN registry were too small for 96 use within a MAIC. As such, the company performed the MAIC using a basket of different 97 treatments from within the HMRN,(8) The exact figures from the MAIC are confidential and 98 cannot be presented here.  99 The company submission stated that, based on the MAIC results, zanubrutinib was superior 100 to the basket of treatments in terms of both progression-free survival (PFS) and overall 101 survival (OS). However, there were several methodological limitations associated with the 102 MAIC. The first, as previously discussed, was that the lack of a common comparator meant 103 that the MAIC was unanchored, which, as previously noted, is inherently open to 104 5  uncertainties. The second major issue highlighted by the EAG related to the limited 105 availability of participant data from the HMRN registry with which to compare and adjust 106 using data from the pooled MAGNOLIA-003 trials. The HMRN registry is noted to have a 107 socio-demographic profile broadly representative of the UK population as a whole.(17) 108 However, the company were only able to match participants using five covariates: age; 109 median time since diagnosis; number of previous lines of therapy; refractoriness to last 110 therapy; and disease progression in the previous two years. Although the HMRN registry is 111 based on UK data, the EAG had concerns surrounding the small number of covariates 112 adjusted for. Despite the results of the MAIC being robust to sensitivity analyses, the EAG 113 were still concerned by the considerable uncertainty present in the analysis. However, the 114 EAG also acknowledged that, due to the lack of information available from the HMRN 115 registry, this was a challenging issue to resolve.  116 The uncertainties that arose within the MAIC subsequently impacted on the assessment of 117 cost-effectiveness. To project the long-term clinical and economic consequences of 118 zanubrutinib for treating relapsed or refractory MZL, the company developed a three 119 mutually exclusive health state partitioned survival model (PSM) in Microsoft Excel. Given 120 the results from the MAIC directly fed into the comparative effectiveness estimates used to 121 estimate the state occupancy of the PSM health states, the uncertainties are propagated 122 through the cost-effectiveness model. The magnitude of the subsequent uncertainty present 123 in the cost-effectiveness model due to the MAIC was difficult to quantify.  124 To potentially ameliorate such concerns with the uncertainty in results across clinical- and 125 cost-effectiveness evidence, with sufficient data, methods such as multilevel network meta-126 regression (ML-NMR), which is designed to synthesise a combination of IPD and aggregate 127 data and can relax the assumption that effect modifiers are balanced across populations, 128 could be used in the future.(18) However, there were limited data to be able to facilitate such 129 methods within this example of zanubrutinib for MZL. As such, the EAG suggested that 130 further comparable evidence with zanubrutinib specifically for patients with R/R MZL could 131 potentially facilitate future MAICs with different treatments. This could include more RWE, 132 though an RCT of zanubrutinib for R/R MZL could also facilitate an anchored MAIC, which 133 would inherently have fewer uncertainties. Collecting longer-term follow-up data, including 134 from an ongoing long-term extension trial of zanubrutinib,(19) may also be able to reduce the 135 uncertainties within the PSM in the future.(8)  136 On 4 September 2024, NICE recommended zanubrutnib for treating MZL in adults who have 137 had at least one anti-CD20-based therapy. 138 4. Key learnings 139 6   There are inherent uncertainties in using unanchored MAICs to assess the clinical 140 effectiveness of treatments for health conditions. These uncertainties may be 141 compounded when potentially limited available information from RWE, such as 142 registry data, does not easily facilitate the matching process. 143  EAGs should be mindful of the key prognostic factors that contribute to disease 144 progression and that may impact on treatment effectiveness and how many of these 145 have been accounted for in MAIC analyses. 146  EAGs should also consider how representative RWE used in analyses such as 147 MAICs are to the current clinical context of a condition.  148 5. Conclusions 149 Where RCTs are unavailable or unsuitable for decision-making, RWE can help reduce 150 uncertainties surrounding intervention effectiveness. Indirect analysis approaches such as 151 MAICs can help to facilitate comparisons between single-arm studies and external controls, 152 though there are several methodological considerations that can significantly reduce 153 certainty in the results. Within future TAs, EAGs should consider what prognostic variables 154 are available for adequate matching of participants and how representative the source of 155 RWE is to the decision-making context. 156  157 Abbreviations 158 EAG = Evidence Assessment Group 159 HMRN = Haematological Malignancy Research Network 160 MAIC = matching-adjusted indirect comparison 161 ML-MNR = multilevel network meta-regression 162 MZL = marginal zone lymphoma 163 NICE = National Institute for Health and Care Excellence 164 PFS = progression-free survival 165 RCT = randomised controlled trial 166 RWE = real-world evidence 167 TA = Technology Appraisal 168 References 169 7  1. Na onal Ins tute for Health and Care Excellence. NICE real-world evidence framework 170 [ECD9] 2022 [updated 23 June 2022. Available from: 171 h ps://www.nice.org.uk/corporate/ecd9/chapter/overview. 172 2. Gomes M, Turner AJ, Sammon C, Dawoud D, Ramagopalan S, Simpson A, et al. Acceptability 173 of Using Real-World Data to Es mate Rela ve Treatment Effects in Health Technology Assessments: 174 Barriers and Future Steps. Value in Health. 2024;27(5):623-32. 175 3. Na onal Ins tute for Health and Care Excellence. Onasemnogene abeparvovec for trea ng 176 spinal muscular atrophy (HST15) 2023 [updated 19 April 2023. Available from: 177 h ps://www.nice.org.uk/guidance/hst15/chapter/4-Considera on-of-the-evidence#impact-of-the-178 new-technology. 179 4. Na onal Ins tute for Health and Care Excellence. Sebelipase alfa for trea ng Wolman 180 disease (HST30) 2024 [updated 10 January 2024. Available from: 181 h ps://www.nice.org.uk/guidance/hst30/chapter/3-Commi ee-discussion#clinical-evidence. 182 5. Na onal Ins tute for Health and Care Excellence. Highly specialised technologies guidance  183 [Available from: h ps://www.nice.org.uk/about/what-we-do/our-programmes/nice-guidance/nice-184 highly-specialised-technologies-guidance. 185 6. Guerra Primo L, Ma hijsse S, Westley T. HTA366 NICE Technology Appraisals Based on Single-186 Arm Trials in 2022 – What Can We Learn from Them? Value in Health. 2023;26(12 Supplement):S391. 187 7. Appiah K, Rizzo M, Sarri G, Hernandez L. Jus fying the source of external comparators in 188 single-arm oncology health technology submissions: a review of NICE and PBAC assessments. Journal 189 of Compara ve Effec veness Research. 2024;13(2). 190 8. Na onal Ins tute for Health and Care Excellence. Zanubru nib for trea ng marginal zone 191 lymphoma a er an -CD20-based treatment (TA1001) 2024 [updated 4 September 2024. Available 192 from: h ps://www.nice.org.uk/guidance/ta1001. 193 9. Signorovitch JE, Sikirica V, Erder MH, Xie J, Lu M, Hodgkins PS, et al. Matching-Adjusted 194 Indirect Comparisons: A New Tool for Timely Compara ve Effec veness Research. Value in Health. 195 2012;15(6):940-7. 196 10. Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support 197 Document 18: Methods for Popula on-Adjusted Indirect Comparisons in Submissions to NICE. 198 Sheffield, UK: Decision Support Unit, ScHARR, University of Sheffield; 2016. 199 11. Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. Methods for Popula on-200 Adjusted Indirect Comparisons in Health Technology Appraisal. Medical Decision Making. 2017;38(2). 201 12. Rubinger L, Ekh ari S, Gazendam A, Bhandari M. Registries: Big data, bigger problems? 202 Injury. 2023;53(Supplement 3):S39-42. 203 13. Cancer Research UK. Low grade NHL 2024 [updated 23 February 2024. Available from: 204 h ps://www.cancerresearchuk.org/about-cancer/non-hodgkin-lymphoma/types/low-205 grade#:~:text=Each%20year%20around%202600%20people,zone%20lymphoma%20in%20the%20UK206 . 207 14. Opat S, Tedeschi A, Linton K, McKay P, Hu B, Chan H, et al. The MAGNOLIA Trial: Zanubru nib, 208 a Next-Genera on Bruton Tyrosine Kinase Inhibitor, Demonstrates Safety and Efficacy in 209 Relapsed/Refractory Marginal Zone Lymphoma Clinical Cancer Research. 2021;27(23):6323-32. 210 15. Phillips T, Chan H, Tam CS, Tedeschi A, Johnston P, Oh SY, et al. Zanubru nib monotherapy in 211 relapsed/refractory indolent non-Hodgkin lymphoma. Blood Advances. 2022;6(11):3472-9. 212 16. Haematological Malignancy Research Network. HMRN  [Available from: h ps://hmrn.org/. 213 17. Roman E, Kane E, Howell D, Lamb M, Bagguley T, Crouch S, et al. Cohort Profile Update: The 214 Haematological Malignancy Research Network (HMRN) UK popula on-based cohorts Interna onal 215 Journal of Epidemiology. 2022;51(3):e87-94. 216 18. Phillippo DM, Dias S, Ades AE, Belger M, Brnabic A, Schacht A, et al. Mul level Network 217 Meta-Regression for Popula on-Adjusted Treatment Comparisons. Journal of the Royal Sta s cal 218 Society Series A: Sta s cs in Society. 2020;183(3):1189-210. 219 8  19. ClinicalTrials.gov. Long-term Extension Study of Zanubru nib (BGB-3111) Regimens in 220 Par cipants With B-cell Malignancies. 2019. 221  222 

Uncertainty in Matching-Adjusted Indirect Comparisons Using Real-World Evidence: Evidence Assessment Group Perspective

https://eprints.ncl.ac.uk/file_store/production/305009/2FBA1BE5-8809-4DE8-881C-E364D28D39B5.pdf

Uncertainty in Matching-Adjusted Indirect Comparisons Using Real-World Evidence: Evidence Assessment Group Perspective

Abstract

Similar works

Full text

Available Versions

Newcastle University E-Prints