Methods for the inclusion of real-world evidence in network meta-analysis

Background Network Meta-Analysis (NMA) is a key component of submissions to reimbursement agencies world-wide, especially when there is limited direct head-to-head evidence for multiple technologies from randomised controlled trials (RCTs). Many NMAs include only data from RCTs. However, real-world evidence (RWE) is also becoming widely recognised as a valuable source of clinical data. This study aims to investigate methods for the inclusion of RWE in NMA and its impact on the level of uncertainty around the effectiveness estimates, with particular interest in effectiveness of fingolimod. Methods A range of methods for inclusion of RWE in evidence synthesis were investigated by applying them to an illustrative example in relapsing remitting multiple sclerosis (RRMS). A literature search to identify RCTs and RWE evaluating treatments in RRMS was conducted. To assess the impact of inclusion of RWE on the effectiveness estimates, Bayesian hierarchical and adapted power prior models were applied. The effect of the inclusion of RWE was investigated by varying the degree of down weighting of this part of evidence by the use of a power prior. Results Whilst the inclusion of the RWE led to an increase in the level of uncertainty surrounding effect estimates in this example, this depended on the method of inclusion adopted for the RWE. ‘Power prior’ NMA model resulted in stable effect estimates for fingolimod yet increasing the width of the credible intervals with increasing weight given to RWE data. The hierarchical NMA models were effective in allowing for heterogeneity between study designs, however, this also increased the level of uncertainty. Conclusion The ‘power prior’ method for the inclusion of RWE in NMAs indicates that the degree to which RWE is taken into account can have a significant impact on the overall level of uncertainty. The hierarchical modelling approach further allowed for accommodating differences between study types. Consequently, further work investigating both empirical evidence for biases associated with individual RWE studies and methods of elicitation from experts on the extent of such biases is warranted

Clinical Evidential and Methodological Challenges of Early Assessments of New Health Technologies

This thesis explores the challenges of assessing the relative effectiveness of new health technologies earlier in their clinical development and the potential implications on health technology assessment (HTA), including health policy decision-making on the basis of economic decision models. Public appeal for rapid access to new medicines has increased pressures on regulators and payers to approve and market products often before appropriate measures of effectiveness are available. First, this thesis identifies the key evidential and methodological issues posed by early or accelerated regulatory approval, as well as any parallels found in the literature for conditional reimbursement and coverage with evidence. A review of international HTA and pharmacoeconomic methods guidelines is performed to draw on cross-country experience in dealing with evidentiary issues in evidence synthesis and cost-effectiveness (Chapter 2). A summary of methods used in HTA relevant to this thesis is provided in Chapter 3. Using three examples from different therapeutic areas, I explore the impact on HTA outcomes of i) subgroup and comparator selection (Chapter 4), ii) specific search strategies to identify indirect evidence for network meta-analysis (Chapter 5), and iii) bias adjustment techniques to include observational data in evidence synthesis (Chapter 6). Each chapter evaluates how the uncertainty in relative clinical estimates influences cost-effectiveness results. Using a simulation approach, Chapter 7 extends the example in Chapter 4—ticagrelor for acute coronary syndromes—to model evolving evidence within the context of HTA. The pivotal trial data is replicated and truncated at different time points, both in terms of follow-up and calendar time, to assess relative treatment effects and costs under different scenarios of ‘early’ HTA. This thesis illustrates how on-going regulatory changes impact clinical evidence considerations in HTA and how existing HTA methods can be adapted to allow for earlier product assessments and ensure timely access to new health technologies

Searching for indirect evidence and extending the network of studies for network meta-analysis: case study in venous thromboembolic events prevention following elective total knee replacement surgery

OBJECTIVE: To evaluate the effect of study identification methods and network size on the relative effectiveness and cost-effectiveness of recommended pharmacological venous thromboembolic events (VTEs) prophylaxis for adult patients undergoing elective total knee replacement surgery in the United Kingdom.METHODS: A stepwise literature search specifically designed to identify indirect evidence was conducted to extend the original clinical review from the latest National Institute for Health and Care Excellence (NICE) VTE technology appraisal. Different network sizes or network orders, based on the successive searches, informed three network meta-analyses (NMAs), which were compared with a replicated base case. The resulting comparative estimates were inputted in an economic model to investigate the effect of network size on cost-effectiveness probabilities.RESULTS: Searches increased the number of indirect comparisons between VTE interventions, progressively widening the relevant network of studies for NMA. Precision around mean relative treatment effects was increased as the network was extended from the base case to first-order NMA, but further extensions had limited effect. Cost-effectiveness analysis results were largely insensitive to variation in clinical inputs from the different NMA orders.CONCLUSIONS: No standard methodology is currently recommended by NICE to identify the most relevant network of studies for NMA. Our study showed that optimizing the identification of studies for NMA can extend the evidence base for analysis and reduce the uncertainty in relative effectiveness estimates. Although in our example network extensions did not affect the acceptability of available treatments in VTE prevention based on cost-effectiveness results, it may in other applications.</p

Methods for the inclusion of real world evidence in network meta-analysis

Background: Network Meta-Analysis (NMA) is a key component of submissions to reimbursement agencies world-wide, especially when there is limited direct head-to-head evidence for multiple technologies from randomised controlled trials (RCTs). Many NMAs include only data from RCTs. However, real-world evidence (RWE) is also becoming widely recognised as a valuable source of clinical data. We investigate methods for the inclusion of RWE in NMA and its impact on the uncertainty around the effectiveness estimates. Methods: A range of methods for inclusion of RWE in evidence synthesis, including Bayesian hierarchical and power prior models, were investigated by applying them to an example in relapsing remitting multiple sclerosis. The effect of the inclusion of RWE was investigated by varying the degree of down weighting of this part of evidence by the use of a power prior. Results: Whilst the inclusion of the RWE led to an increase in the level of uncertainty surrounding effect estimates in this example, this depended on the method of inclusion adopted for the RWE. Power prior NMA model resulted in stable effect estimates for fingolimod yet increasing the width of the credible intervals with increasing weight given to RWE data. The hierarchical NMA models were effective in allowing for heterogeneity between study designs; however, this also increased the level of uncertainty. Conclusion: The power prior approach for the inclusion of RWE in NMAs indicates that the degree to which RWE is taken into account can have a significant impact on the overall level of uncertainty. The hierarchical modelling approach further allowed for accommodating differences between study types. Consequently, further work investigating both empirical evidence for biases associated with individual RWE studies and methods of elicitation from experts on the extent of such biases is warranted.Comment: 24 pages, 3 figure

Use of implicit and explicit bayesian methods in health technology assessment

OBJECTIVES: The aim of this study was to examine the use of implicit and explicit Bayesian methods in health technology assessments and to identify whether this has changed over time. METHODS: A review of all health technology assessment (HTA) reports of secondary research published by the UK National Institute of Health Research (NIHR) between 1997 and 2011. Data were extracted on the use and implementation of Bayesian methods, whether defined as such by the original authors (i.e., explicit) or not (i.e., implicit). RESULTS: A total of 155 of 375 (41 percent) NIHR HTA reports, identified as relevant to this review, contained a Bayesian analysis. Of these, 128 (83 percent) contained an implicit Bayesian analysis, 3 (2 percent) an explicit Bayesian analysis and 24 (15 percent) both implicit and explicit Bayesian analyses. Of the twenty-seven reports that explicitly used Bayes theorem, only six included prior information in the form of (informative) prior distributions. Over time, the percentage of HTA reports that used Bayesian (implicit and/or explicit) methods increased from 0 percent in 1997 to nearly 80 percent in 2011. CONCLUSIONS: This review has shown that there has been an increase in the use of Bayesian methods in HTA, which is likely to be a result of the increase in freely available resources to implement the approach. Areas where Bayesian methods have the potential to advance healthcare evaluations in the future are considered in the discussion

The inclusion of real world evidence in clinical development planning

BACKGROUND: When designing studies it is common to search the literature to investigate variability estimates to use in sample size calculations. Proprietary data of previously designed trials in a particular indication are also used to obtain estimates of variability. Estimates of treatment effects are typically obtained from randomised controlled clinical trials (RCTs). Based on the observed estimates of treatment effect, variability and the minimum clinical relevant difference to detect, the sample size for a subsequent trial is estimated. However, data from real world evidence (RWE) studies, such as observational studies and other interventional studies in patients in routine clinical practice, are not widely used in a systematic manner when designing studies. In this paper, we propose a framework for inclusion of RWE in planning of a clinical development programme. METHODS: In our proposed approach, all evidence, from both RCTs and RWE (i.e. from studies in routine clinical practice), available at the time of designing of a new clinical trial is combined in a Bayesian network meta-analysis (NMA). The results can be used to inform the design of the next clinical trial in the programme. The NMA was performed at key milestones, such as at the end of the phase II trial and prior to the design of key phase III studies. To illustrate the methods, we designed an alternative clinical development programme in multiple sclerosis using RWE through clinical trial simulations. RESULTS: Inclusion of RWE in the NMA and the resulting trial simulations demonstrated that 284 patients per arm were needed to achieve 90% power to detect effects of predetermined size in the TRANSFORMS study. For the FREEDOMS and FREEDOMS II clinical trials, 189 patients per arm were required. Overall there was a reduction in sample size of at least 40% across the three phase III studies, which translated to a time savings of at least 6 months for the undertaking of the fingolimod phase III programme. CONCLUSION: The use of RWE resulted in a reduced sample size of the pivotal phase III studies, which led to substantial time savings compared to the approach of sample size calculations without RWE

Supplementary materials: Adjusting for treatment crossover in the MAVORIC trial: survival in advanced mycosis fungoides and Sezary syndrome

These are peer-reviewed supplementary materials for the article 'Adjusting for treatment crossover in the MAVORIC trial: survival in advanced mycosis fungoides and Sezary syndrome' published in the Journal of Comparative Effectiveness Research.A. List of variables available for analysis B. Methodology to fit survival curves C. IPCW extra detail: Weights Figure 1: Histogram of weights from the IPCW analysisFigure 2: Plot of weights over time from the IPCW analysisD. ITT population RPSFTMIPCWTSEBackground: Relative overall survival (OS) estimates reported in the MAVORIC trial are potentially confounded by a high proportion of patients randomized to vorinostat switching to mogamulizumab; furthermore, vorinostat is not used in clinical practice in the UK. Methods: Three methods were considered for crossover adjustment. Survival post-crossover adjustment was compared with data from the Hospital Episode Statistics (HES) to contextualize estimates. Results: Following adjustment, the OS hazard ratio for mogamulizumab versus vorinostat was 0.42 (95% CI: 0.18, 0.98) using the method considered most appropriate based on an assessment of assumptions and comparison with HES. Conclusions: OS of mogamulizumab relative to vorinostat may be underestimated in MAVORIC due to the presence of crossover. The HES database was used to validate this adjustment.</p