Emulating Target Trials With Real-World Data to Inform Health Technology Assessment: Findings and Lessons From an Application to Emergency Surgery

Objectives: International health technology assessment (HTA) agencies recommend that real-world data (RWD) are used in some circumstances to add to the evidence base about the effectiveness and cost-effectiveness of health interventions. The target trial framework applies the design principles of randomized-controlled trials to RWD and can help alleviate inevitable concerns about bias and design flaws with nonrandomized studies. This article aimed to tackle the lack of guidance and exemplar applications on how this methodology can be applied to RWD to inform HTA decision making./ Methods: We use Hospital Episode Statistics data from England on emergency hospital admissions from 2010 to 2019 to evaluate the cost-effectiveness of emergency surgery for 2 acute gastrointestinal conditions. We draw on the case study to describe the main challenges in applying the target trial framework alongside RWD and provide recommendations for how these can be addressed in practice./ Results: The 4 main challenges when applying the target trial framework to RWD are (1) defining the study population, (2) defining the treatment strategies, (3) establishing time zero (baseline), and (4) adjusting for unmeasured confounding. The recommendations for how to address these challenges, mainly around the incorporation of expert judgment and use of appropriate methods for handling unmeasured confounding, are illustrated within the case study./ Conclusions: The recommendations outlined in this study could help future studies seeking to inform HTA decision processes. These recommendations can complement checklists for economic evaluations and design tools for estimating treatment effectiveness in nonrandomized studies

How does a local instrumental variable method perform across settings with instruments of differing strengths? A simulation study and an evaluation of emergency surgery

Local instrumental variable (LIV) approaches use continuous/multi-valued instrumental variables (IV) to generate consistent estimates of average treatment effects (ATEs) and Conditional Average Treatment Effects (CATEs). There is little evidence on how LIV approaches perform according to the strength of the IV or with different sample sizes. Our simulation study examined the performance of an LIV method, and a two-stage least squares (2SLS) approach across different sample sizes and IV strengths. We considered four 'heterogeneity' scenarios: homogeneity, overt heterogeneity (over measured covariates), essential heterogeneity (unmeasured), and overt and essential heterogeneity combined. In all scenarios, LIV reported estimates with low bias even with the smallest sample size, provided that the instrument was strong. Compared to 2SLS, LIV provided estimates for ATE and CATE with lower levels of bias and Root Mean Squared Error. With smaller sample sizes, both approaches required stronger IVs to ensure low bias. We considered both methods in evaluating emergency surgery (ES) for three acute gastrointestinal conditions. Whereas 2SLS found no differences in the effectiveness of ES according to subgroup, LIV reported that frailer patients had worse outcomes following ES. In settings with continuous IVs of moderate strength, LIV approaches are better suited than 2SLS to estimate policy-relevant treatment effect parameters

Emulating Target Trials With Real-World Data to Inform Health Technology Assessment: Findings and Lessons From an Application to Emergency Surgery.

OBJECTIVES: International health technology assessment (HTA) agencies recommend that real-world data (RWD) are used in some circumstances to add to the evidence base about the effectiveness and cost-effectiveness of health interventions. The target trial framework applies the design principles of randomized-controlled trials to RWD and can help alleviate inevitable concerns about bias and design flaws with nonrandomized studies. This article aimed to tackle the lack of guidance and exemplar applications on how this methodology can be applied to RWD to inform HTA decision making. METHODS: We use Hospital Episode Statistics data from England on emergency hospital admissions from 2010 to 2019 to evaluate the cost-effectiveness of emergency surgery for 2 acute gastrointestinal conditions. We draw on the case study to describe the main challenges in applying the target trial framework alongside RWD and provide recommendations for how these can be addressed in practice. RESULTS: The 4 main challenges when applying the target trial framework to RWD are (1) defining the study population, (2) defining the treatment strategies, (3) establishing time zero (baseline), and (4) adjusting for unmeasured confounding. The recommendations for how to address these challenges, mainly around the incorporation of expert judgment and use of appropriate methods for handling unmeasured confounding, are illustrated within the case study. CONCLUSIONS: The recommendations outlined in this study could help future studies seeking to inform HTA decision processes. These recommendations can complement checklists for economic evaluations and design tools for estimating treatment effectiveness in nonrandomized studies

Variation in the rates of emergency surgery amongst emergency admissions to hospital for common acute conditions.

BACKGROUND: This paper assesses variation in rates of emergency surgery (ES) amongst emergency admissions to hospital in patients with acute appendicitis, cholelithiasis, diverticular disease, abdominal wall hernia, and intestinal obstruction. METHODS: Records of emergency admissions between 1 April 2010 and 31 December 2019 for the five conditions were extracted from Hospital Episode Statistics for 136 acute National Health Service (NHS) trusts in England. Patients who had ES were identified using Office of Population Censuses and Surveys (OPCS) procedure codes, selected by consensus of a clinical panel. The differences in ES rates according to patient characteristics, and unexplained variations across NHS trusts were estimated by multilevel logistic regression, adjusting for year of emergency admission, age, sex, ethnicity, diagnostic subcategories, index of multiple deprivation, number of co-morbidities, and frailty. RESULTS: The cohort sizes ranged from 107 325 (hernia) to 268 253 (appendicitis) patients, and the proportion of patients who received ES from 11.0 per cent (diverticular disease) to 92.3 per cent (appendicitis). Older patients were generally less likely to receive ES, with adjusted odds ratios (ORs) of ES for those aged 75-79 versus those aged 45-49 years: 0.34 (appendicitis), 0.49 (cholelithiasis), 0.87 (hernia), and 0.91 (intestinal obstruction). Patients with diverticular disease aged 75-79 were more likely to receive ES than those aged 45-49 (OR 1.40). Variation in ES rates across NHS trusts remained after case mix adjustment and was greatest for cholelithiasis (trust median 18 per cent, 10th to 90th centile 7-35 per cent). CONCLUSION: For patients presenting as emergency hospital admissions with common acute conditions, variation in ES rates between NHS trusts remained after adjustment for demographic and clinical characteristics. Age was strongly associated with the likelihood of ES receipt for some procedures

Local Instrumental Variable Methods to Address Confounding and Heterogeneity when Using Electronic Health Records: An Application to Emergency Surgery

BACKGROUND: Electronic health records (EHRs) offer opportunities for comparative effectiveness research to inform decision making. However, to provide useful evidence, these studies must address confounding and treatment effect heterogeneity according to unmeasured prognostic factors. Local instrumental variable (LIV) methods can help studies address these challenges, but have yet to be applied to EHR data. This article critically examines a LIV approach to evaluate the cost-effectiveness of emergency surgery (ES) for common acute conditions from EHRs. METHODS: This article uses hospital episodes statistics (HES) data for emergency hospital admissions with acute appendicitis, diverticular disease, and abdominal wall hernia to 175 acute hospitals in England from 2010 to 2019. For each emergency admission, the instrumental variable for ES receipt was each hospital's ES rate in the year preceding the emergency admission. The LIV approach provided individual-level estimates of the incremental quality-adjusted life-years, costs and net monetary benefit of ES, which were aggregated to the overall population and subpopulations of interest, and contrasted with those from traditional IV and risk-adjustment approaches. RESULTS: The study included 268,144 (appendicitis), 138,869 (diverticular disease), and 106,432 (hernia) patients. The instrument was found to be strong and to minimize covariate imbalance. For diverticular disease, the results differed by method; although the traditional approaches reported that, overall, ES was not cost-effective, the LIV approach reported that ES was cost-effective but with wide statistical uncertainty. For all 3 conditions, the LIV approach found heterogeneity in the cost-effectiveness estimates across population subgroups: in particular, ES was not cost-effective for patients with severe levels of frailty. CONCLUSIONS: EHRs can be combined with LIV methods to provide evidence on the cost-effectiveness of routinely provided interventions, while fully recognizing heterogeneity. HIGHLIGHTS: This article addresses the confounding and heterogeneity that arise when assessing the comparative effectiveness from electronic health records (EHR) data, by applying a local instrumental variable (LIV) approach to evaluate the cost-effectiveness of emergency surgery (ES) versus alternative strategies, for patients with common acute conditions (appendicitis, diverticular disease, and abdominal wall hernia).The instrumental variable, the hospital's tendency to operate, was found to be strongly associated with ES receipt and to minimize imbalances in baseline characteristics between the comparison groups.The LIV approach found that, for each condition, there was heterogeneity in the estimates of cost-effectiveness according to baseline characteristics.The study illustrates how an LIV approach can be applied to EHR data to provide cost-effectiveness estimates that recognize heterogeneity and can be used to inform decision making as well as to generate hypotheses for further research

Clinical effectiveness and cost-effectiveness of emergency surgery for adult emergency hospital admissions with common acute gastrointestinal conditions: the ESORT study

Background: Evidence is required on the clinical effectiveness and cost-effectiveness of emergency surgery compared with non-emergency surgery strategies (including medical management, non-surgical procedures and elective surgery) for patients admitted to hospital with common acute gastrointestinal conditions. Objectives: We aimed to evaluate the relative (1) clinical effectiveness of two strategies (i.e. emergency surgery vs. non-emergency surgery strategies) for five common acute conditions presenting as emergency admissions; (2) cost-effectiveness for five common acute conditions presenting as emergency admissions; and (3) clinical effectiveness and cost-effectiveness of the alternative strategies for specific patient subgroups. Methods: The records of adults admitted as emergencies with acute appendicitis, cholelithiasis, diverticular disease, abdominal wall hernia or intestinal obstruction to 175 acute hospitals in England between 1 April 2010 and 31 December 2019 were extracted from Hospital Episode Statistics and linked to mortality data from the Office for National Statistics. Eligibility was determined using International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, diagnosis codes, which were agreed by clinical panel consensus. Patients having emergency surgery were identified from Office of Population Censuses and Surveys procedure codes. The study addressed the potential for unmeasured confounding with an instrumental variable design. The instrumental variable was each hospital’s propensity to use emergency surgery compared with non-emergencysurgery strategies. The primary outcome was the ‘number of days alive and out of hospital’ at 90 days. We reported the relative effectiveness of the alternative strategies overall, and for prespecified subgroups (i.e. age, number of comorbidities and frailty level). The cost-effectiveness analyses used resource use and mortality from the linked data to derive estimates of incremental costs, quality-adjusted life-years and incremental net monetary benefits at 1 year. Results: Cohort sizes were as follows: 268,144 admissions with appendicitis, 240,977 admissions with cholelithiasis, 138,869 admissions with diverticular disease, 106,432 admissions with a hernia and 133,073 admissions with an intestinal obstruction. Overall, at 1 year, the average number of days alive and out of hospitals at 90 days, costs and quality-adjusted life-years were similar following either strategy, after adjusting for confounding. For each of the five conditions, overall, the 95% confidence intervals (CIs) around the incremental net monetary benefit estimates all included zero. For patients with severe frailty, emergency surgery led to a reduced number of days alive and out of hospital and was not cost-effective compared with non-emergency surgery, with incremental net monetary benefit estimates of –£18,727 (95% CI –£23,900 to –£13,600) for appendicitis, –£7700 (95% CI –£13,000 to –£2370) for cholelithiasis, –£9230 (95% CI –£24,300 to £5860) for diverticular disease, –£16,600 (95% CI –£21,100 to –£12,000) for hernias and –£19,300 (95% CI –£25,600 to –£13,000) for intestinal obstructions. For patients who were ‘fit’, emergency surgery was relatively cost-effective, with estimated incremental net monetary benefit estimates of £5180 (95% CI £684 to £9680) for diverticular disease, £2040 (95% CI £996 to £3090) for hernias, £7850 (95% CI £5020 to £10,700) for intestinal obstructions, £369 (95% CI –£728 to £1460) for appendicitis and £718 (95% CI £294 to £1140) for cholelithiasis. Public and patient involvement translation workshop participants emphasised that these findings should be made widely available to inform future decisions about surgery. Limitations: The instrumental variable approach did not eliminate the risk of confounding, and the acute hospital perspective excluded costs to other providers. Conclusions: Neither strategy was more cost-effective overall. For patients with severe frailty, nonemergency surgery strategies were relatively cost-effective. For patients who were fit, emergency surgery was more cost-effective. Future work: For patients with multiple long-term conditions, further research is required to assess the benefits and costs of emergency surgery. Study registration: This study is registered as reviewregistry784

SP3.1.5 Variation and predictors of emergency surgery in emergency hospital admissions from the Emergency Surgery Or Not (ESORT) study

To assess variation in use of emergency surgery (ES) for emergency hospital admissions with acute appendicitis, cholelithiasis, diverticular disease, abdominal wall hernia or intestinal obstruction.Cohorts were extracted from Hospital Episode Statistics for 136 acute NHS trusts in England. Clinical panel consensus defined ES for emergency admissions between 1/4/2010 and 31/12/2019. The association of socio-demographic characteristics with ES use was estimated by multivariable logistic regression, with adjustment for comorbidity, frailty, diagnosis and trust.The cohort sizes ranged from 49,385 (hernia) to 184,777 (appendicitis) patients. ES was less likely for: patients aged over 80, with odds ratios (ORs) across conditions from 0.15 to 0.84 versus those aged under 40; the most deprived, ORs 0.83 to 0.92, versus least deprived; and Asian patients, ORs 0.72 to 0.88, versus White patients. Black patients were less likely to have emergency surgery for appendicitis (OR 0.78) and cholelithiasis (OR 0.78). Females were less likely to have ES for appendicitis (OR 0.94 versus males), but more likely to have surgery for intestinal obstruction (OR 1.29), hernia (OR 1.13) and cholelithiasis (OR 1.22). Unexplained variation in ES across trusts, remained after case-mix adjustment, and was greatest for cholelithiasis (median of 16%, 10 to 90 centile 5%-34%), and hernia (61%, 52%-71%), followed by intestinal obstruction (29%, 24%-36%). appendicitis (93%, 89%-95%), and diverticular disease (15%, 11%-20%).The socio-demographic characteristics of emergency admissions are associated with the likelihood of receiving ES. Variation in ES use between NHS trusts remained after adjustment for demographic and clinical characteristics.RD&E staff can access the full-text of this article by clicking on the 'Additional Link' above and logging in with NHS OpenAthens if prompted.Published version, accepted version (12 month embargo), submitted versio