Bias estimation in study design: a meta-epidemiological analysis of transcatheter versus surgical aortic valve replacement

Background: Paucity of RCTs of non-drug technologies lead to widespread dependence on non-randomized studies. Relationship between nonrandomized study design attributes and biased estimates of treatment effects are poorly understood. Our purpose was to estimate the bias associated with specific nonrandomized study attributes among studies comparing transcatheter aortic valve implantation with surgical aortic valve replacement for the treatment of severe aortic stenosis. Results: We included 6 RCTs and 87 nonrandomized studies. Surgical risk scores were similar for comparison groups in RCTs, but were higher for patients having transcatheter aortic valve implantation in nonrandomized studies. Nonrandomized studies underestimated the benefit of transcatheter aortic valve implantation compared with RCTs. For example, nonrandomized studies without adjustment estimated a higher risk of postoperative mortality for transcatheter aortic valve implantation compared with surgical aortic valve replacement (OR 1.43 [95% CI 1.26 to 1.62]) than high quality RCTs (OR 0.78 [95% CI 0.54 to 1.11). Nonrandomized studies using propensity score matching (OR 1.13 [95% CI 0.85 to 1.52]) and regression modelling (OR 0.68 [95% CI 0.57 to 0.81]) to adjust results estimated treatment effects closer to high quality RCTs. Nonrandomized studies describing losses to follow-up estimated treatment effects that were significantly closer to high quality RCT than nonrandomized studies that did not. Conclusion: Studies with different attributes produce different estimates of treatment effects. Study design attributes related to the completeness of follow-up may explain biased treatment estimates in nonrandomized studies, as in the case of aortic valve replacement where high-risk patients were preferentially selected for the newer (transcatheter) procedure

ROBINS-I: a tool for assessing risk of bias in non-randomized studies of interventions

Non-randomised studies of the effects of interventions are critical to many areas of healthcare evaluation, but their results may be biased. It is therefore important to understand and appraise their strengths and weaknesses. We developed ROBINS-I ("Risk Of Bias In Non-randomised Studies-of Interventions"), a new tool for evaluating risk of bias in estimates of the comparative effectiveness (harm or benefit) of interventions from studies that did not use randomisation to allocate units (individuals or clusters of individuals) to comparison groups. The tool will be particularly useful to those undertaking systematic reviews that include non-randomised studies

An Empirical Study of Bias in Randomized Controlled Trials and Non-randomized Studies of Surgical Interventions

Objectives: The aim of this dissertation was to examine bias in randomized controlled trials (RCTs) and non-randomized studies (NRS) in surgery using the literature evaluating laparoscopy and conventional (i.e. open) surgery for the treatment of colon cancer as a case study. The objectives were 1) to develop a conceptual framework for bias in comparative NRS; 2) to compare effect estimates from NRS with those from RCTs at low risk of bias; 3) to explore the impact of NRS-design attributes on estimates of treatment effect. Methods: The methods included a modified framework synthesis, systematic review of the literature, random-effects meta-analyses, and frequentist and Bayesian meta-regression. The Cochrane Risk of Bias Tool was used to classify trials as Strong RCTs (i.e. low risk of bias) or Typical RCTs (i.e. unclear or high risk of bias). Results: A conceptual framework for bias in comparative NRS was developed and it contains 37 individual sources of bias or “items”. These items were organized within 6 overarching “domains”: selection bias, information bias, performance bias, detection bias, attrition bias, and selective reporting bias. Our analyses revealed that NRS were associated with more extreme estimates of benefit for laparoscopy than Strong RCTs when examining subjective outcomes. The odds ratios from NRS were 36% smaller (i.e. demonstrating more benefit for laparoscopy) than those from Strong RCTs for the outcome post-operative complications (Ratio of Odds Ratios, ROR 0.64, [0.42, 0.97], p=0.04). Similar exaggerated benefit was seen among NRS when assessing length of stay, (Difference in Mean Differences, -2.15 days, [-4.08, -0.21], p=0.03). This pattern was not observed with the objective outcomes peri-operative mortality and number of lymph nodes harvested. Analyses adjusted for period effects and between-study case-mix yielded similar findings. Finally, effect estimates in NRS did not consistently vary according to the presence or absence of nine design characteristics identified from the conceptual framework. Conclusions: We have demonstrated that the results of surgical NRS can be significantly biased as compared with those of low risk of bias RCTs when evaluating subjective outcomes. However, none of the nine NRS-design characteristics examined was consistently associated with biased effect estimates.Ph