Delta inflation: a bias in the design of randomized controlled trials in critical care medicine

Introduction: Mortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown. Methods: We searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for critically ill adults over a ten year period. We abstracted data on the statistical design and results of these trials to compare the predicted delta (delta; the effect size of the therapy compared to control expressed as an absolute mortality reduction) to the observed delta to determine if there is a systematic overestimation of predicted delta that might explain the high prevalence of negative results in these trials. Results: We found 38 trials meeting our inclusion criteria. Only 5/38 (13.2%) of the trials provided justification for the predicted delta. The mean predicted delta among the 38 trials was 10.1% and the mean observed delta was 1.4% (P < 0.0001), resulting in a delta-gap of 8.7%. In only 2/38 (5.3%) of the trials did the observed delta exceed the predicted delta and only 7/38 (18.4%) of the trials demonstrated statistically significant results in the hypothesized direction; these trials had smaller delta-gaps than the remainder of the trials (delta-gap 0.9% versus 10.5%; P < 0.0001). For trials showing non-significant trends toward benefit greater than 3%, large increases in sample size (380% - 1100%) would be required if repeat trials use the observed delta from the index trial as the predicted delta for a follow-up study. Conclusions: Investigators of therapies for critical illness systematically overestimate treatment effect size (delta) during the design of randomized controlled trials. This bias, which we refer to as "delta inflation", is a potential reason that these trials have a high rate of negative results

Results from the national sepsis practice survey: predictions about mortality and morbidity and recommendations for limitation of care orders

Introduction: Critically ill patients and families rely upon physicians to provide estimates of prognosis and recommendations for care. Little is known about patient and clinician factors which influence these predictions. The association between these predictions and recommendations for continued aggressive care is also understudied. Methods: We administered a mail-based survey with simulated clinical vignettes to a random sample of the Critical Care Assembly of the American Thoracic Society. Vignettes represented a patient with septic shock with multi-organ failure with identical APACHE II scores and sepsis-associated organ failures. Vignettes varied by age (50 or 70 years old), body mass index (BMI) (normal or obese) and co-morbidities (none or recently diagnosed stage IIA lung cancer). All subjects received the vignettes with the highest and lowest mortality predictions from pilot testing and two additional, randomly selected vignettes. Respondents estimated outcomes and selected care for each hypothetical patient. Results: Despite identical severity of illness, the range of estimates for hospital mortality (5th to 95th percentile range, 17% to 78%) and for problems with self-care (5th to 95th percentile range, 2% to 74%) was wide. Similar variation was observed when clinical factors (age, BMI, and co-morbidities) were identical. Estimates of hospital mortality and problems with self-care among survivors were significantly higher in vignettes with obese BMIs (4.3% and 5.3% higher, respectively), older age (8.2% and 11.6% higher, respectively), and cancer diagnosis (5.9% and 6.9% higher, respectively). Higher estimates of mortality (adjusted odds ratio 1.29 per 10% increase in predicted mortality), perceived problems with self-care (adjusted odds ratio 1.26 per 10% increase in predicted problems with self-care), and early-stage lung cancer (adjusted odds ratio 5.82) were independently associated with recommendations to limit care. Conclusions: The studied clinical factors were consistently associated with poorer outcome predictions but did not explain the variation in prognoses offered by experienced physicians. These observations raise concern that provided information and the resulting decisions about continued aggressive care may be influenced by individual physician perception. To provide more reliable and accurate estimates of outcomes, tools are needed which incorporate patient characteristics and preferences with physician predictions and practices

Effect sizes in ongoing randomized controlled critical care trials

Abstract Background An important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians’ views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality. Methods We conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations. Results Estimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10–20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5–5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6–10%) (P = 0.27). Conclusions For some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred