Delta inflation: a bias in the design of randomized controlled trials in critical care medicine

Introduction: Mortality is the most widely accepted outcome measure in randomized controlled trials of therapies for critically ill adults, but most of these trials fail to show a statistically significant mortality benefit. The reasons for this are unknown. Methods: We searched five high impact journals (Annals of Internal Medicine, British Medical Journal, JAMA, The Lancet, New England Journal of Medicine) for randomized controlled trials comparing mortality of therapies for critically ill adults over a ten year period. We abstracted data on the statistical design and results of these trials to compare the predicted delta (delta; the effect size of the therapy compared to control expressed as an absolute mortality reduction) to the observed delta to determine if there is a systematic overestimation of predicted delta that might explain the high prevalence of negative results in these trials. Results: We found 38 trials meeting our inclusion criteria. Only 5/38 (13.2%) of the trials provided justification for the predicted delta. The mean predicted delta among the 38 trials was 10.1% and the mean observed delta was 1.4% (P < 0.0001), resulting in a delta-gap of 8.7%. In only 2/38 (5.3%) of the trials did the observed delta exceed the predicted delta and only 7/38 (18.4%) of the trials demonstrated statistically significant results in the hypothesized direction; these trials had smaller delta-gaps than the remainder of the trials (delta-gap 0.9% versus 10.5%; P < 0.0001). For trials showing non-significant trends toward benefit greater than 3%, large increases in sample size (380% - 1100%) would be required if repeat trials use the observed delta from the index trial as the predicted delta for a follow-up study. Conclusions: Investigators of therapies for critical illness systematically overestimate treatment effect size (delta) during the design of randomized controlled trials. This bias, which we refer to as "delta inflation", is a potential reason that these trials have a high rate of negative results

Results from the national sepsis practice survey: predictions about mortality and morbidity and recommendations for limitation of care orders

Introduction: Critically ill patients and families rely upon physicians to provide estimates of prognosis and recommendations for care. Little is known about patient and clinician factors which influence these predictions. The association between these predictions and recommendations for continued aggressive care is also understudied. Methods: We administered a mail-based survey with simulated clinical vignettes to a random sample of the Critical Care Assembly of the American Thoracic Society. Vignettes represented a patient with septic shock with multi-organ failure with identical APACHE II scores and sepsis-associated organ failures. Vignettes varied by age (50 or 70 years old), body mass index (BMI) (normal or obese) and co-morbidities (none or recently diagnosed stage IIA lung cancer). All subjects received the vignettes with the highest and lowest mortality predictions from pilot testing and two additional, randomly selected vignettes. Respondents estimated outcomes and selected care for each hypothetical patient. Results: Despite identical severity of illness, the range of estimates for hospital mortality (5th to 95th percentile range, 17% to 78%) and for problems with self-care (5th to 95th percentile range, 2% to 74%) was wide. Similar variation was observed when clinical factors (age, BMI, and co-morbidities) were identical. Estimates of hospital mortality and problems with self-care among survivors were significantly higher in vignettes with obese BMIs (4.3% and 5.3% higher, respectively), older age (8.2% and 11.6% higher, respectively), and cancer diagnosis (5.9% and 6.9% higher, respectively). Higher estimates of mortality (adjusted odds ratio 1.29 per 10% increase in predicted mortality), perceived problems with self-care (adjusted odds ratio 1.26 per 10% increase in predicted problems with self-care), and early-stage lung cancer (adjusted odds ratio 5.82) were independently associated with recommendations to limit care. Conclusions: The studied clinical factors were consistently associated with poorer outcome predictions but did not explain the variation in prognoses offered by experienced physicians. These observations raise concern that provided information and the resulting decisions about continued aggressive care may be influenced by individual physician perception. To provide more reliable and accurate estimates of outcomes, tools are needed which incorporate patient characteristics and preferences with physician predictions and practices

Analysis of physicians' probability estimates of a medical outcome based on a sequence of events

IMPORTANCE The probability of a conjunction of 2 independent events is the product of the probabilities of the 2 components and therefore cannot exceed the probability of either component; violation of this basic law is called the conjunction fallacy. A common medical decision-making scenario involves estimating the probability of a final outcome resulting from a sequence of independent events; however, little is known about physicians' ability to accurately estimate the overall probability of success in these situations. OBJECTIVE To ascertain whether physicians are able to correctly estimate the overall probability of a medical outcome resulting from 2 independent events. DESIGN, SETTING, AND PARTICIPANTS This survey study consisted of 3 separate substudies, in which 215 physicians were asked via internet-based survey to estimate the probability of success of each of 2 components of a diagnostic or prognostic sequence as well as the overall probability of success of the 2-step sequence. Substudy 1 was performed from April 2 to 4, 2021, substudy 2 from November 2 toll, 2021, and substudy 3 from May 13 to 19, 2021. All physicians were board certified or board eligible in the primary specialty germane to the substudy (ie, obstetrics and gynecology for substudies land 3 and pulmonology for substudy 2), were recruited from a commercial survey service, and volunteered to participate in the study. EXPOSURES Case scenarios presented in an online survey. MAIN OUTCOMES AND MEASURES Respondents were asked to provide their demographic information in addition to 3 probability estimates. The first substudy included a scenario describing a brow presentation discovered during labor; the 2 conjuncts were the probabilities that the brow presentation would resolve and that the delivery would be vaginal. The second substudy involved a diagnostic evaluation of an incidentally discovered pulmonary nodule; the 2 conjuncts were the probabilities that the patient had a malignant condition and that a technically successful transthoracic needle biopsy would reveal a malignant condition. The third substudy included a modification of the first substudy in an attempt to debias the conjunction fallacy prevalent in the first substudy. Respondents' own probability estimates of the individual events were used to calculate the mathematically correct conjunctive probability. RESULTS Among 215 respondents, the mean (SD) age was 54.0 (9.5) years; 142 respondents (66.0%) were male. Data on race and ethnicity were not collected. A total of 168 physicians (78.1%) estimated the probability of the 2-step sequence to be greater than the probability of at least 1 of the 2 component events. Compared with the product of their 2 estimated components, respondents overestimated the combined probability by 12.8% (95% CI, 9.6%-16.1%; P < .001) in substudy 1, 19.8% (95% Cl, 16.6%-23.0%; P < .001) in substudy 2, and 18.0% (95% CI, 13.4%-22.5%; P < .001) in substudy 3, results that were mathematically incoherent (ie, formally illogical and mathematically incorrect). CONCLUSIONS AND RELEVANCE In this survey study of 215 physicians, respondents consistently overestimated the combined probability of 2 events compared with the probability calculated from their own estimates of the individual events. This biased estimation, consistent with the conjunction fallacy, may have substantial implications for diagnostic and prognostic decision-making

Effect sizes in ongoing randomized controlled critical care trials

Abstract Background An important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians’ views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality. Methods We conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations. Results Estimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10–20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5–5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6–10%) (P = 0.27). Conclusions For some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred