Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap-based optimism correction methods

In assessing prediction accuracy of multivariable prediction models, optimism corrections are essential for preventing biased results. However, in most published papers of clinical prediction models, the point estimates of the prediction accuracy measures are corrected by adequate bootstrap-based correction methods, but their confidence intervals are not corrected, e.g., the DeLong's confidence interval is usually used for assessing the C-statistic. These naive methods do not adjust for the optimism bias and do not account for statistical variability in the estimation of parameters in the prediction models. Therefore, their coverage probabilities of the true value of the prediction accuracy measure can be seriously below the nominal level (e.g., 95%). In this article, we provide two generic bootstrap methods, namely (1) location-shifted bootstrap confidence intervals and (2) two-stage bootstrap confidence intervals, that can be generally applied to the bootstrap-based optimism correction methods, i.e., the Harrell's bias correction, 0.632, and 0.632+ methods. In addition, they can be widely applied to various methods for prediction model development involving modern shrinkage methods such as the ridge and lasso regressions. Through numerical evaluations by simulations, the proposed confidence intervals showed favourable coverage performances. Besides, the current standard practices based on the optimism-uncorrected methods showed serious undercoverage properties. To avoid erroneous results, the optimism-uncorrected confidence intervals should not be used in practice, and the adjusted methods are recommended instead. We also developed the R package predboot for implementing these methods (https://github.com/nomahi/predboot). The effectiveness of the proposed methods are illustrated via applications to the GUSTO-I clinical trial

Errors in Relative Risks Reported in Figure 3 in a Network Meta-analysis of Cognitive Behavior Therapy Delivery Formats in Adults with Depression

To the Editor: The authors regret to report that Figure 3B was not represented correctly in their Original Investigation, “Effectiveness and Acceptability of Cognitive Behavior Therapy Delivery Formats in Adults With Depression: A Network Meta-analysis,”1 published in the July 2019 issue. In reviewing the article for a presentation, a coauthor detected the errors. In the original Figure 3B, we had shown the dropouts of care as usual over each of the treatment formats, instead of dropouts of the formats over care as usual. But Figure 3B supposed care as usual to be the common comparator, so the correct representation should give the dropouts of the various formats over care as usual. We therefore had to reverse the relative risks. Because of this adjustment, we had to change a number in the text and add a phrase for context, but none of these changes affect the interpretations or conclusions of the study. Thus, we have requested that our article be corrected.2 The authors apologize for any inconvenience caused. This article was previously corrected on July 17, 2019, to fix a label error in Figure 3B.

Around ten percent of most recent Cochrane reviews included outcomes in their literature search strategy and were associated with potentially exaggerated results: A research-on-research study

[Objectives] To assess the proportion of the recent Cochrane reviews that included outcomes in their literature search strategy, how often they acknowledged these limitations, and how qualitatively different the results of outcomes included and not included in the search strategy were. [Design and Setting] We identified all the Cochrane reviews of the interventions published in 2020 that used a search strategy connecting outcome terms with “AND.” Reviews were defined as acknowledging the limitations of searching for outcomes if they mentioned them in the discussion. We compared the characteristics of outcomes included and not included in the search strategy. [Results] Of the 523 Cochrane reviews published in 2020, 51 (9.8%) included outcomes in their search strategy. Only one review acknowledged it as a limitation. Forty-seven (92%) assessed outcomes not included in the search strategy. Outcomes included in the search strategies tended to include a larger number of studies and show their effects in favor of the intervention. [Conclusions] Around ten percent of the recent Cochrane reviews included outcomes in their search, which may have resulted in more outcomes significantly in favor of the intervention. Reviewers should be more explicit in acknowledging the potential implications of searching for outcomes

Estimating Patient-Specific Relative Benefit of Adding Biologics to Conventional Rheumatoid Arthritis Treatment: An Individual Participant Data Meta-Analysis.

IMPORTANCE Current evidence remains ambiguous regarding whether biologics should be added to conventional treatment of rheumatoid arthritis for specific patients, which may cause potential overuse or treatment delay. OBJECTIVES To estimate the benefit of adding biologics to conventional antirheumatic drugs for the treatment of rheumatoid arthritis given baseline characteristics. DATA SOURCES Cochrane CENTRAL, Scopus, MEDLINE, and the World Health Organization International Clinical Trials Registry Platform were searched for articles published from database inception to March 2, 2022. STUDY SELECTION Randomized clinical trials comparing certolizumab plus conventional antirheumatic drugs with placebo plus conventional drugs were selected. DATA EXTRACTION AND SYNTHESIS Individual participant data of the prespecified outcomes and covariates were acquired from the Vivli database. A 2-stage model was fitted to estimate patient-specific relative outcomes of adding certolizumab vs conventional drugs only. Stage 1 was a penalized logistic regression model to estimate the baseline expected probability of the outcome regardless of treatment using baseline characteristics. Stage 2 was a bayesian individual participant data meta-regression model to estimate the relative outcomes for a particular baseline expected probability. Patient-specific results were displayed interactively on an application based on a 2-stage model. MAIN OUTCOMES AND MEASURES The primary outcome was low disease activity or remission at 3 months, defined by 3 disease activity indexes (ie, Disease Activity Score based on the evaluation of 28 joints, Clinical Disease Activity Index, or Simplified Disease Activity Index). RESULTS Individual participant data were obtained from 3790 patients (2996 female [79.1%] and 794 male [20.9%]; mean [SD] age, 52.7 [12.3] years) from 5 large randomized clinical trials for moderate to high activity rheumatoid arthritis with usable data for 22 prespecified baseline covariates. Overall, adding certolizumab was associated with a higher probability of reaching low disease activity. The odds ratio for patients with an average baseline expected probability of the outcome was 6.31 (95% credible interval, 2.22-15.25). However, the benefits differed in patients with different baseline characteristics. For example, the estimated risk difference was smaller than 10% for patients with either low or high baseline expected probability. CONCLUSIONS AND RELEVANCE In this individual participant data meta-analysis, adding certolizumab was associated with more effectiveness for rheumatoid arthritis in general. However, the benefit was uncertain for patients with low or high baseline expected probability, for whom other evaluations were necessary. The interactive application displaying individual estimates may help with treatment selection