Effective sample size: a measure of individual uncertainty in predictions

Clinical prediction models are estimated using a sample of limited size from the target population, leading to uncertainty in predictions, even when the model is correctly specified. Generally, not all patient profiles are observed uniformly in model development. As a result, sampling uncertainty varies between individual patients' predictions. We aimed to develop an intuitive measure of individual prediction uncertainty. The variance of a patient's prediction can be equated to the variance of the sample mean outcome in n* hypothetical patients with the same predictor values. This hypothetical sample size n* can be interpreted as the number of similar patients n_eff that the prediction is effectively based on, given that the model is correct. For generalised linear models, we derived analytical expressions for the effective sample size. In addition, we illustrated the concept in patients with acute myocardial infarction. In model development, n_eff can be used to balance accuracy versus uncertainty of predictions. In a validation sample, the distribution of n_eff indicates which patients were more and less represented in the development data, and whether predictions might be too uncertain for some to be practically meaningful. In a clinical setting, the effective sample size may facilitate communication of uncertainty about predictions. We propose the effective sample size as a clinically interpretable measure of uncertainty in individual predictions. Its implications should be explored further for the development, validation and clinical implementation of prediction models.Comment: 11 pages, 5 figure

Single-arm studies involving patient-reported outcome data in oncology: a literature review on current practice

Patient-reported outcomes (PROs) are increasingly used in single-arm cancer studies. We reviewed 60 recent publications of single-arm studies of cancer treatment involving PRO data for current practice on design, analysis, reporting, and interpretation. We further examined their handling of potential bias and how they informed decision-making. Most studies (97%) analyzed PROs without stating a predefined research hypothesis. Thirteen studies (22%) used a PRO as a (co)primary endpoint. Definitions of PRO objectives, study population, endpoints, and strategies of handling missing data varied widely. Twenty-three studies (38%) compared the PRO data to external information, most often by using a clinically important difference value; one study used a historical control group. Appropriateness of methods to handle missingness and intercurrent events including death were seldom discussed. Most studies (85%) concluded that PRO results supported treatment. Conducting and reporting of PROs in cancer single-arm studies lacks standards, and a critical discussion of statistical methods and possible biases. These findings will guide the Setting International Standards in Analysing Patient-Reported Outcomes and Quality of Life Data in Cancer Clinical Trials-Innovative Medicines Initiative (SISAQOL-IMI) in developing recommendations for the use of PRO-measures in single arm studies

Discontinuation of Anti-Tumour Necrosis Factor Therapy in Patients with Perianal Fistulizing Crohn's Disease:Individual Participant Data Meta-Analysis of 309 Patients from 12 Studies

BACKGROUND: The risk of relapse after anti-tumour necrosis factor [TNF] therapy discontinuation in Crohn's disease patients with perianal fistulas [pCD] is unclear. We aimed to assess this risk. METHODS: A systematic literature search was conducted to identify cohort studies on the incidence of relapse following anti-TNF discontinuation in pCD patients. Individual participant data were requested from the original study cohorts. Inclusion criteria were age ≥16 years, pCD as a (co)indication for start of anti-TNF therapy, more than three doses, and remission of luminal and pCD at anti-TNF discontinuation. The primary outcome was the cumulative incidence of CD relapse using Kaplan-Meier estimates. Secondary outcomes included response to re-treatment and risk factors associated with relapse as assessed by Cox regression analysis. RESULTS: In total, 309 patients from 12 studies in ten countries were included. The median duration of anti-TNF treatment was 14 months [interquartile range 5.8-32.5]. Most patients were treated for pCD without active luminal disease [89%], received first-line anti-TNF therapy [87%], and continued immunomodulatory therapy following anti-TNF discontinuation [78%]. The overall cumulative incidence of relapse was 36% (95% confidence interval [CI] 25-48%) and 42% [95% CI 32-53%] at 1 and 2 years after anti-TNF discontinuation, respectively. Risk factors for relapse included smoking (hazard ratio [HR] 1.5 [1.0, 2.1]) and history of proctitis (HR 1.7 [1.1, 2.5]). The overall re-treatment response rate was 82%. CONCLUSIONS: This individual participant data meta-analysis, on predominantly patients with pCD without active luminal disease and first-line anti-TNF therapy, shows that over half of patients remain in remission 2 years after anti-TNF discontinuation. Therefore, anti-TNF discontinuation may be considered in this subgroup.</p

Discontinuation of Anti-Tumour Necrosis Factor Therapy in Patients with Perianal Fistulizing Crohn's Disease

The risk of relapse after anti-tumour necrosis factor [TNF] therapy discontinuation in Crohn's disease patients with perianal fistulas [pCD] is unclear. We aimed to assess this risk.A systematic literature search was conducted to identify cohort studies on the incidence of relapse following anti-TNF discontinuation in pCD patients. Individual Participant Data were requested from the original study cohorts. Inclusion criteria were age ≥16 years, pCD as (co)indication for start of anti-TNF therapy, >3 doses, and remission of luminal and pCD at anti-TNF discontinuation. Primary outcome was the cumulative incidence of CD relapse using Kaplan-Meier estimates. Secondary outcomes included response to retreatment and risk factors associated with relapse as assessed by Cox regression analysis.309 patients from 12 studies in 10 countries were included. Median duration of anti-TNF treatment was 14 months [IQR 5.8 - 32.5]. Most patients were treated for pCD without active luminal disease [89%], received first line anti-TNF therapy [87%] and continued immunomodulatory following anti-TNF discontinuation [78%]. Overall cumulative incidence of relapse was 36% [95% CI 25-48%] and 42% [95% CI 32-53%] at 1 and 2 years after anti-TNF discontinuation. Risk factors for relapse included smoking [HR 1.5 (1.0, 2.1)] and history of proctitis [HR 1.7 (1.1, 2.5)]. Overall retreatment response rate was 82%.This IPD-MA, on predominantly patients with pCD without active luminal disease and first line anti-TNF therapy, shows that over half of patients remain in remission 2 years after anti-TNF discontinuation. Therefore, anti-TNF discontinuation may be considered in this subgroup

Validation and update of a prediction model for risk of relapse after cessation of anti-TNF treatment in Crohn's disease

BACKGROUND : Anti-tumor necrosis factor (TNF) therapy is effective for the treatment of Crohn's disease. Cessation may be considered in patients with a low risk of relapse. We aimed to externally validate and update our previously developed prediction model to estimate the risk of relapse after cessation of anti-TNF therapy. METHODS : We performed a retrospective cohort study in 17 Dutch hospitals. Crohn's disease patients in clinical, biochemical or endoscopic remission were included after anti-TNF cessation. Primary outcome was a relapse necessitating treatment. Discrimination and calibration of the previously developed model were assessed. After external validation, the model was updated. The performance of the updated prediction model was assessed in internal-external validation and by using decision curve analysis. RESULTS : 486 patients were included with a median follow-up of 1.7 years. Relapse rates were 35 and 54% after 1 and 2 years. At external validation, the discriminative ability of the prediction model was equal to that found at the development of the model [c-statistic 0.58 (95% confidence interval (CI) 0.54-0.62)], though the model was not well-calibrated on our cohort [calibration slope: 0.52 (0.28-0.76)]. After an update, a c-statistic of 0.60 (0.58-0.63) and calibration slope of 0.89 (0.69-1.09) were reported in internal-external validation. CONCLUSION : Our previously developed and updated prediction model for the risk of relapse after cessation of anti-TNF in Crohn's disease shows reasonable performance. The use of the model may support clinical decision-making to optimize patient selection in whom anti-TNF can be withdrawn. Clinical validation is ongoing in a prospective randomized trial

Validation and update of a prediction model for risk of relapse after cessation of anti-TNF treatment in Crohn's disease

BACKGROUND: Anti-tumor necrosis factor (TNF) therapy is effective for the treatment of Crohn's disease. Cessation may be considered in patients with a low risk of relapse. We aimed to externally validate and update our previously developed prediction model to estimate the risk of relapse after cessation of anti-TNF therapy. METHODS: We performed a retrospective cohort study in 17 Dutch hospitals. Crohn's disease patients in clinical, biochemical or endoscopic remission were included after anti-TNF cessation. Primary outcome was a relapse necessitating treatment. Discrimination and calibration of the previously developed model were assessed. After external validation, the model was updated. The performance of the updated prediction model was assessed in internal-external validation and by using decision curve analysis. RESULTS: 486 patients were included with a median follow-up of 1.7 years. Relapse rates were 35 and 54% after 1 and 2 years. At external validation, the discriminative ability of the prediction model was equal to that found at the development of the model [c-statistic 0.58 (95% confidence interval (CI) 0.54-0.62)], though the model was not well-calibrated on our cohort [calibration slope: 0.52 (0.28-0.76)]. After an update, a c-statistic of 0.60 (0.58-0.63) and calibration slope of 0.89 (0.69-1.09) were reported in internal-external validation. CONCLUSION: Our previously developed and updated prediction model for the risk of relapse after cessation of anti-TNF in Crohn's disease shows reasonable performance. The use of the model may support clinical decision-making to optimize patient selection in whom anti-TNF can be withdrawn. Clinical validation is ongoing in a prospective randomized trial

Single-arm studies involving patient-reported outcome data in oncology : a literature review on current practice

Patient-reported outcomes (PROs) are increasingly used in single-arm cancer studies. We reviewed 60 papers published between 2018 and 2021 of single-arm studies of cancer treatment with PRO data for current practice on design, analysis, reporting, and interpretation. We further examined the studies’ handling of potential bias and how they informed decision making. Most studies (58; 97%) analysed PROs without stating a predefined research hypothesis. 13 (22%) of the 60 studies used a PRO as a primary or co-primary endpoint. Definitions of PRO objectives, study population, endpoints, and missing data strategies varied widely. 23 studies (38%) compared the PRO data with external information, most often by using a clinically important difference value; one study used a historical control group. Appropriateness of methods to handle missing data and intercurrent events (including death) were seldom discussed. Most studies (51; 85%) concluded that PRO results supported treatment. Conducting and reporting of PROs in cancer single-arm studies need standards and a critical discussion of statistical methods and possible biases. These findings will guide the Setting International Standards in Analysing Patient-Reported Outcomes and Quality of Life Data in Cancer Clinical Trials–Innovative Medicines Initiative (SISAQOL-IMI) in developing recommendations for the use of PRO-measures in single-arm studies