The expected value of sample information calculations for external validation of risk prediction models

In designing external validation studies of clinical prediction models, contemporary sample size calculation methods are based on the frequentist inferential paradigm. One of the widely reported metrics of model performance is net benefit (NB), and the relevance of conventional inference around NB as a measure of clinical utility is doubtful. Value of Information methodology quantifies the consequences of uncertainty in terms of its impact on clinical utility of decisions. We introduce the expected value of sample information (EVSI) for validation as the expected gain in NB from conducting an external validation study of a given size. We propose algorithms for EVSI computation, and in a case study demonstrate how EVSI changes as a function of the amount of current information and future study's sample size. Value of Information methodology provides a decision-theoretic lens to the process of planning a validation study of a risk prediction model and can complement conventional methods when designing such studies.Comment: 14 pages, 4 figures, 0 table

Three myths about risk thresholds for prediction models

Acknowledgments This work was developed as part of the international initiative of strengthening analytical thinking for observational studies (STRATOS). The objective of STRATOS is to provide accessible and accurate guidance in the design and analysis of observational studies (http://stratos-initiative.org/). Members of the STRATOS Topic Group ‘Evaluating diagnostic tests and prediction models’ are Gary Collins, Carl Moons, Ewout Steyerberg, Patrick Bossuyt, Petra Macaskill, David McLernon, Ben van Calster, and Andrew Vickers. Funding The study is supported by the Research Foundation-Flanders (FWO) project G0B4716N and Internal Funds KU Leuven (project C24/15/037). Laure Wynants is a post-doctoral fellow of the Research Foundation – Flanders (FWO). The funding bodies had no role in the design of the study, collection, analysis, interpretation of data, nor in writing the manuscript. Contributions LW and BVC conceived the original idea of the manuscript, to which ES, MVS and DML then contributed. DT acquired the data. LW analyzed the data, interpreted the results and wrote the first draft. All authors revised the work, approved the submitted version, and are accountable for the integrity and accuracy of the work.Peer reviewedPublisher PD

Covipendium : information available to support the development of medical countermeasures and interventions against COVID-19

The living paper on the new coronavirus disease (COVID-19) provides a structured compilation of scientific data about the virus, the disease and its control. Its objective is to help scientists identify the most relevant publications on COVID-19 in the mass of information that appears every day. It is also expected to foster a global understanding of disease control and stimulate transdisciplinary initiatives

There is no such thing as a validated prediction model

Background: Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context? Main body: We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models. Conclusion: Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making

A comparison of regression models for static and dynamic prediction of a prognostic outcome during admission in electronic health care records

Objective Hospitals register information in the electronic health records (EHR) continuously until discharge or death. As such, there is no censoring for in-hospital outcomes. We aimed to compare different dynamic regression modeling approaches to predict central line-associated bloodstream infections (CLABSI) in EHR while accounting for competing events precluding CLABSI. Materials and Methods We analyzed data from 30,862 catheter episodes at University Hospitals Leuven from 2012 and 2013 to predict 7-day risk of CLABSI. Competing events are discharge and death. Static models at catheter onset included logistic, multinomial logistic, Cox, cause-specific hazard, and Fine-Gray regression. Dynamic models updated predictions daily up to 30 days after catheter onset (i.e. landmarks 0 to 30 days), and included landmark supermodel extensions of the static models, separate Fine-Gray models per landmark time, and regularized multi-task learning (RMTL). Model performance was assessed using 100 random 2:1 train-test splits. Results The Cox model performed worst of all static models in terms of area under the receiver operating characteristic curve (AUC) and calibration. Dynamic landmark supermodels reached peak AUCs between 0.741-0.747 at landmark 5. The Cox landmark supermodel had the worst AUCs (<=0.731) and calibration up to landmark 7. Separate Fine-Gray models per landmark performed worst for later landmarks, when the number of patients at risk was low. Discussion and Conclusion Categorical and time-to-event approaches had similar performance in the static and dynamic settings, except Cox models. Ignoring competing risks caused problems for risk prediction in the time-to-event framework (Cox), but not in the categorical framework (logistic regression).Comment: 3388 words; 3 figures; 4 table

Screening for data clustering in multicenter studies: the residual intraclass correlation

status: publishe

Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

Objective: Many prediction models for Coronavirus Disease 2019 (COVID-19) have been developed. External validation is mandatory before implementation in the Intensive Care Unit (ICU). We selected and validated prognostic models in the Euregio Intensive Care COVID (EICC) cohort. Study design and setting: In this multinational cohort study, routine data from COVID-19 patients admitted to ICUs within the Euregio Meuse-Rhine were collected from March to August 2020. COVID-19 models were selected based on model type, predictors, outcomes, and reporting. Furthermore, general ICU scores were assessed. Discrimination was assessed by area under the receiver operating characteristic curves (AUCs) and calibration by calibration-in-the-large and calibration plots. A random-effects meta-analysis was used to pool results. Results: 551 patients were admitted. Mean age was 65.4±11.2 years, 29% were female, and ICU mortality was 36%. Nine out of 238 published models were externally validated. Pooled AUCs were between 0.53 and 0.70 and calibration-in-the-large between -9% and 6%. Calibration plots showed generally poor but, for the 4C Mortality score and SEIMC score, moderate calibration. Conclusion: Of the nine prognostic models that were externally validated in the EICC cohort, only two showed reasonable discrimination and moderate calibration. For future pandemics, better models based on routine data are needed to support admission decision-making

The definition of predictor and outcome variables in mortality prediction models: a scoping review and quality of reporting study

Background and objectives: Mortality prediction models are promising tools for guiding clinical decision-making and resource allocation in intensive care units (ICUs). Clearly specified predictor and outcome variables are necessary to enable external validation and safe clinical application of prediction models. The objective of this study was to identify the predictor and outcome variables used in different mortality prediction models in the ICU and investigate their reporting.Methods: For this scoping review, MEDLINE, EMBASE, Web of Science, and the Cochrane Central Register of Controlled Trials were searched. Studies developed within a general ICU population reporting on prediction models with mortality as a primary or secondary outcome were eligible. The selection criteria were adopted from a review by Keuning et al. Predictor and outcome variables, variable characteristics (defined as units, definitions, moments of measurement, and methods of measurement), and publication details (defined as first author, year of publication and title) were extracted from the included studies. Predictor and outcome variable categories were demographics, chronic disease, care logistics, acute diagnosis, clinical examination and physiological derangement, laboratory assessment, additional diagnostics, support and therapy, risk scores, and (mortality) outcomes.Results: A total of 56 mortality prediction models, containing 204 unique predictor and outcome variables, were included. The predictor variables most frequently included in the models were age (40 times), admission type (27 times), and mechanical ventilation (21 times). We observed that single variables were measured with different units, according to different definitions, at a different moment, and with a different method of measurement in different studies. The reporting of the unit was mostly complete (98% overall, 95% in the laboratory assessment category), whereas the definition of the variable (74% overall, 63% in the chronic disease category) and method of measurement (70% overall, 34% in the demographics category) were most often lacking.Conclusion: Accurate and transparent reporting of predictor and outcome variables is paramount to enhance reproducibility, model performance in different contexts, and validity. Since unclarity about the required input data may introduce bias and thereby affect model performance, this study advocates that prognostic ICU models can be improved by transparent and clear reporting of predictor and outcome variables and their characteristics.Keywords: Clinical decision rules; Intensive care unit; Mortality; Mortality prediction models; Prognosis; Public reporting of health-care data

A decomposition of Fisher's information to inform sample size for developing fair and precise clinical prediction models -- part 1:binary outcomes

When developing a clinical prediction model, the sample size of the development dataset is a key consideration. Small sample sizes lead to greater concerns of overfitting, instability, poor performance and lack of fairness. Previous research has outlined minimum sample size calculations to minimise overfitting and precisely estimate the overall risk. However even when meeting these criteria, the uncertainty (instability) in individual-level risk estimates may be considerable. In this article we propose how to examine and calculate the sample size required for developing a model with acceptably precise individual-level risk estimates to inform decisions and improve fairness. We outline a five-step process to be used before data collection or when an existing dataset is available. It requires researchers to specify the overall risk in the target population, the (anticipated) distribution of key predictors in the model, and an assumed 'core model' either specified directly (i.e., a logistic regression equation is provided) or based on specified C-statistic and relative effects of (standardised) predictors. We produce closed-form solutions that decompose the variance of an individual's risk estimate into Fisher's unit information matrix, predictor values and total sample size; this allows researchers to quickly calculate and examine individual-level uncertainty interval widths and classification instability for specified sample sizes. Such information can be presented to key stakeholders (e.g., health professionals, patients, funders) using prediction and classification instability plots to help identify the (target) sample size required to improve trust, reliability and fairness in individual predictions. Our proposal is implemented in software module pmstabilityss. We provide real examples and emphasise the importance of clinical context including any risk thresholds for decision making

A decomposition of Fisher's information to inform sample size for developing fair and precise clinical prediction models -- part 1:binary outcomes

When developing a clinical prediction model, the sample size of the development dataset is a key consideration. Small sample sizes lead to greater concerns of overfitting, instability, poor performance and lack of fairness. Previous research has outlined minimum sample size calculations to minimise overfitting and precisely estimate the overall risk. However even when meeting these criteria, the uncertainty (instability) in individual-level risk estimates may be considerable. In this article we propose how to examine and calculate the sample size required for developing a model with acceptably precise individual-level risk estimates to inform decisions and improve fairness. We outline a five-step process to be used before data collection or when an existing dataset is available. It requires researchers to specify the overall risk in the target population, the (anticipated) distribution of key predictors in the model, and an assumed 'core model' either specified directly (i.e., a logistic regression equation is provided) or based on specified C-statistic and relative effects of (standardised) predictors. We produce closed-form solutions that decompose the variance of an individual's risk estimate into Fisher's unit information matrix, predictor values and total sample size; this allows researchers to quickly calculate and examine individual-level uncertainty interval widths and classification instability for specified sample sizes. Such information can be presented to key stakeholders (e.g., health professionals, patients, funders) using prediction and classification instability plots to help identify the (target) sample size required to improve trust, reliability and fairness in individual predictions. Our proposal is implemented in software module pmstabilityss. We provide real examples and emphasise the importance of clinical context including any risk thresholds for decision making