Linked shrinkage to improve estimation of interaction effects in regression models

We address a classical problem in statistics: adding two-way interaction terms to a regression model. As the covariate dimension increases quadratically, we develop an estimator that adapts well to this increase, while providing accurate estimates and appropriate inference. Existing strategies overcome the dimensionality problem by only allowing interactions between relevant main effects. Building on this philosophy, we implement a softer link between the two types of effects using a local shrinkage model. We empirically show that borrowing strength between the amount of shrinkage for main effects and their interactions can strongly improve estimation of the regression coefficients. Moreover, we evaluate the potential of the model for inference, which is notoriously hard for selection strategies. Large-scale cohort data are used to provide realistic illustrations and evaluations. Comparisons with other methods are provided. The evaluation of variable importance is not trivial in regression models with many interaction terms. Therefore, we derive a new analytical formula for the Shapley value, which enables rapid assessment of individual-specific variable importance scores and their uncertainties. Finally, while not targeting for prediction, we do show that our models can be very competitive to a more advanced machine learner, like random forest, even for fairly large sample sizes. The implementation of our method in RStan is fairly straightforward, allowing for adjustments to specific needs.Comment: 28 pages, 18 figure

Measuring the performance of prediction models to personalize treatment choice.

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect

Using a Smartphone App and Coaching Group Sessions to Promote Residents' Reflection in the Workplace

Item does not contain fulltextPROBLEM: Reflecting on workplace-based experiences is necessary for professional development. However, residents need support to raise their awareness of valuable moments for learning and to thoughtfully analyze those learning moments afterwards. APPROACH: From October to December 2012, the authors held a multidisciplinary six-week postgraduate training module focused on general competencies. Residents were randomly assigned to one of four conditions with varying degrees of reflection support; they were offered (1) a smartphone app, (2) coaching group sessions, (3) a combination of both, or (4) neither type of support. The app allowed participants to capture in real time learning moments as a text note, audio recording, picture, or video. Coaching sessions held every two weeks aimed to deepen participants' reflection on captured learning moments. Questionnaire responses and reflection data were compared between conditions to assess the effects of the app and coaching sessions on intensity and frequency of reflection. OUTCOMES: Sixty-four residents participated. App users reflected more often, captured more learning moments, and reported greater learning progress than nonapp users. Participants who attended coaching sessions were more alert to learning moments and pursued more follow-up learning activities to improve on the general competencies. Those who received both types of support were most alert to these learning moments. NEXT STEPS: A simple mobile app for capturing learning moments shows promise as a tool to support workplace-based learning, especially when combined with coaching sessions. Future research should evaluate these tools on a broader scale and in conjunction with residents' and students' personal digital portfolios

Real-time imputation of missing predictor values in clinical practice

Use of prediction models is widely recommended by clinical guidelines, but usually requires complete information on all predictors that is not always available in daily practice. We describe two methods for real-time handling of missing predictor values when using prediction models in practice. We compare the widely used method of mean imputation (M-imp) to a method that personalizes the imputations by taking advantage of the observed patient characteristics. These characteristics may include both prediction model variables and other characteristics (auxiliary variables). The method was implemented using imputation from a joint multivariate normal model of the patient characteristics (joint modeling imputation; JMI). Data from two different cardiovascular cohorts with cardiovascular predictors and outcome were used to evaluate the real-time imputation methods. We quantified the prediction model's overall performance (mean squared error (MSE) of linear predictor), discrimination (c-index), calibration (intercept and slope) and net benefit (decision curve analysis). When compared with mean imputation, JMI substantially improved the MSE (0.10 vs. 0.13), c-index (0.70 vs 0.68) and calibration (calibration-in-the-large: 0.04 vs. 0.06; calibration slope: 1.01 vs. 0.92), especially when incorporating auxiliary variables. When the imputation method was based on an external cohort, calibration deteriorated, but discrimination remained similar. We recommend JMI with auxiliary variables for real-time imputation of missing values, and to update imputation models when implementing them in new settings or (sub)populations.Comment: 17 pages, 6 figures, to be published in European Heart Journal - Digital Health, accepted for MEMTAB 2020 conferenc

Identifying adults with acute rhinosinusitis in primary care that benefit most from antibiotics : protocol of an individual patient data meta-analysis using multivariable risk prediction modelling

Introduction Acute rhinosinusitis (ARS) is a prime reason for doctor visits and among the conditions with highest antibiotic overprescribing rates in adults. To reduce inappropriate prescribing, we aim to predict the absolute benefit of antibiotic treatment for individual adult patients with ARS by applying multivariable risk prediction methods to individual patient data (IPD) of multiple randomised placebo-controlled trials. Methods and analysis This is an update and re-analysis of a 2008 IPD meta-analysis on antibiotics for adults with clinically diagnosed ARS. First, the reference list of the 2018 Cochrane review on antibiotics for ARS will be reviewed for relevant studies published since 2008. Next, the systematic searches of CENTRAL, MEDLINE and Embase of the Cochrane review will be updated to 1 September 2020. Methodological quality of eligible studies will be assessed using the Cochrane Risk of Bias 2 tool. The primary outcome is cure at 8-15 days. Regression-based methods will be used to model the risk of being cured based on relevant predictors and treatment, while accounting for clustering. Such model allows for risk predictions as a function of treatment and individual patient characteristics and hence gives insight into individualised absolute benefit. Candidate predictors will be based on literature, clinical reasoning and availability. Calibration and discrimination will be evaluated to assess model performance. Resampling techniques will be used to assess internal validation. In addition, internal-external cross-validation procedures will be used to inform on between-study differences and estimate out-of-sample model performance. Secondarily, we will study possible heterogeneity of treatment effect as a function of outcome risk. Ethics and dissemination In this study, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply and official ethical approval is not required. Results will be submitted for publication in international peer-reviewed journals. PROSPERO registration number CRD42020220108.Peer reviewe

The relation between prediction model performance measures and patient selection outcomes for proton therapy in head and neck cancer

Background: Normal-tissue complication probability (NTCP) models predict complication risk in patients receiving radiotherapy, considering radiation dose to healthy tissues, and are used to select patients for proton therapy, based on their expected reduction in risk after proton therapy versus photon radiotherapy (ΔNTCP). Recommended model evaluation measures include area under the receiver operating characteristic curve (AUC), overall calibration (CITL), and calibration slope (CS), whose precise relation to patient selection is still unclear. We investigated how each measure relates to patient selection outcomes. Methods: The model validation and consequent patient selection process was simulated within empirical head and neck cancer patient data. By manipulating performance measures independently via model perturbations, the relation between model performance and patient selection was studied. Results: Small reductions in AUC (-0.02) yielded mean changes in ΔNTCP between 0.9–3.2 %, and single-model patient selection differences between 2–19 %. Deviations (-0.2 or +0.2) in CITL or CS yielded mean changes in ΔNTCP between 0.3–1.4 %, and single-model patient selection differences between 1–10 %. Conclusions: Each measure independently impacts ΔNTCP and patient selection and should thus be assessed in a representative sufficiently large external sample. Our suggested practical model selection approach is considering the model with the highest AUC, and recalibrating it if needed

Risk-based decision making: estimands for sequential prediction under interventions

Prediction models are used amongst others to inform medical decisions on interventions. Typically, individuals with high risks of adverse outcomes are advised to undergo an intervention while those at low risk are advised to refrain from it. Standard prediction models do not always provide risks that are relevant to inform such decisions: e.g., an individual may be estimated to be at low risk because similar individuals in the past received an intervention which lowered their risk. Therefore, prediction models supporting decisions should target risks belonging to defined intervention strategies. Previous works on prediction under interventions assumed that the prediction model was used only at one time point to make an intervention decision. In clinical practice, intervention decisions are rarely made only once: they might be repeated, deferred and re-evaluated. This requires estimated risks under interventions that can be reconsidered at several potential decision moments. In the current work, we highlight key considerations for formulating estimands in sequential prediction under interventions that can inform such intervention decisions. We illustrate these considerations by giving examples of estimands for a case study about choosing between vaginal delivery and cesarean section for women giving birth. Our formalization of prediction tasks in a sequential, causal, and estimand context provides guidance for future studies to ensure that the right question is answered and appropriate causal estimation approaches are chosen to develop sequential prediction models that can inform intervention decisions.Comment: 32 pages, 2 figure

Prognostic and treatment effect modeling in medical research

Prediction models are increasingly used in the medical domain, benefitting from the increased availability of digital healthcare data and accessible software implementations. Two important applications of such models are (i) predicting the likely course of a medical condition (also known as prognosis) and (ii) predicting of the consequences of a particular treatment decision. In both cases, the goal is often to go beyond the prediction of population averages towards more personalized predictions. This dissertation provides a biostatistical perspective on such prediction models for prognosis and treatment effects. Specific methodological contributions include methods to handle missing data points during the validation of prediction models and their application in practice. New developments are also described regarding methods that help strike the right balance between model complexity and the size of the available data. Furthermore, the differences between associative and causal prediction modeling are explored in depth, and guidance is provided for the development of prediction models for predicting personalized treatment effects based on randomized trial data. models for treatment effect heterogeneity based on randomized trial data is provided. Subsequently, the evaluation of the quality of such models is described, including a novel approach to measure prediction performance. Finally, an applied chapter describes the modeling of both prognosis and treatment effect in a context where individual patient level data are available from multiple studies