A multiple imputation strategy for sequential multiple assignment randomized trials

Sequential multiple assignment randomized trials (SMARTs) are increasingly being used to inform clinical and intervention science. In a SMART, each patient is repeatedly randomized over time. Each randomization occurs at a critical decision point in the treatment course. These critical decision points often correspond to milestones in the disease process or other changes in a patient's health status. Thus, the timing and number of randomizations may vary across patients and depend on evolving patient‐specific information. This presents unique challenges when analyzing data from a SMART in the presence of missing data. This paper presents the first comprehensive discussion of missing data issues typical of SMART studies: we describe five specific challenges and propose a flexible imputation strategy to facilitate valid statistical estimation and inference using incomplete data from a SMART. To illustrate these contributions, we consider data from the Clinical Antipsychotic Trial of Intervention and Effectiveness, one of the most well‐known SMARTs to date. Copyright © 2014 John Wiley & Sons, Ltd.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/108622/1/sim6223-sup-0001-SupInfo.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/108622/2/sim6223.pd

Predicting suicide attempts and suicide deaths among adolescents following outpatient visits

BACKGROUND: Few studies report on machine learning models for suicide risk prediction in adolescents and their utility in identifying those in need of further evaluation. This study examined whether a model trained and validated using data from all age groups works as well for adolescents or whether it could be improved. METHODS: We used healthcare data for 1.4 million specialty mental health and primary care outpatient visits among 256,823 adolescents across 7 health systems. The prediction target was 90-day risk of suicide attempt following a visit. We used logistic regression with least absolute shrinkage and selection operator (LASSO) and generalized estimating equations (GEE) to predict risk. We compared performance of three models: an existing model, a recalibrated version of that model, and a newly-learned model. Models were compared using area under the receiver operating curve (AUC), sensitivity, specificity, positive predictive value and negative predictive value. RESULTS: The AUC produced by the existing model for specialty mental health visits estimated in adolescents alone (0.796; [0.789, 0.802]) was not significantly different than the AUC of the recalibrated existing model (0.794; [0.787, 0.80]) or the newly-learned model (0.795; [0.789, 0.801]). Predicted risk following primary care visits was also similar: existing (0.855; [0.844, 0.866]), recalibrated (0.85 [0.839, 0.862]), newly-learned (0.842, [0.829, 0.854]). LIMITATIONS: The models did not incorporate non-healthcare risk factors. The models relied on ICD9-CM codes for diagnoses and outcome measurement. CONCLUSIONS: Prediction models already in operational use by health systems can be reliably employed for identifying adolescents in need of further evaluation

Complex modeling with detailed temporal predictors does not improve health records-based suicide risk prediction

Suicide risk prediction models can identify individuals for targeted intervention. Discussions of transparency, explainability, and transportability in machine learning presume complex prediction models with many variables outperform simpler models. We compared random forest, artificial neural network, and ensemble models with 1500 temporally defined predictors to logistic regression models. Data from 25,800,888 mental health visits made by 3,081,420 individuals in 7 health systems were used to train and evaluate suicidal behavior prediction models. Model performance was compared across several measures. All models performed well (area under the receiver operating curve [AUC]: 0.794-0.858). Ensemble models performed best, but improvements over a regression model with 100 predictors were minimal (AUC improvements: 0.006-0.020). Results are consistent across performance metrics and subgroups defined by race, ethnicity, and sex. Our results suggest simpler parametric models, which are easier to implement as part of routine clinical practice, perform comparably to more complex machine learning methods

Proceedings of the 13th annual conference of INEBRIA

CITATION: Watson, R., et al. 2016. Proceedings of the 13th annual conference of INEBRIA. Addiction Science & Clinical Practice, 11:13, doi:10.1186/s13722-016-0062-9.The original publication is available at https://ascpjournal.biomedcentral.comENGLISH SUMMARY : Meeting abstracts.https://ascpjournal.biomedcentral.com/articles/10.1186/s13722-016-0062-9Publisher's versio

Empirical evaluation of internal validation methods for prediction in large-scale clinical data with rare-event outcomes: a case study in suicide risk prediction

Abstract Background There is increasing interest in clinical prediction models for rare outcomes such as suicide, psychiatric hospitalizations, and opioid overdose. Accurate model validation is needed to guide model selection and decisions about whether and how prediction models should be used. Split-sample estimation and validation of clinical prediction models, in which data are divided into training and testing sets, may reduce predictive accuracy and precision of validation. Using all data for estimation and validation increases sample size for both procedures, but validation must account for overfitting, or optimism. Our study compared split-sample and entire-sample methods for estimating and validating a suicide prediction model. Methods We compared performance of random forest models estimated in a sample of 9,610,318 mental health visits (“entire-sample”) and in a 50% subset (“split-sample”) as evaluated in a prospective validation sample of 3,754,137 visits. We assessed optimism of three internal validation approaches: for the split-sample prediction model, validation in the held-out testing set and, for the entire-sample model, cross-validation and bootstrap optimism correction. Results The split-sample and entire-sample prediction models showed similar prospective performance; the area under the curve, AUC, and 95% confidence interval was 0.81 (0.77–0.85) for both. Performance estimates evaluated in the testing set for the split-sample model (AUC = 0.85 [0.82–0.87]) and via cross-validation for the entire-sample model (AUC = 0.83 [0.81–0.85]) accurately reflected prospective performance. Validation of the entire-sample model with bootstrap optimism correction overestimated prospective performance (AUC = 0.88 [0.86–0.89]). Measures of classification accuracy, including sensitivity and positive predictive value at the 99th, 95th, 90th, and 75th percentiles of the risk score distribution, indicated similar conclusions: bootstrap optimism correction overestimated classification accuracy in the prospective validation set. Conclusions While previous literature demonstrated the validity of bootstrap optimism correction for parametric models in small samples, this approach did not accurately validate performance of a rare-event prediction model estimated with random forests in a large clinical dataset. Cross-validation of prediction models estimated with all available data provides accurate independent validation while maximizing sample size

Variable Selection for Individualized Treatment Rules with Discrete Outcomes

An individualized treatment rule (ITR) is a decision rule that aims to improve individual patients health outcomes by recommending optimal treatments according to patients specific information. In observational studies, collected data may contain many variables that are irrelevant for making treatment decisions. Including all available variables in the statistical model for the ITR could yield a loss of efficiency and an unnecessarily complicated treatment rule, which is difficult for physicians to interpret or implement. Thus, a data-driven approach to select important tailoring variables with the aim of improving the estimated decision rules is crucial. While there is a growing body of literature on selecting variables in ITRs with continuous outcomes, relatively few methods exist for discrete outcomes, which pose additional computational challenges even in the absence of variable selection. In this paper, we propose a variable selection method for ITRs with discrete outcomes. We show theoretically and empirically that our approach has the double robustness property, and that it compares favorably with other competing approaches. We illustrate the proposed method on data from a study of an adaptive web-based stress management tool to identify which variables are relevant for tailoring treatment