Prediction of cardiovascular outcomes with machine learning techniques: application to the Cardiovascular Outcomes in Renal Atherosclerotic Lesions (CORAL) study.

Background: Data derived from the Cardiovascular Outcomes in Renal Atherosclerotic Lesions (CORAL) study were analyzed in an effort to employ machine learning methods to predict the composite endpoint described in the original study. Methods: We identified 573 CORAL subjects with complete baseline data and the presence or absence of a composite endpoint for the study. These data were subjected to several models including a generalized linear (logistic-linear) model, support vector machine, decision tree, feed-forward neural network, and random forest, in an effort to attempt to predict the composite endpoint. The subjects were arbitrarily divided into training and testing subsets according to an 80%:20% distribution with various seeds. Prediction models were optimized within the CARET package of R. Results: The best performance of the different machine learning techniques was that of the random forest method which yielded a receiver operator curve (ROC) area of 68.1%±4.2% (mean ± SD) on the testing subset with ten different seed values used to separate training and testing subsets. The four most important variables in the random forest method were SBP, serum creatinine, glycosylated hemoglobin, and DBP. Each of these variables was also important in at least some of the other methods. The treatment assignment group was not consistently an important determinant in any of the models. Conclusion: Prediction of a composite cardiovascular outcome was difficult in the CORAL population, even when employing machine learning methods. Assignment to either the stenting or best medical therapy group did not serve as an important predictor of composite outcome. Clinical Trial Registration: ClinicalTrials.gov, NCT00081731

Quantification of improvement in risk prediction models

Thesis (Ph.D.)--Boston University, 2012.PLEASE NOTE: Boston University Libraries did not receive an Authorization To Manage form for this thesis or dissertation. It is therefore not openly accessible, though it may be available by request. If you are the author or principal advisor of this work and would like to request open access for it, please contact us at [email protected]. Thank you.The identification of new factors in modeling the probability of a binary outcome is the main challenge for statisticians and clinicians who want to improve risk prediction. This motivates researchers to search for measures that quantify the performance of new markers. There are many commonly used measures that assess the performance of the binary outcome model: logistic R-squares, discrimination slope. area under the ROC (receiver operating characteristic) curve (AUC) and Hosmer-Lemeshow goodness of fit chi-square. However, metrics that work well for model assessment may not be as good for quantifying the usefulness of new risk factors, especially when we add a new predictor to a well performing baseline model. The recently proposed new measures of improvement, the Integrated Discrimination Improvement (IDI) - difference between discrimination slopes - and the Net Reclassification Improvement (NRI), directly address the question of model performance and take it beyond the simple statistical significance of a new risk factor. Since these two measures are new and have not been studied as extensively as the older ones, a question of their interpretation naturally arises. In our research we propose meaningful interpretations to the new measures as well as extensions of these measures that address some of their potential shortcomings. Following the derivation of the maximum R-squared by Nagelkerke, we show how the IDI, which depends on the event rate, could be rescaled by its hypothetical maximum value to reduce this dependence. Furthermore, the IDI metric assumes a uniform distribution for all risk cut-offs. Application of clinically important thresholds prompted us to derive a metric that includes a prior distribution for the cut-off points and assigns different weights to sensitivity and specificity. Similarly, we propose the maximum and rescaled NRI. The latter is based on counting the number of categories by which the risk of a given person moved and guarantees that reclassification tables with equal marginal probabilities will lead to a zero NRI. All developments are investigated employing numerical simulations under the assumption of normality and varying effect sizes of the associations. We also illustrate the proposed concepts using examples from the Framingham Heart Study.2031-01-0

Interpreting Incremental Value of Markers Added to Risk Prediction Models

The discrimination of a risk prediction model measures that model's ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model