Measures of discrimination, reclassification, and calibration for risk prediction models: an exploration in their interrelationships and practical utility and improvement in their estimation

Public health practice and quality of medical care rely heavily on the accuracy, precision, and robustness of risk prediction models. Health care providers use risk prediction models to assess a patient’s risk of developing an event during a specified time frame given the patient’s specific characteristics, and subsequently recommend a course of treatment or preventative action. In public health research, risk prediction models are often constructed with common statistical modeling techniques, such as logistic regression for binary outcomes or Cox proportional hazard regression for time-to-event outcomes, and the performance of the model is assessed through internal or external validation, or some combination. Model validation requires statistical and clinical significance and satisfactory baseline or improvement in model calibration and discrimination: calibration quantifies how close predictions are to observed outcomes while discrimination quantifies the model’s ability to distinguish correctly between events and nonevents. Measures for evaluating these qualities include (but are not limited to) Brier score, calibration-in-the-large, proportion of variation (R2), sensitivity and specificity, area under the receiver operating characteristic curve (AUC), discrimination slope, net reclassification index (NRI), integrated discrimination improvement (IDI), and decision theory analytic measures such as net benefit and relative utility. Among these measures exist several interrelationships under certain assumptions, and their estimation and interpretation is an active area of research. The first two parts of this thesis focus on studying the empirical distributions and improving confidence interval (CI) estimation of ∆AUC, NRI, and IDI for both binary event data and time-to-event data. Through data simulation and the comparison of several CI types derived with bootstrapping techniques, we make recommendations for proper estimation in future work and apply our recommendations to real-life Framingham Heart Study data. The third part of this thesis summarizes the many interrelationships and possible redundancies among the measures listed, extends theoretical formulas assuming normal variables for ∆AUC, NRI, and IDI from nested models to non-nested models and to Brier score, and explores the impact of varying discrimination and calibration assumptions on Yates’ and Sanders’ decomposed versions of Brier score through simulation. Lastly, overall conclusions and future directions are presented at the end

Measures of discrimination, reclassification, and calibration for risk prediction models: an exploration in their interrelationships and practical utility and improvement in their estimation

Public health practice and quality of medical care rely heavily on the accuracy, precision, and robustness of risk prediction models. Health care providers use risk prediction models to assess a patient’s risk of developing an event during a specified time frame given the patient’s specific characteristics, and subsequently recommend a course of treatment or preventative action. In public health research, risk prediction models are often constructed with common statistical modeling techniques, such as logistic regression for binary outcomes or Cox proportional hazard regression for time-to-event outcomes, and the performance of the model is assessed through internal or external validation, or some combination. Model validation requires statistical and clinical significance and satisfactory baseline or improvement in model calibration and discrimination: calibration quantifies how close predictions are to observed outcomes while discrimination quantifies the model’s ability to distinguish correctly between events and nonevents. Measures for evaluating these qualities include (but are not limited to) Brier score, calibration-in-the-large, proportion of variation (R2), sensitivity and specificity, area under the receiver operating characteristic curve (AUC), discrimination slope, net reclassification index (NRI), integrated discrimination improvement (IDI), and decision theory analytic measures such as net benefit and relative utility. Among these measures exist several interrelationships under certain assumptions, and their estimation and interpretation is an active area of research. The first two parts of this thesis focus on studying the empirical distributions and improving confidence interval (CI) estimation of ∆AUC, NRI, and IDI for both binary event data and time-to-event data. Through data simulation and the comparison of several CI types derived with bootstrapping techniques, we make recommendations for proper estimation in future work and apply our recommendations to real-life Framingham Heart Study data. The third part of this thesis summarizes the many interrelationships and possible redundancies among the measures listed, extends theoretical formulas assuming normal variables for ∆AUC, NRI, and IDI from nested models to non-nested models and to Brier score, and explores the impact of varying discrimination and calibration assumptions on Yates’ and Sanders’ decomposed versions of Brier score through simulation. Lastly, overall conclusions and future directions are presented at the end

Association of Parental Obesity and Diabetes Mellitus With Circulating Adipokines in Nonobese Nondiabetic Offspring

BACKGROUND: Adipokines are implicated in the development of obesity-related traits. We hypothesized that nonobese participants without diabetes mellitus (DM) whose parents were obese or had DM would have altered circulating adipokines compared with those without parental history of these conditions. METHODS AND RESULTS: Participants in the community-based Framingham Third Generation cohort who were not obese (body mass index \u3c 30) and not diabetic with both parents in the Framingham Offspring cohort were included in this analysis (n=2034, mean age 40 years, 54% women). Circulating concentrations of fetuin A, RBP4 (retinol binding protein 4), FABP4 (fatty acid binding protein 4), leptin, LEP-R (leptin receptor), and adiponectin were assayed. Parental DM was defined as occurring before age 60 years, and obesity was defined as body mass index \u3e /=30 before age 60 years. General estimating equations were used to compare concentrations of adipokines among participants with 0, 1, or 2 parents affected by obesity or DM (separate analyses for each), adjusting for known correlates of adipokines. Overall, 44% had at least 1 parent who was obese and 15% had parents with DM. Parental obesity was associated with higher serum levels of FABP4 and LEP-R in their offspring (P=0.02 for both). Parental DM was associated with lower adiponectin but higher RBP4 concentrations in offspring (P \u3c /=0.02 for both). CONCLUSIONS: In our community-based sample, a parental history of DM or obesity was associated with an altered adipokine profile in nonobese nondiabetic offspring. Additional studies are warranted to evaluate whether such preclinical biomarker alterations presage future risk of disease

Association of Exhaled Carbon Monoxide With Stroke Incidence and Subclinical Vascular Brain Injury

Background and purposeExhaled carbon monoxide (CO) is associated with cardiometabolic traits, subclinical atherosclerosis, and cardiovascular disease, but its specific relations with stroke are unexplored. We related exhaled CO to magnetic resonance imaging measures of subclinical cerebrovascular disease cross-sectionally and to incident stroke/transient ischemic attack prospectively in the Framingham Offspring study.MethodsWe measured exhaled CO in 3313 participants (age 59±10 years; 53% women), and brain magnetic resonance imaging was available in 1982 individuals (age 58±10 years; 54% women). Participants were analyzed according to tertiles of exhaled CO concentration.ResultsIn age- and sex-adjusted models, the highest tertile of exhaled CO was associated with lower total cerebral brain volumes, higher white-matter hyperintensity volumes, and greater prevalence of silent cerebral infarcts (P<0.05 for all). The results for total cerebral brain volume and white-matter hyperintensity volume were consistent after removing smokers from the sample, and the association with white-matter hyperintensity volume persisted after multivariable adjustment (P=0.04). In prospective analyses (mean follow-up 12.9 years), higher exhaled CO was associated with 67% (second tertile) and 97% (top tertile) increased incidence of stroke/transient ischemic attack relative to the first tertile that served as referent (P<0.01 for both). These results were consistent in nonsmokers and were partially attenuated upon adjustment for vascular risk factors.ConclusionsIn this large, community-based sample of individuals without clinical stroke/transient ischemic attack at baseline, higher exhaled CO was associated with a greater burden of subclinical cerebrovascular disease cross-sectionally and with increased risk of stroke/transient ischemic attack prospectively. Further investigation is necessary to explore the biological mechanisms linking elevated CO with stroke

Adjuvant Gemcitabine Plus Docetaxel Followed by Doxorubicin Versus Observation for High-Grade Uterine Leiomyosarcoma: A Phase III NRG Oncology/Gynecologic Oncology Group Study

Item does not contain fulltex

Association of Ideal Cardiovascular Health With Vascular Brain Injury and Incident Dementia

Background and purposeThe American Heart Association developed the ideal cardiovascular health (CVH) index as a simple tool to promote CVH; yet, its association with brain atrophy and dementia remains unexamined.MethodsOur aim was to investigate the prospective association of ideal CVH with vascular brain injury, including the 10-year risks of incident stroke and dementia, as well as cognitive decline and brain atrophy on magnetic resonance imaging, measured for ≈7 years. We studied 2750 stroke- and dementia-free Framingham Heart Study Offspring cohort participants (mean age, 62±9 years; 45% men). Ideal CVH was quantified on a 7-point scale with 1 point awarded for each of the following: nonsmoking status, ideal body mass index, regular physical activity, healthy diet, as well as optimum blood pressure, cholesterol, and fasting blood glucose. Both recent (baseline) and remote (6.9 years earlier) ideal CVH scores were examined.ResultsRecent ideal CVH was associated with stroke (hazard ratio, 0.80; 95% confidence interval, 0.67-0.95), vascular dementia (hazard ratio, 0.49; 95% confidence interval, 0.30-0.81), frontal brain atrophy (P=0.003), and cognitive decline on tasks measuring visual memory and reasoning (P<0.05). In addition to predicting stroke, vascular dementia, whole-brain atrophy, and cognitive decline, remote ideal CVH was associated with the incidence of all-cause dementia (hazard ratio, 0.80; 95% confidence interval, 0.67-0.97) and Alzheimer disease (hazard ratio, 0.79; 95% confidence interval, 0.64-0.98).ConclusionsAdherence to the American Heart Association's ideal CVH factors and behaviors, particularly in midlife, may protect against cerebrovascular disease and dementia

Cardiometabolic correlates and heritability of fetuin-A, retinol-binding protein 4, and fatty-acid binding protein 4 in the Framingham Heart Study

CONTEXT: Fetuin-A, retinol-binding protein 4 (RBP4), and fatty-acid binding protein 4 (FABP4) are novel biomarkers that may link adiposity to insulin resistance and the metabolic syndrome (MetSyn). OBJECTIVE: The aim of this study was to investigate the correlates of these three adiposity biomarkers in a large community-based sample. DESIGN, SETTING, PARTICIPANTS, AND OUTCOMES: Serum concentrations of fetuin-A, RBP4, and FABP4 were assayed in 3658 participants of the Third Generation Framingham Heart Study cohort (mean age 40 yr, 54% women). We used multivariable regression to cross-sectionally relate biomarkers to insulin resistance, cardiovascular risk factors, and the MetSyn. The genetic contribution to inter-individual variation in biomarker levels was assessed using variance-components analysis. RESULTS: All three biomarkers exhibited sexual dimorphisms (levels higher in women for fetuin-A and FABP4 but greater in men for RBP4) and were associated positively with insulin resistance assessed using the homeostasis model, with high-sensitivity C-reactive protein, and with prevalent MetSyn (P\u3c0.01 for all). The biomarkers showed distinct patterns of association with metabolic risk factors. RBP4 levels were correlated with body mass index only in unadjusted but not in adjusted models. None of the biomarkers were associated with prevalent diabetes in multivariable models. Circulating fetuin-A, RBP4, and FABP4 levels showed modest heritability, ranging from 15-44% (all P\u3c0.0001). CONCLUSIONS: In our large young- to middle-aged community-based sample, we observed that circulating levels of fetuin-A, RBP4, and FABP4 are associated with insulin resistance and with distinct components of MetSyn consistent with the multifactorial pathogenesis of metabolic dysregulation

Adjuvant Gemcitabine Plus Docetaxel Followed by Doxorubicin Versus Observation for High-Grade Uterine Leiomyosarcoma: A Phase III NRG Oncology/Gynecologic Oncology Group Study.

PURPOSE: We conducted a randomized phase III trial to determine whether adjuvant chemotherapy improves survival in women with uterine leiomyosarcoma. METHODS: Women with uterus-confined, high-grade leiomyosarcoma who were confirmed disease free by imaging were randomly assigned to four cycles of gemcitabine plus docetaxel, followed by four cycles of doxorubicin, or to observation. All were followed for evidence of recurrence. The primary end point was overall survival (OS). RESULTS: With international collaboration, 38 of the targeted accrual of 216 patients were enrolled, after which the study was closed by the National Cancer Institute for accrual futility. Twenty patients were assigned to chemotherapy, 18 to observation. Among the 17 patients treated with at least one cycle of chemotherapy, grade 3 or 4 toxicities were observed in 47%; among the 18 patients assigned to observation, one had grade 3 hypertension. There were six deaths (chemotherapy, n = 5; observation, n = 1), all due to disease. The restricted mean survival time for OS was estimated as 34.3 months (95% CI, 25.3 to 43.3 months) in the chemotherapy arm and as 46.4 months (95% CI, 43.6 to 49.1 months) in the observation arm. There were eight recurrences in each arm. The restricted mean survival time for recurrence-free survival was estimated as 18.1 (95% CI, 14.2 to 22.0) months in the chemotherapy arm and as 14.6 months (95% CI, 10.3 to 19.0 months) in the observation arm. Neither survival outcome comparison was considered statistically robust, due to the small sample size. CONCLUSION: Despite international collaboration to test the role of adjuvant chemotherapy in uterine-confined leiomyosarcoma, this study was closed for accrual futility. Although the sample size precludes robust statistical comparison, observed OS and recurrence-free survival data do not show superior outcomes with adjuvant chemotherapy