Towards better clinical prediction models

Clinical prediction models provide risk estimates for the presence of disease (diagnosis) or an event in the future course of disease (prognosis) for individual patients. Although publications that present and evaluate such models are becoming more frequent, the methodology is often suboptimal. We propose that seven steps should be considered in developing prediction models: (i) consideration of the research question and initial data inspection; (ii) coding of predictors; (iii) model specification; (iv) model estimation; (v) evaluation of model performance; (vi) internal validation; and (vii) model presentation. The validity of a prediction model is ideally assessed in fully independent data, where we propose four key measures to evaluate model performance: calibration-in-the-large, or the model intercept (A); calibration slope (B); discrimination, with a concordance statistic (C); and clinical usefulness, with decision-curve analysis (D). As an application, we develop and validate prediction models for 30-day mortality in patients with an acute myocardial infarction. This illustrates the usefulness of the proposed framework to strengthen the methodological rigour and quality for prediction models in cardiovascular research

The Patient Health Questionnaire-9 for detection of major depressive disorder in primary care: consequences of current thresholds in a crosssectional study

Background: There is a need for brief instruments to ascertain the diagnosis of major depressive disorder. In this study, we present the reliability, construct validity and accuracy of the PHQ-9 and PHQ-2 to detect major depressive disorder in primary care.Methods: Cross-sectional analyses within a large prospective cohort study (PREDICT-NL). Data was collected in seven large general practices in the centre of the Netherlands. 1338 subjects were recruited in the general practice waiting room, irrespective of their presenting complaint. The diagnostic accuracy (the area under the ROC curve and sensitivities and specificities for various thresholds) was calculated against a diagnosis of major depressive disorder determined with the Composite International Diagnostic Interview (CIDI).Results: The PHQ-9 showed a high degree of internal consistency (ICC = 0.88) and test-retest reliability (correlation = 0.94). With respect to construct validity, it showed a clear association with functional status measurements, sick days and number of consultations. The discriminative ability was good for the PHQ-9 (area under the ROC curve = 0.87, 95% CI: 0.84-0.90) and the PHQ-2 (ROC area = 0.83, 95% CI 0.80-0.87). Sensitivities at the recommended thresholds were 0.49 for the PHQ-9 at a score of 10 and 0.28 for a categorical algorithm. Adjustment of the threshold and the algorithm improved sensitivities to 0.82 and 0.84 respectively but the specificity decreased from 0.95 to 0.82 (threshold) and from 0.98 to 0.81 (algorithm). Similar results were found for the PHQ-2: the recommended threshold of 3 had a sensitivity of 0.42 and lowering the threshold resulted in an improved sensitivity of 0.81.Conclusion: The PHQ-9 and the PHQ-2 are useful instruments to detect major depressive disorder in primary care, provided a high score is followed by an additional diagnostic work-up. However, often recommended thresholds for the PHQ-9 and the PHQ-2 resulted in many undetected major depressive disorders

Validation of Clinical Prediction Models: Theory and Applications in Testicular Germ Cell Cancer

linical prediction models combine patient characteristics to predict the probability of having a certain disease (diagnosis) or the probability that a particular disease state will occur (prognosis). The predicted probability of the diagnostic or prognostic outcome may assist the clinician in decision making for patient care. Before a prediction model can reliably be applied in clinical practice, the performance of the model in new patients needs to be studied (‘external validity’). This thesis described several theoretical and practical aspects of the external validation of clinical prediction models. The objectives were (i) to describe aspects of model validity and relevant performance measures; (ii) to estimate the power of these performance measures; (iii) to externally validate a prediction model for residual mass histology in testicular cancer; and (iv) to update this model with all available information. Three aspects of model performance are discussed: calibration, discrimination, and clinical usefulness. The external validity of a model does not only depend on the new patients for whom the model is applied, but also on the development process of the model. Therefore, this thesis contained also some illustrations of model development aspects. The development process of the prediction model for residual mass histology in nonseminomatous testicular germ cell cancer was described; the model predicts the probability that a residual retroperitoneal mass contains only benign tissue after cis-platin based chemotherapy for metastatic tumour

Improving prediction models with new markers: A comparison of updating strategies

Background: New markers hold the promise of improving risk prediction for individual patients. We aimed to compare the performance of different strategies to extend a previously developed prediction model with a new marker. Methods: Our motivating example was the extension of a risk calculator for prostate cancer with a new marker that was available in a relatively small dataset. Performance of the strategies was also investigated in simulations. Development, marker and test sets with different sample sizes originating from the same underlying population were generated. A prediction model was fitted using logistic regression in the development set, extended using the marker set and validated in the test set. Extension strategies considered were re-estimating individual regression coefficients, updating of predictions using conditional likelihood ratios (LR) and imputation of marker values in the development set and subsequently fitting a model in the combined development and marker sets. Sample sizes considered for the development and marker set were 500 and 100, 500 and 500, and 100 and 500 patients. Discriminative ability of the extended models was quantified using the concordance statistic (c-statistic) and calibration was quantified using the calibration slope. Results: All strategies led to extended models with increased discrimination (c-statistic increase from 0.75 to 0.80 in test sets). Strategies estimating a large number of parameters (re-estimation of all coefficients and updating using conditional LR) led to overfitting (calibration slope below 1). Parsimonious methods, limiting the number of coefficients to be re-estimated, or applying shrinkage after model revision, limited the amount of overfitting. Combining the development and marker set using imputation of missing marker values approach led to consistently good performing models in all scenarios. Similar results were observed in the motivating example. Conclusion: When the sample with the new marker information is small, parsimonious methods are required to prevent overfitting of a new prediction model. Combining all data with imputation of missing marker values is an attractive option, even if a relatively large marker data set is available

Palliative care consultation services in hospitals in the Netherlands: The design of the COMPASS study

Background: Patients with an advanced incurable disease are often hospitalised for some time during the last phase of life. Care in hospitals is generally focussed at curing disease and prolonging life and may therefore not in all cases adequately address the needs of such patients. We present the COMPASS study, a study on the effects and costs of consultation teams for palliative care in hospitals. This observational study aims to investigate the use, effects and costs of PCT consultation services for hospitalized patients with incurable cancer in the Netherlands. Methods/design: The study consists of 3 parts: 1. A questionnaire, interviews and a focus group discussion to investigate the characteristics of PCT consultation in 12 hospitals. PCTs will register their activities to calculate the costs of PCT consultation. 2. Cancer patients for whom the attending physician would not be surprised that they would die within 12 month will be included in a medical file search in three hospitals. Medical records will be investigated to compare care, treatment and hospital costs between patients with and patients without PCT consultation. 3. In the other nine hospitals, we will perform a longitudinal study, and compare quality of life between 100 patients for whom a PCT was consulted with 200 patients without PCT consultation. Propensity score matching will be used to adjust for differences between both patient groups. Patients will be followed for three months after inclusion. Quality of life will be assessed with the Palliative Outcome Scale, the EuroQol-5d and the EORTC-QLQ-C15 PAL. Satisfaction with care in the hospital is measured with the IN-PATSAT32. The cost impact of PCT consultation will also be explored. Discussion: This is the first multicenter study on PCT consultation in the Netherlands. The study will give valuable insight in the process, effects and costs of PCT consultation in hospitals. It is anticipated that PCT consultation has a positive effect on patients' quality of life and satisfaction with care and will lead to less hospital care costs

No negative impact of Palliative sedation on relatives' experience of the dying phase and their wellbeing after the patient's death: An observational study

Background: Palliative sedation is the widely-used intervention of administering sedating agents to induce a state of unconsciousness to take away a dying patient's perception of otherwise irrelievable symptoms. However, it remains questionable whether this ethically complex intervention is beneficial for patients and whether the associated lack of communication in the last phase of life has a negative impact on relatives' wellbeing. Methods: An observational questionnaire study was conducted among relatives of a consecutive sample of patients who died a non-sudden death in the Erasmus MC Cancer Institute or in the hospice 'Laurens Cadenza' (both in Rotterdam) between 2010 and 2013. Results: Relatives filled in questionnaires regarding 151 patients who had been sedated and 90 patients who had not been sedated. The median time since all patients had passed away was 21 (IQR 14-32) months. No significant differences were found in relatives' assessments of the quality of end-of-life care, patients' quality of life in the last week before death and their quality of dying, between patients who did and did not receive sedation, or in relatives' satisfaction with their own life, their general health and their mental wellbeing after the patient's death. Conclusions: The use of sedation in these patients appears to have no negative effect on bereaved relatives' evaluation of the patient's dying phase, or on their own wellbeing after the patient's death

Incorporating published univariable associations in diagnostic and prognostic modeling

Background: Diagnostic and prognostic literature is overwhelmed with studies reporting univariable predictor-outcome associations. Currently, methods to incorporate such information in the construction of a prediction model are underdeveloped and unfamiliar to many researchers. Methods. This article aims to improve upon an adaptation method originally proposed by Greenland (1987) and Steyerberg (2000) to incorporate previously published univariable associations in the construction of a novel prediction model. The proposed method improves upon the variance estimation component by reconfiguring the adaptation process in established theory and making it more robust. Different variants of the proposed method were tested in a simulation study, where performance was measured by comparing estimated associations with their predefined values according to the Mean Squared Error and coverage of the 90% confidence intervals. Results: Results demonstrate that performance of estimated multivariable associations considerably improves for small datasets where external evidence is included. Although the error of estimated associations decreases with increasing amount of individual participant data, it does not disappear completely, even in very large datasets. Conclusions: The proposed method to aggregate previously published univariable associations with individual participant data in the construction of a novel prediction models outperforms established approaches and is especially worthwhile when relatively limited individual participant data are available

External validation of a referral rule for axial spondyloarthritis in primary care patients with chronic low back pain

Objectives To validate and optimize a referral rule to identify primary care patients with chronic low back pain (CLBP) suspected for axial spondyloarthritis (axSpA). Design Cross-sectional study with data from 19 Dutch primary care practices for development and 38 for validation. Participants Primary care patients aged 18-45 years with CLBP existing more than three months and onset of back pain started before the age of 45 years. Main Outcome The number of axSpA patients according to the ASAS criteria. Methods The referral rule (CaFaSpA referral rule) was developed using 364 CL