External validation of prediction models for pneumonia in primary care patients with lower respiratory tract infection: an individual patient data meta-analysis

Pneumonia remains difficult to diagnose in primary care. Prediction models based on signs and symptoms (S&S) serve to minimize the diagnostic uncertainty. External validation of these models is essential before implementation into routine practice. In this study all published S&S models for prediction of pneumonia in primary care were externally validated in the individual patient data (IPD) of previously performed diagnostic studies

External Validation of Prediction Models for Pneumonia in Primary Care Patients with Lower Respiratory Tract Infection: An Individual Patient Data Meta-Analysis.

BackgroundPneumonia remains difficult to diagnose in primary care. Prediction models based on signs and symptoms (S&S) serve to minimize the diagnostic uncertainty. External validation of these models is essential before implementation into routine practice. In this study all published S&S models for prediction of pneumonia in primary care were externally validated in the individual patient data (IPD) of previously performed diagnostic studies.Methods and findingsS&S models for diagnosing pneumonia in adults presenting to primary care with lower respiratory tract infection and IPD for validation were identified through a systematical search. Six prediction models and IPD of eight diagnostic studies (N total = 5308, prevalence pneumonia 12%) were included. Models were assessed on discrimination and calibration. Discrimination was measured using the pooled Area Under the Curve (AUC) and delta AUC, representing the performance of an individual model relative to the average dataset performance. Prediction models by van Vugt et al. and Heckerling et al. demonstrated the highest pooled AUC of 0.79 (95% CI 0.74-0.85) and 0.72 (0.68-0.76), respectively. Other models by Diehr et al., Singal et al., Melbye et al., and Hopstaken et al. demonstrated pooled AUCs of 0.65 (0.61-0.68), 0.64 (0.61-0.67), 0.56 (0.49-0.63) and 0.53 (0.5-0.56), respectively. A similar ranking was present based on the delta AUCs of the models. Calibration demonstrated close agreement of observed and predicted probabilities in the models by van Vugt et al. and Singal et al., other models lacked such correspondence. The absence of predictors in the IPD on dataset level hampered a systematical comparison of model performance and could be a limitation to the study.ConclusionsThe model by van Vugt et al. demonstrated the highest discriminative accuracy coupled with reasonable to good calibration across the IPD of different study populations. This model is therefore the main candidate for primary care use

The added value of C-reactive protein measurement in diagnosing pneumonia in primary care: a meta-analysis of individual patient data

BACKGROUND C-reactive protein (CRP) is increasingly being included in the diagnostic work-up for community-acquired pneumonia in primary care. Its added diagnostic value beyond signs and symptoms, however, remains unclear. We conducted a meta-analysis of individual patient data to quantify the added value of CRP measurement. METHODS We included studies of the diagnostic accuracy of CRP in adult outpatients with suspected lower respiratory tract infection. We contacted authors of eligible studies for inclusion of data and for additional data as needed. The value of adding CRP measurement to a basic signs-and-symptoms prediction model was assessed. Outcome measures were improvement in discrimination between patients with and without pneumonia in primary care and improvement in risk classification, both within the individual studies and across studies. RESULTS Authors of 8 eligible studies (n = 5308) provided their data sets. In all of the data sets, discrimination between patients with and without pneumonia improved after CRP measurement was added to the prediction model (extended model), with a mean improvement in the area under the curve of 0.075 (range 0.02-0.18). In a hypothetical cohort of 1000 patients, the proportion of patients without pneumonia correctly classified at low risk increased from 28% to 36% in the extended model, and the proportion with pneumonia correctly classified at high risk increased from 63% to 70%. The number of patients with pneumonia classified at low risk did not change (n = 4). Overall, the proportion of patients assigned to the intermediate-risk category decreased from 56% to 51%. INTERPRETATION Adding CRP measurement to the diagnostic work-up for suspected pneumonia in primary care improved the discrimination and risk classification of patients. However, it still left a substantial group of patients classified at intermediate risk, in which clinical decision-making remains challenging

Graphic representation of model performance relative to dataset average AUC, measured as delta AUC.

<p>Each point represents the performance of an individual model relative to the average performance of all models per dataset (deltaAUC, calculated as individual model AUC minus [–] the mean AUC of dataset). The figure shows how the discriminative performance per model, in the datasets in which it could be validated, is compared to the discriminative performance of the other models in that same dataset. For example, we see that the model by van Vugt et al. performs above average in all datasets in which it could be validated (i.e. Graffelman et al., Melbye et al, and Flanders et al). Furthermore, by studying the figure more closely, we can see the order of what model performed best in what dataset. For example, the models by van Vugt et al. and Heckerling et al. perform best in the dataset by Flanders et al., followed by the models by Singal et al., Diehr et al., Melbye et al. and Hopstaken et al.</p

Calibration plots of prediction models clustered per risk group with low (0–10%), intermediate (10–30%) and high (30–100%) predicted probabilities.

<p>Calibration results are presented for each validation dataset where the model could be validated. Plots show how well the predicted probabilities (x-axis) agree with observed probabilities (y-axis). For perfect agreement, the calibration curve falls on the ideal diagonal line (optimal calibration). Two vertical cut-off lines for 10% and 30% risk of pneumonia are depicted. (A) Calibration plot of the model by van Vugt et al. (B) Calibration plot of the model by Singal et al. (C) Calibration plot of the model by Hopstaken et al. (D) Calibration plot of the model by Heckerling et al. (E) Calibration plot of the model by Diehr et al. (F) Calibration plot of the model by Melbye et al.</p

Overview of included prediction models to diagnose pneumonia in a primary care setting and their incorporated predictors.

<p>Overview of included prediction models to diagnose pneumonia in a primary care setting and their incorporated predictors.</p

PRISMA flow diagram of the selection process of IPD used for external validation of prediction models [39].

<p>PRISMA flow diagram of the selection process of IPD used for external validation of prediction models [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0149895#pone.0149895.ref039" target="_blank">39</a>].</p

Discriminative performance of pneumonia prediction models per dataset, measured as Area Under the ROC Curve (AUC) and as pooled AUC in all suited individual patient data (IPD).

<p>Discriminative performance of pneumonia prediction models per dataset, measured as Area Under the ROC Curve (AUC) and as pooled AUC in all suited individual patient data (IPD).</p