Net reclassification improvement: Computation, interpretation, and controversies: A literature review and clinician's guide

The net reclassification improvement (NRI) is an increasingly popular measure for evaluating improvements in risk predictions. This article details a review of 67 publications in high-impact general clinical journals that considered the NRI. Incomplete reporting of NRI methods, incorrect calculation, and common misinterpretations were found. To aid improved applications of the NRI, the article elaborates on several aspects of the computation and interpretation in various settings. Limitations and controversies are discussed, including the effect of miscalibration of prediction models, the use of the continuous NRI and clinical NRI, and the relation with decision analytic measures. A systematic approach toward presenting NRI analysis is proposed: Detail and motivate the methods used for computation of the NRI, use clinically meaningful risk cutoffs for the category-based NRI, report both NRI components, address issues of calibration, and do not interpret the overall NRI as a percentage of the study population reclassified. Promising NRI findings need to be followed with decision analytic or formal costeffectiveness evaluations

Graphical assessment of incremental value of novel markers in prediction models

New markers may improve prediction of diagnostic and prognostic outcomes. We aimed to review options for graphical display and summary measures to assess the predictive value of markers over standard, readily available predictors. We illustrated various approaches using previously published data on 3264 participants from the Framingham Heart Study, where 183 developed coronary heart disease (10-year risk 5.6%). We considered performance measures for the incremental value of adding HDL cholesterol to a prediction model. An initial assessment may consider statistical significance (HR = 0.65, 95% confidence interval 0.53 to 0.80; likelihood ratio p < 0.001), and distributions of predicted risks (densities or box plots) with various summary measures. A range of decision thresholds is considered in predictiveness and receiver operating characteristic curves, where the area under the curve (AUC) increased from 0.762 to 0.774 by adding HDL. We can furthermore focus on reclassification of participants with and without an event in a reclassification graph, with the continuous net reclassification improvement (NRI) as a summary measure. When we focus on one particular decision threshold, the changes in sensitivity and specificity are central. We propose a net reclassification risk graph, which allows us to focus on the number of reclassified persons and their event rates. Summary measures include the binary AUC, the two-category NRI, and decision analytic variants such as the net benefit (NB). Various graphs and summary measures can be used to assess the incremental predictive value of a marker. Important insights for impact on decision making are provided by a simple graph for the net reclassification risk

The Added Value of Percentage of Free to Total Prostate-specific Antigen, PCA3, and a Kallikrein Panel to the ERSPC Risk Calculator for Prostate Cancer in Prescreened Men

Background: Prostate-specific antigen (PSA) testing has limited accuracy for the early detection of prostate cancer (PCa). Objective: To assess the value added by percentage of free to total PSA (%fPSA), prostate cancer antigen 3 (PCA3), and a kallikrein panel (4k-panel) to the European Randomised Study of Screening for Prostate Cancer (ERSPC) multivariable prediction models: risk calculator (RC) 4, including transrectal ultrasound, and RC 4 plus digital rectal examination (4+DRE) for prescreened men. Design, setting, and participants: Participants were invited for rescreening between October 2007 and February 2009 within the Dutch part of the ERSPC study. Biopsies were taken in men with a PSA level ≥3.0 ng/ml or a PCA3 score ≥10. Additional analyses of the 4k-panel were done on serum samples. Outcome measurements and statistical analysis: Outcome was defined as PCa detectable by sextant biopsy. Receiver operating characteristic curve and decision curve analyses were performed to compare the predictive capabilities of %fPSA, PCA3, 4k-panel, the ERSPC RCs, and their combinations in logistic regression models. Results and limitations: PCa was detected in 119 of 708 men. The %fPSA did not perform better univariately or added to the RCs compared with the RCs alone. In 202 men with an elevated PSA, the 4k-panel discriminated better than PCA3 when modelled univariately (area under the curve [AUC]: 0.78 vs 0.62; p = 0.01). The multivariable models with PCA3 or the 4k-panel were equivalent (AUC: 0.80 for RC 4+DRE). In the total population, PCA3 discriminated better than the 4k-panel (univariate AUC: 0.63 vs 0.56; p = 0.05). There was no statistically significant difference between the multivariable model with PCA3 (AUC: 0.73) versus the mode

Comparison of Two Prostate Cancer Risk Calculators that Include the Prostate Health Index

Background: Risk prediction models for prostate cancer (PCa) have become important tools in reducing unnecessary prostate biopsies. The Prostate Health Index (PHI) may increase the predictive accuracy of such models. Objectives: To compare two PCa risk calculators (RCs) that include PHI. Design, setting, and participants: We evaluated the predictive performance of a previously developed PHI-based nomogram and updated versions of the European Randomized Study of Screening for Prostate Cancer (ERSPC) RCs based on digital rectal examination (DRE): RC3 (no prior biopsy) and RC4 (prior biopsy). For the ERSPC updates, the original RCs were recalibrated and PHI was added as a predictor. The PHI-updated ERSPC RCs were compared with the Lughezzani nomogram in 1185 men from four European sites. Outcomes were biopsy-detectable PC and potentially advanced or aggressive PCa, defined as clinical stage >T2b and/or a Gleason score ≥7 (clinically relevant PCa). Results and limitations: The PHI-updated ERSPC models had a combined area under the curve for the receiver operating characteristic (AUC) of 0.72 for all PCa and 0.68 for clinically relevant PCa. For the Lughezzani PHI-based nomogram, AUCs were 0.75 for all PCa and 0.69 for clinically relevant PCa. For men without a prior biopsy, PHI-updated RC3 resulted in AUCs of 0.73 for PCa and 0.66 for clinically relevant PCa. Decision curves confirmed these patterns, although the number of clinically relevant cancers was low. Conclusion: Differences between RCs that include PHI are small. Addition of PHI to an RC leads to further reductions in the rate of unnecessary biopsies when compared to a strategy based on prostate-specific antigen measurement. Patient summary: Risk prediction models for prostate cancer have become important tools in reducing unnecessary prostate biopsies. We compared two risk prediction models for prostate cancer that include the Prostate Health Index. We found that these models are equivalent to each other, and both perform better than the prostate-specific antigen test alone in predicting cancer. Prostate-specific antigen screening reduces prostate cancer mortality but leads to many unnecessary prostate biopsies and overdiagnosis. Inclusion of the Prostate Health Index results in an equivalent increase in predictive ability for both the Lughezzani and the updated European Randomized Study of Screening for Prostate Cancer models