Method for evaluating prediction models that apply the results of randomized trials to individual patients

<p>Abstract</p> <p>Introduction</p> <p>The clinical significance of a treatment effect demonstrated in a randomized trial is typically assessed by reference to differences in event rates at the group level. An alternative is to make individualized predictions for each patient based on a prediction model. This approach is growing in popularity, particularly for cancer. Despite its intuitive advantages, it remains plausible that some prediction models may do more harm than good. Here we present a novel method for determining whether predictions from a model should be used to apply the results of a randomized trial to individual patients, as opposed to using group level results.</p> <p>Methods</p> <p>We propose applying the prediction model to a data set from a randomized trial and examining the results of patients for whom the treatment arm recommended by a prediction model is congruent with allocation. These results are compared with the strategy of treating all patients through use of a net benefit function that incorporates both the number of patients treated and the outcome. We examined models developed using data sets regarding adjuvant chemotherapy for colorectal cancer and Dutasteride for benign prostatic hypertrophy.</p> <p>Results</p> <p>For adjuvant chemotherapy, we found that patients who would opt for chemotherapy even for small risk reductions, and, conversely, those who would require a very large risk reduction, would on average be harmed by using a prediction model; those with intermediate preferences would on average benefit by allowing such information to help their decision making. Use of prediction could, at worst, lead to the equivalent of an additional death or recurrence per 143 patients; at best it could lead to the equivalent of a reduction in the number of treatments of 25% without an increase in event rates. In the Dutasteride case, where the average benefit of treatment is more modest, there is a small benefit of prediction modelling, equivalent to a reduction of one event for every 100 patients given an individualized prediction.</p> <p>Conclusion</p> <p>The size of the benefit associated with appropriate clinical implementation of a good prediction model is sufficient to warrant development of further models. However, care is advised in the implementation of prediction modelling, especially for patients who would opt for treatment even if it was of relatively little benefit.</p

Pre-Treatment Biomarker Levels Improve the Accuracy of Post-Prostatectomy Nomogram for Prediction of Biochemical Recurrence

PURPOSE. We tested the ability of several pre-operative blood-based biomarkers to enhance the accuracy of standard post-operative features for the prediction of biochemical recurrence (BCR) after radical prostatectomy (RP). METHODS. Pre-operative plasma levels of Endoglin, interleukin-6 (IL-6), interleukin-6 soluble receptor (IL-6sR), transforming growth factor-beta 1 (TGF-beta 1), urokinase plasminogen activator (uPA), urokinase plasminogen inhibitor-1 (PAI-1), urokinase plasminogen receptor (uPAR), vascular cell adhesion molecule-1 (VCAM1), and vascular endothelial growth factor (VEGF) were measured using commercially available enzyme immunoassays in 423 consecutive patients treated with RP for clinically localized prostate cancer. Standard postoperative features consisted of surgical margin status, extracapsular extension, seminal vesicle invasion, lymph node involvement, and pathologic Gleason sum. Multivariable modeling was used to explore the gain in the predictive accuracy. The accuracy was quantified by the c-index statistic and was internally validated with 200 bootstrap resamples. RESULTS. Plasma IL-6 (P = 0.03), IL-6sR (P < 0.001), TGF-beta 1 (P = 0.005), and V-CAM1 (P = 0.01) achieved independent predictor status after adjusting for the effects of standard post-operative features. After stepwise backward variable elimination, a model relying on RP Gleason sum, IL-6sR, TGF-beta 1, VCAM1, and uPA improved the predictive accuracy of the standard post-operative model by 4% (86.1% vs. 82.1%, P < 0.001). CONCLUSIONS. Pre-operative plasma biomarkers improved the accuracy of established post-operative prognostic factors of BCR by a significant margin. Incorporation of these biomarkers into standard predictive models may allow more accurate identification of patients who are likely to fail RP thereby allowing more efficient delivery of adjuvant therapy. Prostate 69: 886-894, 2009. (c) 2009 Wiley-Liss, Inc

Comparison of immunohistochemistry with reverse transcription-PCR for the detection of micrometastatic prostate cancer in lymph nodes

The objective is to compare the performance of immunohistochemistry (IHC) with that of reverse transcription (RT)-PCR for detecting clinically significant micrometastases in histopathologically normal archival pelvic lymph nodes (PLN) removed at radical prostatectomy from men with locally advanced nonmetastatic prostate cancer. We stained 1864 fixed, paraffin-embedded PLNs from 199 pT(3)N(0)M(0) prostate cancer patients for prostate-specific antigen (PSA) and cytokeratin. We also assessed human glandular kallikrein (hK2) expression in a subset of 164 patients. In addition, all PLN specimens were assayed for hK2 mRNA using a previously described RT-PCR assay. PSA and cytokeratin were expressed in the same 13 of 199 (7%) cases; hK2 was expressed in 3 of 164 (2%) cases. PSA/cytokeratin and hK2 expression were associated with cancer involvement of seminal vesicles, higher Gleason sum, and a positive RT-PCR-hK2 assay result. In standard postoperative multivariable models, IHC-PSA/IHC-Cytokeratin or IHC-hK2 was associated with prostate cancer progression, development of distant metastases, and prostate cancer-specific survival. However, when RT-PCR-hK2 assay result was added to the models, it was the sole predictor of clinical outcomes. Although IHC-PSA/IHC-Cytokeratin and IHC-hK2 were more specific for identifying patients who would suffer biochemical progression and develop metastases and who would ultimately die of prostate cancer, RT-PCR-hK2 was more sensitive and accurate. Although IHC for PSA, cytokeratin, and hK2 appear to be more specific methods for detecting biologically and clinically significant prostate cancer micrometastases in histopathologically normal PLN, RT-PCR-hK2 appears to be a more sensitive method that maintained a reasonable specificity. In pT(3)N(0) patients, a positive RT-PCR-hK2 assay result when performed on PLN was the strongest predictor of clinical outcomes after radical prostatectomy

The Prostate Health Index Selectively Identifies Clinically Significant Prostate Cancer

Purpose: The Prostate Health Index (phi) is a new test combining total, free and [-2] proPSA into a single score. It was recently approved by the FDA and is now commercially available in the U.S., Europe and Australia. We investigate whether phi improves specificity for detecting clinically significant prostate cancer and can help reduce prostate cancer over diagnosis. Materials and Methods: From a multicenter prospective trial we identified 658 men age 50 years or older with prostate specific antigen 4 to 10 ng/ml and normal digital rectal examination who underwent prostate biopsy. In this population we compared the performance of prostate specific antigen, % free prostate specific antigen, [-2] proPSA and phi to predict biopsy results and, specifically, the presence of clinically significant prostate cancer using multiple criteria. Results: The Prostate Health Index was significantly higher in men with Gleason 7 or greater and "Epstein significant" cancer. On receiver operating characteristic analysis phi had the highest AUC for overall prostate cancer (AUCs phi 0.708, percent free prostate specific antigen 0.648, [-2] proPSA 0.550 and prostate specific antigen 0.516), Gleason 7 or greater (AUCs phi 0.707, percent free prostate specific antigen 0.661, [-2] proPSA 0.558, prostate specific antigen 0.551) and significant prostate cancer (AUCs phi 0.698, percent free prostate specific antigen 0.654, [-2] proPSA 0.550, prostate specific antigen 0.549). At the 90% sensitivity cut point for phi (a score less than 28.6) 30.1% of patients could have been spared an unnecessary biopsy for benign disease or insignificant prostate cancer compared to 21.7% using percent free prostate specific antigen. Conclusions: The new phi test outperforms its individual components of total, free and [-2] proPSA for the identification of clinically significant prostate cancer. Phi may be useful as part of a multivariable approach to reduce prostate biopsies and over diagnosis

A Multicenter Study of [-2]Pro-Prostate Specific Antigen Combined With Prostate Specific Antigen and Free Prostate Specific Antigen for Prostate Cancer Detection in the 2.0 to 10.0 ng/ml Prostate Specific Antigen Range

Purpose: Prostate specific antigen and free prostate specific antigen have limited specificity to detect clinically significant, curable prostate cancer, leading to unnecessary biopsy, and detection and treatment of some indolent tumors. Specificity to detect clinically significant prostate cancer may be improved by [-2]pro-prostate specific antigen. We evaluated [-2]pro-prostate specific antigen, free prostate specific antigen and prostate specific antigen using the formula, ([-2]pro-prostate specific antigen/free prostate specific antigen X prostate specific antigen(1/2)) to enhance specificity to detect overall and high grade prostate cancer. Materials and Methods: We enrolled 892 men with no history of prostate cancer, normal rectal examination, prostate specific antigen 2 to 10 ng/ml and 6-core or greater prostate biopsy in a prospective multi-institutional trial. We examined the relationship of serum prostate specific antigen, free-to-total prostate specific antigen and the prostate health index with biopsy results. Primary end points were specificity and AUC using the prostate health index to detect overall and Gleason 7 or greater prostate cancer on biopsy compared with those of free-to-total prostate specific antigen. Results: In the 2 to 10 ng/ml prostate specific antigen range at 80% to 95% sensitivity the specificity and AUC (0.703) of the prostate health index exceeded those of prostate specific antigen and free-to-total prostate specific antigen. An increasing prostate health index was associated with a 4.7-fold increased risk of prostate cancer and a 1.61-fold increased risk of Gleason score greater than or equal to 4 + 3 = 7 disease on biopsy. The AUC of the index exceeded that of free-to-total prostate specific antigen (0.724 vs 0.670) to discriminate prostate cancer with Gleason 4 or greater + 3 from lower grade disease or negative biopsy. Prostate health index results were not associated with age and prostate volume. Conclusions: The prostate health index may be useful in prostate cancer screening to decrease unnecessary biopsy in men 50 years old or older with prostate specific antigen 2 to 10 ng/ml and negative digital rectal examination with minimal loss in sensitivity