Background: In biomarker discovery studies, markers are ranked for validation using P-values. Standard P-value calculations use normal approximations that may not be valid for small P-values and small sample sizes common in discovery research.
Methods: We compared exact P-values, valid by definition, with normal and logit-normal approximations in a simulated study of 40 cases and 160 controls. The key measure of biomarker performance was sensitivity at 90% specificity. Data for 3000 uninformative markers and 30 true markers were generated randomly, with 10 replications of the simulation. We also analyzed real data on 2371 antibody array markers measured in plasma from 121 cases with ER/PR positive breast cancer and 121 controls.
Results: Using the same discovery criterion, the valid exact P-values lead to discovery of 24 true and 82 false biomarkers while approximate P-values yielded 15 true and 15 false biomarkers (normal approximation) and 20 true and 86 false biomarkers (logit-normal approximation). Moreover, the estimated numbers of true markers among those discovered were substantially incorrect for approximate P-values: normal estimated 0 true markers discovered but found 15; logit-normal estimated 42 but found 20. The exact method estimated 22, close to the actual number of 24 true discoveries. With real data, exact and approximate P-values ranked candidate breast cancer biomarkers very differently.
Conclusions: Exact P-values should be used because they are universally valid. Approximate P-values can lead to inappropriate biomarker selection rules and incorrect conclusions.
Impact: Rigorous data analysis methodology in discovery research may improve the yield of biomarkers that validate clinically
