Skip to main content
Article thumbnail
Location of Repository

Accuracy of logistic models and receiver operating characteristic curves

By Philip Corbett

Abstract

The accuracy of prediction is a commonly studied topic in modern statistics. The\ud performance of a predictor is becoming increasingly more important as real-life\ud decisions axe made on the basis of prediction. In this thesis we investigate the\ud prediction accuracy of logistic models from two different approaches.\ud Logistic regression is often used to discriminate between two groups or populations\ud based on a number of covariates. The receiver operating characteristic\ud (ROC) curve is a commonly used tool (especially in medical statistics) to assess\ud the performance Of such a score or test. By using the same data to fit the logistic\ud regression and calculate the ROC curve we overestimate the performance that\ud the score would give if validated on a sample of future cases. This overestimation\ud is studied and we propose a correction for the ROC curve and the area under the\ud curve. The methods axe illustrated through way of two medical examples and a\ud simulation study, and we show that the overestimation can be quite substantial\ud for small sample sizes.\ud The idea of shrinkage pertains to the notion that by including some prior information\ud about the data under study we can improve prediction. Until now,\ud the study of shrinkage has almost exclusively been concentrated on continuous\ud measurements. We propose a methodology to study shrinkage for logistic regression\ud modelling of categorical data with a binary response. Categorical data\ud with a large number of levels is often grouped for modelling purposes, which\ud discards useful information about the data. By using this information we can\ud apply Bayesian methods to update model parameters and show through examples\ud and simulations that in some circumstances the updated estimates are better\ud predictors than the model

Topics: QA
OAI identifier: oai:wrap.warwick.ac.uk:4114

Suggested articles

Citations

  1. (1989). A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. doi
  2. (1988). A general regression methodology for ROC curve estimation. doi
  3. (1996). A generalized linear model approach for analysing receiver operating characteristic curves. doi
  4. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. doi
  5. (1998). A new strategy for evaluating the impact of epidemiologic risk factors for cancer with respect to melanoma. doi
  6. (1977). An asymptotic equivalence of choice by model by crossvalidation and Akaike's criterion.
  7. (2000). An improved measure for comparing diagnostic tests. doi
  8. (1990). An introduction to generalized linear models. doi
  9. (1970). Analysis of Binary Data. doi
  10. (1989). Applied Logistic Regression. doi
  11. (1978). Basic principles of ROC analysis. doi
  12. (1999). Binormal association - marginal models for ROC analysis. doi
  13. (1990). Categorical Data Analysis. doi
  14. (1978). Choosing between logistic regression and discriminant analysis. doi
  15. (1988). Comparing the areas under two or more correlated receiver operating chaxacteristic curves: a non-parametric approach. doi
  16. (1997). Construction and Assessment of Classification Rules. doi
  17. (1999). Correcting bias due to misclassification in the estimation of logistic regression models. Statistics and Probability Letters, doi
  18. (1987). Cross-validation shrinkage of regression predictors. doi
  19. (1986). Cross-validation, the jackknife and the bootstrap, excess error estimation in forward logistic regression. doi
  20. (1975). Discrete Multivariate Analysis. doi
  21. (1983). Estimating the error rate of a prediction rule: improvement of cross-validation. doi
  22. (2000). Evaluating the predictive performance of habitat models developed using logistic regression. Ecologic . al Modelling, doi
  23. (1983). Generalized Linear Models. doi
  24. (1986). How biased is the apparent error rate of a prediction rule? doi
  25. I (1994a). Assessing classification rules.
  26. (1981). Logistic regression diagnostics. doi
  27. (2000). Maximum likelihood estimation of misclassification rates of binomial regression. doi
  28. (1968). Maximum likelihood estimation of the parameters of signal detection theory -a direct solution. doi
  29. (2001). Measuring diagnostic accuracy of statistical prediction rules. doi
  30. (1993). Medical statistics, a commonsense approach. doi
  31. (1997). Modem Applied Statistics with 35-PL US. doi
  32. (1996). Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy and measuring and reducing errors. doi
  33. (1996). Nonparametric and semi-parametric estimation of the receiver operating characteristic curve. The Annals of Statistics, doi
  34. (1986). On the robustness of shrinkage predictors in regression to differences between past and future data.
  35. (1999). Overfitting and the Stein factor.
  36. (1996). Predicting reoffending for discretionary conditional release. Home Office Research and Statistics Directorate Report 150.
  37. (1990). Predictive value of statistical models. doi
  38. (1993). Receiver operating characteristic (ROC) plots: a fundamentalavaluaitiujg-ol in clinical medicene.
  39. (1984). Regression modelling strategies for improved prognostic prediction. doi
  40. (1983). Regression, prediction and shrinkage (with discussion). doi
  41. (1989). Regularized Discriminant Analysis.
  42. (1994). ROC curves for classification trees. doi
  43. (1997). Sampling variability of non-paxametric estimates of the areas under the receiver operating characteristic curves: an update. doi
  44. (1996). Screening for cutaneous melanoma by self-skin examination. doi
  45. (1986). Sensitivity, specificity and predictive values: a graphical approach. doi
  46. (2000). Shrinkage and Calibration in Multiple Regression. PhD Thesis.
  47. (1971). Signal detectability and medical decision making. doi
  48. (1966). Signal Detection Theory and Psychophysics. doi
  49. (1975). Signal detection theory and ROC analysis. doi
  50. (1998). Slopes of a receiver operating characteristic curve and likelihood ratios for a diagnostic test. doi
  51. (1973). Some comments on cp. doi
  52. (1997). Statistical classification methods in consumer credit scoring: a review. doi
  53. (1980). Statistical significance tests for binormal ROC curves. doi
  54. (1975). The area above the ordinal dominance graph and the area below the the receiver operating characteristic curve. doi
  55. (1991). The axea under the ROC curve and its competitors. Medical Decision Making, doi
  56. (1999). The effectiveness of risk scores: The logit rank plot. doi
  57. (1975). The efficiency of logistic regression compared to normal discriminant analysis. doi
  58. (1982). The jackknife, bootstrap and other resampling plans. doi
  59. (1982). The meaning and use of the area under a receiver operating characteristic curve (ROC) curve. doi
  60. (1988). The robustness of 'binormal' assumptions used in fitting ROC curves. doi
  61. (1986). The use of relative operating characteristic (ROC) curves in test performance evaluation. Archives of pathology and laboratory medicine,
  62. (1996). The use of the 'binormal' model for parametric ROC analysis of quantitative diagnostic tests. doi
  63. (1998). Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. doi
  64. (1991). Verification bias and the one parameter logistic ROC curve - some claxifications. Medical Decision Making, doi

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.