Estimating the Area under a Receiver Operating Characteristic Curve For Repeated Measures Design

The receiver operating characteristic (ROC) curve is widely used for diagnosing as well as for judging the discrimination ability of different statistical models. Although theories about ROC curves have been established and computation methods and computer software are available for cross-sectional design, limited research for estimating ROC curves and their summary statistics has been done for repeated measure designs, which are useful in many applications, such as biological, medical and health services research. Furthermore, there is no published statistical software available that can generate ROC curves and calculate summary statistics of the area under a ROC curve for data from a repeated measures design. Using generalized linear mixed model (GLMM), we estimate the predicted probabilities of the positivity of a disease or condition, and the estimated probability is then used as a bio-marker for constructing the ROC curve and computing the area under the curve. The area under a ROC curve is calculated using the Wilcoxon non-parametric approach by comparing the predicted probability of all discordant pairs of observations. The ROC curve is constructed by plotting a series of pairs of true positive rate (sensitivity) and false positive rate (1- specificity) calculated from varying cuts of positivity escalated by increments of 0.005 in predicted probability. The computation software is written in SAS/IML/MACRO v8 and can be executed in any computer that has a working SAS v8 system with SAS/IML/MACRO.

ROCnReg: An R Package for Receiver Operating Characteristic Curve Inference With and Without Covariates

This paper introduces the package ROCnReg that allows estimating the pooled ROC curve, the covariate-specific ROC curve, and the covariate-adjusted ROC curve by different methods, both from (semi) parametric and nonparametric perspectives and within Bayesian and frequentist paradigms. From the estimated ROC curve (pooled, covariate-specific, or covariate-adjusted), several summary measures of discriminatory accuracy, such as the (partial) area under the ROC curve and the Youden index, can be obtained. The package also provides functions to obtain ROC-based optimal threshold values using several criteria, namely, the Youden index criterion and the criterion that sets a target value for the false positive fraction. For the Bayesian methods, we provide tools for assessing model fit via posterior predictive checks, while the model choice can be carried out via several information criteria. Numerical and graphical outputs are provided for all methods. This is the only package implementing Bayesian procedures for ROC curves

Using the ROC Curve to Measure Association and Evaluate Prediction Accuracy for a Binary Outcome

This review article addresses the ROC curve and its advantage over the odds ratio to measure the association between a continuous variable and a binary outcome. A simple parametric model under the normality assumption and the method of Box-Cox transformation for non-normal data are discussed. Applications of the binormal model and the Box-Cox transformation under both univariate and multivariate inference are illustrated by a comprehensive data analysis tutorial. Finally, a summary and recommendations are given as to the usage of the binormal ROC curve

Estimating the Area under a Receiver Operating Characteristic Curve For Repeated Measures Design

The receiver operating characteristic (ROC) curve is widely used for diagnosing as well as for judging the discrimination ability of different statistical models. Although theories about ROC curves have been established and computation methods and computer software are available for cross-sectional design, limited research for estimating ROC curves and their summary statistics has been done for repeated measure designs, which are useful in many applications, such as biological, medical and health services research. Furthermore, there is no published statistical software available that can generate ROC curves and calculate summary statistics of the area under a ROC curve for data from a repeated measures design. Using generalized linear mixed model (GLMM), we estimate the predicted probabilities of the positivity of a disease or condition, and the estimated probability is then used as a bio-marker for constructing the ROC curve and computing the area under the curve. The area under a ROC curve is calculated using the Wilcoxon non-parametric approach by comparing the predicted probability of all discordant pairs of observations. The ROC curve is constructed by plotting a series of pairs of true positive rate (sensitivity) and false positive rate (1- specificity) calculated from varying cuts of positivity escalated by increments of 0.005 in predicted probability. The computation software is written in SAS/IML/MACRO v8 and can be executed in any computer that has a working SAS v8 system with SAS/IML/MACRO

On efficient assessment of image-quality metrics based on linear model observers

pre-printThis paper is motivated by the problem of image-quality assessment using model observers for the purpose of development and optimization of medical imaging systems. Specifically, we present a study regarding the estimation of the receiver operating characteristic (ROC) curve for the observer and associated summary measures. This study evaluates the statistical advantage that may be gained in ROC estimates of observer performance by assuming that the difference of the class means for the observer ratings is known. Such knowledge is frequently available in image-quality studies employing known-location lesion detection tasks together with linear model observers. The study is carried out by introducing parametric point and confidence interval estimators that incorporate a known difference of class means. An evaluation of the new estimators for the area under the ROC curve establishes that a large reduction in statistical variability can be achieved through incorporation of knowledge of the difference of class means. Namely, the mean 95% AUC confidence interval length can be as much as seven times smaller in some cases. We also examine how knowledge of the difference of class means can be advantageously used to compare the areas under two correlated ROC curves, and observe similar gains

Bayesian nonparametric inference for the covariate-adjusted ROC curve

Accurate diagnosis of disease is of fundamental importance in clinical practice and medical research. Before a medical diagnostic test is routinely used in practice, its ability to distinguish between diseased and nondiseased states must be rigorously assessed through statistical analysis. The receiver operating characteristic (ROC) curve is the most popular used tool for evaluating the discriminatory ability of continuous-outcome diagnostic tests. It has been acknowledged that several factors (e.g., subject-specific characteristics, such as age and/or gender) can affect the test's accuracy beyond disease status. Recently, the covariate-adjusted ROC curve has been proposed and successfully applied as a global summary measure of diagnostic accuracy that takes covariate information into account. We motivate the use of the covariate-adjusted ROC curve and develop a highly robust model based on a combination of B-splines dependent Dirichlet process mixture models and the Bayesian bootstrap. Multiple simulation studies demonstrate the ability of our model to successfully recover the true covariate-adjusted ROC curve and to produce valid inferences in a variety of complex scenarios. Our methods are motivated by and applied to an endocrine study where the main goal is to assess the accuracy of the body mass index, adjusted for age and gender, for predicting clusters of cardiovascular disease risk factors. The R-package AROC, implementing our proposed methods, is provided