Comparative assessment of three common algorithms for estimating the variance of the area under the nonparametric receiver operating characteristic curve

The area under the receiver operating characteristic (ROC) curve is often used to summarize and compare the discriminatory accuracy of a diagnostic test or modality,and to evaluate the predictive power of statistical models for binary outcomes. Parametric maximum likelihood methods for Þtting of the ROC curve provide direct estimates of the area under the ROC curve and its variance. Nonparametric methods, on the other hand, provide estimates of the area under the ROC curve, but do not directly estimate its variance. Three algorithms for computing the variance for the area under the nonparametric ROC curve are commonly used, although ambiguity exists about their behavior under diverse study conditions. Using simulated data, we found similar asymptotic performance between these algorithms when the diagnostic test produces results on a continuous scale, but found notable differences in small samples, and when the diagnostic test yields results on a discrete diagnostic scale. Copyright 2002 by Stata Corporation.receiver operating characteristic (ROC )curve,trapezoidal rule, sensitivity,specificity,discriminatory accuracy,predictive power

Submission of Evidence on Online Violence Against Women to the UN Special Rapporteur on Violence Against Women, its Causes and Consequences, Dr Dubravka Šimonović

Figure S1. B3GALNT2 levels determined by W.B. and ROC curve. a–c Relative mRNA expression of B3GALNT2 in HCC tumor tissues and normal liver tissues obtained from GSE76427, GSE36376, and TCGA-LIHC datasets. d Western blot analysis of B3GALNT2 levels in 24 pairs of HCC tissues. T HCC tumor tissue, N adjacent non-tumor tissue. e ROC curve analysis of the sensitivity and specificity for the predictive value of TNM model, B3GALNT2 expression, and the combination model. (TIFF 546 kb

Estimating the Area under a Receiver Operating Characteristic Curve For Repeated Measures Design

The receiver operating characteristic (ROC) curve is widely used for diagnosing as well as for judging the discrimination ability of different statistical models. Although theories about ROC curves have been established and computation methods and computer software are available for cross-sectional design, limited research for estimating ROC curves and their summary statistics has been done for repeated measure designs, which are useful in many applications, such as biological, medical and health services research. Furthermore, there is no published statistical software available that can generate ROC curves and calculate summary statistics of the area under a ROC curve for data from a repeated measures design. Using generalized linear mixed model (GLMM), we estimate the predicted probabilities of the positivity of a disease or condition, and the estimated probability is then used as a bio-marker for constructing the ROC curve and computing the area under the curve. The area under a ROC curve is calculated using the Wilcoxon non-parametric approach by comparing the predicted probability of all discordant pairs of observations. The ROC curve is constructed by plotting a series of pairs of true positive rate (sensitivity) and false positive rate (1- specificity) calculated from varying cuts of positivity escalated by increments of 0.005 in predicted probability. The computation software is written in SAS/IML/MACRO v8 and can be executed in any computer that has a working SAS v8 system with SAS/IML/MACRO.

An Improved Method for Bandwidth Selection when Estimating ROC Curves

The receiver operating characteristic (ROC) curve is used to describe the performance of a diagnostic test which classifies observations into two groups. We introduce a new method for selecting bandwidths when computing kernel estimates of ROC curves. Our technique allows for interaction between the distributions of each group of observations and gives substantial improvement in MISE over other proposed methods, especially when the two distributions are very different.Bandwidth selection; binary classification; kernel estimator; ROC curve

What is an ROC curve?

The paper by Body et al is concerned with the evaluation of decision aids, which can be used to identify potential acute coronary syndromes (ACS) in the ED. The authors previously developed the Manchester Acute Coronary Syndromes model (MACS) decision aid, which uses several clinical variables and two biomarkers to ‘rule in’ and ‘rule out’ ACS. However, one of the two biomarkers (heart-type fatty acid bindingprotein, H-FABP) is not widely used so a revised decision aid has been developed (Troponin-only Manchester Acute Coronary Syndromes, T-MACS), which include a single biomarker hs-cTnT. In this issue, the authors show how they derive a revised decision aid and describe its performance in a number of independent diagnostic cohort studies. Decision aids (as well as other types of ‘diagnostic tests’) are often evaluated in terms of diagnostic testing parameters such as the area under the receiver operating characteristic (ROC) curve, sensitivity and specificity. In this article, we explain how the ROC analysis is conducted and why it is an essential step towards developing a test with the desirable levels of sensitivity and specificity

Measuring the Discriminative Power of Rating Systems

Assessing the discriminative power of rating systems is an important question to banks and to regulators. In this article we analyze the Cumulative Accuracy Profile (CAP) and the Receiver Operating Characteristic (ROC) which are both commonly used in practice. We give a test-theoretic interpretation for the concavity of the CAP and the ROC curve and demonstrate how this observation can be used for more efficiently exploiting the informational contents of accounting ratios. Furthermore, we show that two popular summary statistics of these concepts, namely the Accuracy Ratio and the area under the ROC curve, contain the same information and we analyse the statistical properties of these measures. We show in detail how to identify accounting ratios with high discriminative power, how to calculate confidence intervals for the area below the ROC curve, and how to test if two rating models validated on the same data set are different. All concepts are illustrated by applications to real data. --Validation,Rating Models,Credit Analysis