Receiver Operating Characteristic (ROC) Analysis

Visual expertise covers a broad range of types of studies and methodologies. Many studies incorporate some measure(s) of observer performance or how well participants perform on a given task. Receiver Operating Characteristic (ROC) analysis is a method commonly used in signal detection tasks (i.e., those in which the observer must decide whether or not a target is present or absent; or must classify a given target as belonging to one category or another), especially those in the medical imaging literature. This frontline paper will review some of the core theoretical underpinnings of ROC analysis, provide an overview of how to conduct an ROC study, and discuss some of the key variants of ROC analysis and their applications

A Linear Regression Framework for Receiver Operating Characteristic(ROC) Curve Analysis

In the field of medical diagnostic testing, the receiver operating characteristics(ROC) curve has long been used as a standard statistical tool to assess the accuracy of tests that yield continuous results. Although previous research in this area focused mostly on estimating the ROC curve, recently it has been recognized that the accuracy of a given test may fluctuate depending on certain factors, which motivates modelling covariate effects on the ROC curve. Comparing the corresponding ROC curves between two or more tests is a special case of covariate effect modelling. In this manuscript, we introduce a linear regression framework to model covariate effect on the ROC curve. We assumes the ROC curve takes a specific parametric form for each covariate level and the covariate effect reflects on the parameters of the curves. The new method provides an unified approach for the ROC curve analysis and it is intuitive and easy to apply. Two real data sets are used to illustrate the new approach

Do We Know Who Will Drop Out? A Review of the Predictors of Dropping out of High School: Precision, Sensitivity and Specificity

The purpose of this study is to review the literature on the most accurate indicators of students at risk of dropping out of high school. We used Relative Operating Characteristic (ROC) analysis to compare the sensitivity and specificity of 110 dropout flags across 36 studies. Our results indicate that 1) ROC analysis provides a means to compare the accuracy of different dropout indicators, 2) the majority of dropout flags to date have high precision yet lack accuracy, 3) longitudinal growth models provided the most accurate flags, while 4) the most accurate cross-sectional flags examine low or failing grades. We provide recommendations for future policy and practice. Keywords: Dropout, dropout characteristics, dropout identification, dropout prediction, dropout research, ROC, relative operating characteristic, receiver operating characteristic, growth mixture models, grades

A Linear Regression Framework for the Receiver Operating Characteristic(ROC) Curve Analysis

The receiver operating characteristic (ROC) curve has been a popular statistical tool for characterizing the discriminating power of a classifier, such as a biomarker or an imaging modality for disease screening or diagnosis. It has been recognized that the accuracy of a given procedure may depend on some underlying factors, such as subject’s demographic characteristics or disease risk factors, among others. Non-parametric- or parametric-based methods tend to be either inefficient or cumbersome when evaluating effect of multiple covariates is the main focus. Here we propose a semi-parametric linear regression framework to model covariate effect. It allows the estimation of sensitivity at given specificity to vary according to the covariates and provides a way to model the area under the ROC curve indirectly. Estimation procedure and asymptotic theory are presented. Extensive simulation studies have been conducted to investigate the validity of the proposed method. We illustrate the new method on a diagnostic test dataset

Algorithm Reliability of Kalman Filter Coefficients Determination for Low-Intensity Electroretinosignal

The estimation method of the algorithm reliability for determination of the Kalman filter coefficients for lowintensity electroretinosignal (ERS) processing is constructed. The estimation method of the algorithm reliability is obtained by modifying the Neumann-Pearson criterion. This allowed using of receiver operating characteristic analysis (ROC-analysis) and determination of the area under ROC-curve characteristics (AUC-characteristics) of the proposed algorithm

On the binormal predictive receiver operating characteristic curve for the joint assessment of positive and negative predictive values

The predictive receiver operating characteristic (PROC) curve is a diagrammatic format with application in the statistical evaluation of probabilistic disease forecasts. The PROC curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for evaluation using metrics defined conditionally on the outcome of the forecast rather than metrics defined conditionally on the actual disease status. Starting from the binormal ROC curve formulation, an overview of some previously published binormal PROC curves is presented in order to place the PROC curve in the context of other methods used in statistical evaluation of probabilistic disease forecasts based on the analysis of predictive values; in particular, the index of separation (PSEP) and the leaf plot. An information theoretic perspective on evaluation is also outlined. Five straightforward recommendations are made with a view to aiding understanding and interpretation of the sometimes-complex patterns generated by PROC curve analysis. The PROC curve and related analyses augment the perspective provided by traditional ROC curve analysis. Here, the binormal ROC model provides the exemplar for investigation of the PROC curve, but potential application extends to analysis based on other distributional models as well as to empirical analysis

Multi-objective optimisation for receiver operating characteristic analysis

Copyright © 2006 Springer-Verlag Berlin Heidelberg. The final publication is available at link.springer.comBook title: Multi-Objective Machine LearningSummary Receiver operating characteristic (ROC) analysis is now a standard tool for the comparison of binary classifiers and the selection operating parameters when the costs of misclassification are unknown. This chapter outlines the use of evolutionary multi-objective optimisation techniques for ROC analysis, in both its traditional binary classification setting, and in the novel multi-class ROC situation. Methods for comparing classifier performance in the multi-class case, based on an analogue of the Gini coefficient, are described, which leads to a natural method of selecting the classifier operating point. Illustrations are given concerning synthetic data and an application to Short Term Conflict Alert