Fast and Unbiased Estimation of Volume Under Ordered Three-Class ROC Surface (VUS) With Continuous or Discrete Measurements

Receiver Operating Characteristic (ROC) surfaces have been studied in the literature essentially during the last decade and are considered as a natural generalization of ROC curves in three-class problems. The volume under the surface (VUS) is useful for evaluating the performance of a trichotomous diagnostic system or a three-class classifier's overall accuracy when the possible disease condition or sample belongs to one of three ordered categories. In the areas of medical studies and machine learning, the VUS of a new statistical model is typically estimated through a sample of ordinal and continuous measurements obtained by some suitable specimens. However, discrete scales of the prediction are also frequently encountered in practice. To deal with such scenario, in this paper, we proposed a unified and efficient algorithm of linearithmic order, based on dynamic programming, for unbiased estimation of the mean and variance of VUS with unidimensional samples drawn from continuous or non-continuous distributions. Monte Carlo simulations verify our theoretical findings and developed algorithms

Statistical evaluation of diagnostic tests under verification bias

The use of diagnostic tests to discriminate between disease classes is becoming more and more popular in medicine, which leads to the urgent need for assessing accuracy of diagnostic tests before their implementation. To do that, a common tool is receiver operating characteristic (ROC) analysis. More precisely, the ROC curve and the area under the ROC curve (AUC) are commonly employed when two disease classes (typically, non-diseased and diseased) are considered, whereas the ROC surface and the volume under the ROC surface (VUS) are frequently used when the disease status has three categories (e.g., non-diseased, intermediate and diseased). In estimating such parameters, we assume that the true disease status of each patient can be determined by means of a gold standard test. In practice, unfortunately, the true disease status could be unavailable for all study subjects, due to the expensiveness or invasiveness of the gold standard test. Thus, often only a subset of patients undergoes disease verification. Statistical evaluations of diagnostic accuracy of a test based only on data from subjects with verified disease status are typically biased. This bias is known as verification bias. Various methods have been developed to adjust for verification bias in estimation of the ROC curve and its area for tests with binary or ordinal or continuous results. For the ROC surface and its volume, verification bias correction methods exist for tests with ordinal responses, but not for continuous tests. In this thesis, we propose several bias--corrected methods for estimating the ROC surface and the VUS of continuous diagnostic tests in presence of verification bias. In particular, these methods are constructed based on imputation and re--weighting techniques, and work well when the missingness mechanism of the true disease status is missing at random or missing not at random. The asymptotic behaviors of the estimators are also studied. To illustrate how to use the methods in real applications, two datasets dealing with epithelial ovarian cancer are considered. To support researchers in carrying out the ROC surface analysis in presence of verification bias, an R package and the corresponding Shiny web application have been created

Grid multi-category response logistic models.

BackgroundMulti-category response models are very important complements to binary logistic models in medical decision-making. Decomposing model construction by aggregating computation developed at different sites is necessary when data cannot be moved outside institutions due to privacy or other concerns. Such decomposition makes it possible to conduct grid computing to protect the privacy of individual observations.MethodsThis paper proposes two grid multi-category response models for ordinal and multinomial logistic regressions. Grid computation to test model assumptions is also developed for these two types of models. In addition, we present grid methods for goodness-of-fit assessment and for classification performance evaluation.ResultsSimulation results show that the grid models produce the same results as those obtained from corresponding centralized models, demonstrating that it is possible to build models using multi-center data without losing accuracy or transmitting observation-level data. Two real data sets are used to evaluate the performance of our proposed grid models.ConclusionsThe grid fitting method offers a practical solution for resolving privacy and other issues caused by pooling all data in a central site. The proposed method is applicable for various likelihood estimation problems, including other generalized linear models

ROC Surfaces in the Presence of Verification Bias

In diagnostic medicine, the Receiver Operating Characteristic (ROC) surface is one of the established tools for assessing the accuracy of a diagnostic test in discriminating three disease states, and the volume under the ROC surface has served as a summary index for diagnostic accuracy. In practice, the selection for definitive disease examination may be based on initial test measurements, and induces verification bias in the assessment. We propose here a nonparametric likelihood-based approach to construct the empirical ROC surface in the presence of differential verification, and to estimate the volume under the ROC surface. Estimators of the standard deviation are derived by both the Fisher\u27s Information and Jack-knife method, and their relative accuracy is evaluated in an extensive simulation study. The methodology is further extended to incorporate discrete baseline covariates in the selection process, and to compare the accuracy of a pair of diagnostic tests. We apply the proposed method to compare the diagnostic accuracy between Mini-Mental State Examination and clinical evaluation of dementia, in discriminating among three disease states of Alzheimer\u27s disease

DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups

Medical researchers endeavor to identify potentially useful biomarkers to develop markerbased screening assays for disease diagnosis and prevention. Useful summary measures which properly evaluate the discriminative ability of diagnostic markers are critical for this purpose. Literature and existing software, for example, R packages nicely cover summary measures for diagnostic markers used for the binary case (e.g., healthy vs. diseased). An intermediate population at an early disease stage usually exists between the healthy and the fully diseased population in many disease processes. Supporting utilities for threegroup diagnostic tests are highly desired and important for identifying patients at the early disease stage for timely treatments. However, application packages which provide summary measures for three ordinal groups are currently lacking. This paper focuses on two summary measures of diagnostic accuracy—volume under the receiver operating characteristic surface and the extended Youden index, with three diagnostic groups. We provide the R package DiagTest3Grp to estimate, under both parametric and nonparametric assumptions, the two summary measures and the associated variances, as well as the optimal cut-points for disease diagnosis. An omnibus test for multiple markers and a Wald test for two markers, on independent or paired samples, are incorporated to compare diagnostic accuracy across biomarkers. Sample size calculation under the normality assumption can be performed in the R package to design future diagnostic studies. A real world application evaluating the diagnostic accuracy of neuropsychological markers for Alzheimer’s disease is used to guide readers through step-by-step implementation of DiagTest3Grp to demonstrate its utility

Nonparametric Predictive Inference for Ordinal Data and Accuracy of Diagnostic Tests

This thesis considers Nonparametric Predictive Inference (NPI) for ordinal data and accuracy of diagnostic tests. We introduce NPI for ordinal data, which are categor- ical data with an ordering of the categories. Such data occur in many application areas, for example medical and social studies. The method uses a latent variable representation of the observations and categories on the real line. Lower and upper probabilities for events involving the next observation are presented, with specic attention to comparison of multiple groups of ordinal data. We introduce NPI for accuracy of diagnostic tests with ordinal outcomes, with the inferences based on data for a disease group and a non-disease group. We intro- duce empirical and NPI lower and upper Receiver Operating Characteristic (ROC) curves and the corresponding areas under the curves. We discuss the use of the Youden index related to the NPI lower and upper ROC curves in order to deter- mine the optimal cut-o point for the test. Finally, we present NPI for assessment of accuracy of diagnostic tests involving three groups of real-valued data. This is achieved by developing NPI lower and upper ROC surfaces and the corresponding volumes under these surfaces, and we also consider the choice of cut-o points for classications based on such diagnostic tests

Non‐parametric predictive inference for the validation of credit rating systems

Credit rating or credit scoring systems are important tools for estimating the obligor's creditworthiness and for providing an indication of the obligor's future status. The discriminatory power of a credit rating or credit scoring system refers to its ex ante ability to distinguish between two or more classes of borrowers. One of the most popular tools for the validation of the power of credit rating or credit scoring models to distinguish between two (or more) classes of borrowers is the receiver operating characteristic (ROC) curve (hypersurface) and its widely used overall summary, the area (hypervolume) under the curve (hypersurface). As the end goal of building such models is to predict and quantify uncertainty about future loans, prediction methods are especially valuable in this context. For this, non‐parametric predictive inference is a promising candidate for such inference as it is a frequentist statistical method that is explicitly aimed at using few modelling assumptions, enabled through the use of lower and upper probabilities to quantify uncertainty. The aim of the paper is to introduce non‐parametric predictive inference for ROC analysis within a banking context, for which novel results on ROC hypersurfaces for more than three groups are presented. Examples are provided to illustrate the method