Fast and Unbiased Estimation of Volume Under Ordered Three-Class ROC Surface (VUS) With Continuous or Discrete Measurements

Receiver Operating Characteristic (ROC) surfaces have been studied in the literature essentially during the last decade and are considered as a natural generalization of ROC curves in three-class problems. The volume under the surface (VUS) is useful for evaluating the performance of a trichotomous diagnostic system or a three-class classifier's overall accuracy when the possible disease condition or sample belongs to one of three ordered categories. In the areas of medical studies and machine learning, the VUS of a new statistical model is typically estimated through a sample of ordinal and continuous measurements obtained by some suitable specimens. However, discrete scales of the prediction are also frequently encountered in practice. To deal with such scenario, in this paper, we proposed a unified and efficient algorithm of linearithmic order, based on dynamic programming, for unbiased estimation of the mean and variance of VUS with unidimensional samples drawn from continuous or non-continuous distributions. Monte Carlo simulations verify our theoretical findings and developed algorithms

Inference in receiver operating characteristic surface analysis via a trinormal model‐based testing approach

Receiver operating characteristic (ROC) analysis is the methodological framework of choice for the assessment of diagnostic markers and classification procedures in general, in both two‐class and multiple‐class classification problems. We focus on the three‐class problem for which inference usually involves formal hypothesis testing using a proxy metric such as the volume under the ROC surface (VUS). In this article, we develop an existing approach from the two‐class ROC framework. We define a hypothesis‐testing procedure that directly compares two ROC surfaces under the assumption of the trinormal model. In the case of the assessment of a single marker, the corresponding ROC surface is compared with the chance plane, that is, to an uninformative marker. A simulation study investigating the proposed tests with existing ones on the basis of the VUS metric follows. Finally, the proposed methodology is applied to a dataset of a panel of pancreatic cancer diagnostic markers. The described testing procedures along with related graphical tools are supported in the corresponding R‐package trinROC, which we have developed for this purpose

DiagTest3Grp: An R Package for Analyzing Diagnostic Tests with Three Ordinal Groups

Medical researchers endeavor to identify potentially useful biomarkers to develop markerbased screening assays for disease diagnosis and prevention. Useful summary measures which properly evaluate the discriminative ability of diagnostic markers are critical for this purpose. Literature and existing software, for example, R packages nicely cover summary measures for diagnostic markers used for the binary case (e.g., healthy vs. diseased). An intermediate population at an early disease stage usually exists between the healthy and the fully diseased population in many disease processes. Supporting utilities for threegroup diagnostic tests are highly desired and important for identifying patients at the early disease stage for timely treatments. However, application packages which provide summary measures for three ordinal groups are currently lacking. This paper focuses on two summary measures of diagnostic accuracy—volume under the receiver operating characteristic surface and the extended Youden index, with three diagnostic groups. We provide the R package DiagTest3Grp to estimate, under both parametric and nonparametric assumptions, the two summary measures and the associated variances, as well as the optimal cut-points for disease diagnosis. An omnibus test for multiple markers and a Wald test for two markers, on independent or paired samples, are incorporated to compare diagnostic accuracy across biomarkers. Sample size calculation under the normality assumption can be performed in the R package to design future diagnostic studies. A real world application evaluating the diagnostic accuracy of neuropsychological markers for Alzheimer’s disease is used to guide readers through step-by-step implementation of DiagTest3Grp to demonstrate its utility

ROC curves and the generalization to multiple classes

The present work focuses on the study and extension of ROC analysis methodology for multiple-class classification problems. In clinical medical research, the need for developing an approach to measure the diagnostic accuracy of biomedical tests in classifying the true status of a patient is a critical point when doing both diagnosis and prognosis. In a two-category classification setting, the ROC analysis is the natural approach and the Area Under the Curve (AUC) is a summary measure of the diagnostic accuracy. However, many real classification problems rely to more than two classes; thus, the ROC manifold generalization of curve and the hypervolume (HUM) generalization of area recently appeared in the literature to address classification problems with more than two classes. Motivated by a real research question arose during a four-class classification study for early detection of colorectal cancer, we review the literature on ROC analysis and on its extension to multiple classes. Then, we develop a new estimator of the accuracy measure of a diagnostic marker. We derive the analytical form of the HUM estimator and the analytical representation of its variance. To assess the performance of the proposed estimator and compare it with the two alternatives existing in the literature, we perform simulation exercises and empirical applications. The first application deals with the topic that initially moved our interest, the early detection of colorectal cancer patients; the second concerns the classification of synovial tissue inflammatory cells, a typical case study in the biostatistics literature. Finally, in the last part of our work, we suggest a statistical method to combine multiple tests for multicategory classification. The novelty of our approach is the use of the classification accuracy (HUM) of the combined marker as the objective function to be maximized. The methodology is evaluated trough a simulation study and two empirical applications

Nonparametric Predictive Inference for Ordinal Data and Accuracy of Diagnostic Tests

This thesis considers Nonparametric Predictive Inference (NPI) for ordinal data and accuracy of diagnostic tests. We introduce NPI for ordinal data, which are categor- ical data with an ordering of the categories. Such data occur in many application areas, for example medical and social studies. The method uses a latent variable representation of the observations and categories on the real line. Lower and upper probabilities for events involving the next observation are presented, with specic attention to comparison of multiple groups of ordinal data. We introduce NPI for accuracy of diagnostic tests with ordinal outcomes, with the inferences based on data for a disease group and a non-disease group. We intro- duce empirical and NPI lower and upper Receiver Operating Characteristic (ROC) curves and the corresponding areas under the curves. We discuss the use of the Youden index related to the NPI lower and upper ROC curves in order to deter- mine the optimal cut-o point for the test. Finally, we present NPI for assessment of accuracy of diagnostic tests involving three groups of real-valued data. This is achieved by developing NPI lower and upper ROC surfaces and the corresponding volumes under these surfaces, and we also consider the choice of cut-o points for classications based on such diagnostic tests

Can Subjective Pain Be Inferred From Objective Physiological Data? Evidence From Patients With Sickle Cell Disease

Patients with sickle cell disease (SCD) experience lifelong struggles with both chronic and acute pain, often requiring medical interventMaion. Pain can be managed with medications, but dosages must balance the goal of pain mitigation against the risks of tolerance, addiction and other adverse effects. Setting appropriate dosages requires knowledge of a patient\u27s subjective pain, but collecting pain reports from patients can be difficult for clinicians and disruptive for patients, and is only possible when patients are awake and communicative. Here we investigate methods for estimating SCD patients\u27 pain levels indirectly using vital signs that are routinely collected and documented in medical records. Using machine learning, we develop both sequential and non-sequential probabilistic models that can be used to infer pain levels or changes in pain from sequences of these physiological measures. We demonstrate that these models outperform null models and that objective physiological data can be used to inform estimates for subjective pain. Author summary: Understanding subjective human pain remains a major challenge. If objective data could be used in place of reported pain levels, it could reduce patient burdens and enable the collection of much larger data sets that could deepen our understanding of causes of pain and allow for accurate forecasting and more effective pain management. Here we apply two machine learning approaches to data from patients with sickle cell disease, who often experience debilitating pain crises. Using vital sign data routinely collected in hospital settings including respiratory rate, heart rate, and blood pressure and amidst the real-world challenges of irregular timing, missing data, and inter-patient variation, we demonstrate that these models outperform baseline models in estimating subjective pain, distinguishing between typical and atypical pain levels, and detecting changes in pain. Once trained, these types of models could be used to improve pain estimates in real time in the absence of direct pain reports

ROC Analysis in Diagnostic Medicine

Ph.DDOCTOR OF PHILOSOPH

Generalization of Kullback-Leibler Divergence for Multi-Stage Diseases: Application to Diagnostic Test Accuracy and Optimal Cut-Points Selection Criterion

The Kullback-Leibler divergence (KL), which captures the disparity between two distributions, has been considered as a measure for determining the diagnostic performance of an ordinal diagnostic test. This study applies KL and further generalizes it to comprehensively measure the diagnostic accuracy test for multi-stage (K \u3e 2) diseases, named generalized total Kullback-Leibler divergence (GTKL). Also, GTKL is proposed as an optimal cut-points selection criterion for discriminating subjects among different disease stages. Moreover, the study investigates a variety of applications of GTKL on measuring the rule-in/out potentials in the single-stage and multi-stage levels. Intensive simulation studies are conducted to compare the performance of GTKL and other diagnostic accuracy measures, such as generalized Youden index (GYI), hypervolume under the manifold (HUM), and maximum absolute determinant (MADET). Furthermore, a comprehensive analysis of a real dataset is performed to illustrate the application of the proposed measure

Risk-Based Comparison of Classification Systems

Performance measures for families of classification system families that rely upon the analysis of receiver operating characteristics (ROCs), such as area under the ROC curve (AUC), often fail to fully address the issue of risk, especially for classification systems involving more than two classes. For the general case, we denote matrices of class prevalences, costs, and class-conditional probabilities, and assume costs are subjectively fixed, acceptable estimates for expected values of class-conditional probabilities exist, and mutual independence between a variable in one such matrix and those of any other matrix. The ROC Risk Functional (RRF), valid for any finite number of classes, has an associated parameter argument, that which specifies a member of a family of classification systems, and which system minimizes Bayes risk over the family. We typify joint distributions for class prevalences over standard simplices by means of uniform and beta distributions, and create a family of classification systems using actual data, testing independence assumptions under two such class prevalence distributions. We minimize risk under two different sets of costs

Statistical evaluation of diagnostic tests under verification bias

The use of diagnostic tests to discriminate between disease classes is becoming more and more popular in medicine, which leads to the urgent need for assessing accuracy of diagnostic tests before their implementation. To do that, a common tool is receiver operating characteristic (ROC) analysis. More precisely, the ROC curve and the area under the ROC curve (AUC) are commonly employed when two disease classes (typically, non-diseased and diseased) are considered, whereas the ROC surface and the volume under the ROC surface (VUS) are frequently used when the disease status has three categories (e.g., non-diseased, intermediate and diseased). In estimating such parameters, we assume that the true disease status of each patient can be determined by means of a gold standard test. In practice, unfortunately, the true disease status could be unavailable for all study subjects, due to the expensiveness or invasiveness of the gold standard test. Thus, often only a subset of patients undergoes disease verification. Statistical evaluations of diagnostic accuracy of a test based only on data from subjects with verified disease status are typically biased. This bias is known as verification bias. Various methods have been developed to adjust for verification bias in estimation of the ROC curve and its area for tests with binary or ordinal or continuous results. For the ROC surface and its volume, verification bias correction methods exist for tests with ordinal responses, but not for continuous tests. In this thesis, we propose several bias--corrected methods for estimating the ROC surface and the VUS of continuous diagnostic tests in presence of verification bias. In particular, these methods are constructed based on imputation and re--weighting techniques, and work well when the missingness mechanism of the true disease status is missing at random or missing not at random. The asymptotic behaviors of the estimators are also studied. To illustrate how to use the methods in real applications, two datasets dealing with epithelial ovarian cancer are considered. To support researchers in carrying out the ROC surface analysis in presence of verification bias, an R package and the corresponding Shiny web application have been created