Measuring the Discriminative Power of Rating Systems

Assessing the discriminative power of rating systems is an important question to banks and to regulators. In this article we analyze the Cumulative Accuracy Profile (CAP) and the Receiver Operating Characteristic (ROC) which are both commonly used in practice. We give a test-theoretic interpretation for the concavity of the CAP and the ROC curve and demonstrate how this observation can be used for more efficiently exploiting the informational contents of accounting ratios. Furthermore, we show that two popular summary statistics of these concepts, namely the Accuracy Ratio and the area under the ROC curve, contain the same information and we analyse the statistical properties of these measures. We show in detail how to identify accounting ratios with high discriminative power, how to calculate confidence intervals for the area below the ROC curve, and how to test if two rating models validated on the same data set are different. All concepts are illustrated by applications to real data. --Validation,Rating Models,Credit Analysis

Problems related to efficacy measurement and analyses

In clinical research it is very common to compare two treatments on the basis of an efficacy variable. More specifically, if X and Y denote the responses of patients on the two treatments A and B, respectively, the quantity P(Y\u3eX) (which can be called the probabilistic index for the Effect Size), is of interest in clinical statistics. The objective of this study is to derive an efficacy measure that would compare two treatments more informatively and objectively compared to the earlier approaches. Kernel density estimation is a useful non-parametric method that has not been well utilized as an applied statistical tool, mainly due to its computational complexity. The current study shows that this method is robust even under correlation structures that arise during the computation of all possible differences. The kernel methods can be applied to the estimation of the ROC (Receiver Operating Characteristic) curve as well as to the implementation of nonparametric regression of ROC. The area under the ROC curve (AUC), which is exactly equal to the quantity P(Y\u3eX), is also explored in this dissertation. The methodology used for this study is easy to generalize to other areas of application

ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the "pessimistic" approach by Fawcett (2006).Comment: 16 pages, 3 figures, cod

A comparative evaluation of 3 different free-form deformable image registration and contour propagation methods for head and neck MRI : the case of parotid changes radiotherapy

Purpose: To validate and compare the deformable image registration and parotid contour propagation process for head and neck magnetic resonance imaging in patients treated with radiotherapy using 3 different approachesthe commercial MIM, the open-source Elastix software, and an optimized version of it. Materials and Methods: Twelve patients with head and neck cancer previously treated with radiotherapy were considered. Deformable image registration and parotid contour propagation were evaluated by considering the magnetic resonance images acquired before and after the end of the treatment. Deformable image registration, based on free-form deformation method, and contour propagation available on MIM were compared to Elastix. Two different contour propagation approaches were implemented for Elastix software, a conventional one (DIR_Trx) and an optimized homemade version, based on mesh deformation (DIR_Mesh). The accuracy of these 3 approaches was estimated by comparing propagated to manual contours in terms of average symmetric distance, maximum symmetric distance, Dice similarity coefficient, sensitivity, and inclusiveness. Results: A good agreement was generally found between the manual contours and the propagated ones, without differences among the 3 methods; in few critical cases with complex deformations, DIR_Mesh proved to be more accurate, having the lowest values of average symmetric distance and maximum symmetric distance and the highest value of Dice similarity coefficient, although nonsignificant. The average propagation errors with respect to the reference contours are lower than the voxel diagonal (2 mm), and Dice similarity coefficient is around 0.8 for all 3 methods. Conclusion: The 3 free-form deformation approaches were not significantly different in terms of deformable image registration accuracy and can be safely adopted for the registration and parotid contour propagation during radiotherapy on magnetic resonance imaging. More optimized approaches (as DIR_Mesh) could be preferable for critical deformations

Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the performance of different methods using well-characterized benchmark datasets, to determine the strengths of each method or to provide recommendations regarding suitable choices of methods for an analysis. However, benchmarking studies must be carefully designed and implemented to provide accurate, unbiased, and informative results. Here, we summarize key practical guidelines and recommendations for performing high-quality benchmarking analyses, based on our experiences in computational biology

Comparing empirical ROC curves using a Java application: CERCUS

Receiver Operating Characteristic (ROC) analysis is a methodology that has gained much popularity in our days, especially in Medicine, since through the ROC curves, it provides a useful tool to evaluate and specify problems in the performance of a diagnostic indicator. The area under empirical ROC curve (AUC) it’s an indicator that can be used to compare two or more ROC curves. This work arose from the necessity of the existence of software that allows the calculation of the necessary measures to compare systems based on ROC curves. Several software, commercial and non-commercial, are available to perform the calculation of the measures associated to the ROC analysis. However, they present some flaws, especially when there is a need to compare independent samples with different dimensions, or also to compare two ROC curves that intersect. In this paper is presented a new application called CERCUS (Comparison of Empirical ROC Curves). This was developed using a programming language (Java) and stands out for the possibility of comparing two or more ROC curves that cross each other. The main objective of CERCUS is the calculation of several ROC estimates using different methods and make the ROC curves comparison, even if there is an intersection, either for independent or paired samples. It also allows the graph representation of the ROC curve in a unitary plan as well the graph of the area between curves in comparison. This paper presents the program’s versatility in data entry, test menus and visualization of graphs and results.This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/201