ROC curves in cost space

The final publication is available at Springer via http://dx.doi.org/10.1007/s10994-013-5328-9ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area under the optimal cost curve. In this paper we present new findings and connections between ROC space and cost space. In particular, we show that ROC curves can be transferred to cost space by means of a very natural threshold choice method, which sets the decision threshold such that the proportion of positive predictions equals the operating condition. We call these new curves rate-driven curves, and we demonstrate that the expected loss as measured by the area under these curves is linearly related to AUC. We show that the rate-driven curves are the genuine equivalent of ROC curves in cost space, establishing a point-point rather than a point-line correspondence. Furthermore, a decomposition of the rate-driven curves is introduced which separates the loss due to the threshold choice method from the ranking loss (Kendall τ distance). We also derive the corresponding curve to the ROC convex hull in cost space; this curve is different from the lower envelope of the cost lines, as the latter assumes only optimal thresholds are chosen.We would like to thank the anonymous referees for their helpful comments. This work was supported by the MEC/MINECO projects CONSOLIDER-INGENIO CSD2007-00022 and TIN 2010-21062-C02-02, GVA project PROMETEO/2008/051, the COST-European Cooperation in the field of Scientific and Technical Research IC0801 AT, and the REFRAME project granted by the European Coordinated Research on Long-term Challenges in Information and Communication Sciences & Technologies ERA-Net (CHIST-ERA), and funded by the Engineering and Physical Sciences Research Council in the UK and the Ministerio de Economia y Competitividad in Spain.Hernández Orallo, J.; Flach ., P.; Ferri Ramírez, C. (2013). ROC curves in cost space. Machine Learning. 93(1):71-91. https://doi.org/10.1007/s10994-013-5328-9S7191931Adams, N., & Hand, D. (1999). Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition, 32(7), 1139–1147.Chang, J., & Yap, C. (1986). A polynomial solution for the potato-peeling problem. Discrete & Computational Geometry, 1(1), 155–182.Drummond, C., & Holte, R. (2000). Explicitly representing expected cost: an alternative to ROC representation. In Knowl. discovery & data mining (pp. 198–207).Drummond, C., & Holte, R. (2006). Cost curves: an improved method for visualizing classifier performance. Machine Learning, 65, 95–130.Elkan, C. (2001). The foundations of cost-sensitive learning. In B. Nebel (Ed.), Proc. of the 17th intl. conf. on artificial intelligence (IJCAI-01) (pp. 973–978).Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874.Fawcett, T., & Niculescu-Mizil, A. (2007). PAV and the ROC convex hull. Machine Learning, 68(1), 97–106.Flach, P. (2003). The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In Machine learning, proceedings of the twentieth international conference (ICML 2003) (pp. 194–201).Flach, P., Hernández-Orallo, J., & Ferri, C. (2011). A coherent interpretation of AUC as a measure of aggregated classification performance. In Proc. of the 28th intl. conference on machine learning, ICML2011.Frank, A., & Asuncion, A. (2010). UCI machine learning repository. http://archive.ics.uci.edu/ml .Hand, D. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning, 77(1), 103–123.Hernández-Orallo, J., Flach, P., & Ferri, C. (2011). Brier curves: a new cost-based visualisation of classifier performance. In Proceedings of the 28th international conference on machine learning, ICML2011.Hernández-Orallo, J., Flach, P., & Ferri, C. (2012). A unified view of performance metrics: translating threshold choice into expected classification loss. Journal of Machine Learning Research, 13, 2813–2869.Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1/2), 81–93. doi: 10.2307/2332226 .Swets, J., Dawes, R., & Monahan, J. (2000). Better decisions through science. Scientific American, 283(4), 82–87

Technical Note: Towards ROC Curves in Cost Space

ROC curves and cost curves are two popular ways of visualising classifier performance, finding appropriate thresholds according to the operating condition, and deriving useful aggregated measures such as the area under the ROC curve (AUC) or the area under the optimal cost curve. In this note we present some new findings and connections between ROC space and cost space, by using the expected loss over a range of operating conditions. In particular, we show that ROC curves can be transferred to cost space by means of a very natural way of understanding how thresholds should be chosen, by selecting the threshold such that the proportion of positive predictions equals the operating condition (either in the form of cost proportion or skew). We call these new curves {ROC Cost Curves}, and we demonstrate that the expected loss as measured by the area under these curves is linearly related to AUC. This opens up a series of new possibilities and clarifies the notion of cost curve and its relation to ROC analysis. In addition, we show that for a classifier that assigns the scores in an evenly-spaced way, these curves are equal to the Brier Curves. As a result, this establishes the first clear connection between AUC and the Brier score

CASP-DM: Context Aware Standard Process for Data Mining

We propose an extension of the Cross Industry Standard Process for Data Mining (CRISPDM) which addresses specific challenges of machine learning and data mining for context and model reuse handling. This new general context-aware process model is mapped with CRISP-DM reference model proposing some new or enhanced outputs

Estudio y Desarrollo de una Librería en R para Evaluar las Prestaciones de un Clasificador

[ES] Los modelos de clasificación se generan por algoritmos de aprendizaje supervisado, que aprenden a través de un conjunto de datos de entrenamiento. Estos modelos establecen relaciones entre las instancias, que les permiten predecir si pertenecen, o no, a un mismo tipo o clase. Cuando los clasificadores se usan en aplicaciones de la vida real como: discriminación de imágenes, diagnósticos en medicina, gestión de las telecomunicaciones, bioinformática, clasificación de texto, detección de fraude en transacciones financieras, etc., se enfrentan a dificultades ocasionadas por la distribución de las clases y/o por los costes de clasificar erróneamente una instancia. Existen algunas herramientas que permiten evaluar las prestaciones de los clasi- ficadores, una de las más usadas debido a la facilidad de su interpretación es la curva ROC, que aunque tiene asociados estadísticos que permiten seleccionar o descartar modelos de acuerdo a su desempeño, no toma en cuenta la distribución de las clases y el coste de clasificación. Para solventar estas limitaciones surgieron las Curvas de Coste. El propósito de este trabajo es realizar un estudio de las herramientas gráficas de evaluación del rendimiento de clasificadores, dando mayor énfasis a las Curvas de Coste y métodos de selección de umbral sobre clasificadores suaves. Como resultado de este trabajo se desarrolla una librería gráfica, en el lenguaje de programación R, que incorpora estas funcionalidades. Además, se incluyen algunos ejemplos del uso de la nueva librería con conjuntos de datos reales y métodos de clasificación conocidos. Estos ejemplos ilustran las ventajas que presenta la utilización de las Curvas de Costes y los métodos de selección de umbral cuando se requiere evaluar el rendimiento de clasificadores en entornos con contextos cambiantes.[EN] Classification models are generated by supervised learning algorithms that learn through a training dataset. These models establish relationships between instances, which allow them to predict whether they belong or not to the same type or class. When classifiers are used in real-life applications, such as image discrimination, medical diagnosis, telecommunications management, bioinformatics, text classification, fraud detection in financial transactions, and others, they face difficulties caused by the distribution of classes and/or the cost of misclassifying an instance. There are some tools that can evaluate the performance of classifiers. In particular, the ROC curve is one of the most used due to its ease of interpretation. Although it has statistical methods that allow to select or exclude models according to their performance, the ROC Curve does not take into account distributions of classes and misclassification costs. The Cost Curves appeared as a solution to overcome these limitations. This paper aims to research graphic tools for performance evaluation of classi- fiers, focused on Cost Curves and threshold choice methods applied to soft classi- fiers. As a result of this analysis, we develop, using the programming language R, a graphical library that incorporates these functionalities. We include some examples using the new library with real datasets and well-known classifiers methods. These examples illustrate the advantages that introduce the use of Cost Curves and threshold choice methods when we want to assess the performance of classi- fiers in environments with changing context.Morillo Alcivar, PA. (2016). Estudio y Desarrollo de una Librería en R para Evaluar las Prestaciones de un Clasificador. http://hdl.handle.net/10251/69212.TFG

Are the statistical tests the best way to deal with the biomarker selection problem?

Statistical tests are a powerful set of tools when applied correctly, but unfortunately the extended misuse of them has caused great concern. Among many other applications, they are used in the detection of biomarkers so as to use the resulting p-values as a reference with which the candidate biomarkers are ranked. Although statistical tests can be used to rank, they have not been designed for that use. Moreover, there is no need to compute any p-value to build a ranking of candidate biomarkers. Those two facts raise the question of whether or not alternative methods which are not based on the computation of statistical tests that match or improve their performances can be proposed. In this paper, we propose two alternative methods to statistical tests. In addition, we propose an evaluation framework to assess both statistical tests and alternative methods in terms of both the performance and the reproducibility. The results indicate that there are alternative methods that can match or surpass methods based on statistical tests in terms of the reproducibility when processing real data, while maintaining a similar performance when dealing with synthetic data. The main conclusion is that there is room for the proposal of such alternative methods.This work is partially supported by the Basque Government (IT1244-19, Elkartek BID3A and Elkartek project 3KIA, KK2020/00049) and the Spanish Ministry of Economy and Competitiveness MINECO (PID2019-104966GB-I00) and a University-Society Project 15/19 (Basque Government and University of the Basque Country UPV/EHU). Ari Urkullu has been supported by the Basque Government through a predoctoral grant (PRE_2013_1_1313, PRE_2014_2_87, PRE_2015_2_0280 and PRE_2016_2_0314). Aritz Perez has been supported by the Basque Government through the BERC 2022-2025 and Elkartek programs and by the Ministry of Science, Innovation and Universities: BCAM Severo Ochoa accreditation SEV-2017-0718. Borja Calvo has been partially supported by the IT1244-19 project and the ELKARTEK program from Basque Government, and the project PID2019-104966GB-I00 from the Spanish Ministry of Economy and Competitiveness

Are the statistical tests the best way to deal with the biomarker selection problem?

Statistical tests are a powerful set of tools when applied correctly, but unfortunately the extended misuse of them has caused great concern. Among many other applications, they are used in the detection of biomarkers so as to use the resulting p-values as a reference with which the candidate biomarkers are ranked. Although statistical tests can be used to rank, they have not been designed for that use. Moreover, there is no need to compute any p-value to build a ranking of candidate biomarkers. Those two facts raise the question of whether or not alternative methods which are not based on the computation of statistical tests that match or improve their performances can be proposed. In this paper, we propose two alternative methods to statistical tests. In addition, we propose an evaluation framework to assess both statistical tests and alternative methods in terms of both the performance and the reproducibility. The results indicate that there are alternative methods that can match or surpass methods based on statistical tests in terms of the reproducibility when processing real data, while maintaining a similar performance when dealing with synthetic data. The main conclusion is that there is room for the proposal of such alternative methods

Characterizing compromise solutions for investors with uncertain risk preferences

[EN] The optimum portfolio selection for an investor with particular preferences was proven to lie on the normalized efficient frontier between two bounds defined by the Ballestero (1998) bounding theorem. A deeper understanding is possible if the decision-maker is provided with visual and quantitative techniques. Here, we derive useful insights as a way to support investor's decision-making through: (i) a new theorem to assess balance of solutions; (ii) a procedure and a new plot to deal with discrete efficient frontiers and uncertain risk preferences; and (iii) two quality metrics useful to predict long-run performance of investors.Work partially funded by projects Collectiveware TIN2015-66863-C2-1-R (MINECO/FEDER) and 2014 SGR 118Salas-Molina, F.; Rodriguez-Aguilar, JA.; Pla Santamaría, D. (2019). Characterizing compromise solutions for investors with uncertain risk preferences. Operational Research. 19(3):661-677. https://doi.org/10.1007/s12351-017-0309-6S661677193Amiri M, Ekhtiari M, Yazdani M (2011) Nadir compromise programming: a model for optimization of multi-objective portfolio problem. Expert Syst Appl 38(6):7222–7226Ballestero E (1998) Approximating the optimum portfolio for an investor with particular preferences. J Oper Res Soc 49:998–1000Ballestero E (2007) Compromise programming: a utility-based linear-quadratic composite metric from the trade-off between achievement and balanced (non-corner) solutions. Eur J Oper Res 182(3):1369–1382Ballestero E, Pla-Santamaria D (2004) Selecting portfolios for mutual funds. Omega 32(5):385–394Ballestero E, Pla-Santamaria D, Garcia-Bernabeu A, Hilario A (2015) Portfolio selection by compromise programming. In: Ballestero E, Pérez-Gladish B, Garcia-Bernabeu A (eds) Socially responsible investment. A multi-criteria decision making approach, vol 219. Springer, Switzerland, pp 177–196Ballestero E, Romero C (1996) Portfolio selection: a compromise programming solution. J Oper Res Soc 47(11):1377–1386Ballestero E, Romero C (1998) Multiple criteria decision making and its applications to economic problems. Kluwer Academic Publishers, BerlinBilbao-Terol A, Pérez-Gladish B, Arenas-Parra M, Rodríguez-Uría MV (2006) Fuzzy compromise programming for portfolio selection. Appl Math Comput 173(1):251–264Bravo M, Ballestero E, Pla-Santamaria D (2012) Evaluating fund performance by compromise programming with linear-quadratic composite metric: an actual case on the caixabank in spain. J Multi-Criteria Decis Anal 19(5–6):247–255Ehrgott M, Klamroth K, Schwehm C (2004) An MCDM approach to portfolio optimization. Eur J Oper Res 155(3):752–770Fawcett T (2006) An introduction to ROC analysis. Pattern Recognit Lett 27(8):861–874Hernández-Orallo J, Flach P, Ferri C (2013) ROC curves in cost space. Mach Learn 93(1):71–91Markowitz H (1952) Portfolio selection. J Finance 7(1):77–91Pla-Santamaria D, Bravo M (2013) Portfolio optimization based on downside risk: a mean-semivariance efficient frontier from dow jones blue chips. Ann Oper Res 205(1):189–201Ringuest JL (1992) Multiobjective optimization: behavioral and computational considerations. Springer Science & Business Media, BerlinSteuer RE, Qi Y, Hirschberger M (2007) Suitable-portfolio investors, nondominated frontier sensitivity, and the effect of multiple objectives on standard portfolio selection. Ann Oper Res 152(1):297–317Xidonas P, Mavrotas G, Krintas T, Psarras J, Zopounidis C (2012) Multicriteria portfolio management. Springer, BerlinYu P-L (1973) A class of solutions for group decision problems. Manag Sci 19(8):936–946Yu P-L (1985) Multiple criteria decision making: concepts, techniques and extensions. Plenum Press, BerlinZeleny M (1982) Multiple criteria decision making. McGraw-Hill, New Yor