Evaluation of Performance Measures for Classifiers Comparison

The selection of the best classification algorithm for a given dataset is a very widespread problem, occuring each time one has to choose a classifier to solve a real-world problem. It is also a complex task with many important methodological decisions to make. Among those, one of the most crucial is the choice of an appropriate measure in order to properly assess the classification performance and rank the algorithms. In this article, we focus on this specific task. We present the most popular measures and compare their behavior through discrimination plots. We then discuss their properties from a more theoretical perspective. It turns out several of them are equivalent for classifiers comparison purposes. Futhermore. they can also lead to interpretation problems. Among the numerous measures proposed over the years, it appears that the classical overall success rate and marginal rates are the more suitable for classifier comparison task

Large-Scale Detection of Non-Technical Losses in Imbalanced Data Sets

Non-technical losses (NTL) such as electricity theft cause significant harm to our economies, as in some countries they may range up to 40% of the total electricity distributed. Detecting NTLs requires costly on-site inspections. Accurate prediction of NTLs for customers using machine learning is therefore crucial. To date, related research largely ignore that the two classes of regular and non-regular customers are highly imbalanced, that NTL proportions may change and mostly consider small data sets, often not allowing to deploy the results in production. In this paper, we present a comprehensive approach to assess three NTL detection models for different NTL proportions in large real world data sets of 100Ks of customers: Boolean rules, fuzzy logic and Support Vector Machine. This work has resulted in appreciable results that are about to be deployed in a leading industry solution. We believe that the considerations and observations made in this contribution are necessary for future smart meter research in order to report their effectiveness on imbalanced and large real world data sets.Comment: Proceedings of the Seventh IEEE Conference on Innovative Smart Grid Technologies (ISGT 2016

Development and Evaluation of an Interdisciplinary Periodontal Risk Prediction Tool Using a Machine Learning Approach

Periodontitis (PD) is a major public health concern which profoundly affects oral health and concomitantly, general health of the population worldwide. Evidence-based research continues to support association between PD and systemic diseases such as diabetes and hypertension, among others. Notably PD also represents a modifiable risk factor that may reduce the onset and progression of some systemic diseases, including diabetes. Due to lack of oral screening in medical settings, this population does not get flagged with the risk of developing PD. This study sought to develop a PD risk assessment model applicable at clinical point-of-care (POC) by comparing performance of five supervised machine learning (ML) algorithms: Naïve Bayes, Logistic Regression, Support Vector Machine, Artificial Neural Network and Decision Tree, for modeling risk by retrospectively interrogating clinical data collected across seven different models of care (MOC) within the interdisciplinary settings. Risk assessment modeling was accomplished using Waikato Environment for Knowledge Analysis (WEKA) open-sourced tool, which supported comparative assessment of the relative performance of the five ML algorithms when applied to risk prediction. To align with current conventions for clinical classification of disease severity, predicting PD risk was treated as a ‘classification problem’, where patients were sorted into two categories based on disease severity and ‘low risk PD’ was defined as no or mild gum disease (‘controls’) or ‘high risk PD’ defined as moderate to severe disease (‘cases’). To assess the predictive performance of models, the study compared performance of ML algorithms applying analysis of recall, specificity, area under the curve, precision, F-measure and Matthew’s correlation coefficient (MCC) and receiver operating characteristic (ROC) curve. A tenfold-cross validation was performed. External validation of the resultant models was achieved by creating validation data subsets applying random selection of approximately 10% of each class of data proportionately. Findings from this study have prognostic implications for assessing PD risk. Models evolved in the present study have translational value in that they can be incorporated into the Electronic Health Record (EHR) to support POC screening. Additionally, the study has defined relative performance of PD risk prediction models across various MOC environments. Moreover, these findings have established the power ML application can serve to create a decision support tool for dental providers in assessing PD status, severity and inform treatment decisions. Further, such risk scores could also inform medical providers regarding the need for patient referrals and management of comorbid conditions impacted by presence of oral disease such as PD. Finally, this study illustrates the benefit of the integrated medical and dental care delivery environment for detecting risk of periodontitis at a stage when implementation of proven interventions could delay and even prevent disease progression. Keywords: Periodontitis, Risk Assessment, Interprofessional Relations, Machine learning, Electronic Health Records, Decision Support System

Evaluation Measures for Models Assessment over Imbalanced Data Sets

Imbalanced data learning is one of the challenging problems in data mining; among this matter, founding the right model assessment measures is almost a primary research issue. Skewed class distribution causes a misreading of common evaluation measures as well it lead a biased classification. This article presents a set of alternative for imbalanced data learning assessment, using a combined measures (G-means, likelihood ratios, Discriminant power, F-Measure Balanced Accuracy, Youden index, Matthews correlation coefficient), and graphical performance assessment (ROC curve, Area Under Curve, Partial AUC, Weighted AUC, Cumulative Gains Curve and lift chart, Area Under Lift AUL), that aim to provide a more credible evaluation. We analyze the applications of these measures in churn prediction models evaluation, a well known application of imbalanced data Keywords: imbalanced data, Model assessment, accuracy , G-means, likelihood ratios, F-Measure, Youden index, Matthews correlation coefficient, ROC, AUC, P-AUC,W-AUC, Lift, AU