Master your Metrics with Calibration

Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subpopulations/subperiods in a dataset. In this paper, we propose a way to calibrate the metrics so that they can be made invariant to the prior. We conduct a large number of experiments on balanced and imbalanced data to assess the behavior of calibrated metrics and show that they improve interpretability and provide a better control over what is really measured. We describe specific real-world use-cases where calibration is beneficial such as, for instance, model monitoring in production, reporting, or fairness evaluation.Comment: Presented at IDA202

Scenario-based requirements elicitation for user-centric explainable AI

Explainable Artificial Intelligence (XAI) develops technical explanation methods and enable interpretability for human stakeholders on why Artificial Intelligence (AI) and machine learning (ML) models provide certain predictions. However, the trust of those stakeholders into AI models and explanations is still an issue, especially domain experts, who are knowledgeable about their domain but not AI inner workings. Social and user-centric XAI research states it is essential to understand the stakeholder’s requirements to provide explanations tailored to their needs, and enhance their trust in working with AI models. Scenario-based design and requirements elicitation can help bridge the gap between social and operational aspects of a stakeholder early before the adoption of information systems and identify its real problem and practices generating user requirements. Nevertheless, it is still rarely explored the adoption of scenarios in XAI, especially in the domain of fraud detection to supporting experts who are about to work with AI models. We demonstrate the usage of scenario-based requirements elicitation for XAI in a fraud detection context, and develop scenarios derived with experts in banking fraud. We discuss how those scenarios can be adopted to identify user or expert requirements for appropriate explanations in his daily operations and to make decisions on reviewing fraudulent cases in banking. The generalizability of the scenarios for further adoption is validated through a systematic literature review in domains of XAI and visual analytics for fraud detection

Classification of multi-class imbalanced data streams using a dynamic data-balancing technique

The performance of classification algorithms with imbalanced streaming data depends upon efficient re-balancing strategy for learning tasks. The difficulty becomes more elevated with multi-class highly imbalanced streaming data. In this paper, we investigate the multi-class imbalance problem in data streams and develop an adaptive framework to cope with imbalanced data scenarios. The proposed One-Vs-All Adaptive Window re-Balancing with Retain Knowledge (OVA-AWBReK) classification framework will combine OVA binarization with Automated Re-balancing Strategy (ARS) using Racing Algorithm (RA). We conducted experiments on highly imbalanced datasets to demonstrate the use of the proposed OVA-AWBReK framework. The results show that OVA-AWBReK framework can enhance the classification performance of the multi-class highly imbalanced data

When is undersampling effective in unbalanced classification tasks?

info:eu-repo/semantics/publishe

Anti-proliferative and pro-apoptotic effects of short-term inhibition of telomerase in vivo and in human malignant b cells xenografted in zebrafish

Besides its canonical role in stabilizing telomeres, telomerase reverse transcriptase (TERT) may promote tumor growth/progression through extra-telomeric functions. Our previous in vitro studies demonstrated that short-term TERT inhibition by BIBR1532 (BIBR), an inhibitor of TERT catalytic activity, negatively impacts cell proliferation and viability via telomeres\u2019 length-independent mechanism. Here we evaluate the anti-proliferative and pro-apoptotic effects of short-term telomerase inhibition in vivo in wild-type (wt) and tert mutant (terthu3430/hu3430; tert 12/ 12) zebrafish embryos, and in malignant human B cells xenografted in casper zebrafish embryos. Short-term Tert inhibition by BIBR in wt embryos reduced cell proliferation, induced an accumulation of cells in S-phase and ultimately led to apoptosis associated with the activation of DNA damage response; all these effects were unrelated to telomere shortening/dysfunction. BIBR treatment showed no effects in tert 12/ 12 embryos. Xenografted untreated malignant B cells proliferated in zebrafish embryos, while BIBR pretreated cells constantly decreased and were significantly less than those in the controls from 24 to up to 72 h after xenotransplantation. Additionally, xenografted tumor cells, treated with BIBR prior-or post-transplantation, displayed a significant higher apoptotic rate compared to untreated control cells. In conclusion, our data demonstrate that short-term telomerase inhibition impairs proliferation and viability in vivo and in human malignant B cells xenografted in zebrafish, thus supporting therapeutic applications of TERT inhibitors in human malignancies

Graph-based fraud detection with the free energy distance

This paper investigates a real-world application of the free energy distance between nodes of a graph by proposing an improved extension of the existing Fraud Detection System named APATE. It relies on a new way of computing the free energy distance based on paths of increasing length, and scaling on large, sparse, graphs. This new approach is assessed on a real-world large-scale e-commerce payment transactions dataset obtained from a major Belgian credit card issuer. Our results show that the free-energy based approach reduces the computation time by one half while maintaining state-ofthe art performance in term of Precision@100 on fraudulent card prediction