Master your Metrics with Calibration

Machine learning models deployed in real-world applications are often evaluated with precision-based metrics such as F1-score or AUC-PR (Area Under the Curve of Precision Recall). Heavily dependent on the class prior, such metrics make it difficult to interpret the variation of a model's performance over different subpopulations/subperiods in a dataset. In this paper, we propose a way to calibrate the metrics so that they can be made invariant to the prior. We conduct a large number of experiments on balanced and imbalanced data to assess the behavior of calibrated metrics and show that they improve interpretability and provide a better control over what is really measured. We describe specific real-world use-cases where calibration is beneficial such as, for instance, model monitoring in production, reporting, or fairness evaluation.Comment: Presented at IDA202

POS0724 GENDER DIFFERENCES IN THROMBOTIC PRIMARY ANTIPHOSPHOLIPID SYNDROME IN A LARGE COHORT OF PATIENTS FROM FOUR EUROPEAN CENTERS

Background:Autoimmune diseases occur more frequently in females and their course and severity can be affected by gender. Antiphospholipid syndrome (APS) is a systemic autoimmune disorder in which antiphospholipid antibodies (aPL) exert a pathogenic role resulting in vascular thrombosis and/or pregnancy morbidities. Data about gender differences in thrombotic APS (t-APS) are still scarce1,2.Objectives:To evaluate the differences in frequency, disease expression and severity between females and males affected by primary t-APS.Methods:Retrospective study enrolling subjects with a formal diagnosis of primary APS (Miyakis 2006) with vascular thrombosis at onset. Women who presented with obstetric events as first aPL-related manifestation were excluded. All the patients were followed from 1967 to 2019 in four European centers: three French centers and one Italian center.Results:The study included 433 patients (68% females, 32% males). Median age at t-APS onset [31 (24-46) vs 41 (29-53) years, p<0.001] and at diagnosis [34 (27-50) vs 46 (34-57) years, p<0.001] was significantly lower in females.The most common presenting manifestations were venous thrombosis (60%) followed by arterial events (37%) and catastrophic APS (3%). Venous events were more frequent in women as compared to men (64% vs 51% p:0.012 OR:1.7 [1.1-2.5]). Sites of venous thrombosis included: limbs (35%), pulmonary (17%), cerebral (3%), portal and inferior cava (2%) and retinal (1%) veins, without gender differences. The arterial events were more frequent among men (43% vs 34% p:0.053). Strokes (27%) and myocardial infarctions (4%) were the most frequent manifestations, followed by thrombosis of limbs (2%), retina (2%) and abdominal organs (1%). Noteworthy, only men presented with visceral ischemia.During the follow-up, new thrombosis occurred in 41% of patients (179/433). 33% out of them had at least two episodes and these occurred especially among males (22% vs 10% p:0.001 OR:2.5 [1.3-4.8]). New events were mostly of the same type, but ⅓ of patients presented a switch from venous to arterial side and viceversa, with no gender differences.Complete aPL profile was available in 357 subjects: 33% had single aPL positivity, 24% double positivity and 43% triple positivity, with no differences between women and men. About 80% of the patients had a concomitant risk factor (RF) for thrombosis. Established cardiovascular RFs were more represented among men as shown in table 1. In women, estrogenic exposure was the main RFs, present in almost 40% of them.Table 1.MALESn= 137FEMALESn= 296POR [IC 95%]Traditional cardiovascular RFs, n (%)Smoke66 (48)81 (27)<0.0012.5 [1.6-3.8]Arterial hypertension59 (43)75 (25)<0.0012.2 [1.5-3.4]Dyslipidemia52 (38)72 (24)0.0041.9 [1.2-2.9]Diabetes16 (12)15 (5)0.0142.5 [1.8-5.1]Obesity13 (10)38 (13)nsOther thrombophilic factors, n (%)Estrogenic stimuli*0116 (39)-Trauma / surgery / immobilization21 (15)32 (11)nsCongenital thrombophilia9/94 (10)33/204 (16)nsData were compared using contingency tables, p value was calculated with Chi-Squared or Fisher exact test. *= hormonal therapy, pregnancy, post-partumConclusion:This gender-oriented analysis of patients with primary t-APS showed that women had the first vascular event at a younger age and mostly on the venous side, while men presented mainly with arterial events, later in life and suffered from more recurrent events. No differences were observed in the distribution of the aPL profile. The different frequency of arterial and venous events in the two groups could be attributed mainly to the presence of additional RFs rather than to biological gender-specific issues. However, it should be underlined that some RFs, such as the use of estrogens or classic cardiovascular RFs, are exclusive or more represented in one gender rather than the other, making it difficult to assess the link of causality between gender and manifestations of t-APS.References:[1]JF de Carvalho. Rheumatol Int. 2011.[2]LJ Jara. Lupus. 2005.Disclosure of Interests:None declare

Scenario-based requirements elicitation for user-centric explainable AI

Explainable Artificial Intelligence (XAI) develops technical explanation methods and enable interpretability for human stakeholders on why Artificial Intelligence (AI) and machine learning (ML) models provide certain predictions. However, the trust of those stakeholders into AI models and explanations is still an issue, especially domain experts, who are knowledgeable about their domain but not AI inner workings. Social and user-centric XAI research states it is essential to understand the stakeholder’s requirements to provide explanations tailored to their needs, and enhance their trust in working with AI models. Scenario-based design and requirements elicitation can help bridge the gap between social and operational aspects of a stakeholder early before the adoption of information systems and identify its real problem and practices generating user requirements. Nevertheless, it is still rarely explored the adoption of scenarios in XAI, especially in the domain of fraud detection to supporting experts who are about to work with AI models. We demonstrate the usage of scenario-based requirements elicitation for XAI in a fraud detection context, and develop scenarios derived with experts in banking fraud. We discuss how those scenarios can be adopted to identify user or expert requirements for appropriate explanations in his daily operations and to make decisions on reviewing fraudulent cases in banking. The generalizability of the scenarios for further adoption is validated through a systematic literature review in domains of XAI and visual analytics for fraud detection

Pelvic mass, ascites, hydrothorax: A malignant or benign condition? Meigs syndrome with high levels of CA 125

Introduction: Abdominal-pelvic mass, ascites and pleural effusion are suggestive of malignant metastatic ovarian cancer. This triad is also present in a rare benign condition called Meigs syndrome. Rarely this condition is associated with an increased CA 125 level. Case report: A 62-year-old woman with a history of abdominal pain underwent an ultrasound (US) examination and a chest X-ray. The imaging revealed the presence of a large pelvic mass and ascites with a monolateral pleural effusion and a high level of the tumor marker CA 125. The patient underwent a total abdominal hysterectomy, salpingoophorectomy, removal of the pelvic mass, pelvic lymphadenectomy and peritoneal biopsies. The histology showed an ovarian fibrothecoma. Discussion: The US analysis according to international ovarian tumor analysis simple rules revealed "inconclusive results"; the logistic regression model LR2 and Adnex suggested a high risk of malignancy. The presence of ascites and the size of the lesion associated with a high level of CA 125 affected the correct assessment of the risk of malignancy, exposing the patient to overtreatment Conclusions: Meigs syndrome is characterized by the resolution of symptoms after surgical removal of the pelvic mass. However, it mimics the clinical picture of a malignant metastatic ovarian cancer. Clinicians have to exclude ovarian cancer and recognize the syndrome to reduce inappropriate procedures

Classification of multi-class imbalanced data streams using a dynamic data-balancing technique

The performance of classification algorithms with imbalanced streaming data depends upon efficient re-balancing strategy for learning tasks. The difficulty becomes more elevated with multi-class highly imbalanced streaming data. In this paper, we investigate the multi-class imbalance problem in data streams and develop an adaptive framework to cope with imbalanced data scenarios. The proposed One-Vs-All Adaptive Window re-Balancing with Retain Knowledge (OVA-AWBReK) classification framework will combine OVA binarization with Automated Re-balancing Strategy (ARS) using Racing Algorithm (RA). We conducted experiments on highly imbalanced datasets to demonstrate the use of the proposed OVA-AWBReK framework. The results show that OVA-AWBReK framework can enhance the classification performance of the multi-class highly imbalanced data

Rituximab for rapidly progressive juvenile systemic sclerosis

Objective: Juvenile systemic sclerosis (JSSc) with rapidly progressive course is a life-threatening condition associated with a poor prognosis. Recently, rituximab (RTX) has been shown to be a promising treatment for adult patients with SSc. We present a series of four patients with rapidly progressive JSSc successfully treated with RTX. Methods: Clinical, laboratory and functional parameters were collected from four patients with rapidly progressive JSSc treated with RTX for at least 1 year. All patients underwent four yearly courses of i.v. RTX 375 mg/m2 on day 0 and 14, at 3-month intervals. Low dose oral prednisone and MMF were also administered. Data were recorded at baseline and every 6 months and included pulmonary and myocardial function parameters, muscular, vascular and skin changes. The Juvenile Systemic Sclerosis Severity Score (J4S) estimated the overall disease severity over time. Results: Four patients (three males, one female), aged 8-17 years, entered the study. Three patients presented with prevalent cardiac involvement, one with severe pulmonary involvement. After 1 year of RTX treatment, all patients showed significant improvement of J4S, Raynaud's phenomenon and cutaneous involvement. Among those with prevalent cardiac involvement, two showed an improvement of the myocardial function (left ventricular ejection fraction [EF] +37% and +19%, respectively) and in the third arrhythmias disappeared. The patient with severe pulmonary involvement showed a significant improvement of the respiratory function (forced vital capacity +46%, forced expiratory volume in 1 s +33%, diffusing capacity of the lung for carbon monoxide [DLCO] +30%). No major side effects were reported. Conclusions: Our data suggest that a combination of RTX and MMF is effective in arresting the rapid progression of JSSc

Scalable machine learning techniques for Highly Imbalanced Credit Card Fraud Detection: A Comparative Study

In the real world of credit card fraud detection, due to a minority of fraud related transactions, has created a class imbalance problem. With the increase of transactions at massive scale, the imbalanced data is immense and has created a challenging issue on how well Machine Learning (ML) techniques can scale up to efficiently learn to detect fraud from the massive incoming data and to respond faster with high prediction accuracy and reduced misclassification costs. This paper is based on experiments that compared several popular ML techniques and investigated their suitability as a “scalable algorithm” when working with highly imbalanced massive or “Big” datasets. The experiments were conducted on two highly imbalanced datasets using Random Forest, Balanced Bagging Ensemble, and Gaussian Naïve Bayes. We observed that many detection algorithms performed well with medium-sized dataset but struggled to maintain similar predictions when it is massive

When is undersampling effective in unbalanced classification tasks?

info:eu-repo/semantics/publishe