Human-Interpretable Explanations for Black-Box Machine Learning Models: An Application to Fraud Detection

Machine Learning (ML) has been increasingly used to aid humans making high-stakes decisions in a wide range of areas, from public policy to criminal justice, education, healthcare, or financial services. However, it is very hard for humans to grasp the rationale behind every ML model’s prediction, hindering trust in the system. The field of Explainable Artificial Intelligence (XAI) emerged to tackle this problem, aiming to research and develop methods to make those “black-boxes” more interpretable, but there is still no major breakthrough. Additionally, the most popular explanation methods — LIME and SHAP — produce very low-level feature attribution explanations, being of limited usefulness to personas without any ML knowledge. This work was developed at Feedzai, a fintech company that uses ML to prevent financial crime. One of the main Feedzai products is a case management application used by fraud analysts to review suspicious financial transactions flagged by the ML models. Fraud analysts are domain experts trained to look for suspicious evidence in transactions but they do not have ML knowledge, and consequently, current XAI methods do not suit their information needs. To address this, we present JOEL, a neural network-based framework to jointly learn a decision-making task and associated domain knowledge explanations. JOEL is tailored to human-in-the-loop domain experts that lack deep technical ML knowledge, providing high-level insights about the model’s predictions that very much resemble the experts’ own reasoning. Moreover, by collecting the domain feedback from a pool of certified experts (human teaching), we promote seamless and better quality explanations. Lastly, we resort to semantic mappings between legacy expert systems and domain taxonomies to automatically annotate a bootstrap training set, overcoming the absence of concept-based human annotations. We validate JOEL empirically on a real-world fraud detection dataset, at Feedzai. We show that JOEL can generalize the explanations from the bootstrap dataset. Furthermore, obtained results indicate that human teaching is able to further improve the explanations prediction quality.A Aprendizagem de Máquina (AM) tem sido cada vez mais utilizada para ajudar os humanos a tomar decisões de alto risco numa vasta gama de áreas, desde política até à justiça criminal, educação, saúde e serviços financeiros. Porém, é muito difícil para os humanos perceber a razão da decisão do modelo de AM, prejudicando assim a confiança no sistema. O campo da Inteligência Artificial Explicável (IAE) surgiu para enfrentar este problema, visando desenvolver métodos para tornar as “caixas-pretas” mais interpretáveis, embora ainda sem grande avanço. Além disso, os métodos de explicação mais populares — LIME and SHAP — produzem explicações de muito baixo nível, sendo de utilidade limitada para pessoas sem conhecimento de AM. Este trabalho foi desenvolvido na Feedzai, a fintech que usa a AM para prevenir crimes financeiros. Um dos produtos da Feedzai é uma aplicação de gestão de casos, usada por analistas de fraude. Estes são especialistas no domínio treinados para procurar evidências suspeitas em transações financeiras, contudo não tendo o conhecimento em AM, os métodos de IAE atuais não satisfazem as suas necessidades de informação. Para resolver isso, apresentamos JOEL, a framework baseada em rede neuronal para aprender conjuntamente a tarefa de tomada de decisão e as explicações associadas. A JOEL é orientada a especialistas de domínio que não têm conhecimento técnico profundo de AM, fornecendo informações de alto nível sobre as previsões do modelo, que muito se assemelham ao raciocínio dos próprios especialistas. Ademais, ao recolher o feedback de especialistas certificados (ensino humano), promovemos explicações contínuas e de melhor qualidade. Por último, recorremos a mapeamentos semânticos entre sistemas legados e taxonomias de domínio para anotar automaticamente um conjunto de dados, superando a ausência de anotações humanas baseadas em conceitos. Validamos a JOEL empiricamente em um conjunto de dados de detecção de fraude do mundo real, na Feedzai. Mostramos que a JOEL pode generalizar as explicações aprendidas no conjunto de dados inicial e que o ensino humano é capaz de melhorar a qualidade da previsão das explicações

ConceptDistil: Model-Agnostic Distillation of Concept Explanations

Concept-based explanations aims to fill the model interpretability gap for non-technical humans-in-the-loop. Previous work has focused on providing concepts for specific models (eg, neural networks) or data types (eg, images), and by either trying to extract concepts from an already trained network or training self-explainable models through multi-task learning. In this work, we propose ConceptDistil, a method to bring concept explanations to any black-box classifier using knowledge distillation. ConceptDistil is decomposed into two components:(1) a concept model that predicts which domain concepts are present in a given instance, and (2) a distillation model that tries to mimic the predictions of a black-box model using the concept model predictions. We validate ConceptDistil in a real world use-case, showing that it is able to optimize both tasks, bringing concept-explainability to any black-box model.Comment: ICLR 2022 PAIR2Struct Worksho

On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods

Machine Learning (ML) models now inform a wide range of human decisions, but using ``black box'' models carries risks such as relying on spurious correlations or errant data. To address this, researchers have proposed methods for supplementing models with explanations of their predictions. However, robust evaluations of these methods' usefulness in real-world contexts have remained elusive, with experiments tending to rely on simplified settings or proxy tasks. We present an experimental study extending a prior explainable ML evaluation experiment and bringing the setup closer to the deployment setting by relaxing its simplifying assumptions. Our empirical study draws dramatically different conclusions than the prior work, highlighting how seemingly trivial experimental design choices can yield misleading results. Beyond the present experiment, we believe this work holds lessons about the necessity of situating the evaluation of any ML method and choosing appropriate tasks, data, users, and metrics to match the intended deployment contexts

Fish Probiotics: Cell Surface Properties of Fish Intestinal Lactobacilli and <i>Escherichia coli</i>

The properties of intestinal bacteria/probiotics, such as cell surface hydrophobicity (CSH), auto-aggregation, and biofilm formation ability, play an important role in shaping the relationship between the bacteria and the host. The current study aimed to investigate the cell surface properties of fish intestinal bacteria and probiotics. Microbial adhesion to hydrocarbons was tested according to Kos and coauthors. The aggregation abilities of the investigated strains were studied as described by Collado and coauthors. The ability of bacterial isolates to form a biofilm was determined by performing a qualitative analysis using crystal violet staining based on the attachment of bacteria to polystyrene. These studies prove that bacterial cell surface hydrophobicity (CSH) is associated with the growth medium, and the effect of the growth medium on CSH is species-specific and likely also strain-specific. Isolates of intestinal lactobacilli from fish (Salmo ischchan) differed from isolates of non-fish/shrimp origin in the relationship between auto-aggregation and biofilm formation. Average CSH levels for fish lactobacilli and E. coli might were lower compared to those of non-fish origin, which may affect the efficiency of non-fish probiotics use in fisheries due to the peculiarities of the hosts’ aquatic lifestyles

Extended Microbiological Characterization of Göttingen Minipigs in the Context of Xenotransplantation: Detection and Vertical Transmission of Hepatitis E Virus

Xenotransplantation has been proposed as a solution to the shortage of suitable human donors. Pigs are currently favoured as donor animals for xenotransplantation of cells, including islet cells, or organs. To reduce the xenotransplantation-associated risk of infection of the recipient the pig donor should be carefully characterised. Göttingen minipigs from Ellegaard are often used for biomedical research and are regularly tested by their vendor for the presence of numerous bacteria, fungi, viruses and parasites. However, screening for some pathogens transmittable to humans had not been performed.The presence of microorganisms was examined in Göttingen Minipigs by PCR methods. Since zoonotic transmission of porcine hepatitis E virus HEV to humans has been demonstrated, extended search for HEV was considered as a priority. RNA from sera, islet and other cells from 40 minipigs were examined for HEV using different real-time reverse transcription (RT)-PCRs, among them two newly established. In addition, sera were examined by Western blot analysis using two recombinant capsid proteins of HEV as antigens. HEV RNA was not detected in pigs older than one year including gilts, but it was detected in the sera of three of ten animals younger than 1 year. Furthermore, HEV was also detected in the sera of three sows six days after delivery and their offspring, indicating vertical transmission of the virus. PCR amplicons were cloned, sequenced and the viruses were found to belong to the HEV genotype (gt) 3/4. Anti-HEV immunoglobulins G were detected in one sow and maternal antibodies in her six day old piglet. Since Göttingen minipigs were negative for many xenotransplantation-relevant microorganisms, they can now be classified as safe. HEV may be eliminated from the Ellegaard herd by selection of negative animals and/or by treatment of the animals