13 research outputs found
The Grammar of Interactive Explanatory Model Analysis
The growing need for in-depth analysis of predictive models leads to a series
of new methods for explaining their local and global properties. Which of these
methods is the best? It turns out that this is an ill-posed question. One
cannot sufficiently explain a black-box machine learning model using a single
method that gives only one perspective. Isolated explanations are prone to
misunderstanding, which inevitably leads to wrong or simplistic reasoning. This
problem is known as the Rashomon effect and refers to diverse, even
contradictory interpretations of the same phenomenon. Surprisingly, the
majority of methods developed for explainable machine learning focus on a
single aspect of the model behavior. In contrast, we showcase the problem of
explainability as an interactive and sequential analysis of a model. This paper
presents how different Explanatory Model Analysis (EMA) methods complement each
other and why it is essential to juxtapose them together. The introduced
process of Interactive EMA (IEMA) derives from the algorithmic side of
explainable machine learning and aims to embrace ideas developed in cognitive
sciences. We formalize the grammar of IEMA to describe potential human-model
dialogues. IEMA is implemented in the human-centered framework that adopts
interactivity, customizability and automation as its main traits. Combined,
these methods enhance the responsible approach to predictive modeling.Comment: 17 pages, 10 figures, 3 table
Adversarial Attacks and Defenses in Explainable Artificial Intelligence: A Survey
Explainable artificial intelligence (XAI) methods are portrayed as a remedy
for debugging and trusting statistical and deep learning models, as well as
interpreting their predictions. However, recent advances in adversarial machine
learning (AdvML) highlight the limitations and vulnerabilities of
state-of-the-art explanation methods, putting their security and
trustworthiness into question. The possibility of manipulating, fooling or
fairwashing evidence of the model's reasoning has detrimental consequences when
applied in high-stakes decision-making and knowledge discovery. This survey
provides a comprehensive overview of research concerning adversarial attacks on
explanations of machine learning models, as well as fairness metrics. We
introduce a unified notation and taxonomy of methods facilitating a common
ground for researchers and practitioners from the intersecting research fields
of AdvML and XAI. We discuss how to defend against attacks and design robust
interpretation methods. We contribute a list of existing insecurities in XAI
and outline the emerging research directions in adversarial XAI (AdvXAI).
Future work should address improving explanation methods and evaluation
protocols to take into account the reported safety issues.Comment: A shorter version of this paper was presented at the IJCAI 2023
Workshop on Explainable A
Towards Evaluating Explanations of Vision Transformers for Medical Imaging
As deep learning models increasingly find applications in critical domains
such as medical imaging, the need for transparent and trustworthy
decision-making becomes paramount. Many explainability methods provide insights
into how these models make predictions by attributing importance to input
features. As Vision Transformer (ViT) becomes a promising alternative to
convolutional neural networks for image classification, its interpretability
remains an open research question. This paper investigates the performance of
various interpretation methods on a ViT applied to classify chest X-ray images.
We introduce the notion of evaluating faithfulness, sensitivity, and complexity
of ViT explanations. The obtained results indicate that Layerwise relevance
propagation for transformers outperforms Local interpretable model-agnostic
explanations and Attention visualization, providing a more accurate and
reliable representation of what a ViT has actually learned. Our findings
provide insights into the applicability of ViT explanations in medical imaging
and highlight the importance of using appropriate evaluation criteria for
comparing them.Comment: Accepted by XAI4CV Workshop at CVPR 202
Performance is not enough: the story told by a Rashomon quartet
Predictive modelling is often reduced to finding the best model that
optimizes a selected performance measure. But what if the second-best model
describes the data in a completely different way? What about the third-best? Is
it possible that the equally effective models describe different relationships
in the data? Inspired by Anscombe's quartet, this paper introduces a Rashomon
quartet, a four models built on synthetic dataset which have practically
identical predictive performance. However, their visualization reveals distinct
explanations of the relation between input variables and the target variable.
The illustrative example aims to encourage the use of visualization to compare
predictive models beyond their performance
SurvSHAP(t): Time-dependent explanations of machine learning survival models
Machine and deep learning survival models demonstrate similar or even
improved time-to-event prediction capabilities compared to classical
statistical learning methods yet are too complex to be interpreted by humans.
Several model-agnostic explanations are available to overcome this issue;
however, none directly explain the survival function prediction. In this paper,
we introduce SurvSHAP(t), the first time-dependent explanation that allows for
interpreting survival black-box models. It is based on SHapley Additive
exPlanations with solid theoretical foundations and a broad adoption among
machine learning practitioners. The proposed methods aim to enhance precision
diagnostics and support domain experts in making decisions. Experiments on
synthetic and medical data confirm that SurvSHAP(t) can detect variables with a
time-dependent effect, and its aggregation is a better determinant of the
importance of variables for a prediction than SurvLIME. SurvSHAP(t) is
model-agnostic and can be applied to all models with functional output. We
provide an accessible implementation of time-dependent explanations in Python
at http://github.com/MI2DataLab/survshap
Hospital Length of Stay Prediction Based on Multi-modal Data towards Trustworthy Human-AI Collaboration in Radiomics
To what extent can the patient's length of stay in a hospital be predicted
using only an X-ray image? We answer this question by comparing the performance
of machine learning survival models on a novel multi-modal dataset created from
1235 images with textual radiology reports annotated by humans. Although
black-box models predict better on average than interpretable ones, like Cox
proportional hazards, they are not inherently understandable. To overcome this
trust issue, we introduce time-dependent model explanations into the human-AI
decision making process. Explaining models built on both: human-annotated and
algorithm-extracted radiomics features provides valuable insights for
physicians working in a hospital. We believe the presented approach to be
general and widely applicable to other time-to-event medical use cases. For
reproducibility, we open-source code and the TLOS dataset at
https://github.com/mi2datalab/xlungs-trustworthy-los-prediction.Comment: Accepted at International Conference on Artificial Intelligence in
Medicine (AIME 2023
Explainable AI with counterfactual paths
Explainable AI (XAI) is an increasingly important area of research in machine
learning, which in principle aims to make black-box models transparent and
interpretable. In this paper, we propose a novel approach to XAI that uses
counterfactual paths generated by conditional permutations. Our method provides
counterfactual explanations by identifying alternative paths that could have
led to different outcomes. The proposed method is particularly suitable for
generating explanations based on counterfactual paths in knowledge graphs. By
examining hypothetical changes to the input data in the knowledge graph, we can
systematically validate the behaviour of the model and examine the features or
combination of features that are most important to the model's predictions. Our
approach provides a more intuitive and interpretable explanation for the
model's behaviour than traditional feature weighting methods and can help
identify and mitigate biases in the model
Manipulating SHAP via Adversarial Data Perturbations (Student Abstract)
We introduce a model-agnostic algorithm for manipulating SHapley Additive exPlanations (SHAP) with perturbation of tabular data. It is evaluated on predictive tasks from healthcare and financial domains to illustrate how crucial is the context of data distribution in interpreting machine learning models. Our method supports checking the stability of the explanations used by various stakeholders apparent in the domain of responsible AI; moreover, the result highlights the explanations' vulnerability that can be exploited by an adversary
Responsible Prediction Making of COVID-19 Mortality (Student Abstract)
For high-stakes prediction making, the Responsible Artificial Intelligence (RAI) is more important than ever. It builds upon Explainable Artificial Intelligence (XAI) to advance the efforts in providing fairness, model explainability, and accountability to the AI systems. During the literature review of COVID-19 related prognosis and diagnosis, we found out that most of the predictive models are not faithful to the RAI principles, which can lead to biassed results and wrong reasoning. To solve this problem, we show how novel XAI techniques boost transparency, reproducibility and quality of models