1,288 research outputs found
Adversarial Attacks and Defenses in Explainable Artificial Intelligence: A Survey
Explainable artificial intelligence (XAI) methods are portrayed as a remedy
for debugging and trusting statistical and deep learning models, as well as
interpreting their predictions. However, recent advances in adversarial machine
learning (AdvML) highlight the limitations and vulnerabilities of
state-of-the-art explanation methods, putting their security and
trustworthiness into question. The possibility of manipulating, fooling or
fairwashing evidence of the model's reasoning has detrimental consequences when
applied in high-stakes decision-making and knowledge discovery. This survey
provides a comprehensive overview of research concerning adversarial attacks on
explanations of machine learning models, as well as fairness metrics. We
introduce a unified notation and taxonomy of methods facilitating a common
ground for researchers and practitioners from the intersecting research fields
of AdvML and XAI. We discuss how to defend against attacks and design robust
interpretation methods. We contribute a list of existing insecurities in XAI
and outline the emerging research directions in adversarial XAI (AdvXAI).
Future work should address improving explanation methods and evaluation
protocols to take into account the reported safety issues.Comment: A shorter version of this paper was presented at the IJCAI 2023
Workshop on Explainable A
To Explain or Not to Explain?—Artificial Intelligence Explainability in Clinical Decision Support Systems
Explainability for artificial intelligence (AI) in medicine is a hotly debated topic. Our paper presents a review of the key arguments in favor and against explainability for AI-powered Clinical Decision Support System (CDSS) applied to a concrete use case, namely an AI-powered CDSS currently used in the emergency call setting to identify patients with life-threatening cardiac arrest. More specifically, we performed a normative analysis using socio-technical scenarios to provide a nuanced account of the role of explainability for CDSSs for the concrete use case, allowing for abstractions to a more general level. Our analysis focused on three layers: technical considerations, human factors, and the designated system role in decision-making. Our findings suggest that whether explainability can provide added value to CDSS depends on several key questions: technical feasibility, the level of validation in case of explainable algorithms, the characteristics of the context in which the system is implemented, the designated role in the decision-making process, and the key user group(s). Thus, each CDSS will require an individualized assessment of explainability needs and we provide an example of how such an assessment could look like in practice
Don't trust your eyes: on the (un)reliability of feature visualizations
How do neural networks extract patterns from pixels? Feature visualizations
attempt to answer this important question by visualizing highly activating
patterns through optimization. Today, visualization methods form the foundation
of our knowledge about the internal workings of neural networks, as a type of
mechanistic interpretability. Here we ask: How reliable are feature
visualizations? We start our investigation by developing network circuits that
trick feature visualizations into showing arbitrary patterns that are
completely disconnected from normal network behavior on natural input. We then
provide evidence for a similar phenomenon occurring in standard, unmanipulated
networks: feature visualizations are processed very differently from standard
input, casting doubt on their ability to "explain" how neural networks process
natural images. We underpin this empirical finding by theory proving that the
set of functions that can be reliably understood by feature visualization is
extremely small and does not include general black-box neural networks.
Therefore, a promising way forward could be the development of networks that
enforce certain structures in order to ensure more reliable feature
visualizations
On the Robustness of Explanations of Deep Neural Network Models: A Survey
Explainability has been widely stated as a cornerstone of the responsible and
trustworthy use of machine learning models. With the ubiquitous use of Deep
Neural Network (DNN) models expanding to risk-sensitive and safety-critical
domains, many methods have been proposed to explain the decisions of these
models. Recent years have also seen concerted efforts that have shown how such
explanations can be distorted (attacked) by minor input perturbations. While
there have been many surveys that review explainability methods themselves,
there has been no effort hitherto to assimilate the different methods and
metrics proposed to study the robustness of explanations of DNN models. In this
work, we present a comprehensive survey of methods that study, understand,
attack, and defend explanations of DNN models. We also present a detailed
review of different metrics used to evaluate explanation methods, as well as
describe attributional attack and defense methods. We conclude with lessons and
take-aways for the community towards ensuring robust explanations of DNN model
predictions.Comment: Under Review ACM Computing Surveys "Special Issue on Trustworthy AI
Measuring Perceived Trust in XAI-Assisted Decision-Making by Eliciting a Mental Model
This empirical study proposes a novel methodology to measure users’ perceived trust in an Explainable Artificial Intelligence (XAI) model. To do so, users’ mental models are elicited using Fuzzy Cognitive Maps (FCMs). First, we exploit an interpretable Machine Learning (ML) model to classify suspected COVID-19 patients into positive or negative cases. Then, Medical Experts (MEs) conduct a diagnostic decision-making task based on their knowledge and the predictions and interpretations provided by the XAI model. In order to evaluate the impact of interpretations on perceived trust, explanation satisfaction attributes are rated by MEs through a survey. Then, they are considered as FCM’s concepts to determine their influences on each other and, ultimately, on the perceived trust. Moreover, to consider MEs’ mental subjectivity, fuzzy linguistic variables are used to determine the strength of influences. After reaching the steady state of FCMs, a quantified value is obtained to measure the perceived trust of each ME. The results show that the quantified values can determine whether MEs trust or distrust the XAI model. We analyze this behavior by comparing the quantified values with MEs’ performance in completing diagnostic tasks
Measuring Perceived Trust in XAI-Assisted Decision-Making by Eliciting a Mental Model
This empirical study proposes a novel methodology to measure users' perceived trust in an Explainable Artificial Intelligence (XAI) model. To do so, users' mental models are elicited using Fuzzy Cognitive Maps (FCMs). First, we exploit an interpretable Machine Learning (ML) model to classify suspected COVID-19 patients into positive or negative cases. Then, Medical Experts' (MEs) conduct a diagnostic decision-making task based on their knowledge and then prediction and interpretations provided by the XAI model. In order to evaluate the impact of interpretations on perceived trust, explanation satisfaction attributes are rated by MEs through a survey. Then, they are considered as FCM's concepts to determine their influences on each other and, ultimately, on the perceived trust. Moreover, to consider MEs' mental subjectivity, fuzzy linguistic variables are used to determine the strength of influences. After reaching the steady state of FCMs, a quantified value is obtained to measure the perceived trust of each ME. The results show that the quantified values can determine whether MEs trust or distrust the XAI model. We analyze this behavior by comparing the quantified values with MEs' performance in completing diagnostic tasks
IMPACT OF EXPLAINABLE AI ON COGNITIVE LOAD: INSIGHTS FROM AN EMPIRICAL STUDY
While the emerging research field of explainable artificial intelligence (XAI) claims to address the lack of explainability in high-performance machine learning models, in practice XAI research targets developers rather than actual end-users. Unsurprisingly, end-users are unwilling to use XAI-based decision support systems. Similarly, there is scarce interdisciplinary research on end-users’ behavior during XAI explanations usage, rendering it unknown how explanations may impact cognitive load and further affect end-user performance. Therefore, we conducted an empirical study with 271 prospective physicians, measuring their cognitive load, task performance, and task time for distinct implementation-independent XAI explanation types using a COVID-19 use case. We found that these explanation types strongly influence end-users’ cognitive load, task performance, and task time. Based on these findings, we classified the explanation types in a mental efficiency matrix, ranking local XAI explanation types as best, and thereby providing recommendations for future applications and implications for sociotechnical XAI research
Impact Of Explainable AI On Cognitive Load: Insights From An Empirical Study
While the emerging research field of explainable artificial intelligence
(XAI) claims to address the lack of explainability in high-performance machine
learning models, in practice, XAI targets developers rather than actual
end-users. Unsurprisingly, end-users are often unwilling to use XAI-based
decision support systems. Similarly, there is limited interdisciplinary
research on end-users' behavior during XAI explanations usage, rendering it
unknown how explanations may impact cognitive load and further affect end-user
performance. Therefore, we conducted an empirical study with 271 prospective
physicians, measuring their cognitive load, task performance, and task time for
distinct implementation-independent XAI explanation types using a COVID-19 use
case. We found that these explanation types strongly influence end-users'
cognitive load, task performance, and task time. Further, we contextualized a
mental efficiency metric, ranking local XAI explanation types best, to provide
recommendations for future applications and implications for sociotechnical XAI
research.Comment: Thirty-first European Conference on Information Systems (ECIS 2023
- …