648 research outputs found
Improving Trust in Deep Neural Networks with Nearest Neighbors
Deep neural networks are used increasingly for perception and decision-making in UAVs. For example, they can be used to recognize objects from images and decide what actions the vehicle should take. While deep neural networks can perform very well at complex tasks, their decisions may be unintuitive to a human operator. When a human disagrees with a neural network prediction, due to the black box nature of deep neural networks, it can be unclear whether the system knows something the human does not or whether the system is malfunctioning. This uncertainty is problematic when it comes to ensuring safety. As a result, it is important to develop technologies for explaining neural network decisions for trust and safety. This paper explores a modification to the deep neural network classification layer to produce both a predicted label and an explanation to support its prediction. Specifically, at test time, we replace the final output layer of the neural network classifier by a k-nearest neighbor classifier. The nearest neighbor classifier produces 1) a predicted label through voting and 2) the nearest neighbors involved in the prediction, which represent the most similar examples from the training dataset. Because prediction and explanation are derived from the same underlying process, this approach guarantees that the explanations are always relevant to the predictions. We demonstrate the approach on a convolutional neural network for a UAV image classification task. We perform experiments using a forest trail image dataset and show empirically that the hybrid classifier can produce intuitive explanations without loss of predictive performance compared to the original neural network. We also show how the approach can be used to help identify potential issues in the network and training process
Explainability for Machine Learning Models: From Data Adaptability to User Perception
This thesis explores the generation of local explanations for already
deployed machine learning models, aiming to identify optimal conditions for
producing meaningful explanations considering both data and user requirements.
The primary goal is to develop methods for generating explanations for any
model while ensuring that these explanations remain faithful to the underlying
model and comprehensible to the users.
The thesis is divided into two parts. The first enhances a widely used
rule-based explanation method. It then introduces a novel approach for
evaluating the suitability of linear explanations to approximate a model.
Additionally, it conducts a comparative experiment between two families of
counterfactual explanation methods to analyze the advantages of one over the
other. The second part focuses on user experiments to assess the impact of
three explanation methods and two distinct representations. These experiments
measure how users perceive their interaction with the model in terms of
understanding and trust, depending on the explanations and representations.
This research contributes to a better explanation generation, with potential
implications for enhancing the transparency, trustworthiness, and usability of
deployed AI systems.Comment: PhD Thesi
Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
This survey presents a comprehensive review of current literature on Explainable Artificial Intelligence (XAI) methods for cyber security applications. Due to the rapid development of Internet-connected systems and Artificial Intelligence in recent years, Artificial Intelligence including Machine Learning and Deep Learning has been widely utilized in the fields of cyber security including intrusion detection, malware detection, and spam filtering. However, although Artificial Intelligence-based approaches for the detection and defense of cyber attacks and threats are more advanced and efficient compared to the conventional signature-based and rule-based cyber security strategies, most Machine Learning-based techniques and Deep Learning-based techniques are deployed in the “black-box” manner, meaning that security experts and customers are unable to explain how such procedures reach particular conclusions. The deficiencies of transparencies and interpretability of existing Artificial Intelligence techniques would decrease human users’ confidence in the models utilized for the defense against cyber attacks, especially in current situations where cyber attacks become increasingly diverse and complicated. Therefore, it is essential to apply XAI in the establishment of cyber security models to create more explainable models while maintaining high accuracy and allowing human users to comprehend, trust, and manage the next generation of cyber defense mechanisms. Although there are papers reviewing Artificial Intelligence applications in cyber security areas and the vast literature on applying XAI in many fields including healthcare, financial services, and criminal justice, the surprising fact is that there are currently no survey research articles that concentrate on XAI applications in cyber security. Therefore, the motivation behind the survey is to bridge the research gap by presenting a detailed and up-to-date survey of XAI approaches applicable to issues in the cyber security field. Our work is the first to propose a clear roadmap for navigating the XAI literature in the context of applications in cyber security
It is not "accuracy vs. explainability" -- we need both for trustworthy AI systems
We are witnessing the emergence of an AI economy and society where AI
technologies are increasingly impacting health care, business, transportation
and many aspects of everyday life. Many successes have been reported where AI
systems even surpassed the accuracy of human experts. However, AI systems may
produce errors, can exhibit bias, may be sensitive to noise in the data, and
often lack technical and judicial transparency resulting in reduction in trust
and challenges in their adoption. These recent shortcomings and concerns have
been documented in scientific but also in general press such as accidents with
self driving cars, biases in healthcare, hiring and face recognition systems
for people of color, seemingly correct medical decisions later found to be made
due to wrong reasons etc. This resulted in emergence of many government and
regulatory initiatives requiring trustworthy and ethical AI to provide accuracy
and robustness, some form of explainability, human control and oversight,
elimination of bias, judicial transparency and safety. The challenges in
delivery of trustworthy AI systems motivated intense research on explainable AI
systems (XAI). Aim of XAI is to provide human understandable information of how
AI systems make their decisions. In this paper we first briefly summarize
current XAI work and then challenge the recent arguments of accuracy vs.
explainability for being mutually exclusive and being focused only on deep
learning. We then present our recommendations for the use of XAI in full
lifecycle of high stakes trustworthy AI systems delivery, e.g. development,
validation and certification, and trustworthy production and maintenance
Explanation Needs in App Reviews: Taxonomy and Automated Detection
Explainability, i.e. the ability of a system to explain its behavior to
users, has become an important quality of software-intensive systems. Recent
work has focused on methods for generating explanations for various algorithmic
paradigms (e.g., machine learning, self-adaptive systems). There is relatively
little work on what situations and types of behavior should be explained. There
is also a lack of support for eliciting explainability requirements. In this
work, we explore the need for explanation expressed by users in app reviews. We
manually coded a set of 1,730 app reviews from 8 apps and derived a taxonomy of
Explanation Needs. We also explore several approaches to automatically identify
Explanation Needs in app reviews. Our best classifier identifies Explanation
Needs in 486 unseen reviews of 4 different apps with a weighted F-score of 86%.
Our work contributes to a better understanding of users' Explanation Needs.
Automated tools can help engineers focus on these needs and ultimately elicit
valid Explanation Needs
Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
As systems based on opaque Artificial Intelligence (AI) continue to flourish
in diverse real-world applications, understanding these black box models has
become paramount. In response, Explainable AI (XAI) has emerged as a field of
research with practical and ethical benefits across various domains. This paper
not only highlights the advancements in XAI and its application in real-world
scenarios but also addresses the ongoing challenges within XAI, emphasizing the
need for broader perspectives and collaborative efforts. We bring together
experts from diverse fields to identify open problems, striving to synchronize
research agendas and accelerate XAI in practical applications. By fostering
collaborative discussion and interdisciplinary cooperation, we aim to propel
XAI forward, contributing to its continued success. Our goal is to put forward
a comprehensive proposal for advancing XAI. To achieve this goal, we present a
manifesto of 27 open problems categorized into nine categories. These
challenges encapsulate the complexities and nuances of XAI and offer a road map
for future research. For each problem, we provide promising research directions
in the hope of harnessing the collective intelligence of interested
stakeholders
A clinician’s guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML)
Developing functional machine learning (ML)-based models to address unmet clinical needs requires unique considerations for optimal clinical utility. Recent debates about the rigours, transparency, explainability, and reproducibility of ML models, terms which are defined in this article, have raised concerns about their clinical utility and suitability for integration in current evidence-based practice paradigms. This featured article focuses on increasing the literacy of ML among clinicians by providing them with the knowledge and tools needed to understand and critically appraise clinical studies focused on ML. A checklist is provided for evaluating the rigour and reproducibility of the four ML building blocks: data curation, feature engineering, model development, and clinical deployment. Checklists like this are important for quality assurance and to ensure that ML studies are rigourously and confidently reviewed by clinicians and are guided by domain knowledge of the setting in which the findings will be applied. Bridging the gap between clinicians, healthcare scientists, and ML engineers can address many shortcomings and pitfalls of ML-based solutions and their potential deployment at the bedside
- …