18 research outputs found
Towards Interpretable Deep Learning Models for Knowledge Tracing
As an important technique for modeling the knowledge states of learners, the
traditional knowledge tracing (KT) models have been widely used to support
intelligent tutoring systems and MOOC platforms. Driven by the fast
advancements of deep learning techniques, deep neural network has been recently
adopted to design new KT models for achieving better prediction performance.
However, the lack of interpretability of these models has painfully impeded
their practical applications, as their outputs and working mechanisms suffer
from the intransparent decision process and complex inner structures. We thus
propose to adopt the post-hoc method to tackle the interpretability issue for
deep learning based knowledge tracing (DLKT) models. Specifically, we focus on
applying the layer-wise relevance propagation (LRP) method to interpret
RNN-based DLKT model by backpropagating the relevance from the model's output
layer to its input layer. The experiment results show the feasibility using the
LRP method for interpreting the DLKT model's predictions, and partially
validate the computed relevance scores from both question level and concept
level. We believe it can be a solid step towards fully interpreting the DLKT
models and promote their practical applications in the education domain
Evaluating Explanation Methods for Deep Learning in Security
Deep learning is increasingly used as a building block of security systems.
Unfortunately, neural networks are hard to interpret and typically opaque to
the practitioner. The machine learning community has started to address this
problem by developing methods for explaining the predictions of neural
networks. While several of these approaches have been successfully applied in
the area of computer vision, their application in security has received little
attention so far. It is an open question which explanation methods are
appropriate for computer security and what requirements they need to satisfy.
In this paper, we introduce criteria for comparing and evaluating explanation
methods in the context of computer security. These cover general properties,
such as the accuracy of explanations, as well as security-focused aspects, such
as the completeness, efficiency, and robustness. Based on our criteria, we
investigate six popular explanation methods and assess their utility in
security systems for malware detection and vulnerability discovery. We observe
significant differences between the methods and build on these to derive
general recommendations for selecting and applying explanation methods in
computer security.Comment: IEEE European Symposium on Security and Privacy, 202
Optimizing Explanations by Network Canonization and Hyperparameter Search
Explainable AI (XAI) is slowly becoming a key component for many AI
applications. Rule-based and modified backpropagation XAI approaches however
often face challenges when being applied to modern model architectures
including innovative layer building blocks, which is caused by two reasons.
Firstly, the high flexibility of rule-based XAI methods leads to numerous
potential parameterizations. Secondly, many XAI methods break the
implementation-invariance axiom because they struggle with certain model
components, e.g., BatchNorm layers. The latter can be addressed with model
canonization, which is the process of re-structuring the model to disregard
problematic components without changing the underlying function. While model
canonization is straightforward for simple architectures (e.g., VGG, ResNet),
it can be challenging for more complex and highly interconnected models (e.g.,
DenseNet). Moreover, there is only little quantifiable evidence that model
canonization is beneficial for XAI. In this work, we propose canonizations for
currently relevant model blocks applicable to popular deep neural network
architectures,including VGG, ResNet, EfficientNet, DenseNets, as well as
Relation Networks. We further suggest a XAI evaluation framework with which we
quantify and compare the effect sof model canonization for various XAI methods
in image classification tasks on the Pascal-VOC and ILSVRC2017 datasets, as
well as for Visual Question Answering using CLEVR-XAI. Moreover, addressing the
former issue outlined above, we demonstrate how our evaluation framework can be
applied to perform hyperparameter search for XAI methods to optimize the
quality of explanations
xxAI - Beyond Explainable AI
This is an open access book.
Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans.
Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed.
After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science.https://digitalcommons.unomaha.edu/isqafacbooks/1000/thumbnail.jp
xxAI - Beyond Explainable AI
This is an open access book. Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans. Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed. After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science
Machine Learning Interpretability in Malware Detection
The ever increasing processing power of modern computers, as well as the increased availability of large and complex data sets, has led to an explosion in machine learning research. This has led to increasingly complex machine learning algorithms, such as Convolutional Neural Networks, with increasingly complex applications, such as malware detection. Recently, malware authors have become increasingly successful in bypassing traditional malware detection methods, partly due to advanced evasion techniques such as obfuscation and server-side polymorphism. Further, new programming paradigms such as fileless malware, that is malware that exist only in the main memory (RAM) of the infected host, add to the challenges faced with modern day malware detection. This has led security specialists to turn to machine learning to augment their malware detection systems. However, with this new technology comes new challenges. One of these challenges is the need for interpretability in machine learning. Machine learning interpretability is the process of giving explanations of a machine learning model\u27s predictions to humans. Rather than trying to understand everything that is learnt by the model, it is an attempt to find intuitive explanations which are simple enough and provide relevant information for downstream tasks. Cybersecurity analysts always prefer interpretable solutions because of the need to fine tune these solutions. If malware analysts can\u27t interpret the reason behind a misclassification, they will not accept the non-interpretable or black box detector. In this thesis, we provide an overview of machine learning and discuss its roll in cyber security, the challenges it faces, and potential improvements to current approaches in the literature. We showcase its necessity as a result of new computing paradigms by implementing a proof of concept fileless malware with JavaScript. We then present techniques for interpreting machine learning based detectors which leverage n-gram analysis and put forward a novel and fully interpretable approach for malware detection which uses convolutional neural networks. We also define a novel approach for evaluating the robustness of a machine learning based detector
xxAI - Beyond Explainable AI
This is an open access book. Statistical machine learning (ML) has triggered a renaissance of artificial intelligence (AI). While the most successful ML models, including Deep Neural Networks (DNN), have developed better predictivity, they have become increasingly complex, at the expense of human interpretability (correlation vs. causality). The field of explainable AI (xAI) has emerged with the goal of creating tools and models that are both predictive and interpretable and understandable for humans. Explainable AI is receiving huge interest in the machine learning and AI research communities, across academia, industry, and government, and there is now an excellent opportunity to push towards successful explainable AI applications. This volume will help the research community to accelerate this process, to promote a more systematic use of explainable AI to improve models in diverse applications, and ultimately to better understand how current explainable AI methods need to be improved and what kind of theory of explainable AI is needed. After overviews of current methods and challenges, the editors include chapters that describe new developments in explainable AI. The contributions are from leading researchers in the field, drawn from both academia and industry, and many of the chapters take a clear interdisciplinary approach to problem-solving. The concepts discussed include explainability, causability, and AI interfaces with humans, and the applications include image processing, natural language, law, fairness, and climate science
Explainable AI for Bioinformatics: Methods, Tools, and Applications
Artificial intelligence(AI) systems based on deep neural networks (DNNs) and
machine learning (ML) algorithms are increasingly used to solve critical
problems in bioinformatics, biomedical informatics, and precision medicine.
However, complex DNN or ML models that are unavoidably opaque and perceived as
black-box methods, may not be able to explain why and how they make certain
decisions. Such black-box models are difficult to comprehend not only for
targeted users and decision-makers but also for AI developers. Besides, in
sensitive areas like healthcare, explainability and accountability are not only
desirable properties of AI but also legal requirements -- especially when AI
may have significant impacts on human lives. Explainable artificial
intelligence (XAI) is an emerging field that aims to mitigate the opaqueness of
black-box models and make it possible to interpret how AI systems make their
decisions with transparency. An interpretable ML model can explain how it makes
predictions and which factors affect the model's outcomes. The majority of
state-of-the-art interpretable ML methods have been developed in a
domain-agnostic way and originate from computer vision, automated reasoning, or
even statistics. Many of these methods cannot be directly applied to
bioinformatics problems, without prior customization, extension, and domain
adoption. In this paper, we discuss the importance of explainability with a
focus on bioinformatics. We analyse and comprehensively overview of
model-specific and model-agnostic interpretable ML methods and tools. Via
several case studies covering bioimaging, cancer genomics, and biomedical text
mining, we show how bioinformatics research could benefit from XAI methods and
how they could help improve decision fairness