Explanations of Black-Box Model Predictions by Contextual Importance and Utility

The significant advances in autonomous systems together with an immensely wider application domain have increased the need for trustable intelligent systems. Explainable artificial intelligence is gaining considerable attention among researchers and developers to address this requirement. Although there is an increasing number of works on interpretable and transparent machine learning algorithms, they are mostly intended for the technical users. Explanations for the end-user have been neglected in many usable and practical applications. In this work, we present the Contextual Importance (CI) and Contextual Utility (CU) concepts to extract explanations that are easily understandable by experts as well as novice users. This method explains the prediction results without transforming the model into an interpretable one. We present an example of providing explanations for linear and non-linear models to demonstrate the generalizability of the method. CI and CU are numerical values that can be represented to the user in visuals and natural language form to justify actions and explain reasoning for individual instances, situations, and contexts. We show the utility of explanations in car selection example and Iris flower classification by presenting complete (i.e. the causes of an individual prediction) and contrastive explanation (i.e. contrasting instance against the instance of interest). The experimental results show the feasibility and validity of the provided explanation methods

Explainable Artificial Intelligence for Human Decision Support System in the Medical Domain

In this paper, we present the potential of Explainable Artificial Intelligence methods for decision support in medical image analysis scenarios. Using three types of explainable methods applied to the same medical image data set, we aimed to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). In vivo gastral images obtained by a video capsule endoscopy (VCE) were the subject of visual explanations, with the goal of increasing health professionals’ trust in black-box predictions. We implemented two post hoc interpretable machine learning methods, called Local Interpretable Model-Agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP), and an alternative explanation approach, the Contextual Importance and Utility (CIU) method. The produced explanations were assessed by human evaluation. We conducted three user studies based on explanations provided by LIME, SHAP and CIU. Users from different non-medical backgrounds carried out a series of tests in a web-based survey setting and stated their experience and understanding of the given explanations. Three user groups (n = 20, 20, 20) with three distinct forms of explanations were quantitatively analyzed. We found that, as hypothesized, the CIU-explainable method performed better than both LIME and SHAP methods in terms of improving support for human decision-making and being more transparent and thus understandable to users. Additionally, CIU outperformed LIME and SHAP by generating explanations more rapidly. Our findings suggest that there are notable differences in human decision-making between various explanation support settings. In line with that, we present three potential explainable methods that, with future improvements in implementation, can be generalized to different medical data sets and can provide effective decision support to medical experts

Explainable artificial intelligence for Healthcare applications using Random Forest Classifier with LIME and SHAP

With the advances in computationally efficient artificial Intelligence (AI) techniques and their numerous applications in our everyday life, there is a pressing need to understand the computational details hidden in black box AI techniques such as most popular machine learning and deep learning techniques; through more detailed explanations. The origin of explainable AI (xAI) is coined from these challenges and recently gained more attention by the researchers by adding explainability comprehensively in traditional AI systems. This leads to develop an appropriate framework for successful applications of xAI in real life scenarios with respect to innovations, risk mitigation, ethical issues and logical values to the users. In this book chapter, an in-depth analysis of several xAI frameworks and methods including LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) are provided. Random Forest Classifier as black box AI is used on a publicly available Diabetes symptoms dataset with LIME and SHAP for better interpretations. The results obtained are interesting in terms of transparency, valid and trustworthiness in diabetes disease prediction.Comment: Chapter-6: Accepted Book Chapter in: Transparent, Interpretable and Explainable AI Systems, BK Tripathy & Hari Seetha (Editors), CRC Press, May 202

Explainable digital forensics AI: Towards mitigating distrust in AI-based digital forensics analysis using interpretable models

The present level of skepticism expressed by courts, legal practitioners, and the general public over Artificial Intelligence (AI) based digital evidence extraction techniques has been observed, and understandably so. Concerns have been raised about closed-box AI models’ transparency and their suitability for use in digital evidence mining. While AI models are firmly rooted in mathematical, statistical, and computational theories, the argument has centered on their explainability and understandability, particularly in terms of how they arrive at certain conclusions. This paper examines the issues with closed-box models; the goals; and methods of explainability/interpretability. Most importantly, recommendations for interpretable AI-based digital forensics (DF) investigation are proposed

Designing for mathematical abstraction

Our focus is on the design of systems (pedagogical, technical, social) that encourage mathematical abstraction, a process we refer to as designing for abstraction. In this paper, we draw on detailed design experiments from our research on children's understanding about chance and distribution to re-present this work as a case study in designing for abstraction. Through the case study, we elaborate a number of design heuristics that we claim are also identifiable in the broader literature on designing for mathematical abstraction. Our previous work on the micro-evolution of mathematical knowledge indicated that new mathematical abstractions are routinely forged in activity with available tools and representations, coordinated with relatively naïve unstructured knowledge. In this paper, we identify the role of design in steering the micro-evolution of knowledge towards the focus of the designer's aspirations. A significant finding from the current analysis is the identification of a heuristic in designing for abstraction that requires the intentional blurring of the key mathematical concepts with the tools whose use might foster the construction of that abstraction. It is commonly recognized that meaningful design constructs emerge from careful analysis of children's activity in relation to the designer's own framework for mathematical abstraction. The case study in this paper emphasizes the insufficiency of such a model for the relationship between epistemology and design. In fact, the case study characterises the dialectic relationship between epistemological analysis and design, in which the theoretical foundations of designing for abstraction and for the micro-evolution of mathematical knowledge can co-emerge. © 2010 Springer Science+Business Media B.V

Towards an interpretable advertisement click prediction

In the new era of computational advertising, Click prediction models are used to anticipate a click response and guide marketers’ decisions about whom to target and how to personalize. The more prediction tasks achieve impressive performances, the more trust is put in black models to make important decisions in a business domain. Due to the complexity and lack of transparency of the models, posterior explanation methods are needed to identify features’ contributions that envision a global explanation of the model. This thesis develops an advertisement Click prediction using a supervised machine learning framework and uses the KernelSHAP method to provide feature importance insights on the predictions made by the model. The thesis aims to answer: 1) how can marketers integrate Click prediction in their businesses; 2) which are the most impactful features for a click response; 3) how advertisement categories differ from an overall model. For that purpose, is used a publicly available ADS dataset (2016) to train a neural network. The results showed the overall model performance is not substantially different from a segmented categories performance. The output of KernelSHAP showed that even though the visual content is impactful for the likelihood of a Click for all models, each category has its own feature importance pattern influenced by the product the category promotes. The performance metrics presented a high ratio of true negative to true positive due to a class imbalance problem. To mitigate the cost of misclassification is suggested an individual analysis that better fit target business model.Na era da publicidade computacional, os modelos de previsão Clique são utilizados para antecipar uma resposta de clique e guiar decisões­chave como a quem publicitar. Quanto mais preciso é o desempenho, mais confiança é depositada em modelos para tomar decisões importantes num domínio empresarial. Devido à complexidade e falta de transparência dos modelos, são necessários métodos de explicação posteriores para identificar como cada atributo contribuiu para a previsão. Esta tese desenvolve um modelo aprendizagem automática que prevê o Clique em anúncios e utiliza o método KernelSHAP que explica quais os atributos mais relevantes à previsão. A tese tem como objetivo responder a: 1) como podem os profissionais de publicidade integrar a previsão de clique nos seus negócios; 2) quais são as características mais impactantes para obter um Clique; 3) como as categorias de publicidade diferem de um modelo global. Para esse efeito, é utilizado um conjunto de dados publicamente disponível ­ ADS(2016)­ para treinar uma rede neural. Os resultados mostraram que o desempenho global do modelo não é substancialmente diferente do desempenho segmentado de categorias. Os resultados do KernelSHAP mostraram que, embora o conteúdo visual seja impactante para a probabilidade de um Clique em todos os modelos, cada categoria tem o seu próprio padrão de atributos mais importantes para a classificação. Estes são influenciados pelo produto que a categoria promove. As métricas de avaliação apresentam uma discrepância entre previsões corretas entre classes. Para mitigar o custo de uma classificação errada, sugere­se uma análise individual que melhor se ajuste ao modelo de negócio

Design and Evaluation of User-Centered Explanations for Machine Learning Model Predictions in Healthcare

Challenges in interpreting some high-performing models present complications in applying machine learning (ML) techniques to healthcare problems. Recently, there has been rapid growth in research on model interpretability; however, approaches to explaining complex ML models are rarely informed by end-user needs and user evaluations of model interpretability are lacking, especially in healthcare. This makes it challenging to determine what explanation approaches might enable providers to understand model predictions in a comprehensible and useful way. Therefore, I aimed to utilize clinician perspectives to inform the design of explanations for ML-based prediction tools and improve the adoption of these systems in practice. In this dissertation, I proposed a new theoretical framework for designing user-centered explanations for ML-based systems. I then utilized the framework to propose explanation designs for predictions from a pediatric in-hospital mortality risk model. I conducted focus groups with healthcare providers to obtain feedback on the proposed designs, which was used to inform the design of a user-centered explanation. The user-centered explanation was evaluated in a laboratory study to assess its effect on healthcare provider perceptions of the model and decision-making processes. The results demonstrated that the user-centered explanation design improved provider perceptions of utilizing the predictive model in practice, but exhibited no significant effect on provider accuracy, confidence, or efficiency in making decisions. Limitations of the evaluation study design, including a small sample size, may have affected the ability to detect an impact on decision-making. Nonetheless, the predictive model with the user-centered explanation was positively received by healthcare providers, and demonstrated a viable approach to explaining ML model predictions in healthcare. Future work is required to address the limitations of this study and further explore the potential benefits of user-centered explanation designs for predictive models in healthcare. This work contributes a new theoretical framework for user-centered explanation design for ML-based systems that is generalizable outside the domain of healthcare. Moreover, the work provides meaningful insights into the role of model interpretability and explanation in healthcare while advancing the discussion on how to effectively communicate ML model information to healthcare providers