Evaluating Explanations for Stagnation

In this paper, we evaluate four explanations for economic stagnation that have been proposed in the literature: coordination failures, ineffective mix of occupational choices, insufficient human capital accumulation, and politico-economic considerations. We calibrate models that embody these explanations in the context of the stagnant economies of sub-Saharan Africa. The methodology of calibration is ideally suited for this evaluation, given the paucity of high-quality data, the high degree of model nonlinearity, and the need for conducting counterfactual policy experiments. In addition to studying how closely and robustly these models capture the African situation, we examine the quantitative aspects of their policy implications. We find that calibrations that yield multiple equilibria -- one prosperity and the other stagnation -- are not particularly robust. This tempers optimism about the efficacy of one-shot or temporary development policies suggested by models with multiplicity. However, the calibrated models indicate that small policy interventions are sufficient to trigger development in stagnant economies.Coordination failure, Occupational choice, Human capital accumulation, Political economy, Economic Development, Calibration.

Simple sentences, substitutions, and mistaken evaluations

Many competent speakers initially judge that (i) is true and (ii) is false, though they know that (iii) is true. (i) Superman leaps more tall buildings than Clark Kent. (ii) Superman leaps more tall buildings than Superman. (iii) Superman is identical with Clark Kent. Semantic explanations of these intuitions say that (i) and (ii) really can differ in truthvalue. Pragmatic explanations deny this, and say that the intuitions are due to misleading implicatures. This paper argues that both explanations are incorrect. (i) and (ii) cannot differ in truth-value, yet the intuitions are not due to implicatures, but rather to mistakes in evaluating (i) and (ii)

Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval

While there have been many proposals on making AI algorithms explainable, few have attempted to evaluate the impact of AI-generated explanations on human performance in conducting human-AI collaborative tasks. To bridge the gap, we propose a Twenty-Questions style collaborative image retrieval game, Explanation-assisted Guess Which (ExAG), as a method of evaluating the efficacy of explanations (visual evidence or textual justification) in the context of Visual Question Answering (VQA). In our proposed ExAG, a human user needs to guess a secret image picked by the VQA agent by asking natural language questions to it. We show that overall, when AI explains its answers, users succeed more often in guessing the secret image correctly. Notably, a few correct explanations can readily improve human performance when VQA answers are mostly incorrect as compared to no-explanation games. Furthermore, we also show that while explanations rated as "helpful" significantly improve human performance, "incorrect" and "unhelpful" explanations can degrade performance as compared to no-explanation games. Our experiments, therefore, demonstrate that ExAG is an effective means to evaluate the efficacy of AI-generated explanations on a human-AI collaborative task.Comment: 2019 AAAI Conference on Human Computation and Crowdsourcin

Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations

Post-hoc explanations of machine learning models are crucial for people to understand and act on algorithmic predictions. An intriguing class of explanations is through counterfactuals, hypothetical examples that show people how to obtain a different prediction. We posit that effective counterfactual explanations should satisfy two properties: feasibility of the counterfactual actions given user context and constraints, and diversity among the counterfactuals presented. To this end, we propose a framework for generating and evaluating a diverse set of counterfactual explanations based on determinantal point processes. To evaluate the actionability of counterfactuals, we provide metrics that enable comparison of counterfactual-based methods to other local explanation methods. We further address necessary tradeoffs and point to causal implications in optimizing for counterfactuals. Our experiments on four real-world datasets show that our framework can generate a set of counterfactuals that are diverse and well approximate local decision boundaries, outperforming prior approaches to generating diverse counterfactuals. We provide an implementation of the framework at https://github.com/microsoft/DiCE.Comment: 13 page

Evaluating explanations of artificial intelligence decisions : the explanation quality rubric and survey

The use of Artificial Intelligence (AI) algorithms is growing rapidly (Vilone & Longo, 2020). With this comes an increasing demand for reliable, robust explanations of AI decisions. There is a pressing need for a way to evaluate their quality. This thesis examines these research questions: What would a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations look like? How can a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations be created? Can a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations be used to improve explanations? Current Explainable Artificial Intelligence (XAI) research lacks an accepted, widely employed method for evaluating AI explanations. This thesis offers a method for creating a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations. It uses this to create an evaluation methodology, the XQ Rubric and XQ Survey. The XQ Rubric and Survey are then employed to improve explanations of AI decisions. The thesis asks what constitutes a good explanation in the context of XAI. It provides: 1. a model of good explanation for use in XAI research 2. a method of gathering non-expert evaluations of XAI explanations 3. an evaluation scheme for non-experts to employ in assessing XAI explanations (XQ Rubric and XQ Survey). The thesis begins with a literature review, primarily an exploration of previous attempts to evaluate XAI explanations formally. This is followed by an account of the development and iterative refinement of a solution to the problem, the eXplanation Quality Rubric (XQ Rubric). A Design Science methodology was used to guide the XQ Rubric and XQ Survey development. The thesis limits itself to XAI explanations appropriate for non-experts. It proposes and tests an evaluation rubric and survey method that is both stable and robust: that is, readily usable and consistently reliable in a variety of XAI-explanation tasks.Doctor of Philosoph