442,024 research outputs found
Evaluating Explanations for Stagnation
In this paper, we evaluate four explanations for economic stagnation that have been proposed in the literature: coordination failures, ineffective mix of occupational choices, insufficient human capital accumulation, and politico-economic considerations. We calibrate models that embody these explanations in the context of the stagnant economies of sub-Saharan Africa. The methodology of calibration is ideally suited for this evaluation, given the paucity of high-quality data, the high degree of model nonlinearity, and the need for conducting counterfactual policy experiments. In addition to studying how closely and robustly these models capture the African situation, we examine the quantitative aspects of their policy implications. We find that calibrations that yield multiple equilibria -- one prosperity and the other stagnation -- are not particularly robust. This tempers optimism about the efficacy of one-shot or temporary development policies suggested by models with multiplicity. However, the calibrated models indicate that small policy interventions are sufficient to trigger development in stagnant economies.Coordination failure, Occupational choice, Human capital accumulation, Political economy, Economic Development, Calibration.
Simple sentences, substitutions, and mistaken evaluations
Many competent speakers initially judge that (i) is true and (ii) is false, though
they know that (iii) is true.
(i) Superman leaps more tall buildings than Clark Kent.
(ii) Superman leaps more tall buildings than Superman.
(iii) Superman is identical with Clark Kent.
Semantic explanations of these intuitions say that (i) and (ii) really can differ in truthvalue.
Pragmatic explanations deny this, and say that the intuitions are due to misleading
implicatures. This paper argues that both explanations are incorrect. (i) and (ii) cannot
differ in truth-value, yet the intuitions are not due to implicatures, but rather to mistakes
in evaluating (i) and (ii)
Can You Explain That? Lucid Explanations Help Human-AI Collaborative Image Retrieval
While there have been many proposals on making AI algorithms explainable, few
have attempted to evaluate the impact of AI-generated explanations on human
performance in conducting human-AI collaborative tasks. To bridge the gap, we
propose a Twenty-Questions style collaborative image retrieval game,
Explanation-assisted Guess Which (ExAG), as a method of evaluating the efficacy
of explanations (visual evidence or textual justification) in the context of
Visual Question Answering (VQA). In our proposed ExAG, a human user needs to
guess a secret image picked by the VQA agent by asking natural language
questions to it. We show that overall, when AI explains its answers, users
succeed more often in guessing the secret image correctly. Notably, a few
correct explanations can readily improve human performance when VQA answers are
mostly incorrect as compared to no-explanation games. Furthermore, we also show
that while explanations rated as "helpful" significantly improve human
performance, "incorrect" and "unhelpful" explanations can degrade performance
as compared to no-explanation games. Our experiments, therefore, demonstrate
that ExAG is an effective means to evaluate the efficacy of AI-generated
explanations on a human-AI collaborative task.Comment: 2019 AAAI Conference on Human Computation and Crowdsourcin
Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations
Post-hoc explanations of machine learning models are crucial for people to
understand and act on algorithmic predictions. An intriguing class of
explanations is through counterfactuals, hypothetical examples that show people
how to obtain a different prediction. We posit that effective counterfactual
explanations should satisfy two properties: feasibility of the counterfactual
actions given user context and constraints, and diversity among the
counterfactuals presented. To this end, we propose a framework for generating
and evaluating a diverse set of counterfactual explanations based on
determinantal point processes. To evaluate the actionability of
counterfactuals, we provide metrics that enable comparison of
counterfactual-based methods to other local explanation methods. We further
address necessary tradeoffs and point to causal implications in optimizing for
counterfactuals. Our experiments on four real-world datasets show that our
framework can generate a set of counterfactuals that are diverse and well
approximate local decision boundaries, outperforming prior approaches to
generating diverse counterfactuals. We provide an implementation of the
framework at https://github.com/microsoft/DiCE.Comment: 13 page
Evaluating explanations of artificial intelligence decisions : the explanation quality rubric and survey
The use of Artificial Intelligence (AI) algorithms is growing rapidly (Vilone & Longo, 2020). With this comes an increasing demand for reliable, robust explanations of AI decisions. There is a pressing need for a way to evaluate their quality. This thesis examines these research questions: What would a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations look like? How can a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations be created? Can a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations be used to improve explanations? Current Explainable Artificial Intelligence (XAI) research lacks an accepted, widely employed method for evaluating AI explanations. This thesis offers a method for creating a rigorous, empirically justified, human-centred scheme for evaluating AI-decision explanations. It uses this to create an evaluation methodology, the XQ Rubric and XQ Survey. The XQ Rubric and Survey are then employed to improve explanations of AI decisions. The thesis asks what constitutes a good explanation in the context of XAI. It provides: 1. a model of good explanation for use in XAI research 2. a method of gathering non-expert evaluations of XAI explanations 3. an evaluation scheme for non-experts to employ in assessing XAI explanations (XQ Rubric and XQ Survey). The thesis begins with a literature review, primarily an exploration of previous attempts to evaluate XAI explanations formally. This is followed by an account of the development and iterative refinement of a solution to the problem, the eXplanation Quality Rubric (XQ Rubric). A Design Science methodology was used to guide the XQ Rubric and XQ Survey development. The thesis limits itself to XAI explanations appropriate for non-experts. It proposes and tests an evaluation rubric and survey method that is both stable and robust: that is, readily usable and consistently reliable in a variety of XAI-explanation tasks.Doctor of Philosoph
- …