Looking under the hood: tools for diagnosing your question answering engine

Journal ArticleIn this paper we analyze two question answering tasks : the TREC-8 question answering task and a set of reading comprehension exams. First, we show that Q/A systems perform better when there are multiple answer opportunities per question. Next, we analyze common approaches to two subproblems: term overlap for answer sentence identification, and answer typing for short answer extraction. We present general tools for analyzing the strengths and limitations of techniques for these subproblems. Our results quantify the limitations of both term overlap and answer typing to distinguish between competing answer candidates

How to Evaluate your Question Answering System Every Day and Still Get Real Work Done

In this paper, we report on Qaviar, an experimental automated evaluation system for question answering applications. The goal of our research was to find an automatically calculated measure that correlates well with human judges' assessment of answer correctness in the context of question answering tasks. Qaviar judges the response by computing recall against the stemmed content words in the human-generated answer key. It counts the answer correct if it exceeds agiven recall threshold. We determined that the answer correctness predicted by Qaviar agreed with the human 93% to 95% of the time. 41 question-answering systems were ranked by both Qaviar and human assessors, and these rankings correlated with a Kendall's Tau measure of 0.920, compared to a correlation of 0.956 between human assessors on the same data.Comment: 6 pages, 3 figures, to appear in Proceedings of the Second International Conference on Language Resources and Evaluation (LREC 2000

Looking Under the Hood : Tools for Diagnosing your Question Answering Engine

In this paper we analyze two question answering tasks : the TREC-8 question answering task and a set of reading comprehension exams. First, we show that Q/A systems perform better when there are multiple answer opportunities per question. Next, we analyze common approaches to two subproblems: term overlap for answer sentence identification, and answer typing for short answer extraction. We present general tools for analyzing the strengths and limitations of techniques for these subproblems. Our results quantify the limitations of both term overlap and answer typing to distinguish between competing answer candidates.Comment: Revision of paper appearing in the Proceedings of the Workshop on Open-Domain Question Answerin

A discourse-based approach for Arabic question answering

The treatment of complex questions with explanatory answers involves searching for arguments in texts. Because of the prominent role that discourse relations play in reflecting text-producers’ intentions, capturing the underlying structure of text constitutes a good instructor in this issue. From our extensive review, a system for automatic discourse analysis that creates full rhetorical structures in large scale Arabic texts is currently unavailable. This is due to the high computational complexity involved in processing a large number of hypothesized relations associated with large texts. Therefore, more practical approaches should be investigated. This paper presents a new Arabic Text Parser oriented for question answering systems dealing with لماذا “why” and كيف “how to” questions. The Text Parser presented here considers the sentence as the basic unit of text and incorporates a set of heuristics to avoid computational explosion. With this approach, the developed question answering system reached a significant improvement over the baseline with a Recall of 68% and MRR of 0.62

Adoption and effects of software engineering best practices in machine learning

Algorithms and the Foundations of Software technolog

EMPIRICAL METHODS FOR FINE-GRAINED OPINION EXTRACTION FROM TEXT

Opinions are everywhere. The op/ed pages of newspapers, political blogs, and consumer websites like epinions.com are just some examples of the textual opinions available to readers. And there are many consumers who are interested in following these opinions - intelligence analysts who track the opinions of foreign countries, public relation firms who want to ensure positive opinions for their clients, pollsters who want to know the public's opinions about politicians, and companies who want to know customers' opinions about their products. The problem faced by all of these consumers of opinion is that there is such a wealth of text to process that it is hard to read it all. Central to processing the opinions in these text will be solving two specific problems - identifying expressions of opinion, and identifying their hierarchical structure. We demonstrate solutions involving empirical natural language processing techniques. Although empirical, data-driven methods such as these have become the norm in natural language processing, little work has been done in analyzing their impact on the reproducibility, efficiency, and effectiveness of research. We address two specific problems in this area. We introduce a lightweight computational workflow system to improve the reproducibility and efficiency of machine learning and natural language processing experiments. And we investigate the process of feature generation, setting out desiderata for an ideal process and exploring the effectiveness of several alternatives. Both are investigated in the context of the natural language learning tasks set out earlier

CS 141-01, Computer Science I: Programming Fundamentals, Spring 2010

This syllabus was submitted to the Rhodes College Office of Academic Affairs by the course instructor. Uploaded by Archives RSA Josephine Hill.CS 141 offers an introduction to the fundamental principles of programming, abstraction, and design. This course teaches you how to think as a computer scientist, by teaching the process of building abstractions to hide implementation details, and of controlling the intellectual complexity of designing large software systems by decomposing problems into simpler sub-problems

CS 465-01, Artificial Intelligence, Fall 2009

This syllabus was submitted to the Rhodes College Office of Academic Affairs by the course instructor. Uploaded by Archives RSA Josephine Hill.This course provides an introduction to the major subareas and current research directions in artificial intelligence. The course will cover the following topics (not necessarily in this order): • Search • Game-playing • Logical reasoning • Probabilistic reasoning • Machine learning • Natural language processing Depending on time and/or interest, we may also discuss additional topics such as robotics, vision, or planning

CS 141-01/02, Computer Science I: Programming Fundamentals, Fall 2009

This syllabus was submitted to the Rhodes College Office of Academic Affairs by the course instructor. Uploaded by Archives RSA Josephine Hill.CS 141 offers an introduction to the fundamental principles of programming, abstraction, and design. This course teaches you how to think as a computer scientist, by teaching the process of building abstractions to hide implementation details, and of controlling the intellectual complexity of designing large software systems by decomposing problems into simpler sub-problems