Visual Question Answering: A Survey of Methods and Datasets

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.Comment: 25 page

Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

We present a new kind of question answering dataset, OpenBookQA, modeled after open book exams for assessing human understanding of a subject. The open book that comes with our questions is a set of 1329 elementary level science facts. Roughly 6000 questions probe an understanding of these facts and their application to novel situations. This requires combining an open book fact (e.g., metals conduct electricity) with broad common knowledge (e.g., a suit of armor is made of metal) obtained from other sources. While existing QA datasets over documents or knowledge bases, being generally self-contained, focus on linguistic understanding, OpenBookQA probes a deeper understanding of both the topic---in the context of common knowledge---and the language it is expressed in. Human performance on OpenBookQA is close to 92%, but many state-of-the-art pre-trained QA methods perform surprisingly poorly, worse than several simple neural baselines we develop. Our oracle experiments designed to circumvent the knowledge retrieval bottleneck demonstrate the value of both the open book and additional facts. We leave it as a challenge to solve the retrieval problem in this multi-hop setting and to close the large gap to human performance.Comment: Published as conference long paper at EMNLP 201

Improving Retrieval-Based Question Answering with Deep Inference Models

Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex science questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multi-choice question answering in the science domain. For each question-answer pair, we use standard retrieval-based models to find relevant candidate contexts and decompose the main problem into two different sub-problems. First, assign correctness scores for each candidate answer based on the context using retrieval models from Lucene. Second, we use deep learning architectures to compute if a candidate answer can be inferred from some well-chosen context consisting of sentences retrieved from the knowledge base. In the end, all these solvers are combined using a simple neural network to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy.Comment: 8 pages, 2 figures, 8 tables, accepted at IJCNN 201

Mobile Phone Text Processing and Question-Answering

Mobile phone text messaging between mobile users and information services is a growing area of Information Systems. Users may require the service to provide an answer to queries, or may, in wikistyle, want to contribute to the service by texting in some information within the service’s domain of discourse. Given the volume of such messaging it is essential to do the processing through an automated service. Further, in the case of repeated use of the service, the quality of such a response has the potential to benefit from a dynamic user profile that the service can build up from previous texts of the same user. This project will investigate the potential for creating such intelligent mobile phone services and aims to produce a computational model to enable their efficient implementation. To make the project feasible, the scope of the automated service is considered to lie within a limited domain of, for example, information about entertainment within a specific town centre. The project will assume the existence of a model of objects within the domain of discourse, hence allowing the analysis of texts within the context of a user model and a domain model. Hence, the project will involve the subject areas of natural language processing, language engineering, machine learning, knowledge extraction, and ontological engineering

Text 4 Health: Addressing Consumer Health Information Needs via Text Reference Service

This study seeks to provide empirical evidence about how health-related questions are answered in text reference service in order to further the understanding of how to best use texting as a reference service venue to fulfill people’s health information needs. Two hundred health reference transactions from My Info Quest, the first nation-wide collaborative text reference service, were analyzed identify the types of questions, length of transactions, question-answering behavior, and information sources used in the transactions. Findings indicate that texting-based health reference transactions are usually brief, and cover a wide variety of topics. The most popular questions are those seeking general factual information about human body, medical/health conditions, diseases, or medical concepts/jargons. Great variance is discovered among the question-answering behavior, with only a little more than half of the answers containing citation to information sources. The study will inform the practice of health reference service via texting, and help libraries make evidence-based decisions on establishing service policies and procedures, providing training for librarians, and ultimately implementing the service successfully

PeerWise - The Marmite of Veterinary Student Learning

PeerWise is a free online student-centred collaborative learning tool with which students anonymously author, answer, and evaluate multiple choice questions (MCQs). Features such as commenting on questions, rating questions and comments, and appearing on leaderboards, can encourage healthy competition, engage students in reflection and debate, and enhance their communication skills. PeerWise has been used in diverse subject areas but never previously in Veterinary Medicine. The Veterinary undergraduates at the University of Glasgow are a distinct cohort; academically gifted and often highly strategic in their learning due to time pressures and volume of course material. In 2010-11 we introduced PeerWise into 1st year Veterinary Biomolecular Sciences in the Glasgow Bachelor of Veterinary Medicine and Surgery programme. To scaffold PeerWise use, a short interactive session introduced students to the tool and to the basic principles of good MCQ authorship. Students were asked to author four and answer forty MCQs throughout the academic year. Participation was encouraged by an allocation of up to 5% of the final year mark and inclusion of studentauthored questions in the first summative examination. Our analysis focuses on engagement of the class with the\ud tool and their perceptions of its use. All 141 students in the class engaged with PeerWise and the majority contributed beyond that which was stipulated. Student engagement with PeerWise prior to a summative exam was positively correlated to exam score, yielding a relationship that was highly significant (p<0.001). Student perceptions of PeerWise were predominantly positive with explicit recognition of its value as a learning and revision tool, and more than two thirds of the class in agreement that question authoring and answering reinforced their learning. There was clear polarisation of views, however, and those students who did not like PeerWise were vociferous in their dislike, the biggest criticism being lack of moderation by staff