Visual Question Answering: A Survey of Methods and Datasets

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.Comment: 25 page

Plan Synthesis for Knowledge and Action Bases

We study plan synthesis for a variant of Knowledge and Action Bases (KABs), a rich, dynamic framework, where states are description logic (DL) knowledge bases (KBs) whose extensional part is manipulated by actions that possibly introduce new objects from an infinite domain. We show that plan existence over KABs is undecidable even under severe restrictions. We then focus on state-bounded KABs, a class for which plan existence is decidable, and provide sound and complete plan synthesis algorithms, which combine techniques based on standard planning, DL query answering, and finite-state abstraction. All results hold for any DL with decidable query answering. We finally show that for lightweight DLs, plan synthesis can be compiled into standard ADL planning

Synthesizing and executing plans in Knowledge and Action Bases

We study plan synthesis for a variant of Knowledge and Action Bases (KABs). KABs have been recently introduced as a rich, dynamic framework where states are full-fledged description logic (DL) knowledge bases (KBs) whose extensional part is manipulated by actions that can introduce new objects from an infinite domain. We show that, in general, plan existence over KABs is undecidable even under severe restrictions. We then focus on the class of state-bounded KABs, for which plan existence is decidable, and we provide sound and complete plan synthesis algorithms, through a novel combination of techniques based on standard planning, DL query answering, and finite-state abstractions. All results hold for any DL with decidable query answering. We finally show that for lightweight DLs, plan synthesis can be compiled into standard ADL planning. © 2016, CEUR-WS. All rights reserved

Analytic frameworks for assessing dialogic argumentation in online learning environments

Over the last decade, researchers have developed sophisticated online learning environments to support students engaging in argumentation. This review first considers the range of functionalities incorporated within these online environments. The review then presents five categories of analytic frameworks focusing on (1) formal argumentation structure, (2) normative quality, (3) nature and function of contributions within the dialog, (4) epistemic nature of reasoning, and (5) patterns and trajectories of participant interaction. Example analytic frameworks from each category are presented in detail rich enough to illustrate their nature and structure. This rich detail is intended to facilitate researchers’ identification of possible frameworks to draw upon in developing or adopting analytic methods for their own work. Each framework is applied to a shared segment of student dialog to facilitate this illustration and comparison process. Synthetic discussions of each category consider the frameworks in light of the underlying theoretical perspectives on argumentation, pedagogical goals, and online environmental structures. Ultimately the review underscores the diversity of perspectives represented in this research, the importance of clearly specifying theoretical and environmental commitments throughout the process of developing or adopting an analytic framework, and the role of analytic frameworks in the future development of online learning environments for argumentation

Combining link and content-based information in a Bayesian inference model for entity search

An architectural model of a Bayesian inference network to support entity search in semantic knowledge bases is presented. The model supports the explicit combination of primitive data type and object-level semantics under a single computational framework. A flexible query model is supported capable to reason with the availability of simple semantics in querie

Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering

Many vision and language tasks require commonsense reasoning beyond data-driven image and natural language processing. Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image. Current state-of-the-art systems attempted to solve the task using deep neural architectures and achieved promising performance. However, the resulting systems are generally opaque and they struggle in understanding questions for which extra knowledge is required. In this paper, we present an explicit reasoning layer on top of a set of penultimate neural network based systems. The reasoning layer enables reasoning and answering questions where additional knowledge is required, and at the same time provides an interpretable interface to the end users. Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based engine to reason over a basket of inputs: visual relations, the semantic parse of the question, and background ontological knowledge from word2vec and ConceptNet. Experimental analysis of the answers and the key evidential predicates generated on the VQA dataset validate our approach.Comment: 9 pages, 3 figures, AAAI 201

NETMAT: A knowledge-based grid system analysis tool

The increasing expansion of electric power systems renders the power system operator's task increasingly complex. The integration into energy management systems of further analytical algorithms implies that more data has to be analysed by the control engineer. For these reasons and many others, more sophisticated tools are required by power engineers to ease the pressure under which they perform their task. The advent of knowledge-based systems has led to a new approach to the problem. The combination of expert systems and numerical algorithms can be advantageously exploited to assist the power system engineer in operating the system. This paper presents the development of a knowledge-based tool for grid system analysis. The tool, NETMAT (NETwork Modelling AssistanT) , is to be used to analyse the impact of grid system maintenance and modification procedures and of new generating plants on power utilities, and in particular on their ability to generate and sell electricity. NE TMAT consists of a number of numerical applications interfaced to an expert system shell through specific problem domain knowledge bases. Results are presented based on the use of the IEEE-30 busbar network as a test network

Decidable Reasoning in Terminological Knowledge Representation Systems

Terminological knowledge representation systems (TKRSs) are tools for designing and using knowledge bases that make use of terminological languages (or concept languages). We analyze from a theoretical point of view a TKRS whose capabilities go beyond the ones of presently available TKRSs. The new features studied, often required in practical applications, can be summarized in three main points. First, we consider a highly expressive terminological language, called ALCNR, including general complements of concepts, number restrictions and role conjunction. Second, we allow to express inclusion statements between general concepts, and terminological cycles as a particular case. Third, we prove the decidability of a number of desirable TKRS-deduction services (like satisfiability, subsumption and instance checking) through a sound, complete and terminating calculus for reasoning in ALCNR-knowledge bases. Our calculus extends the general technique of constraint systems. As a byproduct of the proof, we get also the result that inclusion statements in ALCNR can be simulated by terminological cycles, if descriptive semantics is adopted.Comment: See http://www.jair.org/ for any accompanying file