12 research outputs found
Evaluating Semantic Parsing against a Simple Web-based Question Answering Model
Semantic parsing shines at analyzing complex natural language that involves
composition and computation over multiple pieces of evidence. However, datasets
for semantic parsing contain many factoid questions that can be answered from a
single web document. In this paper, we propose to evaluate semantic
parsing-based question answering models by comparing them to a question
answering baseline that queries the web and extracts the answer only from web
snippets, without access to the target knowledge-base. We investigate this
approach on COMPLEXQUESTIONS, a dataset designed to focus on compositional
language, and find that our model obtains reasonable performance (35 F1
compared to 41 F1 of state-of-the-art). We find in our analysis that our model
performs well on complex questions involving conjunctions, but struggles on
questions that involve relation composition and superlatives.Comment: *sem 201
Multimodal Neural Databases
The rise in loosely-structured data available through text, images, and other
modalities has called for new ways of querying them. Multimedia Information
Retrieval has filled this gap and has witnessed exciting progress in recent
years. Tasks such as search and retrieval of extensive multimedia archives have
undergone massive performance improvements, driven to a large extent by recent
developments in multimodal deep learning. However, methods in this field remain
limited in the kinds of queries they support and, in particular, their
inability to answer database-like queries. For this reason, inspired by recent
work on neural databases, we propose a new framework, which we name Multimodal
Neural Databases (MMNDBs). MMNDBs can answer complex database-like queries that
involve reasoning over different input modalities, such as text and images, at
scale. In this paper, we present the first architecture able to fulfill this
set of requirements and test it with several baselines, showing the limitations
of currently available models. The results show the potential of these new
techniques to process unstructured data coming from different modalities,
paving the way for future research in the area. Code to replicate the
experiments will be released at
https://github.com/GiovanniTRA/MultimodalNeuralDatabase
K-LITE: Learning Transferable Visual Models with External Knowledge
Recent state-of-the-art computer vision systems are trained from natural
language supervision, ranging from simple object category names to descriptive
captions. This free form of supervision ensures high generality and usability
of the learned visual models, based on extensive heuristics on data collection
to cover as many visual concepts as possible. Alternatively, learning with
external knowledge about images is a promising way which leverages a much more
structured source of supervision. In this paper, we propose K-LITE
(Knowledge-augmented Language-Image Training and Evaluation), a simple strategy
to leverage external knowledge to build transferable visual systems: In
training, it enriches entities in natural language with WordNet and Wiktionary
knowledge, leading to an efficient and scalable approach to learning image
representations that can understand both visual concepts and their knowledge;
In evaluation, the natural language is also augmented with external knowledge
and then used to reference learned visual concepts (or describe new ones) to
enable zero-shot and few-shot transfer of the pre-trained models. We study the
performance of K-LITE on two important computer vision problems, image
classification and object detection, benchmarking on 20 and 13 different
existing datasets, respectively. The proposed knowledge-augmented models show
significant improvement in transfer learning performance over existing methods.Comment: Preprint. The first three authors contribute equall
Improvements to GeoQA, a Question Answering system for Geospatial Questions
Η παρούσα εργασία αποτελεί μια προσπάθεια για συγκέντρωση, μελέτη και σύγκριση
συστημάτων απάντησης ερωτήσεων όπως τα QUINT, TEMPO και NEQA και του σκελετού
συστημάτων απάντησης ερωτήσεων Frankenstein. Η μελέτη επικεντρώνεται στην
απάντηση ερωτήσεων σε γεωχωρικά δεδομένα και πιο στο σύστημα GeoQA. Το σύστημα
αυτό έχει προταθεί πρόσφατα και ειναι το πρώτο σύστημα απάντησης ερωτήσεων πάνω
σε συνδεδεμένα γεωχωρικά δεδομένα βασιζόμενο σε πρότυπα. Βελτιώνουμε το
παραπάνω σύστημα χρησιμοποιώντας τα δεδομένα για το σχήμα των βάσεων γνώσης
του, προσθέτοντας πρότυπα για πιο σύνθετες ερωτήσεις και αναπτύσσοντας το
υποσύστημα για την επεξεργασία φυσικής γλώσσας.We study the question-answering GeoQA which was proposed recently. GeoQA is the first
template-based question answering system for linked geospatial data. We improve this
system by exploiting the data schema information of the kb’s it’s using, adding more
templates for more complex queries and by improving the natural language processing
module in order to recognize the patterns. The current work is also an attempt to
concentrate, study and compare some other question-answering systems like QUINT,
Qanary methodology and Frankenstein framework for question answering systems
Recommended from our members
PowerAqua: Open Question Answering on the Semantic Web
With the rapid growth of semantic information in the Web, the processes of searching and querying these very large amounts of heterogeneous content have become increasingly challenging. This research tackles the problem of supporting users in querying and exploring information across multiple and heterogeneous Semantic Web (SW) sources.
A review of literature on ontology-based Question Answering reveals the limitations of existing technology. Our approach is based on providing a natural language Question Answering interface for the SW, PowerAqua. The realization of PowerAqua represents a considerable advance with respect to other systems, which restrict their scope to an ontology-specific or homogeneous fraction of the publicly available SW content. To our knowledge, PowerAqua is the only system that is able to take advantage of the semantic data available on the Web to interpret and answer user queries posed in natural language. In particular, PowerAqua is uniquely able to answer queries by combining and aggregating information, which can be distributed across heterogeneous semantic resources.
Here, we provide a complete overview of our work on PowerAqua, including: the research challenges it addresses; its architecture; the techniques we have realised to map queries to semantic data, to integrate partial answers drawn from different semantic resources and to rank alternative answers; and the evaluation studies we have performed, to assess the performance of PowerAqua. We believe our experiences can be extrapolated to a variety of end-user applications that wish to open up to large scale and heterogeneous structured datasets, to be able to exploit effectively what possibly is the greatest wealth of data in the history of Artificial Intelligence
Neural Methods for Effective, Efficient, and Exposure-Aware Information Retrieval
Neural networks with deep architectures have demonstrated significant
performance improvements in computer vision, speech recognition, and natural
language processing. The challenges in information retrieval (IR), however, are
different from these other application areas. A common form of IR involves
ranking of documents--or short passages--in response to keyword-based queries.
Effective IR systems must deal with query-document vocabulary mismatch problem,
by modeling relationships between different query and document terms and how
they indicate relevance. Models should also consider lexical matches when the
query contains rare terms--such as a person's name or a product model
number--not seen during training, and to avoid retrieving semantically related
but irrelevant results. In many real-life IR tasks, the retrieval involves
extremely large collections--such as the document index of a commercial Web
search engine--containing billions of documents. Efficient IR methods should
take advantage of specialized IR data structures, such as inverted index, to
efficiently retrieve from large collections. Given an information need, the IR
system also mediates how much exposure an information artifact receives by
deciding whether it should be displayed, and where it should be positioned,
among other results. Exposure-aware IR systems may optimize for additional
objectives, besides relevance, such as parity of exposure for retrieved items
and content publishers. In this thesis, we present novel neural architectures
and methods motivated by the specific needs and challenges of IR tasks.Comment: PhD thesis, Univ College London (2020
Algorithms for assessing the quality and difficulty of multiple choice exam questions
Multiple Choice Questions (MCQs) have long been the backbone of standardized
testing in academia and industry. Correspondingly, there is a constant need for the
authors of MCQs to write and refine new questions for new versions of standardized
tests as well as to support measuring performance in the emerging massive open online
courses, (MOOCs). Research that explores what makes a question difficult, or what
questions distinguish higher-performing students from lower-performing students can
aid in the creation of the next generation of teaching and evaluation tools.
In the automated MCQ answering component of this thesis, algorithms query for
definitions of scientific terms, process the returned web results, and compare the returned
definitions to the original definition in the MCQ. This automated method for
answering questions is then augmented with a model, based on human performance
data from crowdsourced question sets, for analysis of question difficulty as well as
the discrimination power of the non-answer alternatives. The crowdsourced question
sets come from PeerWise, an open source online college-level question authoring and
answering environment.
The goal of this research is to create an automated method to both answer and
assesses the difficulty of multiple choice inverse definition questions in the domain of
introductory biology. The results of this work suggest that human-authored question
banks provide useful data for building gold standard human performance models. The
methodology for building these performance models has value in other domains that
test the difficulty of questions and the quality of the exam takers