686 research outputs found
A Survey on Recognizing Textual Entailment as an NLP Evaluation
Recognizing Textual Entailment (RTE) was proposed as a unified evaluation
framework to compare semantic understanding of different NLP systems. In this
survey paper, we provide an overview of different approaches for evaluating and
understanding the reasoning capabilities of NLP systems. We then focus our
discussion on RTE by highlighting prominent RTE datasets as well as advances in
RTE dataset that focus on specific linguistic phenomena that can be used to
evaluate NLP systems on a fine-grained level. We conclude by arguing that when
evaluating NLP systems, the community should utilize newly introduced RTE
datasets that focus on specific linguistic phenomena.Comment: 1st Workshop on Evaluation and Comparison for NLP systems (Eval4NLP)
at EMNLP 2020; 18 page
Understanding comparative questions and retrieving argumentative answers
Making decisions is an integral part of everyday life, yet it can be a difficult and complex process. While peoples’ wants and needs are unlimited, resources are often scarce, making it necessary to research the possible alternatives and weigh the pros and cons before making a decision. Nowadays, the Internet has become the main source of information when it comes to comparing alternatives, making search engines the primary means for collecting new information. However, relying only on term matching is not sufficient to adequately address requests for comparisons. Therefore, search systems should go beyond this approach to effectively address comparative information needs. In this dissertation, I explore from different perspectives how search systems can respond to comparative questions. First, I examine approaches to identifying comparative questions and study their underlying information needs. Second, I investigate a methodology to identify important constituents of comparative questions like the to-be-compared options and to detect the stance of answers towards these comparison options. Then, I address ambiguous comparative search queries by studying an interactive clarification search interface. And finally, addressing answering comparative questions, I investigate retrieval approaches that consider not only the topical relevance of potential answers but also account for the presence of arguments towards the comparison options mentioned in the questions. By addressing these facets, I aim to provide a comprehensive understanding of how to effectively satisfy the information needs of searchers seeking to compare different alternatives
Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA
We present BYOKG, a universal question-answering (QA) system that can operate
on any knowledge graph (KG), requires no human-annotated training data, and can
be ready to use within a day -- attributes that are out-of-scope for current
KGQA systems. BYOKG draws inspiration from the remarkable ability of humans to
comprehend information present in an unseen KG through exploration -- starting
at random nodes, inspecting the labels of adjacent nodes and edges, and
combining them with their prior world knowledge. In BYOKG, exploration
leverages an LLM-backed symbolic agent that generates a diverse set of
query-program exemplars, which are then used to ground a retrieval-augmented
reasoning procedure to predict programs for arbitrary questions. BYOKG is
effective over both small- and large-scale graphs, showing dramatic gains in QA
accuracy over a zero-shot baseline of 27.89 and 58.02 F1 on GrailQA and MetaQA,
respectively. On GrailQA, we further show that our unsupervised BYOKG
outperforms a supervised in-context learning method, demonstrating the
effectiveness of exploration. Lastly, we find that performance of BYOKG
reliably improves with continued exploration as well as improvements in the
base LLM, notably outperforming a state-of-the-art fine-tuned model by 7.08 F1
on a sub-sampled zero-shot split of GrailQA
Scalable, Efficient and Precise Natural Language Processing in the Semantic Web
The Internet of Things (IoT) is an emerging phenomenon in the public space. Users with accessibility needs could especially benefit from these “smart” devices if they were able to interact with them through speech. This thesis presents a Compositional Semantics and framework for developing extensible and expressive Natural Language Query Interfaces to the Semantic Web, addressing privacy and auditability needs in the process. This could be particularly useful in healthcare or legal applications, where confidentiality of information is a key concer
FinnFN 1.0: The Finnish frame semantic database
The article describes the process of creating a Finnish language FrameNet or FinnFN, based on the original English language FrameNet hosted at the International Computer Science Institute in Berkeley, California. We outline the goals and results relating to the FinnFN project and especially to the creation of the FinnFrame corpus. The main aim of the project was to test the universal applicability of frame semantics by annotating real Finnish using the same frames and annotation conventions as in the original Berkeley FrameNet project. From Finnish newspaper corpora, 40,721 sentences were automatically retrieved and manually annotated as example sentences evoking certain frames. This became the FinnFrame corpus. Applying the Berkeley FrameNet annotation conventions to the Finnish language required some modifications due to Finnish morphology, and a convention for annotating individual morphemes within words was introduced for phenomena such as compounding, comparatives and case endings. Various questions about cultural salience across the two languages arose during the project, but problematic situations occurred only in a few examples, which we also discuss in the article. The article shows that, barring a few minor instances, the universality hypothesis of frames is largely confirmed for languages as different as Finnish and English.Peer reviewe
- …