32 research outputs found
Question Answering over Curated and Open Web Sources
The last few years have seen an explosion of research on the topic of automated question answering (QA), spanning the communities of information retrieval, natural language processing, and artificial intelligence. This tutorial would cover the highlights of this really active period of growth for QA to give the audience a grasp over the families of algorithms that are currently being used. We partition research contributions by the underlying source from where answers are retrieved: curated knowledge graphs, unstructured text, or hybrid corpora. We choose this dimension of partitioning as it is the most discriminative when it comes to algorithm design. Other key dimensions are covered within each sub-topic: like the complexity of questions addressed, and degrees of explainability and interactivity introduced in the systems. We would conclude the tutorial with the most promising emerging trends in the expanse of QA, that would help new entrants into this field make the best decisions to take the community forward. Much has changed in the community since the last tutorial on QA in SIGIR 2016, and we believe that this timely overview will indeed benefit a large number of conference participants
Conversational Question Answering over Passages by Leveraging Word Proximity Networks
Question answering (QA) over text passages is a problem of long-standing
interest in information retrieval. Recently, the conversational setting has
attracted attention, where a user asks a sequence of questions to satisfy her
information needs around a topic. While this setup is a natural one and similar
to humans conversing with each other, it introduces two key research
challenges: understanding the context left implicit by the user in follow-up
questions, and dealing with ad hoc question formulations. In this work, we
demonstrate CROWN (Conversational passage ranking by Reasoning Over Word
Networks): an unsupervised yet effective system for conversational QA with
passage responses, that supports several modes of context propagation over
multiple turns. To this end, CROWN first builds a word proximity network (WPN)
from large corpora to store statistically significant term co-occurrences. At
answering time, passages are ranked by a combination of their similarity to the
question, and coherence of query terms within: these factors are measured by
reading off node and edge weights from the WPN. CROWN provides an interface
that is both intuitive for end-users, and insightful for experts for
reconfiguration to individual setups. CROWN was evaluated on TREC CAsT data,
where it achieved above-median performance in a pool of neural methods.Comment: SIGIR 2020 Demonstration
Evaluating Mixed-initiative Conversational Search Systems via User Simulation
Clarifying the underlying user information need by asking clarifying
questions is an important feature of modern conversational search system.
However, evaluation of such systems through answering prompted clarifying
questions requires significant human effort, which can be time-consuming and
expensive. In this paper, we propose a conversational User Simulator, called
USi, for automatic evaluation of such conversational search systems. Given a
description of an information need, USi is capable of automatically answering
clarifying questions about the topic throughout the search session. Through a
set of experiments, including automated natural language generation metrics and
crowdsourcing studies, we show that responses generated by USi are both inline
with the underlying information need and comparable to human-generated answers.
Moreover, we make the first steps towards multi-turn interactions, where
conversational search systems asks multiple questions to the (simulated) user
with a goal of clarifying the user need. To this end, we expand on currently
available datasets for studying clarifying questions, i.e., Qulac and ClariQ,
by performing a crowdsourcing-based multi-turn data acquisition. We show that
our generative, GPT2-based model, is capable of providing accurate and natural
answers to unseen clarifying questions in the single-turn setting and discuss
capabilities of our model in the multi-turn setting. We provide the code, data,
and the pre-trained model to be used for further research on the topic
Towards Building Economic Models of Conversational Search
Various conceptual and descriptive models of conversational search have been
proposed in the literature -- while useful, they do not provide insights into
how interaction between the agent and user would change in response to the
costs and benefits of the different interactions. In this paper, we develop two
economic models of conversational search based on patterns previously observed
during conversational search sessions, which we refer to as: Feedback First
where the agent asks clarifying questions then presents results, and Feedback
After where the agent presents results, and then asks follow up questions. Our
models show that the amount of feedback given/requested depends on its
efficiency at improving the initial or subsequent query and the relative cost
of providing said feedback. This theoretical framework for conversational
search provides a number of insights that can be used to guide and inform the
development of conversational search agents. However, empirical work is needed
to estimate the parameters in order to make predictions specific to a given
conversational search setting.Comment: To appear in ECIR 202