14,413 research outputs found
Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search
In recent years, the proliferation of smart mobile devices has lead to the
gradual integration of search functionality within mobile platforms. This has
created an incentive to move away from the "ten blue links'' metaphor, as
mobile users are less likely to click on them, expecting to get the answer
directly from the snippets. In turn, this has revived the interest in Question
Answering. Then, along came chatbots, conversational systems, and messaging
platforms, where the user needs could be better served with the system asking
follow-up questions in order to better understand the user's intent. While
typically a user would expect a single response at any utterance, a system
could also return multiple options for the user to select from, based on
different system understandings of the user's intent. However, this possibility
should not be overused, as this practice could confuse and/or annoy the user.
How to produce good variable-length lists, given the conflicting objectives of
staying short while maximizing the likelihood of having a correct answer
included in the list, is an underexplored problem. It is also unclear how to
evaluate a system that tries to do that. Here we aim to bridge this gap. In
particular, we define some necessary and some optional properties that an
evaluation measure fit for this purpose should have. We further show that
existing evaluation measures from the IR tradition are not entirely suitable
for this setup, and we propose novel evaluation measures that address it
satisfactorily.Comment: 4 pages, in Proceeding of SIGIR 201
An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric
Many evaluation metrics have been defined to evaluate the effectiveness
ad-hoc retrieval and search result diversification systems. However, it is
often unclear which evaluation metric should be used to analyze the performance
of retrieval systems given a specific task. Axiomatic analysis is an
informative mechanism to understand the fundamentals of metrics and their
suitability for particular scenarios. In this paper, we define a
constraint-based axiomatic framework to study the suitability of existing
metrics in search result diversification scenarios. The analysis informed the
definition of Rank-Biased Utility (RBU) -- an adaptation of the well-known
Rank-Biased Precision metric -- that takes into account redundancy and the user
effort associated to the inspection of documents in the ranking. Our
experiments over standard diversity evaluation campaigns show that the proposed
metric captures quality criteria reflected by different metrics, being suitable
in the absence of knowledge about particular features of the scenario under
study.Comment: Original version: 10 pages. Preprint of full paper to appear at
SIGIR'18: The 41st International ACM SIGIR Conference on Research &
Development in Information Retrieval, July 8-12, 2018, Ann Arbor, MI, USA.
ACM, New York, NY, US
QCBA: Postoptimization of Quantitative Attributes in Classifiers based on Association Rules
The need to prediscretize numeric attributes before they can be used in
association rule learning is a source of inefficiencies in the resulting
classifier. This paper describes several new rule tuning steps aiming to
recover information lost in the discretization of numeric (quantitative)
attributes, and a new rule pruning strategy, which further reduces the size of
the classification models. We demonstrate the effectiveness of the proposed
methods on postoptimization of models generated by three state-of-the-art
association rule classification algorithms: Classification based on
Associations (Liu, 1998), Interpretable Decision Sets (Lakkaraju et al, 2016),
and Scalable Bayesian Rule Lists (Yang, 2017). Benchmarks on 22 datasets from
the UCI repository show that the postoptimized models are consistently smaller
-- typically by about 50% -- and have better classification performance on most
datasets
- …