4,546 research outputs found
Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search
In recent years, the proliferation of smart mobile devices has lead to the
gradual integration of search functionality within mobile platforms. This has
created an incentive to move away from the "ten blue links'' metaphor, as
mobile users are less likely to click on them, expecting to get the answer
directly from the snippets. In turn, this has revived the interest in Question
Answering. Then, along came chatbots, conversational systems, and messaging
platforms, where the user needs could be better served with the system asking
follow-up questions in order to better understand the user's intent. While
typically a user would expect a single response at any utterance, a system
could also return multiple options for the user to select from, based on
different system understandings of the user's intent. However, this possibility
should not be overused, as this practice could confuse and/or annoy the user.
How to produce good variable-length lists, given the conflicting objectives of
staying short while maximizing the likelihood of having a correct answer
included in the list, is an underexplored problem. It is also unclear how to
evaluate a system that tries to do that. Here we aim to bridge this gap. In
particular, we define some necessary and some optional properties that an
evaluation measure fit for this purpose should have. We further show that
existing evaluation measures from the IR tradition are not entirely suitable
for this setup, and we propose novel evaluation measures that address it
satisfactorily.Comment: 4 pages, in Proceeding of SIGIR 201
On Horizontal and Vertical Separation in Hierarchical Text Classification
Hierarchy is a common and effective way of organizing data and representing
their relationships at different levels of abstraction. However, hierarchical
data dependencies cause difficulties in the estimation of "separable" models
that can distinguish between the entities in the hierarchy. Extracting
separable models of hierarchical entities requires us to take their relative
position into account and to consider the different types of dependencies in
the hierarchy. In this paper, we present an investigation of the effect of
separability in text-based entity classification and argue that in hierarchical
classification, a separation property should be established between entities
not only in the same layer, but also in different layers. Our main findings are
the followings. First, we analyse the importance of separability on the data
representation in the task of classification and based on that, we introduce a
"Strong Separation Principle" for optimizing expected effectiveness of
classifiers decision based on separation property. Second, we present
Hierarchical Significant Words Language Models (HSWLM) which capture all, and
only, the essential features of hierarchical entities according to their
relative position in the hierarchy resulting in horizontally and vertically
separable models. Third, we validate our claims on real-world data and
demonstrate that how HSWLM improves the accuracy of classification and how it
provides transferable models over time. Although discussions in this paper
focus on the classification problem, the models are applicable to any
information access tasks on data that has, or can be mapped to, a hierarchical
structure.Comment: Full paper (10 pages) accepted for publication in proceedings of ACM
SIGIR International Conference on the Theory of Information Retrieval
(ICTIR'16
Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing
GUI testing checks if a software system behaves as expected when users
interact with its graphical interface, e.g., testing specific functionality or
validating relevant use case scenarios. Currently, deciding what to test at
this high level is a manual task since automated GUI testing tools target lower
level adequacy metrics such as structural code coverage or activity coverage.
We propose DroidAgent, an autonomous GUI testing agent for Android, for
semantic, intent-driven automation of GUI testing. It is based on Large
Language Models and support mechanisms such as long- and short-term memory.
Given an Android app, DroidAgent sets relevant task goals and subsequently
tries to achieve them by interacting with the app. Our empirical evaluation of
DroidAgent using 15 apps from the Themis benchmark shows that it can set up and
perform realistic tasks, with a higher level of autonomy. For example, when
testing a messaging app, DroidAgent created a second account and added a first
account as a friend, testing a realistic use case, without human intervention.
On average, DroidAgent achieved 61% activity coverage, compared to 51% for
current state-of-the-art GUI testing techniques. Further, manual analysis shows
that 317 out of the 374 autonomously created tasks are realistic and relevant
to app functionalities, and also that DroidAgent interacts deeply with the apps
and covers more features.Comment: 10 page
Unbiased Comparative Evaluation of Ranking Functions
Eliciting relevance judgments for ranking evaluation is labor-intensive and
costly, motivating careful selection of which documents to judge. Unlike
traditional approaches that make this selection deterministically,
probabilistic sampling has shown intriguing promise since it enables the design
of estimators that are provably unbiased even when reusing data with missing
judgments. In this paper, we first unify and extend these sampling approaches
by viewing the evaluation problem as a Monte Carlo estimation task that applies
to a large number of common IR metrics. Drawing on the theoretical clarity that
this view offers, we tackle three practical evaluation scenarios: comparing two
systems, comparing systems against a baseline, and ranking systems. For
each scenario, we derive an estimator and a variance-optimizing sampling
distribution while retaining the strengths of sampling-based evaluation,
including unbiasedness, reusability despite missing data, and ease of use in
practice. In addition to the theoretical contribution, we empirically evaluate
our methods against previously used sampling heuristics and find that they
generally cut the number of required relevance judgments at least in half.Comment: Under review; 10 page
Data-driven evaluation metrics for heterogeneous search engine result pages
Evaluation metrics for search typically assume items are homoge- neous. However, in the context of web search, this assumption does not hold. Modern search engine result pages (SERPs) are composed of a variety of item types (e.g., news, web, entity, etc.), and their influence on browsing behavior is largely unknown. In this paper, we perform a large-scale empirical analysis of pop- ular web search queries and investigate how different item types influence how people interact on SERPs. We then infer a user brows- ing model given people’s interactions with SERP items – creating a data-driven metric based on item type. We show that the proposed metric leads to more accurate estimates of: (1) total gain, (2) total time spent, and (3) stopping depth – without requiring extensive parameter tuning or a priori relevance information. These results suggest that item heterogeneity should be accounted for when de- veloping metrics for SERPs. While many open questions remain concerning the applicability and generalizability of data-driven metrics, they do serve as a formal mechanism to link observed user behaviors directly to how performance is measured. From this approach, we can draw new insights regarding the relationship be- tween behavior and performance – and design data-driven metrics based on real user behavior rather than using metrics reliant on some hypothesized model of user browsing behavior
Constrained surprise search for content generation
In procedural content generation, it is often desirable
to create artifacts which not only fulfill certain playability
constraints but are also able to surprise the player with unexpected
potential uses. This paper applies a divergent evolutionary
search method based on surprise to the constrained problem of
generating balanced and efficient sets of weapons for the Unreal
Tournament III shooter game. The proposed constrained surprise
search algorithm ensures that pairs of weapons are sufficiently
balanced and effective while also rewarding unexpected uses of
these weapons during game simulations with artificial agents.
Results in the paper demonstrate that searching for surprise can
create functionally diverse weapons which require new gameplay
patterns of weapon use in the game.This work has been supported, in part, by the FP7 Marie
Curie CIG project AutoGameDesign (project no: 630665) and
the Horizon 2020 project CrossCult (project no: 693150).peer-reviewe
- …