36 research outputs found
Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Research community evaluations in information retrieval, such as NIST's Text
REtrieval Conference (TREC), build reusable test collections by pooling
document rankings submitted by many teams. Naturally, the quality of the
resulting test collection thus greatly depends on the number of participating
teams and the quality of their submitted runs. In this work, we investigate: i)
how the number of participants, coupled with other factors, affects the quality
of a test collection; and ii) whether the quality of a test collection can be
inferred prior to collecting relevance judgments from human assessors.
Experiments conducted on six TREC collections illustrate how the number of
teams interacts with various other factors to influence the resulting quality
of test collections. We also show that the reusability of a test collection can
be predicted with high accuracy when the same document collection is used for
successive years in an evaluation campaign, as is common in TREC.Comment: Accepted as a full paper at iConference 202
BroDyn’18: Workshop on analysis of broad dynamic topics over social media
This book constitutes the refereed proceedings of the 40th European Conference on IR Research, ECIR 2018, held in Grenoble, France, in March 2018.
The 39 full papers and 39 short papers presented together with 6 demos, 5 workshops and 3 tutorials, were carefully reviewed and selected from 303 submissions. Accepted papers cover the state of the art in information retrieval including topics such as: topic modeling, deep learning, evaluation, user behavior, document representation, recommendation systems, retrieval methods, learning and classication, and micro-blogs
EveTAR: Building a Large-Scale Multi-Task Test Collection over Arabic Tweets
This article introduces a new language-independent approach for creating a
large-scale high-quality test collection of tweets that supports multiple
information retrieval (IR) tasks without running a shared-task campaign. The
adopted approach (demonstrated over Arabic tweets) designs the collection
around significant (i.e., popular) events, which enables the development of
topics that represent frequent information needs of Twitter users for which
rich content exists. That inherently facilitates the support of multiple tasks
that generally revolve around events, namely event detection, ad-hoc search,
timeline generation, and real-time summarization. The key highlights of the
approach include diversifying the judgment pool via interactive search and
multiple manually-crafted queries per topic, collecting high-quality
annotations via crowd-workers for relevancy and in-house annotators for
novelty, filtering out low-agreement topics and inaccessible tweets, and
providing multiple subsets of the collection for better availability. Applying
our methodology on Arabic tweets resulted in EveTAR , the first
freely-available tweet test collection for multiple IR tasks. EveTAR includes a
crawl of 355M Arabic tweets and covers 50 significant events for which about
62K tweets were judged with substantial average inter-annotator agreement
(Kappa value of 0.71). We demonstrate the usability of EveTAR by evaluating
existing algorithms in the respective tasks. Results indicate that the new
collection can support reliable ranking of IR systems that is comparable to
similar TREC collections, while providing strong baseline results for future
studies over Arabic tweets