54 research outputs found
ir_metadata: An Extensible Metadata Schema for IR Experiments
The information retrieval (IR) community has a strong tradition of making the
computational artifacts and resources available for future reuse, allowing the
validation of experimental results. Besides the actual test collections, the
underlying run files are often hosted in data archives as part of conferences
like TREC, CLEF, or NTCIR. Unfortunately, the run data itself does not provide
much information about the underlying experiment. For instance, the single run
file is not of much use without the context of the shared task's website or the
run data archive. In other domains, like the social sciences, it is good
practice to annotate research data with metadata. In this work, we introduce
ir_metadata - an extensible metadata schema for TREC run files based on the
PRIMAD model. We propose to align the metadata annotations to PRIMAD, which
considers components of computational experiments that can affect
reproducibility. Furthermore, we outline important components and information
that should be reported in the metadata and give evidence from the literature.
To demonstrate the usefulness of these metadata annotations, we implement new
features in repro_eval that support the outlined metadata schema for the use
case of reproducibility studies. Additionally, we curate a dataset with run
files derived from experiments with different instantiations of PRIMAD
components and annotate these with the corresponding metadata. In the
experiments, we cover reproducibility experiments that are identified by the
metadata and classified by PRIMAD. With this work, we enable IR researchers to
annotate TREC run files and improve the reuse value of experimental artifacts
even further.Comment: Resource pape
One-Shot Labeling for Automatic Relevance Estimation
Dealing with unjudged documents ("holes") in relevance assessments is a
perennial problem when evaluating search systems with offline experiments.
Holes can reduce the apparent effectiveness of retrieval systems during
evaluation and introduce biases in models trained with incomplete data. In this
work, we explore whether large language models can help us fill such holes to
improve offline evaluations. We examine an extreme, albeit common, evaluation
setting wherein only a single known relevant document per query is available
for evaluation. We then explore various approaches for predicting the relevance
of unjudged documents with respect to a query and the known relevant document,
including nearest neighbor, supervised, and prompting techniques. We find that
although the predictions of these One-Shot Labelers (1SL) frequently disagree
with human assessments, the labels they produce yield a far more reliable
ranking of systems than the single labels do alone. Specifically, the strongest
approaches can consistently reach system ranking correlations of over 0.86 with
the full rankings over a variety of measures. Meanwhile, the approach
substantially increases the reliability of t-tests due to filling holes in
relevance assessments, giving researchers more confidence in results they find
to be significant. Alongside this work, we release an easy-to-use software
package to enable the use of 1SL for evaluation of other ad-hoc collections or
systems.Comment: SIGIR 202
Improving accountability in recommender systems research through reproducibility
Reproducibility is a key requirement for scientific progress. It allows the reproduction of the works of others, and, as a consequence, to fully trust the reported claims and results. In this work, we argue that, by facilitating reproducibility of recommender systems experimentation, we indirectly address the issues of accountability and transparency in recommender systems research from the perspectives of practitioners, designers, and engineers aiming to assess the capabilities of published research works. These issues have become increasingly prevalent in recent literature. Reasons for this include societal movements around intelligent systems and artificial intelligence striving toward fair and objective use of human behavioral data (as in Machine Learning, Information Retrieval, or Human–Computer Interaction). Society has grown to expect explanations and transparency standards regarding the underlying algorithms making automated decisions for and around us. This work surveys existing definitions of these concepts and proposes a coherent terminology for recommender systems research, with the goal to connect reproducibility to accountability. We achieve this by introducing several guidelines and steps that lead to reproducible and, hence, accountable experimental workflows and research. We additionally analyze several instantiations of recommender system implementations available in the literature and discuss the extent to which they fit in the introduced framework. With this work, we aim to shed light on this important problem and facilitate progress in the field by increasing the accountability of researchThis work has been funded by the Ministerio de Ciencia, Innovación y Universidades (reference: PID2019-108965GB-I00
EntropyRank: Unsupervised Keyphrase Extraction via Side-Information Optimization for Language Model-based Text Compression
We propose an unsupervised method to extract keywords and keyphrases from
texts based on a pre-trained language model (LM) and Shannon's information
maximization. Specifically, our method extracts phrases having the highest
conditional entropy under the LM. The resulting set of keyphrases turns out to
solve a relevant information-theoretic problem: if provided as side
information, it leads to the expected minimal binary code length in compressing
the text using the LM and an entropy encoder. Alternately, the resulting set is
an approximation via a causal LM to the set of phrases that minimize the
entropy of the text when conditioned upon it. Empirically, the method provides
results comparable to the most commonly used methods in various keyphrase
extraction benchmark challenges
TripJudge: a relevance judgement test collection for TripClick health retrieval
Computer Systems, Imagery and Medi
A Call for Standardization and Validation of Text Style Transfer Evaluation
Text Style Transfer (TST) evaluation is, in practice, inconsistent.
Therefore, we conduct a meta-analysis on human and automated TST evaluation and
experimentation that thoroughly examines existing literature in the field. The
meta-analysis reveals a substantial standardization gap in human and automated
evaluation. In addition, we also find a validation gap: only few automated
metrics have been validated using human experiments. To this end, we thoroughly
scrutinize both the standardization and validation gap and reveal the resulting
pitfalls. This work also paves the way to close the standardization and
validation gap in TST evaluation by calling out requirements to be met by
future research.Comment: Accepted to Findings of ACL 202
University of Amsterdam at CLEF 2020:Notebook for the Touché Lab on Argument Retrieval at CLEF 2020
MMEAD: MS MARCO Entity Annotations and Disambiguations
MMEAD, or MS MARCO Entity Annotations and Disambiguations, is a resource for
entity links for the MS MARCO datasets. We specify a format to store and share
links for both document and passage collections of MS MARCO. Following this
specification, we release entity links to Wikipedia for documents and passages
in both MS MARCO collections (v1 and v2). Entity links have been produced by
the REL and BLINK systems. MMEAD is an easy-to-install Python package, allowing
users to load the link data and entity embeddings effortlessly. Using MMEAD
takes only a few lines of code. Finally, we show how MMEAD can be used for IR
research that uses entity information. We show how to improve recall@1000 and
MRR@10 on more complex queries on the MS MARCO v1 passage dataset by using this
resource. We also demonstrate how entity expansions can be used for interactive
search applications
Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction
Keyphrase extraction (KPE) is an important task in Natural Language
Processing for many scenarios, which aims to extract keyphrases that are
present in a given document. Many existing supervised methods treat KPE as
sequential labeling, span-level classification, or generative tasks. However,
these methods lack the ability to utilize keyphrase information, which may
result in biased results. In this study, we propose Diff-KPE, which leverages
the supervised Variational Information Bottleneck (VIB) to guide the text
diffusion process for generating enhanced keyphrase representations. Diff-KPE
first generates the desired keyphrase embeddings conditioned on the entire
document and then injects the generated keyphrase embeddings into each phrase
representation. A ranking network and VIB are then optimized together with rank
loss and classification loss, respectively. This design of Diff-KPE allows us
to rank each candidate phrase by utilizing both the information of keyphrases
and the document. Experiments show that Diff-KPE outperforms existing KPE
methods on a large open domain keyphrase extraction benchmark, OpenKP, and a
scientific domain dataset, KP20K.Comment: 10 pages, 2 figure
- …