346 research outputs found
Explainable Information Retrieval: A Survey
Explainable information retrieval is an emerging research area aiming to make
transparent and trustworthy information retrieval systems. Given the increasing
use of complex machine learning models in search systems, explainability is
essential in building and auditing responsible information retrieval models.
This survey fills a vital gap in the otherwise topically diverse literature of
explainable information retrieval. It categorizes and discusses recent
explainability methods developed for different application domains in
information retrieval, providing a common framework and unifying perspectives.
In addition, it reflects on the common concern of evaluating explanations and
highlights open challenges and opportunities.Comment: 35 pages, 10 figures. Under revie
Policy-Aware Unbiased Learning to Rank for Top-k Rankings
Counterfactual Learning to Rank (LTR) methods optimize ranking systems using
logged user interactions that contain interaction biases. Existing methods are
only unbiased if users are presented with all relevant items in every ranking.
There is currently no existing counterfactual unbiased LTR method for top-k
rankings. We introduce a novel policy-aware counterfactual estimator for LTR
metrics that can account for the effect of a stochastic logging policy. We
prove that the policy-aware estimator is unbiased if every relevant item has a
non-zero probability to appear in the top-k ranking. Our experimental results
show that the performance of our estimator is not affected by the size of k:
for any k, the policy-aware estimator reaches the same retrieval performance
while learning from top-k feedback as when learning from feedback on the full
ranking. Lastly, we introduce novel extensions of traditional LTR methods to
perform counterfactual LTR and to optimize top-k metrics. Together, our
contributions introduce the first policy-aware unbiased LTR approach that
learns from top-k feedback and optimizes top-k metrics. As a result,
counterfactual LTR is now applicable to the very prevalent top-k ranking
setting in search and recommendation.Comment: SIGIR 2020 full conference pape
On Elastic Language Models
Large-scale pretrained language models have achieved compelling performance
in a wide range of language understanding and information retrieval tasks.
Knowledge distillation offers an opportunity to compress a large language model
to a small one, in order to reach a reasonable latency-performance tradeoff.
However, for scenarios where the number of requests (e.g., queries submitted to
a search engine) is highly variant, the static tradeoff attained by the
compressed language model might not always fit. Once a model is assigned with a
static tradeoff, it could be inadequate in that the latency is too high when
the number of requests is large or the performance is too low when the number
of requests is small. To this end, we propose an elastic language model
(ElasticLM) that elastically adjusts the tradeoff according to the request
stream. The basic idea is to introduce a compute elasticity to the compressed
language model, so that the tradeoff could vary on-the-fly along scalable and
controllable compute. Specifically, we impose an elastic structure to enable
ElasticLM with compute elasticity and design an elastic optimization to learn
ElasticLM under compute elasticity. To serve ElasticLM, we apply an elastic
schedule. Considering the specificity of information retrieval, we adapt
ElasticLM to dense retrieval and reranking and present ElasticDenser and
ElasticRanker respectively. Offline evaluation is conducted on a language
understanding benchmark GLUE; and several information retrieval tasks including
Natural Question, Trivia QA, and MS MARCO. The results show that ElasticLM
along with ElasticDenser and ElasticRanker can perform correctly and
competitively compared with an array of static baselines. Furthermore, online
simulation with concurrency is also carried out. The results demonstrate that
ElasticLM can provide elastic tradeoffs with respect to varying request stream.Comment: 27 pages, 11 figures, 9 table
Denmark's Participation in the Search Engine TREC COVID-19 Challenge: Lessons Learned about Searching for Precise Biomedical Scientific Information on COVID-19
This report describes the participation of two Danish universities,
University of Copenhagen and Aalborg University, in the international search
engine competition on COVID-19 (the 2020 TREC-COVID Challenge) organised by the
U.S. National Institute of Standards and Technology (NIST) and its Text
Retrieval Conference (TREC) division. The aim of the competition was to find
the best search engine strategy for retrieving precise biomedical scientific
information on COVID-19 from the largest, at that point in time, dataset of
curated scientific literature on COVID-19 -- the COVID-19 Open Research Dataset
(CORD-19). CORD-19 was the result of a call to action to the tech community by
the U.S. White House in March 2020, and was shortly thereafter posted on Kaggle
as an AI competition by the Allen Institute for AI, the Chan Zuckerberg
Initiative, Georgetown University's Center for Security and Emerging
Technology, Microsoft, and the National Library of Medicine at the US National
Institutes of Health. CORD-19 contained over 200,000 scholarly articles (of
which more than 100,000 were with full text) about COVID-19, SARS-CoV-2, and
related coronaviruses, gathered from curated biomedical sources. The TREC-COVID
challenge asked for the best way to (a) retrieve accurate and precise
scientific information, in response to some queries formulated by biomedical
experts, and (b) rank this information decreasingly by its relevance to the
query.
In this document, we describe the TREC-COVID competition setup, our
participation to it, and our resulting reflections and lessons learned about
the state-of-art technology when faced with the acute task of retrieving
precise scientific information from a rapidly growing corpus of literature, in
response to highly specialised queries, in the middle of a pandemic
Adapting Learned Sparse Retrieval for Long Documents
Learned sparse retrieval (LSR) is a family of neural retrieval methods that
transform queries and documents into sparse weight vectors aligned with a
vocabulary. While LSR approaches like Splade work well for short passages, it
is unclear how well they handle longer documents. We investigate existing
aggregation approaches for adapting LSR to longer documents and find that
proximal scoring is crucial for LSR to handle long documents. To leverage this
property, we proposed two adaptations of the Sequential Dependence Model (SDM)
to LSR: ExactSDM and SoftSDM. ExactSDM assumes only exact query term
dependence, while SoftSDM uses potential functions that model the dependence of
query terms and their expansion terms (i.e., terms identified using a
transformer's masked language modeling head).
Experiments on the MSMARCO Document and TREC Robust04 datasets demonstrate
that both ExactSDM and SoftSDM outperform existing LSR aggregation approaches
for different document length constraints. Surprisingly, SoftSDM does not
provide any performance benefits over ExactSDM. This suggests that soft
proximity matching is not necessary for modeling term dependence in LSR.
Overall, this study provides insights into handling long documents with LSR,
proposing adaptations that improve its performance.Comment: SIGIR 202
- …