6 research outputs found
CEQE: Contextualized Embeddings for Query Expansion
In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes query-focused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch
Event-Centric Query Expansion in Web Search
In search engines, query expansion (QE) is a crucial technique to improve
search experience. Previous studies often rely on long-term search log mining,
which leads to slow updates and is sub-optimal for time-sensitive news
searches. In this work, we present Event-Centric Query Expansion (EQE), a novel
QE system that addresses these issues by mining the best expansion from a
significant amount of potential events rapidly and accurately. This system
consists of four stages, i.e., event collection, event reformulation, semantic
retrieval and online ranking. Specifically, we first collect and filter news
headlines from websites. Then we propose a generation model that incorporates
contrastive learning and prompt-tuning techniques to reformulate these
headlines to concise candidates. Additionally, we fine-tune a dual-tower
semantic model to function as an encoder for event retrieval and explore a
two-stage contrastive training approach to enhance the accuracy of event
retrieval. Finally, we rank the retrieved events and select the optimal one as
QE, which is then used to improve the retrieval of event-related documents.
Through offline analysis and online A/B testing, we observe that the EQE system
significantly improves many metrics compared to the baseline. The system has
been deployed in Tencent QQ Browser Search and served hundreds of millions of
users. The dataset and baseline codes are available at
https://open-event-hub.github.io/eqe .Comment: ACL 2023 Industry Trac
Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval
Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models,
have shown the usefulness of expanding and reweighting the users' initial
queries using information occurring in an initial set of retrieved documents,
known as the pseudo-relevant set. Recently, dense retrieval -- through the use
of neural contextual language models such as BERT for analysing the documents'
and queries' contents and computing their relevance scores -- has shown a
promising performance on several information retrieval tasks still relying on
the traditional inverted index for identifying documents relevant to a query.
Two different dense retrieval families have emerged: the use of single embedded
representations for each passage and query (e.g. using BERT's [CLS] token), or
via multiple representations (e.g. using an embedding for each token of the
query and document). In this work, we conduct the first study into the
potential for multiple representation dense retrieval to be enhanced using
pseudo-relevance feedback. In particular, based on the pseudo-relevant set of
documents identified using a first-pass dense retrieval, we extract
representative feedback embeddings (using KMeans clustering) -- while ensuring
that these embeddings discriminate among passages (based on IDF) -- which are
then added to the query representation. These additional feedback embeddings
are shown to both enhance the effectiveness of a reranking as well as an
additional dense retrieval operation. Indeed, experiments on the MSMARCO
passage ranking dataset show that MAP can be improved by upto 26% on the TREC
2019 query set and 10% on the TREC 2020 query set by the application of our
proposed ColBERT-PRF method on a ColBERT dense retrieval approach.Comment: 10 page
CEQE: Contextualized Embeddings for Query Expansion
In this work we leverage recent advances in context-sensitive language models to improve the task of query expansion. Contextualized word representation models, such as ELMo and BERT, are rapidly replacing static embedding models. We propose a new model, Contextualized Embeddings for Query Expansion (CEQE), that utilizes query-focused contextualized embedding vectors. We study the behavior of contextual representations generated for query expansion in ad-hoc document retrieval. We conduct our experiments on probabilistic retrieval models as well as in combination with neural ranking models. We evaluate CEQE on two standard TREC collections: Robust and Deep Learning. We find that CEQE outperforms static embedding-based expansion methods on multiple collections (by up to 18% on Robust and 31% on Deep Learning on average precision) and also improves over proven probabilistic pseudo-relevance feedback (PRF) models. We further find that multiple passes of expansion and reranking result in continued gains in effectiveness with CEQE-based approaches outperforming other approaches. The final model incorporating neural and CEQE-based expansion score achieves gains of up to 5% in P@20 and 2% in AP on Robust over the state-of-the-art transformer-based re-ranking model, Birch
Pretrained Transformers for Text Ranking: BERT and Beyond
The goal of text ranking is to generate an ordered list of texts retrieved
from a corpus in response to a query. Although the most common formulation of
text ranking is search, instances of the task can also be found in many natural
language processing applications. This survey provides an overview of text
ranking with neural network architectures known as transformers, of which BERT
is the best-known example. The combination of transformers and self-supervised
pretraining has been responsible for a paradigm shift in natural language
processing (NLP), information retrieval (IR), and beyond. In this survey, we
provide a synthesis of existing work as a single point of entry for
practitioners who wish to gain a better understanding of how to apply
transformers to text ranking problems and researchers who wish to pursue work
in this area. We cover a wide range of modern techniques, grouped into two
high-level categories: transformer models that perform reranking in multi-stage
architectures and dense retrieval techniques that perform ranking directly.
There are two themes that pervade our survey: techniques for handling long
documents, beyond typical sentence-by-sentence processing in NLP, and
techniques for addressing the tradeoff between effectiveness (i.e., result
quality) and efficiency (e.g., query latency, model and index size). Although
transformer architectures and pretraining techniques are recent innovations,
many aspects of how they are applied to text ranking are relatively well
understood and represent mature techniques. However, there remain many open
research questions, and thus in addition to laying out the foundations of
pretrained transformers for text ranking, this survey also attempts to
prognosticate where the field is heading