360 research outputs found
Balancing Speed and Quality in Online Learning to Rank for Information Retrieval
In Online Learning to Rank (OLTR) the aim is to find an optimal ranking model
by interacting with users. When learning from user behavior, systems must
interact with users while simultaneously learning from those interactions.
Unlike other Learning to Rank (LTR) settings, existing research in this field
has been limited to linear models. This is due to the speed-quality tradeoff
that arises when selecting models: complex models are more expressive and can
find the best rankings but need more user interactions to do so, a requirement
that risks frustrating users during training. Conversely, simpler models can be
optimized on fewer interactions and thus provide a better user experience, but
they will converge towards suboptimal rankings. This tradeoff creates a
deadlock, since novel models will not be able to improve either the user
experience or the final convergence point, without sacrificing the other. Our
contribution is twofold. First, we introduce a fast OLTR model called Sim-MGD
that addresses the speed aspect of the speed-quality tradeoff. Sim-MGD ranks
documents based on similarities with reference documents. It converges rapidly
and, hence, gives a better user experience but it does not converge towards the
optimal rankings. Second, we contribute Cascading Multileave Gradient Descent
(C-MGD) for OLTR that directly addresses the speed-quality tradeoff by using a
cascade that enables combinations of the best of two worlds: fast learning and
high quality final convergence. C-MGD can provide the better user experience of
Sim-MGD while maintaining the same convergence as the state-of-the-art MGD
model. This opens the door for future work to design new models for OLTR
without having to deal with the speed-quality tradeoff.Comment: CIKM 2017, Proceedings of the 2017 ACM on Conference on Information
and Knowledge Managemen
Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval
State-of-the-art neural (re)rankers are notoriously data hungry which - given
the lack of large-scale training data in languages other than English - makes
them rarely used in multilingual and cross-lingual retrieval settings. Current
approaches therefore typically transfer rankers trained on English data to
other languages and cross-lingual setups by means of multilingual encoders:
they fine-tune all the parameters of a pretrained massively multilingual
Transformer (MMT, e.g., multilingual BERT) on English relevance judgments and
then deploy it in the target language. In this work, we show that two
parameter-efficient approaches to cross-lingual transfer, namely Sparse
Fine-Tuning Masks (SFTMs) and Adapters, allow for a more lightweight and more
effective zero-shot transfer to multilingual and cross-lingual retrieval tasks.
We first train language adapters (or SFTMs) via Masked Language Modelling and
then train retrieval (i.e., reranking) adapters (SFTMs) on top while keeping
all other parameters fixed. At inference, this modular design allows us to
compose the ranker by applying the task adapter (or SFTM) trained with source
language data together with the language adapter (or SFTM) of a target
language. Besides improved transfer performance, these two approaches offer
faster ranker training, with only a fraction of parameters being updated
compared to full MMT fine-tuning. We benchmark our models on the CLEF-2003
benchmark, showing that our parameter-efficient methods outperform standard
zero-shot transfer with full MMT fine-tuning, while enabling modularity and
reducing training times. Further, we show on the example of Swahili and Somali
that, for low(er)-resource languages, our parameter-efficient neural re-rankers
can improve the ranking of the competitive machine translation-based ranker
Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity
While deep learning (DL) models are state-of-the-art in text and image
domains, they have not yet consistently outperformed Gradient Boosted Decision
Trees (GBDTs) on tabular Learning-To-Rank (LTR) problems. Most of the recent
performance gains attained by DL models in text and image tasks have used
unsupervised pretraining, which exploits orders of magnitude more unlabeled
data than labeled data. To the best of our knowledge, unsupervised pretraining
has not been applied to the LTR problem, which often produces vast amounts of
unlabeled data.
In this work, we study whether unsupervised pretraining of deep models can
improve LTR performance over GBDTs and other non-pretrained models. By
incorporating simple design choices--including SimCLR-Rank, an LTR-specific
pretraining loss--we produce pretrained deep learning models that consistently
(across datasets) outperform GBDTs (and other non-pretrained rankers) in the
case where there is more unlabeled data than labeled data. This performance
improvement occurs not only on average but also on outlier queries. We base our
empirical conclusions off of experiments on (1) public benchmark tabular LTR
datasets, and (2) a large industry-scale proprietary ranking dataset. Code is
provided at https://anonymous.4open.science/r/ltr-pretrain-0DAD/README.md.Comment: ICML-MFPL 2023 Workshop Ora
Selecting which Dense Retriever to use for Zero-Shot Search
We propose the new problem of choosing which dense retrieval model to use
when searching on a new collection for which no labels are available, i.e. in a
zero-shot setting. Many dense retrieval models are readily available. Each
model however is characterized by very differing search effectiveness -- not
just on the test portion of the datasets in which the dense representations
have been learned but, importantly, also across different datasets for which
data was not used to learn the dense representations. This is because dense
retrievers typically require training on a large amount of labeled data to
achieve satisfactory search effectiveness in a specific dataset or domain.
Moreover, effectiveness gains obtained by dense retrievers on datasets for
which they are able to observe labels during training, do not necessarily
generalise to datasets that have not been observed during training. This is
however a hard problem: through empirical experimentation we show that methods
inspired by recent work in unsupervised performance evaluation with the
presence of domain shift in the area of computer vision and machine learning
are not effective for choosing highly performing dense retrievers in our setup.
The availability of reliable methods for the selection of dense retrieval
models in zero-shot settings that do not require the collection of labels for
evaluation would allow to streamline the widespread adoption of dense
retrieval. This is therefore an important new problem we believe the
information retrieval community should consider. Implementation of methods,
along with raw result files and analysis scripts are made publicly available at
https://www.github.com/anonymized
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
We propose a neural sequence-to-sequence model for direction following, a
task that is essential to realizing effective autonomous agents. Our
alignment-based encoder-decoder model with long short-term memory recurrent
neural networks (LSTM-RNN) translates natural language instructions to action
sequences based upon a representation of the observable world state. We
introduce a multi-level aligner that empowers our model to focus on sentence
"regions" salient to the current world state by using multiple abstractions of
the input sentence. In contrast to existing methods, our model uses no
specialized linguistic resources (e.g., parsers) or task-specific annotations
(e.g., seed lexicons). It is therefore generalizable, yet still achieves the
best results reported to-date on a benchmark single-sentence dataset and
competitive results for the limited-training multi-sentence setting. We analyze
our model through a series of ablations that elucidate the contributions of the
primary components of our model.Comment: To appear at AAAI 2016 (and an extended version of a NIPS 2015
Multimodal Machine Learning workshop paper
Stationary Algorithmic Balancing For Dynamic Email Re-Ranking Problem
Email platforms need to generate personalized rankings of emails that satisfy
user preferences, which may vary over time. We approach this as a
recommendation problem based on three criteria: closeness (how relevant the
sender and topic are to the user), timeliness (how recent the email is), and
conciseness (how brief the email is). We propose MOSR (Multi-Objective
Stationary Recommender), a novel online algorithm that uses an adaptive control
model to dynamically balance these criteria and adapt to preference changes. We
evaluate MOSR on the Enron Email Dataset, a large collection of real emails,
and compare it with other baselines. The results show that MOSR achieves better
performance, especially under non-stationary preferences, where users value
different criteria more or less over time. We also test MOSR's robustness on a
smaller down-sampled dataset that exhibits high variance in email
characteristics, and show that it maintains stable rankings across different
samples. Our work offers novel insights into how to design email re-ranking
systems that account for multiple objectives impacting user satisfaction.Comment: Published in KDD'2
- …