2 research outputs found
Mitigating the Position Bias of Transformer Models in Passage Re-Ranking
Supervised machine learning models and their evaluation strongly depends on
the quality of the underlying dataset. When we search for a relevant piece of
information it may appear anywhere in a given passage. However, we observe a
bias in the position of the correct answer in the text in two popular Question
Answering datasets used for passage re-ranking. The excessive favoring of
earlier positions inside passages is an unwanted artefact. This leads to three
common Transformer-based re-ranking models to ignore relevant parts in unseen
passages. More concerningly, as the evaluation set is taken from the same
biased distribution, the models overfitting to that bias overestimate their
true effectiveness. In this work we analyze position bias on datasets, the
contextualized representations, and their effect on retrieval results. We
propose a debiasing method for retrieval datasets. Our results show that a
model trained on a position-biased dataset exhibits a significant decrease in
re-ranking effectiveness when evaluated on a debiased dataset. We demonstrate
that by mitigating the position bias, Transformer-based re-ranking models are
equally effective on a biased and debiased dataset, as well as more effective
in a transfer-learning setting between two differently biased datasets.Comment: Accepted at ECIR 2021 (Full paper track
Modularized Transfomer-based Ranking Framework
Recent innovations in Transformer-based ranking models have advanced the
state-of-the-art in information retrieval. However, these Transformers are
computationally expensive, and their opaque hidden states make it hard to
understand the ranking process. In this work, we modularize the Transformer
ranker into separate modules for text representation and interaction. We show
how this design enables substantially faster ranking using offline pre-computed
representations and light-weight online interactions. The modular design is
also easier to interpret and sheds light on the ranking process in Transformer
rankers