1 research outputs found
Evaluating Temporal Persistence Using Replicability Measures
In real-world Information Retrieval (IR) experiments, the Evaluation
Environment (EE) is exposed to constant change. Documents are added, removed,
or updated, and the information need and the search behavior of users is
evolving. Simultaneously, IR systems are expected to retain a consistent
quality. The LongEval Lab seeks to investigate the longitudinal persistence of
IR systems, and in this work, we describe our participation. We submitted runs
of five advanced retrieval systems, namely a Reciprocal Rank Fusion (RRF)
approach, ColBERT, monoT5, Doc2Query, and E5, to both sub-tasks. Further, we
cast the longitudinal evaluation as a replicability study to better understand
the temporal change observed. As a result, we quantify the persistence of the
submitted runs and see great potential in this evaluation method.Comment: To be published in Proceedings of the Working Notes of CLEF 2023 -
Conference and Labs of the Evaluation Forum, Thessaloniki, Greece 18 - 21,
202