38 research outputs found
ir_metadata: An Extensible Metadata Schema for IR Experiments
The information retrieval (IR) community has a strong tradition of making the
computational artifacts and resources available for future reuse, allowing the
validation of experimental results. Besides the actual test collections, the
underlying run files are often hosted in data archives as part of conferences
like TREC, CLEF, or NTCIR. Unfortunately, the run data itself does not provide
much information about the underlying experiment. For instance, the single run
file is not of much use without the context of the shared task's website or the
run data archive. In other domains, like the social sciences, it is good
practice to annotate research data with metadata. In this work, we introduce
ir_metadata - an extensible metadata schema for TREC run files based on the
PRIMAD model. We propose to align the metadata annotations to PRIMAD, which
considers components of computational experiments that can affect
reproducibility. Furthermore, we outline important components and information
that should be reported in the metadata and give evidence from the literature.
To demonstrate the usefulness of these metadata annotations, we implement new
features in repro_eval that support the outlined metadata schema for the use
case of reproducibility studies. Additionally, we curate a dataset with run
files derived from experiments with different instantiations of PRIMAD
components and annotate these with the corresponding metadata. In the
experiments, we cover reproducibility experiments that are identified by the
metadata and classified by PRIMAD. With this work, we enable IR researchers to
annotate TREC run files and improve the reuse value of experimental artifacts
even further.Comment: Resource pape
Priors for Diversity and Novelty on Neural Recommender Systems
[Abstract] PRIN is a neural based recommendation method that allows the incorporation of item prior information into the recommendation process. In this work we study how the system behaves in terms of novelty and diversity under different configurations of item prior probability estimations. Our results show the versatility of the framework and how its behavior can be adapted to the desired properties, whether accuracy is preferred or diversity and novelty are the desired properties, or how a balance can be achieved with the proper selection of prior estimations.Ministerio de Ciencia, Innovación y Universidades; RTI2018-093336-B-C22Xunta de Galicia; GPC ED431B 2019/03Xunta de Galicia; ED431G/01Ministerio de Ciencia, Innovación y Universidades; FPU17/03210Ministerio de Ciencia, Innovación y Universidades; FPU014/0172
Local and global query expansion for hierarchical complex topics
In this work we study local and global methods for query expansion for multifaceted complex topics. We study word-based and entity-based expansion methods and extend these approaches to complex topics using fine-grained expansion on different elements of the hierarchical query structure. For a source of hierarchical complex topics we use the TREC Complex Answer Retrieval (CAR) benchmark data collection. We find that leveraging the hierarchical topic structure is needed for both local and global expansion methods to be effective. Further, the results demonstrate that entity-based expansion methods show significant gains over word-based models alone, with local feedback providing the largest improvement. The results on the CAR paragraph retrieval task demonstrate that expansion models that incorporate both the hierarchical query structure and entity-based expansion result in a greater than 20% improvement over word-based expansion approaches
How to Perform Reproducible Experiments in the ELLIOT Recommendation Framework: Data Processing, Model Selection, and Performance Evaluation
Recommender Systems have shown to be an efective way to alleviate the over-choice problem and provide
accurate and tailored recommendations. However, the impressive number of proposed recommendation
algorithms, splitting strategies, evaluation protocols, metrics, and tasks, has made rigorous experimental
evaluation particularly challenging. ELLIOT is a comprehensive recommendation framework that aims
to run and reproduce an entire experimental pipeline by processing a simple confguration fle. The
framework loads, flters, and splits the data considering a vast set of strategies. Then, it optimizes
hyperparameters for several recommendation algorithms, selects the best models, compares them with
the baselines, computes metrics spanning from accuracy to beyond-accuracy, bias, and fairness, and
conducts statistical analysis. The aim is to provide researchers a tool to ease all the experimental
evaluation phases (and make them reproducible), from data reading to results collection. ELLIOT is
freely available on GitHub at https://github.com/sisinflab/ellio
Improving accountability in recommender systems research through reproducibility
Reproducibility is a key requirement for scientific progress. It allows the reproduction of the works of others, and, as a consequence, to fully trust the reported claims and results. In this work, we argue that, by facilitating reproducibility of recommender systems experimentation, we indirectly address the issues of accountability and transparency in recommender systems research from the perspectives of practitioners, designers, and engineers aiming to assess the capabilities of published research works. These issues have become increasingly prevalent in recent literature. Reasons for this include societal movements around intelligent systems and artificial intelligence striving toward fair and objective use of human behavioral data (as in Machine Learning, Information Retrieval, or Human–Computer Interaction). Society has grown to expect explanations and transparency standards regarding the underlying algorithms making automated decisions for and around us. This work surveys existing definitions of these concepts and proposes a coherent terminology for recommender systems research, with the goal to connect reproducibility to accountability. We achieve this by introducing several guidelines and steps that lead to reproducible and, hence, accountable experimental workflows and research. We additionally analyze several instantiations of recommender system implementations available in the literature and discuss the extent to which they fit in the introduced framework. With this work, we aim to shed light on this important problem and facilitate progress in the field by increasing the accountability of researchThis work has been funded by the Ministerio de Ciencia, Innovación y Universidades (reference: PID2019-108965GB-I00
MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion
Query expansion is a commonly-used technique in many search systems to better
represent users' information needs with additional query terms. Existing
studies for this task usually propose to expand a query with retrieved or
generated contextual documents. However, both types of methods have clear
limitations. For retrieval-based methods, the documents retrieved with the
original query might not be accurate enough to reveal the search intent,
especially when the query is brief or ambiguous. For generation-based methods,
existing models can hardly be trained or aligned on a particular corpus, due to
the lack of corpus-specific labeled data. In this paper, we propose a novel
Large Language Model (LLM) based mutual verification framework for query
expansion, which alleviates the aforementioned limitations. Specifically, we
first design a query-query-document generation pipeline, which can effectively
leverage the contextual knowledge encoded in LLMs to generate sub-queries and
corresponding documents from multiple perspectives. Next, we employ a mutual
verification method for both generated and retrieved contextual documents,
where 1) retrieved documents are filtered with the external contextual
knowledge in generated documents, and 2) generated documents are filtered with
the corpus-specific knowledge in retrieved documents. Overall, the proposed
method allows retrieved and generated documents to complement each other to
finalize a better query expansion. We conduct extensive experiments on three
information retrieval datasets, i.e., TREC-DL-2020, TREC-COVID, and MSMARCO.
The results demonstrate that our method outperforms other baselines
significantly