iQPP: A Benchmark for Image Query Performance Prediction

To date, query performance prediction (QPP) in the context of content-based image retrieval remains a largely unexplored task, especially in the query-by-example scenario, where the query is an image. To boost the exploration of the QPP task in image retrieval, we propose the first benchmark for image query performance prediction (iQPP). First, we establish a set of four data sets (PASCAL VOC 2012, Caltech-101, ROxford5k and RParis6k) and estimate the ground-truth difficulty of each query as the average precision or the precision@k, using two state-of-the-art image retrieval models. Next, we propose and evaluate novel pre-retrieval and post-retrieval query performance predictors, comparing them with existing or adapted (from text to image) predictors. The empirical results show that most predictors do not generalize across evaluation scenarios. Our comprehensive experiments indicate that iQPP is a challenging benchmark, revealing an important research gap that needs to be addressed in future work. We release our code and data as open source at https://github.com/Eduard6421/iQPP, to foster future research.Comment: Accepted at SIGIR 202

Investigating Retrieval Method Selection with Axiomatic Features

We consider algorithm selection in the context of ad-hoc information retrieval. Given a query and a pair of retrieval methods, we propose a meta-learner that predicts how to combine the methods' relevance scores into an overall relevance score. Inspired by neural models' different properties with regard to IR axioms, these predictions are based on features that quantify axiom-related properties of the query and its top ranked documents. We conduct an evaluation on TREC Web Track data and find that the meta-learner often significantly improves over the individual methods. Finally, we conduct feature and query weight analyses to investigate the meta-learner's behavior

A Machine Learning Approach to SPARQL Query Performance Prediction

International audienceIn this paper we address the problem of predicting SPARQL query performance. We use machine learning techniques to learn SPARQL query performance from previously executed queries. Traditional approaches for estimating SPARQL query cost are based on statistics about the underlying data. However, in many use-cases involving querying Linked Data, statistics about the underlying data are often missing. Our approach does not require any statistics about the underlying RDF data, which makes it ideal for the Linked Data scenario. We show how to model SPARQL queries as feature vectors, and use k-nearest neighbors regression and Support Vector Machine with the nu-SVR kernel to accurately predict SPARQL query execution time

Performance Prediction for Conversational Search Using Perplexities of Query Rewrites

We consider query performance prediction (QPP) task for conversational search (CS), i.e., to estimate the retrieval quality for queries in multi-turn conversations. We reuse QPP methods from ad-hoc search for CS by feeding them self-contained query rewrites generated by T5. Our experiments on three CS datasets show that (i) lower query rewriting quality may lead to worse QPP performance, and (ii) incorporating query rewriting quality (as measured by perplexity) improves the effectiveness of QPP methods for CS if the query rewriting quality is limited. Our implementation is publicly available at https://github.com/ChuanMeng/QPP4CS.</p

Estimating Cardinalities with Deep Sketches

We introduce Deep Sketches, which are compact models of databases that allow us to estimate the result sizes of SQL queries. Deep Sketches are powered by a new deep learning approach to cardinality estimation that can capture correlations between columns, even across tables. Our demonstration allows users to define such sketches on the TPC-H and IMDb datasets, monitor the training process, and run ad-hoc queries against trained sketches. We also estimate query cardinalities with HyPer and PostgreSQL to visualize the gains over traditional cardinality estimators.Comment: To appear in SIGMOD'1

A Comparative Study and Analysis of Query Performance Prediction Algorithms to Improve their Reproducibility

Una delle sfide principali nella valutazione all’interno dell’Information Retrieval è rappresentata dal costo richiesto dalla valutazione stessa, sia online che offline. Pertanto, negli ultimi anni diversi sforzi sono stati dedicati al compito svolto dalla Query Performance Prediction (QPP). QPP ha come obiettivo quello di stimare la qualità di un sistema quando viene utilizzato per recuperare documenti in risposta a una data query, basandosi su diverse fonti di informazione come la query, i documenti o i punteggi di similarità forniti dal sistema di Information Retrieval. Negli ultimi anni sono stati progettati diversi modelli di QPP pre e post-retrieval, ma raramente sono stati testati nelle stesse condizioni sperimentali. L’obiettivo del nostro lavoro è molteplice: sviluppare una struttura unificante che includa diversi approcci QPP presenti nello stato dell’arte e usare tale struttura per valutare la riproducibilità degli approcci QPP implementati. I nostri risultati illustrano che siamo in grado di raggiungere un alto grado di riproducibilità, con quattordici metodi diversi riprodotti correttamente e risultati di performance paragonabili a quelli originali.One of the primary challenges in Information Retrieval evaluation is represented by the cost of carrying out either online or offline evaluation. Therefore, in recent years several endeavors have been devoted to the Query Performance Prediction (QPP) task. QPP aims to estimate the quality of a system when used to retrieve documents in response to a given query, relying on different sources of information such as the query, the documents or the similarity scores provided by the Information Retrieval system. In the last years several pre and post-retrieval QPP models have been designed, but rarely tested under the same experimental conditions. The objective of our work is multifold: we develop a unifying framework that includes several state-of-the-art QPP approaches and use such framework to assess the reproducibility of such QPP approaches. Our findings illustrate that we are able to achieve a high degree of reproducibility, with fourteen different methods correctly reproduced and performance results comparable to the original ones