We introduce pytrec_eval, a Python interface to the tree_eval information
retrieval evaluation toolkit. pytrec_eval exposes the reference implementations
of trec_eval within Python as a native extension. We show that pytrec_eval is
around one order of magnitude faster than invoking trec_eval as a sub process
from within Python. Compared to a native Python implementation of NDCG,
pytrec_eval is twice as fast for practically-sized rankings. Finally, we
demonstrate its effectiveness in an application where pytrec_eval is combined
with Pyndri and the OpenAI Gym where query expansion is learned using
Q-learning.Comment: SIGIR '18. The 41st International ACM SIGIR Conference on Research &
Development in Information Retrieva