The Effect of Cross-Lingual Pooling on Evaluation

The purpose of this study is to examine whether there is an effect on the relative evaluation of the IR systems using the relevance judgments made by the pooling method and additional interactive searches. Relevance judgments of NTCIR-1&2 were made using the following steps: (1) collecting candidates for relevant documents by using the pooling method, (2) judging candidate documents by human assessors, (3)collecting additional candidates by recall-oriented interactive searches for search topics with more than 100 relevant documents to improve the exhaustiveness of the relevance judgments, and (4)judging the additional candidates. For the purpose of the study we carried out experiments using the relevance judgments and search results submitted for the test of the 2nd NTCIR Workshop. First, we evaluated the search results using the final relevance judgments � of NTCIR-2 and �   Á, that is, the � without the unique relevant documents found by the additional interactive searches Á. Second, we made pools from the search results in each of the sub-tasks and evaluated the search results using the relevance judgments in the pools. Almost the same rankings were produced by all the relevance judgments. Therefore our results verified the reliability of the evaluation using test collection based on pooling