Abstract The fifth QA campaign at CLEF [1], having its first edition in 2003, offered not only a main task but an Answer Validation Exercise (AVE) [2], which continued last year’s pilot, and a new pilot: the Question Answering on Speech Transcripts (QAST) [3, 15]. The main task was characterized by the focus on cross-linguality, while covering as many European languages as possible. As novelty, some QA pairs were grouped in clusters. Every cluster was characterized by a topic (not given to participants). The questions from a cluster possibly contain co-references between one of them and the others. Finally, the need for searching answers in web formats was satisfied by introducing Wikipedia 1 as document corpus. The results and the analyses reported by the participants suggest that the introduction of Wikipedia and the topic related questions led to a drop in systems’ performance. 

Anselmo Peñas

Bogdan Sacaleanu

Christelle Ayache

Corina Forascu

Danilo Giampiccolo

Jesús Herrera

Pamela Forner

Paulo Rocha

Petya Osenova

Richard Sutcliffe

Valentin Jijkoun

The fifth QA campaign at CLEF, the first having been held in 2006. was characterized by continuity with the past and at the same time by innovation. In fact, topics were introduced, under which a number of Question-Answer pairs could be grouped in clusters, containing also co-references between them. Moreover, the systems were given the possibility to search for answers in Wikipedia. In addition to the main task, two other tasks were offered, namely the Answer Validation Exercise (AVE), which continued last year’s successful pilot, and QUAST, aimed at evaluating the task of Question Answering in Speech Transcription. As general remark, it must be said that the task proved to be more difficult than expected, as in comparison with last year’s results the Best Overall Accuracy dropped from 49,47 % to 41,75 % in the multi-lingual subtasks, and, more significantly, from 68,95 % to 54 % in the monolingual subtasks

Dan Cristea

CiteSeerX

Overview of the CLEF 2007 multilingual question answering track

Giampiccolo, Danilo

Forner, Pamela

Peñas, Anselmo

Ayache, Christelle

Cristea, Dan

Jijkoun, Valentin

Osenova, Petya

Rocha, Paulo

Sacaleanu, Bogdan

Sutcliffe, Richard

Repositório Comum

English

Overview of the CLEF 2007 Multilingual Question Answering Track

The fifth QA campaign at CLEF, the first having been held in 2006. was characterized by continuity with the past and at the same time by innovation. In fact, topics were introduced, under which a number of Question-Answer pairs could be grouped in clusters, containing also co-references between them. Moreover, the systems were given the possibility to search for answers in Wikipedia. In addition to the main task, two other tasks were offered, namely the Answer Validation Exercise (AVE), which continued last year’s successful pilot, and QUAST, aimed at evaluating the task of Question Answering in Speech Transcription. The Question Answering Real Time was also proposed again, after the success obtained in 2006. As general remark, it must be said that the task proved to be more difficult than expected, as in comparison with last year’s results the Best Overall Accuracy dropped from 49 % to 41,75 % in the multi-lingual subtasks, and, more significantly, from 68 % to 54 % in the monolingual subtasks.. 

http://ceur-ws.org/Vol-1173/CLEF2007wn-QACLEF-GiampiccoloEt2007.pdf

Overview of the clef 2007 multilingual question answering track

Overview of the clef 2007 multilingual question answering track

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Repositório Comum

CiteSeerX