2,099 research outputs found
iCLEF 2006 Overview: Searching the Flickr WWW photo-sharing repository
This paper summarizes the task design for iCLEF 2006 (the CLEF interactive track).
Compared to previous years, we have proposed a radically new task: searching images
in a naturally multilingual database, Flickr, which has millions of photographs shared
by people all over the planet, tagged and described in a wide variety of languages.
Participants are expected to build a multilingual search front-end to Flickr (using
Flickrâs search API) and study the behaviour of the users for a given set of searching
tasks. The emphasis is put on studying the process, rather than evaluating its outcome
iCLEF at Sheffield
Sheffieldâs contribution to the interactive cross language information retrieval track took the approach of comparing usersâ abilities to judge the relevance of machine translated French documents against ones written in the usersâ native language: English. Conducting such an experiment is challenging, and the issues surrounding the experimental design are discussed. Experimental results strongly suggest that users are just as capable of judging relevance of translated documents as they are for documents in their native languag
Observing Users - Designing clarity a case study on the user-centred design of a cross-language information retrieval system
This paper presents a case study of the development of an interface to a novel and complex form of document retrieval: searching for texts written in foreign languages based on native language queries. Although the underlying technology for achieving such a search is relatively well understood, the appropriate interface design is not. A study involving users (with such searching needs) from the start of the design process is described covering initial examination of user needs and tasks; preliminary
design and testing of interface components; building, testing, and further refining an interface; before
finally conducting usability tests of the system. Lessons are learned at every stage of the process leading to a much more informed view of how such an interface should be built
Using COTS Search Engines and Custom Query Strategies at CLEF
This paper presents a system for bilingual information retrieval using commercial off-the-shelf search engines (COTS). Several custom query construction, expansion and translation strategies are compared. We present the experiments and the corresponding results for the CLEF 2004 event
GeoCLEF 2006: the CLEF 2006 Ccross-language geographic information retrieval track overview
After being a pilot track in 2005, GeoCLEF advanced to be a regular track within CLEF 2006. The
purpose of GeoCLEF is to test and evaluate cross-language geographic information retrieval (GIR): retrieval for
topics with a geographic specification. For GeoCLEF 2006, twenty-five search topics were defined by the
organizing groups for searching English, German, Portuguese and Spanish document collections. Topics were
translated into English, German, Portuguese, Spanish and Japanese. Several topics in 2006 were significantly
more geographically challenging than in 2005. Seventeen groups submitted 149 runs (up from eleven groups and
117 runs in GeoCLEF 2005). The groups used a variety of approaches, including geographic bounding boxes,
named entity extraction and external knowledge bases (geographic thesauri and ontologies and gazetteers)
Evaluation campaigns and TRECVid
The TREC Video Retrieval Evaluation (TRECVid) is an
international benchmarking activity to encourage research
in video information retrieval by providing a large test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TRECVid completed its fifth annual cycle at the end of 2005 and in 2006 TRECVid will involve almost 70 research organizations, universities and other consortia. Throughout its existence, TRECVid has benchmarked both interactive and automatic/manual searching for shots from within a video
corpus, automatic detection of a variety of semantic and
low-level video features, shot boundary detection and the
detection of story boundaries in broadcast TV news. This
paper will give an introduction to information retrieval (IR) evaluation from both a user and a system perspective, highlighting that system evaluation is by far the most prevalent type of evaluation carried out. We also include a summary of TRECVid as an example of a system evaluation benchmarking campaign and this allows us to discuss whether
such campaigns are a good thing or a bad thing. There are
arguments for and against these campaigns and we present
some of them in the paper concluding that on balance they
have had a very positive impact on research progress
Concept hierarchy across languages in text-based image retrieval: a user evaluation
The University of Sheffield participated in Interactive ImageCLEF 2005 with a comparative user
evaluation of two interfaces: one displaying search results as a list, the other organizing retrieved images into
a hierarchy of concepts displayed on the interface as an interactive menu. Data was analysed with respect to
effectiveness (number of images retrieved), efficiency (time needed) and user satisfaction (opinions from
questionnaires). Effectiveness and efficiency were calculated at both 5 minutes (CLEF condition) and at final
time. The list was marginally more effective than the menu at 5 minutes (no statistical significance) but the
two were equal at final time showing the menu needs more time to be effectively used. The list was more efficient
at both 5 minutes and final time, although the difference was not statistically significant. Users preferred
the menu (75% vs. 25% for the list) indicating it to be an interesting and engaging feature. An inspection
of the logs showed that 11% of effective terms (i.e. no stop-words, single terms) were not translated and
that another 5% were ill translations. Some of those terms were used by all participants and were fundamental
for some of the tasks. Non translated and ill translated terms negatively affected the search, hierarchy generation
and, results display. More work has to be carried out to test the system under different setting, e.g. using
a dictionary instead of MT that appears to be ineffective in translating usersâ queries that rarely are
grammatically correct. The evaluation also indicated directions for a new interface design that allows the user
to check query translation (in both input and output) and that incorporates visual content image retrieval to
improve result organization
User experiments with the Eurovision cross-language image retrieval system
In this paper we present Eurovision, a text-based system for cross-language (CL) image retrieval.
The system is evaluated by multilingual users for two search tasks with the system configured in
English and five other languages. To our knowledge this is the first published set of user
experiments for CL image retrieval. We show that: (1) it is possible to create a usable multilingual
search engine using little knowledge of any language other than English, (2) categorizing images
assists the user's search, and (3) there are differences in the way users search between the proposed
search tasks. Based on the two search tasks and user feedback, we describe important aspects of
any CL image retrieval system
Benchmarking News Recommendations in a Living Lab
Most user-centric studies of information access systems in literature suffer from unrealistic settings or limited numbers of users who participate in the study. In order to address this issue, the idea of a living lab has been promoted. Living labs allow us to evaluate research hypotheses using a large number of users who satisfy their information need in a real context. In this paper, we introduce a living lab on news recommendation in real time. The living lab has first been organized as News Recommendation Challenge at ACM RecSysâ13 and then as campaign-style evaluation lab NEWSREEL at CLEFâ14. Within this lab, researchers were asked to provide news article recommendations to millions of users in real time. Different from user studies which have been performed in a laboratory, these users are following their own agenda. Consequently, laboratory bias on their behavior can be neglected. We outline the living lab scenario and the experimental setup of the two benchmarking events. We argue that the living lab can serve as reference point for the implementation of living labs for the evaluation of information access systems
- âŠ