36 research outputs found
iCLEF at Sheffield
Sheffield’s contribution to the interactive cross language information retrieval track took the approach of comparing users’ abilities to judge the relevance of machine translated French documents against ones written in the users’ native language: English. Conducting such an experiment is challenging, and the issues surrounding the experimental design are discussed. Experimental results strongly suggest that users are just as capable of judging relevance of translated documents as they are for documents in their native languag
Is query translation a distinct task from search?
INTRODUCTION
The University of Sheffield participated in iCLEF 2002 using, as a test-bed, the prototype under
development in the Clarity project. Clarity is an EU funded project aimed at developing a system for
cross-language information retrieval for so-called low density languages, those with few translation
resources. Currently translation between English and Finnish is supported; soon Swedish will be added
and in the near future Latvian and Lithuanian.
Clarity is being developed in a user-centred way with user involvement from the beginning. The design
of the first user interface was based on current best practise, particular attention was paid to empirical
evidence for a specific design choice. Six paper-based interface mock-ups representing important
points in the cross-language search task were generated and presented for user assessment as a part of
an extensive user study. The study (reported in Petrelli et al. 2002) was conducted to understand users
and uses of cross-language information retrieval systems. Many different techniques were applied:
contextual enquiry, interviews, questionnaires, informal evaluation of existing cross-language
technology, and participatory design sessions with the interface mock-ups mentioned above. As a
result, a user class profile was sketched and a long list of user requirements was compiled. As a followup,
a redesign session took place and the new system was designed for users whoknow the language(s) they are searching (polyglots);
• search for writing (journalists, translators business analysts);
• have limited searching skills;
• know the topic in advance or will learn/read on it while searching;
• use many languages in the same search session and often swap between them.
New system features were listed as important and the user interface was redesigned. Considering the
result of the study the new interface allowed the user to dynamically change the language setting from
query to query, hid the query translation and showed the retrieved set as ranked list primary.
Despite the fact that this new design was considered to be more effective, a comparison between the
first layout based on the relevant literature and the new one based on the user study was considered an
important research question. In particular, the choice of hiding the query translation was considered an
important design decision, against the common agreement to allow and support the user in controlling
the system actions. Thus the participation of Sheffield in iCLEF was organized around the idea of
checking if the user should validate the query translation before the search is run or instead if the
system should perform the translation and search in a single step without any user’s supervision
iCLEF 2006 Overview: Searching the Flickr WWW photo-sharing repository
This paper summarizes the task design for iCLEF 2006 (the CLEF interactive track).
Compared to previous years, we have proposed a radically new task: searching images
in a naturally multilingual database, Flickr, which has millions of photographs shared
by people all over the planet, tagged and described in a wide variety of languages.
Participants are expected to build a multilingual search front-end to Flickr (using
Flickr’s search API) and study the behaviour of the users for a given set of searching
tasks. The emphasis is put on studying the process, rather than evaluating its outcome
The effects of topic familiarity on user search behavior in question answering systems
This paper reports on experiments that attempt
to characterize the relationship between users
and their knowledge of the search topic in a
Question Answering (QA) system. It also
investigates user search behavior with respect
to the length of answers presented by a QA
system. Two lengths of answers were
compared; snippets (one to two sentences of
text) and exact answers. A user test was
conducted, 92 factoid questions were judged
by 44 participants, to explore the participants’
preferences, feelings and opinions about QA
system tasks. The conclusions drawn from the
results were that participants preferred and
obtained higher accuracy in finding answers
from the snippets set. However, accuracy
varied according to users’ topic familiarity;
users were only substantially helped by the
wider context of a snippet if they were already
familiar with the topic of the question, without
such familiarity, users were about as accurate
at locating answers from the snippets as they
were in exact set
Users' effectiveness and satisfaction for image retrieval
This paper presents results from an initial user
study exploring the relationship between system
effectiveness as quantified by traditional
measures such as precision and recall, and users’
effectiveness and satisfaction of the results. The
tasks involve finding images for recall-based
tasks. It was concluded that no direct relationship
between system effectiveness and users’
performance could be proven (as shown by
previous research). People learn to adapt to a
system regardless of its effectiveness. This study
recommends that a combination of attributes
(e.g. system effectiveness, user performance and
satisfaction) is a more effective way to evaluate
interactive retrieval systems. Results of this
study also reveal that users are more concerned
with accuracy than coverage of the search
results
Multilingual interactive experiments with Flickr
This paper presents a proposal for iCLEF 2006, the interactive track
of the CLEF cross-language evaluation campaign. In the past, iCLEF has
addressed applications such as information retrieval and question answering. However, for 2006 the focus
has turned to text-based image retrieval from Flickr. We describe
Flickr, the challenges this kind of collection presents to
cross-language researchers, and suggest initial iCLEF tasks
An Investigation on Text-Based Cross-Language Picture Retrieval Effectiveness through the Analysis of User Queries
Purpose: This paper describes a study of the queries generated from a user experiment for cross-language information retrieval (CLIR) from a historic image archive. Italian speaking users generated 618 queries for a set of known-item search tasks. The queries generated by user’s interaction with the system have been analysed and the results used to suggest recommendations for the future development of cross-language retrieval systems for digital image libraries.
Methodology: A controlled lab-based user study was carried out using a prototype Italian-English image retrieval system. Participants were asked to carry out searches for 16 images provided to them, a known-item search task. User’s interactions with the system were recorded and queries were analysed manually quantitatively and qualitatively.
Findings: Results highlight the diversity in requests for similar visual content and the weaknesses of Machine Translation for query translation. Through the manual translation of queries we show the benefits of using high-quality translation resources. The results show the individual characteristics of user’s whilst performing known-item searches and the overlap obtained between query terms and structured image captions, highlighting the use of user’s search terms for objects within the foreground of an image.
Limitations and Implications: This research looks in-depth into one case of interaction and one image repository. Despite this limitation, the discussed results are likely to be valid across other languages and image repository.
Value: The growing quantity of digital visual material in digital libraries offers the potential to apply techniques from CLIR to provide cross-language information access services. However, to develop effective systems requires studying user’s search behaviours, particularly in digital image libraries. The value of this paper is in the provision of empirical evidence to support recommendations for effective cross-language image retrieval system design.</p
Observing Users - Designing clarity a case study on the user-centred design of a cross-language information retrieval system
This paper presents a case study of the development of an interface to a novel and complex form of document retrieval: searching for texts written in foreign languages based on native language queries. Although the underlying technology for achieving such a search is relatively well understood, the appropriate interface design is not. A study involving users (with such searching needs) from the start of the design process is described covering initial examination of user needs and tasks; preliminary
design and testing of interface components; building, testing, and further refining an interface; before
finally conducting usability tests of the system. Lessons are learned at every stage of the process leading to a much more informed view of how such an interface should be built
An analysis of machine translation errors on the effectiveness of an Arabic-English QA system
The aim of this paper is to investigate
how much the effectiveness of a Question
Answering (QA) system was affected
by the performance of Machine
Translation (MT) based question translation.
Nearly 200 questions were selected
from TREC QA tracks and ran through a
question answering system. It was able to
answer 42.6% of the questions correctly
in a monolingual run. These questions
were then translated manually from English
into Arabic and back into English using
an MT system, and then re-applied to
the QA system. The system was able to
answer 10.2% of the translated questions.
An analysis of what sort of translation error
affected which questions was conducted,
concluding that factoid type
questions are less prone to translation error
than others