Initial specification of the evaluation tasks "Use cases to bridge validation and benchmarking" PROMISE Deliverable 2.1

Evaluation of multimedia and multilingual information access systems needs to be performed from a usage oriented perspective. This document outlines use cases from the three use case domains of the PROMISE project and gives some initial pointers to how their respective characteristics can be extrapolated to determine and guide evaluation activities, both with respect to benchmarking and to validation of the usage hypotheses. The use cases will be developed further during the course of the evaluation activities and workshops projected to occur in coming CLEF conferences

A global approach to digital library evaluation towards quality interoperability

This paper describes some of the key research works related to my PhD thesis. The goal is the development of a global approach to digital library (DL) evaluation towards quality interoperability. DL evaluation has a vital role to play in building DLs, and in understanding and enhancing their role in society. Responding to two parallel research needs, the project is grouped around two tracks. Track one covers the theoretical approach, and provides an integrated evaluation model which overcomes the fragmentation of quality assessments; track two covers the experimental side, which has been undertaken through a comparative analysis of different DL evaluation methodologies, relating them to the conceptual framework. After presenting the problem dentition, current background and related work, this paper enumerates a set of research questions and hypotheses that I would like to address, and outlines the research methodology, focusing on a proposed evaluation framework and on the lessons learned from the case studies

Evaluation of a personalized digital library based on cognitive styles: Adaptivity vs. adaptability

Personalization can be addressed by adaptability and adaptivity, which have different advantages and disadvantages. This study investigates how digital library users react to these two techniques. More specifically, we develop a personalized digital library to suit the needs of different cognitive styles based on the findings of our previous work (Frias-Martinez, et al., in press). The personalized digital library includes two versions: adaptive version and adaptable version. The results showed that users not only performed better in the adaptive version, but also they perceived more positively to the adaptive version. In addition, cognitive styles have great effects on users’ responses to adaptability and adaptivity. These results provide guidance for designers to select suitable techniques to develop personalized digital libraries

On Birthing Dancing Stars: The Need for Bounded Chaos in Information Interaction

While computers causing chaos is acommon social trope, nearly the entirety of the history of computing is dedicated to generating order. Typical interactive information retrieval tasks ask computers to support the traversal and exploration of large, complex information spaces. The implicit assumption is that they are to support users in simplifying the complexity (i.e. in creating order from chaos). But for some types of task, particularly those that involve the creative application or synthesis of knowledge or the creation of new knowledge, this assumption may be incorrect. It is increasingly evident that perfect order—and the systems we create with it—support highly-structured information tasks well, but provide poor support for less-structured tasks.We need digital information environments that help create a little more chaos from order to spark creative thinking and knowledge creation. This paper argues for the need for information systems that offerwhat we term ‘bounded chaos’, and offers research directions that may support the creation of such interface

An examination of automatic video retrieval technology on access to the contents of an historical video archive

Purpose – This paper aims to provide an initial understanding of the constraints that historical video collections pose to video retrieval technology and the potential that online access offers to both archive and users. Design/methodology/approach – A small and unique collection of videos on customs and folklore was used as a case study. Multiple methods were employed to investigate the effectiveness of technology and the modality of user access. Automatic keyframe extraction was tested on the visual content while the audio stream was used for automatic classification of speech and music clips. The user access (search vs browse) was assessed in a controlled user evaluation. A focus group and a survey provided insight on the actual use of the analogue archive. The results of these multiple studies were then compared and integrated (triangulation). Findings – The amateur material challenged automatic techniques for video and audio indexing, thus suggesting that the technology must be tested against the material before deciding on a digitisation strategy. Two user interaction modalities, browsing vs searching, were tested in a user evaluation. Results show users preferred searching, but browsing becomes essential when the search engine fails in matching query and indexed words. Browsing was also valued for serendipitous discovery; however the organisation of the archive was judged cryptic and therefore of limited use. This indicates that the categorisation of an online archive should be thought of in terms of users who might not understand the current classification. The focus group and the survey showed clearly the advantage of online access even when the quality of the video surrogate is poor. The evidence gathered suggests that the creation of a digital version of a video archive requires a rethinking of the collection in terms of the new medium: a new archive should be specially designed to exploit the potential that the digital medium offers. Similarly, users' needs have to be considered before designing the digital library interface, as needs are likely to be different from those imagined. Originality/value – This paper is the first attempt to understand the advantages offered and limitations held by video retrieval technology for small video archives like those often found in special collections

Information seeking retrieval, reading and storing behaviour of library users

In the interest of digital libraries, it is advisable that designers be aware of the potential behaviour of the users of such a system. There are two distinct parts under investigation, the interaction between traditional libraries involving the seeking and retrieval of relevant material, and the reading and storage behaviours ensuing. Through this analysis, the findings could be incorporated into digital library facilities. There has been copious amounts of research on information seeking leading to the development of behavioural models to describe the process. Often research on the information seeking practices of individuals is based on the task and field of study. The information seeking model, presented by Ellis et al. (1993), characterises the format of this study where it is used to compare various research on the information seeking practices of groups of people (from academics to professionals). It is found that, although researchers do make use of library facilities, they tend to rely heavily on their own collections and primarily use the library as a source for previously identified information, browsing and interloan. It was found that there are significant differences in user behaviour between the groups analysed. When looking at the reading and storage of material it was hard to draw conclusions, due to the lack of substantial research and information on the topic. However, through the use of reading strategies, a general idea on how readers behave can be developed. Designers of digital libraries can benefit from the guidelines presented here to better understand their audience

Users' trust in information resources in the Web environment: a status report

This study has three aims; to provide an overview of the ways in which trust is either assessed or asserted in relation to the use and provision of resources in the Web environment for research and learning; to assess what solutions might be worth further investigation and whether establishing ways to assert trust in academic information resources could assist the development of information literacy; to help increase understanding of how perceptions of trust influence the behaviour of information users

What Are They Doing Anyway?: Library as Place and Student Use of a University Library

Objective - To determine student use of library spaces, the authors recorded student location and behaviors within the Library, to inform future space design. Methods - The case study method was used with both quantitative and qualitative measures. The authors had two objectives to guide this assessment of library spaces: 1) To determine what library spaces are being used by students and whether students are working individually, communally, or collaboratively and 2) To determine whether students use these spaces for learning activities and/or social engagement. Results - After data collection and analysis, the authors determined students are using individual or communal spaces almost equally as compared with collaborative group spaces. Data also revealed peak area usage and times. Conclusion - Observed student individual and social work habits indicate further need for spaces with ample electrical outlets and moveable tables. Further study is recommended to see whether additional seating and renovated spaces continue to enhance informal learning communities at URI and whether the Library is becoming a “third place” on campus

Incorporating Online Instruction in Academic Libraries: Getting Ahead of the Curve

A sea change in higher education is shaping the way many libraries deliver instruction to their students and faculty. Years of technological innovation and changes in the way that people discover and use information has made online instruction an essential part of a library\u27s teaching and learning program. In order to evaluate our library\u27s online instruction program and to determine its future goals, we analyzed the technology, pedagogical models, organizational structures, administrative supports, and partnerships we would need in order to succeed. Our findings may be useful for libraries reassessing their own online instruction programs

Learning object metadata surrogates in search result interfaces: user evaluation, design and content

The purpose of this research was to evaluate user interaction with learning object metadata surrogates both in terms of content and presentation. The main objectives of this study were: (1) to review the literature on learning object metadata and user-centred evaluation of metadata surrogates in the context of cognitive information retrieval (including user-centred relevance and usability research); (2) to develop a framework for the evaluation of user interaction with learning object metadata surrogates in search result interfaces; (3) to investigate the usability of metadata surrogates in search result interfaces of learning object repositories (LORs) in terms of various presentation aspects (such as amount of information, structure and highlighting of query terms) as a means for facilitating the user relevance judgment process; (4) to investigate in-depth the type of content that should be included in learning object metadata surrogates in order to facilitate the process of relevance judgment; (5) to provide a set of recommendations—guidelines for the design of learning object metadata surrogates in search result interfaces both in terms of content and presentation. [Continues.