Reflections on Mira : interactive evaluation in information retrieval

Evaluation in information retrieval (IR) has focussed largely on noninteractive evaluation of text retrieval systems. This is increasingly at odds with how people use modern IR systems: in highly interactive settings to access linked, multimedia information. Furthermore, this approach ignores potential improvements through better interface design. In 1996 the Commission of the European Union Information Technologies Programme, funded a three year working group, Mira, to discuss and advance research in the area of evaluation frameworks for interactive and multimedia IR applications. Led by Keith van Rijsbergen, Steve Draper and myself from Glasgow University, this working group brought together many of the leading researchers in the evaluation domain from both the IR and human computer interaction (HCI) communities. This paper presents my personal view of the main lines of discussion that took place throughout Mira: importing and adapting evaluation techniques from HCI, evaluating at different levels as appropriate, evaluating against different types of relevance and the new challenges that drive the need for rethinking the old evaluation approaches. The paper concludes that we need to consider more varied forms of evaluation to complement engine evaluation

Feasibility of a UK community-based, eTherapy mental health service in Greater Manchester: repeated-measures and between-groups study of ‘Living Life to the Full Interactive’, ‘Sleepio’ and ‘Breaking Free Online’ at ‘Self Help Services’

Objectives: There is increasing evidence to support the effectiveness of eTherapies for mental health, although limited data have been reported from community-based services. Therefore, this service evaluation reports on feasibility and outcomes from an eTherapy mental health service. Setting: ‘Self Help Services’, an Increasing Access to Psychological Therapies (IAPT) eTherapy service in Greater Manchester. Participants: 1068 service users referred to the service for secondary care for their mental health difficulties. Interventions: Participants were triaged into one of three eTherapy programmes: ‘Living Life to the Full Interactive’ for low mood, stress and anxiety; ‘Sleepio’ for insomnia; and ‘Breaking Free Online’ for substance misuse, depending on clinical need. Primary outcomes measures: Standardised psychometric assessments of depression, anxiety and social functioning, collected as part of the IAPT Minimum Data Set, were conducted at baseline and post-treatment. Results: Data indicated baseline differences, with the Breaking Free Online group having higher scores for depression and anxiety than the Living Life to the Full Interactive (depression CI 1.27 to 3.21, p<0.0001; anxiety CI 077 to 1.72, p<0.0001) and Sleepio (depression CI 1.19 to 4.52, p<0.0001; anxiety CI 2.16 to 5.23, p<0.0001) groups. Promising improvements in mental health scores were found within all three groups (all p<0.0001), as were significant reductions in numbers of service users reaching clinical threshold scores for mental health difficulties (p<0.0001). Number of days of engagement was not related to change from baseline for the Living Life to the Full or Sleepio programmes but was associated with degree of change for Breaking Free Online. Conclusion: Data presented provide evidence for feasibility of this eTherapy delivery model in supporting service users with a range of mental health difficulties and suggest that eTherapies may be a useful addition to treatment offering in community-based services

An evaluation of scaffolding for virtual interactive tutorials

Scaffolding refers to a temporary support framework used during construction. Applied to teaching and learning it describes measures to support a learner to become confident and self-reliant in a subject. In a Web environment scaffolding features need to replace the instructor. We discuss our approach to Web-based scaffolding based on the cognitive apprenticeship and activity theories. We suggest a set of four scaffold types that have made our scaffolding-supported virtual interactive tutorial successful. We present a novel evaluation approach for virtual tutorials that is embedded into an iterative, evolutionary instructional design

Using data visualization in creativity workshops: a new tool in the designer's kit

Creativity workshops have proved effective in drawing out unexpected requirements and giving form to participants' novel ideas. Here, we introduce a new addition to the workshop designer's toolkit: interactive data visualization, used as stimuli to prompt insight and inspire creativity. We first describe a pilot study in which we compare the effectiveness of two different styles of data visualization. Here we found that a less ambiguous style was more effective in supporting idea generation. Following this, we report a case study in which we employ data visualization within a service design workshop, where participants gain insights that are later realized in design ideas

Data mining technology for the evaluation of web-based teaching and learning systems

Instructional design for Web-based teaching and learning environments causes problems for two reasons. Firstly, virtual forms of teaching and learning result in little or no direct contact between instructor and learner, making the evaluation of course effectiveness difficult. Secondly, the Web as a relatively new teaching and learning medium still requires more research into learning processes with this technology. We propose data mining – techniques to discover and extract knowledge from a database – as a tool to support the analysis of student learning processes and the evaluation of the effectiveness and usability of Web-based courses. We present and illustrate different data mining techniques for the evaluation of Web-based teaching and learning systems

Assessing fun: young children as evaluators of interactive systems.

In this paper, we describe an exploratory study on the challenges of conducting usability tests with very young children aged 3 to 4 years old (nursery age) and the differences when working with older children aged 5 to 6 years old (primary school). A pilot study was conducted at local nursery and primary schools to understand and experience the challenges working with young children interacting with computer products. We report on the studies and compare the experiences of working with children of different age groups in evaluation studies of interactive systems

DEUCE : a test-bed for evaluating ESL competence criteria

This paper describes work in progress to apply a Web-based facility for evaluating differing criteria for English language competence. The proposed system, Discriminated Evaluation of User's Competence with English (DEUCE), addresses the problem of determining the efficacy of individual criteria for competence in English as a Second Language (ESL). We describe the rationale, design and application of DEUCE and outline its potential as a discriminator for ESL competence criteria and as a basis for low cost mass ESL competence testing

The evaluation of educational service integration in integrated virtual courses

The effectiveness of an integrated virtual course is determined by factors such as the navigability of the system. We argue that in a virtual course, which offers different educational services for different learning activities, the integration of services is a good indicator for the effectiveness of a virtual course infrastructure. We develop a set of metrics to measure the degree of integration of a virtual course. We combine structural metrics and an analysis of the student usage of the system in order to measure integration

Evaluation Strategy for the Re-Development of the Displays and Visitor Facilities at the Museum and Art Gallery, Kelvingrove

No abstract available