Reflections on Mira : interactive evaluation in information retrieval

Evaluation in information retrieval (IR) has focussed largely on noninteractive evaluation of text retrieval systems. This is increasingly at odds with how people use modern IR systems: in highly interactive settings to access linked, multimedia information. Furthermore, this approach ignores potential improvements through better interface design. In 1996 the Commission of the European Union Information Technologies Programme, funded a three year working group, Mira, to discuss and advance research in the area of evaluation frameworks for interactive and multimedia IR applications. Led by Keith van Rijsbergen, Steve Draper and myself from Glasgow University, this working group brought together many of the leading researchers in the evaluation domain from both the IR and human computer interaction (HCI) communities. This paper presents my personal view of the main lines of discussion that took place throughout Mira: importing and adapting evaluation techniques from HCI, evaluating at different levels as appropriate, evaluating against different types of relevance and the new challenges that drive the need for rethinking the old evaluation approaches. The paper concludes that we need to consider more varied forms of evaluation to complement engine evaluation

Health Figures: An Open Source JavaScript Library for Health Data Visualization

The way we look at data has a great impact on how we can understand it, particularly when the data is related to health and wellness. Due to the increased use of self-tracking devices and the ongoing shift towards preventive medicine, better understanding of our health data is an important part of improving the general welfare of the citizens. Electronic Health Records, self-tracking devices and mobile applications provide a rich variety of data but it often becomes difficult to understand. We implemented the hFigures library inspired on the hGraph visualization with additional improvements. The purpose of the library is to provide a visual representation of the evolution of health measurements in a complete and useful manner. We researched the usefulness and usability of the library by building an application for health data visualization in a health coaching program. We performed a user evaluation with Heuristic Evaluation, Controlled User Testing and Usability Questionnaires. In the Heuristics Evaluation the average response was 6.3 out of 7 points and the Cognitive Walkthrough done by usability experts indicated no design or mismatch errors. In the CSUQ usability test the system obtained an average score of 6.13 out of 7, and in the ASQ usability test the overall satisfaction score was 6.64 out of 7. We developed hFigures, an open source library for visualizing a complete, accurate and normalized graphical representation of health data. The idea is based on the concept of the hGraph but it provides additional key features, including a comparison of multiple health measurements over time. We conducted a usability evaluation of the library as a key component of an application for health and wellness monitoring. The results indicate that the data visualization library was helpful in assisting users in understanding health data and its evolution over time.Comment: BMC Medical Informatics and Decision Making 16.1 (2016

Automated Game Design Learning

While general game playing is an active field of research, the learning of game design has tended to be either a secondary goal of such research or it has been solely the domain of humans. We propose a field of research, Automated Game Design Learning (AGDL), with the direct purpose of learning game designs directly through interaction with games in the mode that most people experience games: via play. We detail existing work that touches the edges of this field, describe current successful projects in AGDL and the theoretical foundations that enable them, point to promising applications enabled by AGDL, and discuss next steps for this exciting area of study. The key moves of AGDL are to use game programs as the ultimate source of truth about their own design, and to make these design properties available to other systems and avenues of inquiry.Comment: 8 pages, 2 figures. Accepted for CIG 201

Improving situation awareness of a single human operator interacting with multiple unmanned vehicles: first results

In the context of the supervision of one or several unmanned vehicles by a human operator, the design of an adapted user interface is a major challenge. Therefore, in the context of an existing experimental set up composed of a ground station and heterogeneous unmanned ground and air vehicles we aim at redesigning the human-robot interactions to improve the operator's situation awareness. We base our new design on a classical user centered approach

Knowledge-Based Aircraft Automation: Managers Guide on the use of Artificial Intelligence for Aircraft Automation and Verification and Validation Approach for a Neural-Based Flight Controller

The ultimate goal of this report was to integrate the powerful tools of artificial intelligence into the traditional process of software development. To maintain the US aerospace competitive advantage, traditional aerospace and software engineers need to more easily incorporate the technology of artificial intelligence into the advanced aerospace systems being designed today. The future goal was to transition artificial intelligence from an emerging technology to a standard technology that is considered early in the life cycle process to develop state-of-the-art aircraft automation systems. This report addressed the future goal in two ways. First, it provided a matrix that identified typical aircraft automation applications conducive to various artificial intelligence methods. The purpose of this matrix was to provide top-level guidance to managers contemplating the possible use of artificial intelligence in the development of aircraft automation. Second, the report provided a methodology to formally evaluate neural networks as part of the traditional process of software development. The matrix was developed by organizing the discipline of artificial intelligence into the following six methods: logical, object representation-based, distributed, uncertainty management, temporal and neurocomputing. Next, a study of existing aircraft automation applications that have been conducive to artificial intelligence implementation resulted in the following five categories: pilot-vehicle interface, system status and diagnosis, situation assessment, automatic flight planning, and aircraft flight control. The resulting matrix provided management guidance to understand artificial intelligence as it applied to aircraft automation. The approach taken to develop a methodology to formally evaluate neural networks as part of the software engineering life cycle was to start with the existing software quality assurance standards and to change these standards to include neural network development. The changes were to include evaluation tools that can be applied to neural networks at each phase of the software engineering life cycle. The result was a formal evaluation approach to increase the product quality of systems that use neural networks for their implementation

A study of methods of evaluating multimedia materials for language learning

There is a long tradition of paper based materials evaluation in ELT, but at this juncture, a scarcity of studies on ELT Multimedia (MM) materials evaluation. Such studies as have been undertaken have tended to adopt the perspective of the materials developer rather than the end user. But there have been no developed studies of evaluation methods which could be adopted by potential users. Despite calls being made for systematic evaluation, not many proposals have been developed, and there has been little exploration of potential best practice, or of the 'goodness of fit' between methods and evaluation puposes. This study aims to investigate evaluation methods in order to establish best practice in the evaluation of multimedia CALL applications, with a focus on learners' experience and opinions, and with the aim of enabling potential teacher-users of CALL materials to gauge the suitability of materials for their learners. Chapter one of this thesis provides a rationale for this study and an overview of the background to this research. Chapter two presents a review of literature undertaken in four domains: educational evaluation and research methods; Human Computer Interaction (HCI) usability evaluation methods; ELT materials evaluation; and studies ofCALL materials in use. Chapter three focuses on the design and conduct of the study by explaining how some methods of evaluation were trialled in a pilot study and four were selected for adoption and analysis in the main study. The methods selected were Foeu.r Groups, Rttrospective Protorolr, PLUM and SUM! Questionnaires and Activity Monitonitg. To determine the qualities and limitations of these methods, a set of criteria was developed from the literature on software usability evaluation methods in HCI and a broader literature on educational evaluation and research methods. The four data study chapters (4-7) each discuss one of the chosen methods and descnbe how the method was ,6perationalised in an evaluation of learner responses to multimedia software. The final chapter draws together the discussion of the findings and presents different proposals for best practice. The focus in the discussion of findings is on how the chosen methods performed according to the set of criteria. The findings confirm that focus groups and questionnaires are quick and efficient methods whereas retrospective protocols and activity monitoring provide more detailed and protracted data. Teacher evaluators can be guided by the objectives of their evaluation to explore different combinations ofthese methods. Participants in the pilot were 12 ESL students from the University of Warwick and in the main study 45 Freshman/sophomore students from a university in Pakistan. The materials used to operationalise methods were the EASE CD-ROMS listening to Lectures and Seminar Skills 1: Presentations. The research contributes to the field in undertaking an in-depth and extensive study of evaluation methods applicable to CALL materials, whi~ adopt a leamer-centred perspective, and conform to sound principles within educational evaluation, yet which draw on practice in the field of HCI, since this expertise is so relevant in the rapid development of multimedia materials for use in ELT. Moreover, by developing the composite set of core criteria this study has created a tool which practitioners in the field can use to select most appropriate methods for their particular evaluation purposes

A novel approach towards usability studies for visual search tasks in graphical user interface applications using the activity theory approach

The field of Human Computer Interaction still strives for a generalized model of visual search tasks (icon search, menu search, text search, label search, search through hypertext and feature recognition). The existing models of visual search, in spite of being impressive, are limited under certain perspectives due to lack of generality. The thesis tries to provide a holistic approach for the modeling of visual search tasks in graphical user interfaces from the Activity Theory (AT) perspective with the aim of rendering a theoretical bridge between HCI and Psychology. A detailed review of literature from the variegated discipline contributing to the study of Visual Search revealed the presence of gray areas, which can be partially addressed by the Activity Theory approach. The case study uses thinking aloud Protocol Analysis technique for analyzing the complex interaction of behavior, cognition and motor action, which manifest in these tasks. The results have been analyzed and possible modifications have been identified. Interestingly, it is observed that Activity Theory can provide substantial theoretical support to aid Usability Testing Techniques