Principles in Patterns (PiP) : Project Evaluation Synthesis

Evaluation activity found the technology-supported approach to curriculum design and approval developed by PiP to demonstrate high levels of user acceptance, promote improvements to the quality of curriculum designs, render more transparent and efficient aspects of the curriculum approval and quality monitoring process, demonstrate process efficacy and resolve a number of chronic information management difficulties which pervaded the previous state. The creation of a central repository of curriculum designs as the basis for their management as "knowledge assets", thus facilitating re-use and sharing of designs and exposure of tacit curriculum design practice, was also found to be highly advantageous. However, further process improvements remain possible and evidence of system resistance was found in some stakeholder groups. Recommendations arising from the findings and conclusions include the need to improve data collection surrounding the curriculum approval process so that the process and human impact of C-CAP can be monitored and observed. Strategies for improving C-CAP acceptance among the "late majority", the need for C-CAP best practice guidance, and suggested protocols on the knowledge management of curriculum designs are proposed. Opportunities for further process improvements in institutional curriculum approval, including a re-engineering of post-faculty approval processes, are also recommended

Usability Evaluation in Virtual Environments: Classification and Comparison of Methods

Virtual environments (VEs) are a relatively new type of human-computer interface in which users perceive and act in a three-dimensional world. The designers of such systems cannot rely solely on design guidelines for traditional two-dimensional interfaces, so usability evaluation is crucial for VEs. We present an overview of VE usability evaluation. First, we discuss some of the issues that differentiate VE usability evaluation from evaluation of traditional user interfaces such as GUIs. We also present a review of VE evaluation methods currently in use, and discuss a simple classification space for VE usability evaluation methods. This classification space provides a structured means for comparing evaluation methods according to three key characteristics: involvement of representative users, context of evaluation, and types of results produced. To illustrate these concepts, we compare two existing evaluation approaches: testbed evaluation [Bowman, Johnson, & Hodges, 1999], and sequential evaluation [Gabbard, Hix, & Swan, 1999]. We conclude by presenting novel ways to effectively link these two approaches to VE usability evaluation

eStorys: A visual storyboard system supporting back-channel communication for emergencies

This is the post-print version of the final paper published in Journal of Visual Languages & Computing. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2010 Elsevier B.V.In this paper we present a new web mashup system for helping people and professionals to retrieve information about emergencies and disasters. Today, the use of the web during emergencies, is confirmed by the employment of systems like Flickr, Twitter or Facebook as demonstrated in the cases of Hurricane Katrina, the July 7, 2005 London bombings, and the April 16, 2007 shootings at Virginia Polytechnic University. Many pieces of information are currently available on the web that can be useful for emergency purposes and range from messages on forums and blogs to georeferenced photos. We present here a system that, by mixing information available on the web, is able to help both people and emergency professionals in rapidly obtaining data on emergency situations by using multiple web channels. In this paper we introduce a visual system, providing a combination of tools that demonstrated to be effective in such emergency situations, such as spatio/temporal search features, recommendation and filtering tools, and storyboards. We demonstrated the efficacy of our system by means of an analytic evaluation (comparing it with others available on the web), an usability evaluation made by expert users (students adequately trained) and an experimental evaluation with 34 participants.Spanish Ministry of Science and Innovation and Universidad Carlos III de Madrid and Banco Santander

Toward Building Self-Sustaining Groups in PCR-based Tasks through Implicit Coordination: The Case of Heuristic Evaluation

Usability flaws found in the later stages of the software development process can be extremely costly to resolve. Accordingly, usability evaluation (UE) is an important, albeit usually expensive, part of development. We report on how the inexpensive UE method of heuristic evaluation (HE) can benefit from collaborative software (CSW), implicit coordination, and principles from collaboration engineering. In our study, 439 novice participants were trained in HE methods and then performed HE. Our results show that traditional nominal HE groups can experience implicit coordination through the collaborative software features of group memory and group awareness. One of the key results is that CSW groups had less duplication of effort than traditional nominal groups; these differences were magnified as group size increased from three to six members. Furthermore, because they coordinated less, traditional nominal groups performed more work in the overall process of HE. We attribute the reduction in duplication for CSW-supported groups to the implicit coordination available to them; CSW-supported groups could see violations input by other group members, but could not directly discuss the violations. These findings not only show the power of implicit coordination in groups, but should dramatically change how HE is conducted. These results may also extend to other evaluation tasks, such as software inspection and usability assessment tasks

Layered evaluation of interactive adaptive systems : framework and formative methods

Peer reviewedPostprin

Applying a User-centred Approach to Interactive Visualization Design

Analysing users in their context of work and finding out how and why they use different information resources is essential to provide interactive visualisation systems that match their goals and needs. Designers should actively involve the intended users throughout the whole process. This chapter presents a user-centered approach for the design of interactive visualisation systems. We describe three phases of the iterative visualisation design process: the early envisioning phase, the global specification hase, and the detailed specification phase. The whole design cycle is repeated until some criterion of success is reached. We discuss different techniques for the analysis of users, their tasks and domain. Subsequently, the design of prototypes and evaluation methods in visualisation practice are presented. Finally, we discuss the practical challenges in design and evaluation of collaborative visualisation environments. Our own case studies and those of others are used throughout the whole chapter to illustrate various approaches

Academic methods for usability evaluation of serious games: a systematic review

In the last years, there has been an increasing interest in the design of video games as a tool for education, training, health promotion, socialization, etc. Usability, which is a key factor in any video game, becomes even more important in these so-called Bserious games^, where the users’ special characteristics should be considered, and the game efficacy depends on the users’ adherence and engagement. However, evaluation of the usability of this kind of games requires a redefinition of techniques, methods and even terminology. In this paper, we elicit six research questions and conduct a systematic review of the scientific literature, which resulted in the selection of 187 papers that contained the most relevant responses. The conclusions of this systematic review illustrate the general status of current academic usability evaluations of these games and the main trends in the selection of methodologies and how are they applied. This view may be a very valuable foundation for future researchMinisterio de Ciencia e Innovación PROCUR@-IPT-2011-1038-90000

Glitchspace:teaching programming through puzzles in cyberspace

There is an increasing need to address the player experience in games-based learning. Whilst games offer enormous potential as learning experiences, the balance between entertainment and education must be carefully designed and delivered. Successful commercial games tend to focus gameplay above any educational aspects. In contrast, games designed for educational purposes have a habit of sacrificing entertainment for educational value which can result in a decline in player engagement. For both, the player experience is critical as it can have a profound effect on both the commercial success of the game and in delivering the educational engagement. As part of an Interface-funded research project Abertay University worked with the independent games company, Space Budgie, to enhance the user experience of their educational game Glitchspace. The game aimed to teach basic coding principles and terminology in an entertaining way. The game sets the player inside a Mondrian-inspired cyberspace world where to progress the player needs to reprogramme the world around them to solve puzzles. The main objective of the academic-industry collaborative project was to analyse the user experience (UX) of the game to increase its educational value for a standalone educational version. The UX design focused on both pragmatic and hedonic qualities such playability, usability and the psychological impact of the game. The empirical study of the UX design allowed all parties to develop a deeper understanding of how the game was being played and the initial reactions to the game by the player. The core research question that the study sought to answer was whether when designing an educational game, UX design could improve philosophical concepts like motivation and engagement to foster better learning experiences.</p