A Nine Month Report on Progress Towards a Framework for Evaluating Advanced Search Interfaces considering Information Retrieval and Human Computer Interaction

This is a nine month progress report detailing my research into supporting users in their search for information, where the questions, results or even thei

Scoping analytical usability evaluation methods: A case study

Analytical usability evaluation methods (UEMs) can complement empirical evaluation of systems: for example, they can often be used earlier in design and can provide accounts of why users might experience difficulties, as well as what those difficulties are. However, their properties and value are only partially understood. One way to improve our understanding is by detailed comparisons using a single interface or system as a target for evaluation, but we need to look deeper than simple problem counts: we need to consider what kinds of accounts each UEM offers, and why. Here, we report on a detailed comparison of eight analytical UEMs. These eight methods were applied to it robotic arm interface, and the findings were systematically compared against video data of the arm ill use. The usability issues that were identified could be grouped into five categories: system design, user misconceptions, conceptual fit between user and system, physical issues, and contextual ones. Other possible categories such as User experience did not emerge in this particular study. With the exception of Heuristic Evaluation, which supported a range of insights, each analytical method was found to focus attention on just one or two categories of issues. Two of the three "home-grown" methods (Evaluating Multimodal Usability and Concept-based Analysis of Surface and Structural Misfits) were found to occupy particular niches in the space, whereas the third (Programmable User Modeling) did not. This approach has identified commonalities and contrasts between methods and provided accounts of why a particular method yielded the insights it did. Rather than considering measures such as problem count or thoroughness, this approach has yielded insights into the scope of each method

An analytical inspection framework for evaluating the search tactics and user profiles supported by information seeking interfaces

Searching is something we do everyday both in digital and physical environments. Whether we are searching for books in a library or information on the web, search is becoming increasingly important. For many years, however, the standard for search in software has been to provide a keyword search box that has, over time, been embellished with query suggestions, Boolean operators, and interactive feedback. More recent research has focused on designing search interfaces that better support exploration and learning. Consequently, the aim of this research has been to develop a framework that can reveal to designers how well their search interfaces support different styles of searching behaviour.The primary contribution of this research has been to develop a usability evaluation method, in the form of a lightweight analytical inspection framework, that can assess both search designs and fully implemented systems. The framework, called Sii, provides three types of analyses: 1) an analysis of the amount of support the different features of a design provide; 2) an analysis of the amount of support provided for 32 known search tactics; and 3) an analysis of the amount of support provided for 16 different searcher profiles, such as those who are finding, browsing, exploring, and learning. The design of the framework was validated by six independent judges, and the results were positively correlated against the results of empirical user studies. Further, early investigations showed that Sii has a learning curve that begins at around one and a half hours, and, when using identical analysis results, different evaluators produce similar design revisions.For Search experts, building interfaces for their systems, Sii provides a Human-Computer Interaction evaluation method that addresses searcher needs rather than system optimisation. For Human-Computer Interaction experts, designing novel interfaces that provide search functions, Sii provides the opportunity to assess designs using the knowledge and theories generated by the Information Seeking community. While the research reported here is under controlled environments, future work is planned that will investigate the use of Sii by independent practitioners on their own projects

Systematic evaluation of design choices for software development tools

[Abstract]: Most design and evaluation of software tools is based on the intuition and experience of the designers. Software tool designers consider themselves typical users of the tools that they build and tend to subjectively evaluate their products rather than objectively evaluate them using established usability methods. This subjective approach is inadequate if the quality of software tools is to improve and the use of more systematic methods is advocated. This paper summarises a sequence of studies that show how user interface design choices for software development tools can be evaluated using established usability engineering techniques. The techniques used included guideline review, predictive modelling and experimental studies with users

Applicability of HCI Techniques to Systems Interface Design

PhDThis thesis seeks to identify reasons why HCI techniques are unsuitable for application in real world design projects. User-oriented systems design and evaluation require that many considerations such as the psychology of users, the applications and target tasks be born in mind simultaneously. A selection of influential HCI design and evaluative techniques from HCI research literature are reviewed and characterised in terms of their analytic scope. Two studies of systems designers' approaches to user-oriented design and evaluation were carried out in order to gain a clearer picture of the design process as it occurs in applied and commercial projects. It was found that designers frequently lack adequate information about users, carrying Out, at best, informal user-evaluations of prototypes. Most notably HCI design and evaluative techniques, of the type common in the literature, are not being used in applied and commercial design practice. They seem to be complex, often limited in scope, and possessed of inadequate or unrepresentative views of the design process within which they might be applied. It was noted that design practice is highly varied with only a small number of common goal directed classes of activity being identified. These together with observed user-oriented information sources and design constraints provide a useful schema for viewing applied and commercial design practice. A further study of HCI specialists' practice in commercial environments was undertaken, in order to identify particular user-oriented design approaches and HCI techniques suitable for application in practice. The specialists were able to describe desirable, and undesirable properties of the techniques they used which made it possible to identify a list of specific desirable features for HCI techniques. A framework for assessing applicability of HCI techniques was developed from the findings of the thesis. This is demonstrated using an example project from the design studies and may prove valuable in supporting design, evaluation, critiquing and selection of HCI techniques

Multi-modal usability evaluation.

Research into the usability of multi-modal systems has tended to be device-led, with a resulting lack of theory about multi-modal interaction and how it might differ from more conventional interaction. This is compounded by a confusion over the precise definition of modality between the various disciplines within the HCI community, how modalities can be effectively classified, and their usability properties. There is a consequent lack of appropriate methodologies and notations to model such interactions and assess the usability implications of these interfaces. The role of expertise and craft skill in using HCI techniques is also poorly understood. This thesis proposes a new definition of modality, and goes on to identify issues of importance to multi-modal usability, culminating in the development of a new methodology to support the identification of such usability issues. It additionally explores the role of expertise and craft skill in using usability modelling techniques to assess usability issues. By analysing the problems inherent in current definitions and approaches, as well as issues relevant to cognitive science, a clear understanding of both the requirements for a suitable definition of modality and the salient usability issues are obtained. A novel definition of modality, based on the three elements of sense, information form and temporal nature is proposed. Further, an associated taxonomy is produced, which categorises modalities within the sensory dimension as visual, acoustic and haptic. This taxonomy classifies modalities within the information form dimension as lexical, symbolic or concrete, and classifies the temporal form dimension modalities as discrete, continuous, or dynamic. This results in a twenty-seven cell taxonomy, with each cell representing one taxon, indicating one particular type of modality. This is a faceted classification system, with the modality named after the intersection of the categories, building the category names into a compound modality name. The issues surrounding modality are examined and refined into the concepts of modality types, properties and clashes. Modalities are identified as belonging to either the system or the user, and being expressive or receptive in type. Various properties are described based on issues of granularity and redundancy. The five different types of clashes are described. Problems relating to the modelling of multi-modal interaction are examined by means of a motivating case study based on a portion of an interface for a robotic arm. The effectiveness of five modelling techniques, STN, CW, CPM-GOMS, PUM and Z, in representing multi-modal issues are assessed. From this, and using the collated definition, taxonomy and theory, a new methodology, Evaluating Multi-modal Usability (EMU), is developed. This is applied to a previous case study of the robotic arm to assess its application and coverage. Both the definition and EMU are used by students in a case study to test the definition and methodology's effectiveness, and to examine the leverage such an approach may give. The results shows that modalities can be successfully identified within an interactive context, and that usability issues can be described. Empirical video data of the robotic arm in use is used to confirm the issues identified by the previous analyses, and to identify new issues. A rational re-analysis of the six approaches (STN, CW, CPM-GOMS, PUM, Z and EMU) is conducted in order to distinguish between issues identified through craft skill, based on general HCI expertise and familiarity with the problem, and issues identified due to the core of the method for each approach. This is to gain a realistic understanding of the validity of claims made by each method, and to identify how else issues might be identified, and the consequent implications. Craft skill is found to have a wider role than anticipated, and the importance of expertise in using such approaches emphasised. From the case study and the re-analyses the implications for EMU are examined, and suggestions made for future refinement. The main contributions of this thesis are the new definition, taxonomy and theory, which significantly contribute to the theoretical understanding of multi-modal usability, helping to resolve existing confusion in this area. The new methodology, EMU, is a useful technique for examining interfaces for multi-modal usability issues, although some refinement is required. The importance of craft skill in the identification of usability issues has been explicitly explored, with implications for future work on usability modelling and the training of practitioners in such techniques

Towards an activity-oriented design method for HCI research and practice

During the last two decades, designing for usability has been the focus of attention when developing computer systems. However, the dynamic nature of human use of computer systems has meant that designing for 'usefulness' or 'fitness for purpose' is increasingly becoming the primary concern for systems developers. Central to this concern are issues underpinned by the social context in which a computer user operates.Within the field of Human-Computer Interaction (HCI), this situation led to a search for appropriate theories for conceptualising these design concerns. Whilst Activity Theory has been identified as a suitable framework for conceptualising these user perspectives, the lack of a standard methodology for applying it to HCI research and practice has meant that many systems developers have failed to benefit from the richness of this framework. The objective of this thesis was therefore to develop an Activity Theory based methodology for HCI research and practice. This thesis, contributes the `Activity-Oriented Design Methodology' (AODM) both as a practical and analytical methodology for using Activity Theory within HCI design. AODM incorporates four methodological tools namely:- The `Eight-Step-Model'- The `Activity Notation'- The technique of `Generating Research Questions'- The technique of `Mapping AODM Operational Processes'AODM tools were constructed from empirical work carried out as part of this research. Empirical analysis of work practices in two organisations was conducted for a period of two years using Activity Theory. This empirical work formed the basis for validating AODM. AODM tools support the systems design processes of gathering, analysing and communicating (through modelling) research and design insights from an Activity Theory perspective. It is argued that AODM provides a valuable practical and analytical methodology for operationalising Activity Theory within HCI so as to support early phases of systems design: namely, requirements capture and evaluation