Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing

The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants' subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems.Comment: 10 pages, 8 figures, published in ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17

Human Factors Standards and the Hard Human Factor Problems: Observations on Medical Usability Standards

With increasing variety and sophistication of computer-based medical devices, and more diverse users and use environments, usability is essential, especially to ensure safety. Usability standards and guidelines play an important role. We reviewed several, focusing on the IEC 62366 and 60601 sets. It is plausible that these standards have reduced risks for patients, but we raise concerns regarding: (1) complex design trade-offs that are not addressed, (2) a focus on user interface design (e.g., making alarms audible) to the detriment of other human factors (e.g., ensuring users actually act upon alarms they hear), and (3) some definitions and scope restrictions that may create “blind spots”. We highlight potential related risks, e.g. that clear directives on “easier to understand” risks, though useful, may preclude mitigating other, more “difficult” ones; but ask to what extent these negative effects can be avoided by standard writers, given objective constraints. Our critique is motivated by current research and incident reports, and considers standards from other domains and countries. It is meant to highlight problems, relevant to designers, standards committees, and human factors researchers, and to trigger discussion about the potential and limits of standards

Criteria for assessment of welfare impacts within regional planning of transport

Regional transport system plans present the wide framework for local transport planning, and are the means of applying the national high-level objectives regionally. Transport system plans are usually created interactively with the regional land use planning. Prediction and appraisal of the potential impacts of the alternatives under discussion form an important part of the planning process. This paper presents Finnish experience in defining criteria for assessing the potential welfare impacts of a transport plan or policy. These criteria are to be used co-operatively by the planners and decision-makers. An important issue is the link between the formation of the objectives and the actual evaluation of impacts, as well as the underlying values and views that influence the selection of planning methods and practises. The criteria were drafted based on a literature review and preliminary discussions with national transport authorities. The preliminary criteria were selected and put into order by using the multi-criteria method MACBETH. The process involved active co-operation with the national transport authorities, as well as with other interest groups. The draft criteria were prioritised based on the opinions of selected representatives of national and regional authorities and transport planners. In this paper we focus on the priorisation process of the criteria, as well as discuss the validity and usability of the criteria created. Furthermore, the suitability of multi-criteria analysis in the context of regional transport planning is discussed, in addition to the consistency requirements between the criteria and the multi-criteria model. The criteria have been created in an ongoing Finnish study that aims at supporting regional transport planning, concerning the impacts that changes in the transport system may have on the welfare of the communities and individuals. Furthermore, the suitability of certain Multi-Criteria Decision Aid (MCDA) methods in creating consensus in the planning process is tested.

Designing novel applications inspired by emerging media technologies

The field of Human-Computer Interaction provides a number of useful tools and methods for obtaining information on end-users and their usage context to inform the design of computer systems, yet relatively little is known on how to go about designing for a completely novel application where there is no user base, no existing practice of use available at the start. The success of the currently available HCI methodology that focuses on understanding users’ needs and establishing requirements is well-deserved in making computing applications usable in terms of fitting them to end-users’ usage contexts. However, too much emphasis on identifying user needs tends to stifle other more exploratory design activities where new types of applications are invented in order to discover or create new activities currently not practiced. In this paper, we argue that a great starting point of novel application design is not the problem space (trying to rigorously define the user requirements) but the solution space (trying to leverage emerging computational technologies and growing design knowledge for various interaction platforms), and we build a foundation for a pragmatic design methodology supported by the authors’ extensive experience in designing novel applications inspired by emerging media technologies

Digital libraries in a clinical setting: Friend or foe?

Clinical requirements for quick accessibility to reputable, up-to-date information have increased the importance of web accessible digital libraries for this user community. To understand the social and organisational impacts of ward-accessible digital libraries (DLs) for clinicians, we conducted a study of clinicians. perceptions of electronic information resources within a large London based hospital. The results highlight that although these resources appear to be a relatively innocuous means of information provision (i.e. no sensitive data) social and organisational issues can impede effective technology deployment. Clinical social structures, which produce information. and technology. hoarding behaviours can result from poor training, support and DL usability

Towards a kansei-based user modeling methodology for eco-design

We propose here to highlight the benefits of building a framework linking Kansei Design (KD), User Centered Design (UCD) and Eco-design, as the correlation between these fields is barely explored in research at the current time. Therefore, we believe Kansei Design could serve the goal of achieving more sustainable products by setting up an accurate understanding of the user in terms of ecological awareness, and consequently enhancing performance in the Eco-design process. In the same way, we will consider the means-end chain approach inspired from marketing research, as it is useful for identifying ecological values, mapping associated functions and defining suitable design solutions. Information gathered will serve as entry data for conducting scenario-based design, and supporting the development of an Eco-friendly User Centered Design methodology (EcoUCD).ANR-ECOUS

Interaction platform-orientated perspective in designing novel applications

The lack of HCI offerings in the invention of novel software applications and the bias of design knowledge towards desktop GUI make it difficult for us to design for novel scenarios and applications that leverage emerging computational technologies. These include new media platforms such as mobiles, interactive TV, tabletops and large multi-touch walls on which many of our future applications will operate. We argue that novel application design should come not from user-centred requirements engineering as in developing a conventional application, but from understanding the interaction characteristics of the new platforms. Ensuring general usability for a particular interaction platform without rigorously specifying envisaged usage contexts helps us to design an artifact that does not restrict the possible application contexts and yet is usable enough to help brainstorm its more exact place for future exploitation

Improving Usability of Interactive Graphics Specification and Implementation with Picking Views and Inverse Transformations

Specifying and programming graphical interactions are difficult tasks, notably because designers have difficulties to express the dynamics of the interaction. This paper shows how the MDPC architecture improves the usability of the specification and the implementation of graphical interaction. The architecture is based on the use of picking views and inverse transforms from the graphics to the data. With three examples of graphical interaction, we show how to express them with the architecture, how to implement them, and how this improves programming usability. Moreover, we show that it enables implementing graphical interaction without a scene graph. This kind of code prevents from errors due to cache consistency management

Intangible trust requirements - how to fill the requirements trust "gap"?

Previous research efforts have been expended in terms of the capture and subsequent instantiation of "soft" trust requirements that relate to HCI usability concerns or in relation to "hard" tangible security requirements that primarily relate to security a ssurance and security protocols. Little direct focus has been paid to managing intangible trust related requirements per se. This 'gap' is perhaps most evident in the public B2C (Business to Consumer) E- Systems we all use on a daily basis. Some speculative suggestions are made as to how to fill the 'gap'. Visual card sorting is suggested as a suitable evaluative tool; whilst deontic logic trust norms and UML extended notation are the suggested (methodologically invariant) means by which software development teams can perhaps more fully capture hence visualize intangible trust requirements