Evaluation Strategies for HCI Toolkit Research

Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementation of interactive systems. For publication, the HCI community typically expects toolkit research to include an evaluation component. The problem is that toolkit evaluation is challenging, as it is often unclear what ‘evaluating’ a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and discussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strategies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy

ThermoPixels:Toolkit for Personalizing Arousal-based Interfaces through Hybrid Crafting

Much research has shown the potential of affective interfaces to support people reflect on, and understand their bodily responses. Yet, people find it difficult to engage with, and understand their biodata which they have limited prior experience with. Building on affective interfaces and material-centered design, we developed ThermoPixels, a toolkit including thermochromic and heating materials, as well as galvanic skin response sensors for creating representations of physiological arousal. Within 10 workshops, 20 participants created personalized representations of physiological arousal and its real-time changes using the toolkit. We report on participants’ material exploration, their experience of creating shapes and the use of colors for emotional awareness and regulation. Reflecting on our findings, we discuss embodied exploration and creative expression, the value of technology in emotion regulation and its social context, and the importance of understanding material limitations for effective sense-making

OWidgets: A toolkit to enable smell-based experience design

Interactive technologies are transforming the ways in which people experience, interact and share information. Advances in technology have made it possible to generate real and virtual environments with breath-taking graphics and high-fidelity audio. However, without stimulating the other senses such as touch and smell, and even taste in some cases, such experiences feel hollow and fictitious; they lack realism. One of the main stumbling blocks for progress towards creating truly compelling multisensory experiences is the lack of appropriate tools and guidance for designing beyond audio-visual applications. Here we focus particularly on the sense of smell and how smell-based design can be enabled to create novel user experiences. We present a design toolkit for smell (i.e., OWidgets). The toolkit consists of a graphical user interface and the underlying software framework. The framework uses two main components: a Mapper and Scheduler facilitating the device-independent replication of olfactory experiences. We discuss how our toolkit reduces the complexity of designing with smell and enables a creative exploration based on specific design features. We conclude by reflecting on future directions to extend the toolkit and integrate it into the wider audio-visual ecosystem

MoPeDT: A Modular Head-Mounted Display Toolkit to Conduct Peripheral Vision Research

Peripheral vision plays a significant role in human perception and orientation. However, its relevance for human-computer interaction, especially head-mounted displays, has not been fully explored yet. In the past, a few specialized appliances were developed to display visual cues in the periphery, each designed for a single specific use case only. A multi-purpose headset to exclusively augment peripheral vision did not exist yet. We introduce MoPeDT: Modular Peripheral Display Toolkit, a freely available, flexible, reconfigurable, and extendable headset to conduct peripheral vision research. MoPeDT can be built with a 3D printer and off-the-shelf components. It features multiple spatially configurable near-eye display modules and full 3D tracking inside and outside the lab. With our system, researchers and designers may easily develop and prototype novel peripheral vision interaction and visualization techniques. We demonstrate the versatility of our headset with several possible applications for spatial awareness, balance, interaction, feedback, and notifications. We conducted a small study to evaluate the usability of the system. We found that participants were largely not irritated by the peripheral cues, but the headset's comfort could be further improved. We also evaluated our system based on established heuristics for human-computer interaction toolkits to show how MoPeDT adapts to changing requirements, lowers the entry barrier for peripheral vision research, and facilitates expressive power in the combination of modular building blocks.Comment: Accepted IEEE VR 2023 conference pape

DiLogics: Creating Web Automation Programs With Diverse Logics

Knowledge workers frequently encounter repetitive web data entry tasks, like updating records or placing orders. Web automation increases productivity, but translating tasks to web actions accurately and extending to new specifications is challenging. Existing tools can automate tasks that perform the same logical trace of UI actions (e.g., input text in each field in order), but do not support tasks requiring different executions based on varied input conditions. We present DiLogics, a programming-by-demonstration system that utilizes NLP to assist users in creating web automation programs that handle diverse specifications. DiLogics first semantically segments input data to structured task steps. By recording user demonstrations for each step, DiLogics generalizes the web macros to novel but semantically similar task requirements. Our evaluation showed that non-experts can effectively use DiLogics to create automation programs that fulfill diverse input instructions. DiLogics provides an efficient, intuitive, and expressive method for developing web automation programs satisfying diverse specifications

Creative Toolkits for TIPS

We present a survey of toolkits employed in research workshop approaches within TIPS (Trust, Identity, Privacy and Security) domains. Our survey was developed within wider design research to develop prototypes that support people in evaluating whether to trust that an online actor’s identity is not recently faked, and that a service they are registering personal information with is legitimate; and a subsequent project involving a tool that invites people to reflect on the cumulative risks of sharing apparently harmless personal information online. The radically multidisciplinary nature of both these TIPS projects has determined that we create a research space to promote exchange to, as design researchers, better understand the ‘opaque’ immediate and longer term implications of our proposed services and invite cross-disciplinary discussion towards interdisciplinary understandings. This paper is intended as an at-a-glance resource, or indeed toolkit, for researchers from a range of disciplinary backgrounds working on TIPS research to inform on various different material engagements, with research stakeholders, through creative workshop approaches. Our survey focused on the literature from Design (especially Participatory Design and Codesign), Human Computer Interaction (HCI) and cybersecurity. It comprises 27 papers or toolkit examples organised across: review papers; example toolkits; case studies reporting relevant toolkit use; applied toolkits for learning/knowledge exchange; research toolkits focused on demonstrating a methodological-conceptual approach (some problematising emergent or near-future technologies); and two papers that straddled the latter two categories, focusing on future practical application. We begin with an overview of our rationale and method before presenting each group of texts in a table alongside a summary discussion. We go on to discuss the various material components, affordances and terminology of the toolkits along with core concerns often left out of the reporting of research; before going on to recognise toolkits not such much as a tool that identify and fix things but as a lose collection of readily available resources, used in particular socio-approaches that together help surface techno-relational vulnerabilities and contingencies in TIPS-related discourse