Augmenting communication technologies with non-primary sensory modalities

Humans combine their senses to enhance the world around them. While computers have evolved to reflect these sensory demands, only the primary senses of vision and audition (and to an extent, touch) are used in modern communication. This thesis investigated how additional information, such as emotion and navigational assistance, might be communicated using technology-based implementations of sensory displays that output the non-primary modalities of smell, vibrotactile touch, and thermo-touch. This thesis explored using a portable atomiser sprayer to deliver emotional information via smell to mobile phone users, a ring-shaped device worn on the finger to display emotional information using vibration and colours, and an array of thermoelectric coolers worn on the arm to create temperature sensations. Additionally, this thesis explored two methods of signalling temperature using the thermal implementation, and finally, used it in a controlled study to augment the perceived emotion of text messages using temperature. There were challenges with using some of these implementations to display information. Smells produced with the scent technology were ambiguous and highly cognitive, and poor delivery to the user produced undesirable cross-adaption effects when smells lingered and mixed in the environment. The device used to communicate vibrotactile and colour lighting cues neutralized emotions in text messages. Furthermore, temperature pattern discrimination using the thermal implementation was difficult due to non-linear interaction effects that occurred on the skin’s surface, as well as latency resulting from the thermal neurological pathway and the technology used to heat and cool the skin. However, the thermal implementation enabled more accurate user discrimination between thermal signals than what a single stimulator design provided. Furthermore, the utility of continuous thermal feedback, in the context of spatial navigation, was demonstrated, which improved user performance compared to when the user was not presented with any thermal information. Finally, temperature was demonstrated to elicit arousal reactions across subjects using the thermal implementation, and could augment the arousal of text messages, especially when the content of the message was strongly neutral. However, no similar statistical significance was observed with valence, demonstrating the complex implications of using thermal cues to convey emotional information

Connecting Couples in Long-Distance Relationships : Towards Unconventional Computer-Mediated Emotional Communication Systems

The number of couples who find themselves in a long-distance relationship (LDR) is increasing for a wide range of reasons, such as overseas employment, academic pursuits, military duty, and similar circumstances. With the myriad of communication channels enabled by the low cost and ubiquity of computer-mediated communication technologies, couples in LDRs are able to stay in touch with each other around the globe. However, recent studies have revealed that the mainstream communication tools are inadequate to support the full spectrum of communication needed in intimate relationships. Emotional communication is one of the fundamental needs in close relationships, as it forms an important part of intimacy. This dissertation argues that there is a gap between what is known about LDR couples’ needs in research and what has been implemented for them in practice. The aim of this work is to bridge this gap by mediating emotional communication through unconventional user interfaces that use interaction solutions outside of the scope of their conventional use, with a particular focus on couples who sustain a committed LDR. Here, taking research through design as a core approach, a variety of qualitative methods were employed to seek answers to the research questions. This dissertation includes eight case studies, each of which is dedicated to answering its corresponding research question(s). Study I presents a systematic literature review which explored the current state of the art and identified the design opportunities. Study II introduces a series of co-design activities with five couples in LDRs to reveal the needs and challenges of users in an LDR. Studies III and IV propose two functional prototypes for unconventional communication systems to connect couples in LDRs. Study V showcases 12 design concepts of wearables created by the participants to support their own LDR. Study VI describes how four low-resolution prototypes created for mediating LDRs by the participants in the workshop would be used in real-world contexts. Studies VII and VIII each present a novel design tool to be used as a scaffold when designing communication systems for supporting LDRs: specifically, a conceptual design framework and a card-based design toolkit. This dissertation contributes new knowledge to the field of human-computer interaction through design interventions. It showcases a spectrum of practices which can be seen as a first step towards mediating emotional communication for couples in LDRs using unconventional communication systems. The findings comprise theoretical and empirical insights—derived from the eight case studies in which the author identified design opportunities and design considerations—relating to how couples in LDRs can be better supported by unconventional computer-mediated emotional communication systems

A Framework For Abstracting, Designing And Building Tangible Gesture Interactive Systems

This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power.Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone. Poiché l’interazione gestuale tangibile è un campo di studio relativamente recente, questa tesi presenta un framework (quadro teorico) che ha lo scopo di assistere lavori futuri in questo campo. Il Framework per l’Interazione Gestuale Tangibile fornisce supporto su tre livelli. Per prima cosa, aiuta a riflettere da un punto di vista teorico sui diversi tipi di gesti tangibili che possono essere eseguiti fisicamente, grazie a una tassonomia basata su tre componenti (muovere, tenere, toccare) e attributi addizionali, e che possono essere concepiti semanticamente, grazie a una tassonomia di tutti i costrutti semantici che permettono di associare dei significati ai gesti tangibili. In secondo luogo, il framework proposto aiuta a concepire nuovi sistemi interattivi basati su gesti tangibili e a ideare nuove interazioni basate su gesti con gli oggetti, attraverso linee guida per la definizione di gesti tangibili e una selezione delle migliore pratiche per i differenti campi di applicazione. Infine, il framework aiuta a implementare nuovi sistemi interattivi basati su gesti tangibili, permettendo di scegliere tra quattro differenti approcci tecnologici (incarnato e integrato negli oggetti, indossabile, distribuito nell’ambiente, o ibrido) e fornendo una guida generale per la scelta tra questi differenti approcci. Come applicazione di questo framework, questa tesi presenta anche sette sistemi interattivi basati su gesti tangibili, realizzati per tre differenti campi di applicazione: l’interazione con i sistemi di infotainment degli autoveicoli, la comunicazione interpersonale delle emozioni, e l’interazione nella casa intelligente. Per il primo campo di applicazione, sono stati progettati, sviluppati e testati quattro differenti sistemi che usano gesti tangibili effettuati sul volante come modalità di interazione con il sistema di infotainment. Per il secondo campo di applicazione, è stata concepita e sviluppata una lampada antropomorfica in grado di riconoscere i gesti tipici dell’interazione interpersonale. Per lo stesso campo di applicazione, un secondo sistema, basato su una maglietta intelligente, riconosce quando due persone si abbracciano e ricompensa questo gesto con uno scambio di informazioni digitali. Infine, per l’interazione nella casa intelligente, è stata investigata la realizzazione di uno smart watch per il riconoscimento di gesti eseguiti con oggetti tenuti nella mano. L’analisi dei sistemi interattivi esistenti basati su gesti tangibili permette di dimostrare che il framework ha un buon potere descrittivo e valutativo. Le applicazioni sviluppate durante la tesi mostrano che il framework proposto ha anche un valido potere generativo

An aesthetics of touch: investigating the language of design relating to form

How well can designers communicate qualities of touch? This paper presents evidence that they have some capability to do so, much of which appears to have been learned, but at present make limited use of such language. Interviews with graduate designer-makers suggest that they are aware of and value the importance of touch and materiality in their work, but lack a vocabulary to fully relate to their detailed explanations of other aspects such as their intent or selection of materials. We believe that more attention should be paid to the verbal dialogue that happens in the design process, particularly as other researchers show that even making-based learning also has a strong verbal element to it. However, verbal language alone does not appear to be adequate for a comprehensive language of touch. Graduate designers-makers’ descriptive practices combined non-verbal manipulation within verbal accounts. We thus argue that haptic vocabularies do not simply describe material qualities, but rather are situated competences that physically demonstrate the presence of haptic qualities. Such competencies are more important than groups of verbal vocabularies in isolation. Design support for developing and extending haptic competences must take this wide range of considerations into account to comprehensively improve designers’ capabilities

Practical, appropriate, empirically-validated guidelines for designing educational games

There has recently been a great deal of interest in the potential of computer games to function as innovative educational tools. However, there is very little evidence of games fulfilling that potential. Indeed, the process of merging the disparate goals of education and games design appears problematic, and there are currently no practical guidelines for how to do so in a coherent manner. In this paper, we describe the successful, empirically validated teaching methods developed by behavioural psychologists and point out how they are uniquely suited to take advantage of the benefits that games offer to education. We conclude by proposing some practical steps for designing educational games, based on the techniques of Applied Behaviour Analysis. It is intended that this paper can both focus educational games designers on the features of games that are genuinely useful for education, and also introduce a successful form of teaching that this audience may not yet be familiar with

Internet of Things enabled sedentary behaviour change in office workers: development and feasibility of a novel intervention (WorkMyWay)

Sedentary behaviour (SB) without breaks is associated with adverse health outcomes. The prevalence of prolonged sitting at work among office workers makes a case for SB interventions to target this setting and population. Everyday mundane objects augmented with microelectronics and ubiquitous computing represent a novel mode of delivery for behaviour change interventions enabled by the Internet of Things (IoT). However, there is insufficient research to guide the design of interventions delivered with smart objects. This research addresses this gap by developing WorkMyWay, a workplace SB intervention delivered with IoT-enabled office objects (e.g. smart water bottles and cups), and evaluating its feasibility and acceptability in an 8-week “in-the-wild” study. This thesis made 4 contributions across the disciplines of behavioural medicine and human-computer interactions (HCI). The first contribution is the development of the WorkMyWay intervention, which is informed by findings from a systematic scoping review of prior research in this field (Chapter 3), a diary-probed interview study with 20 office workers (Chapter 4), and a series of technology audit, prototyping, human-centred design, and requirement engineering processes (Chapter 5). Findings from the feasibility study (Chapter 6) suggest that despite technical issues with the data connection, participants perceive high value of WorkMyWay in changing their SB. The intervention is potentially implementable in office-based workplaces, as long as connectivity issues are fixed. Recommendations are made on improvements and a series of future studies in accordance with the Medical Research Council’s guidance on complex intervention development and evaluation. Second, this thesis deepens the theoretical understanding of SB change, by following the Behaviour Change Wheel framework (including the COM-B model, theoretical domain framework, and taxonomies of Behaviour Change Techniques (BCT)) throughout intervention design and evaluation. The intervention contents are specified using the BCT taxonomies (Chapter 5) and informed by the first published COM-B analysis of office worker’s prolonged sitting behaviour at work (Chapter 4). This allows the feasibility study (Chapter 6) to contribute to theory development by matching the interview questions and psychological measures (e.g. strength of habit) with the BCTs (e.g. action planning, prompts and cues) and associated theoretical underpinnings (e.g. goal accessibility). It also allows implementation issues to be considered in light of how well those theories and theory-informed BCTs can work in real-life settings. Third, this thesis makes a methodological contribution by documenting an interdisciplinary approach to develop a digital behaviour change intervention and a model for applying and developing theories of behaviour change in the wild. This helps address the challenge identified in Chapter 3, by bridging the gap between HCI and behavioural medicine, and catalyse the process of feeding technological innovations downstream to health practice and intervention research. Fourth, this research contributes to the HCI literature by proposing a 2×2 matrix framework to guide the design of technology for sustainable behaviour change. On one hand, the framework unifies some of the existing visions and concepts about ubiquitous computing and applies them to the context of behaviour change, by considering the type of cognitive process (automatic versus reflective, based on the dual process model) through which a persuasive design influences the behaviour. For another, the framework considers the required dosage of their technology intervention to maintain the behaviour, or the distribution of changes between the physical world and the human cognition