Multi-Moji: Combining Thermal, Vibrotactile and Visual Stimuli to Expand the Affective Range of Feedback

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of emotions within two-dimensional valencearousal space. This paper is the first to systematically combine multiple modalities to expand the available affective range. Three studies were conducted: Study 1 measured the emotionality of vibrotactile feedback by itself; Study 2 measured the perceived emotional content of three bimodal combinations: vibrotactile + thermal, vibrotactile + visual and visual + thermal. Study 3 then combined all three modalities. Results show that combining modalities increases the available range of emotional states, particularly in the problematic top-right and bottom-left quadrants of the dimensional model. We also provide a novel lookup resource for designers to identify stimuli to convey a range of emotions

A Controlled Study on Evaluation of Thermal Stimulation Influence on Affective Measures of Uninformed Individuals

Although the relationship between temperature and emotional states has been investigated in the field of haptics, it remains unknown if, or in what direction, temperature affects emotional states. We approach this question at the intersection of haptics and psychology using a custom-built thermal device and emotional responses based on photos from the International Affective Picture System (IAPS) library. Unlike past research, this study incorporates deception and a control (i.e., neutral temperature) condition. One hundred and twenty naive subjects reported their emotional responses to fifty-six images varying on normative arousal and valence ratings while being exposed to a cool~(30{\deg}C), neutral (33{\deg}C), or warm (36{\deg}C) temperature applied to the upper back. Participants exposed to warm temperatures reported higher arousal ratings in some image categories than participants exposed to neutral or cool temperatures. Valence ratings were decreased in warm conditions compared to neutral conditions. The emotion wheel was used as a complementary method of affective response measurement, and exploratory analysis methods were implemented. Although the valence and arousal showed statistical significance, the emotion wheel results did not demonstrate any significant differences between the temperature conditions

The effect of thermal stimuli on the emotional perception of images

Thermal stimulation is a feedback channel that has the potential to influence the emotional response of people to media such as images. While previous work has demonstrated that thermal stimuli might have an effect on the emotional perception of images, little is understood about the exact emotional responses different thermal properties and presentation techniques can elicit towards images. This paper presents two user studies that investigate the effect thermal stimuli parameters (e.g. intensity) and timing of thermal stimuli presentation have on the emotional perception of images. We found that thermal stimulation increased valence and arousal in images with low valence and neutral to low arousal. Thermal augmentation of images also reduced valence and arousal in high valence and arousal images. We discovered that depending on when thermal augmentation is presented, it can either be used to create anticipation or enhance the inherent emotion an image is capable of evoking

Using thermal stimuli to enhance photo-sharing in social media

Limited work has been undertaken to show how the emotive ability of thermal stimuli can be used for interaction purposes. One potential application area is using thermal stimuli to influence emotions in images shared online such as social media platforms. This paper presents a two-part study, which examines how the documented emotive property of thermal stimuli can be applied to enhance social media images. Participants in part-one supplied images from their personal collection or social media profiles, and were asked to augment each image with thermal stimuli based on the emotions they wanted to enhance or reduce. Part-one participants were interviewed to understand the effects they wanted augmented images to have. In part-two, these augmented images were perceived by a different set of participants in a simulated social media interface. Results showed strong agreement between the emotions augmented images were designed to evoke and the emotions they actually evoked as perceived by part-two participants. Participants in part-one selected thermal stimuli augmentation intended to modulate valence and arousal in images as a way of enhancing the realism of the images augmented. Part-two results indicate this was achieved as participants perceived thermal stimuli augmentation reduced valence in negative images and modulated valence and arousal in positive images

Using thermal stimuli to influence affect in different picture display sizes

The ability of images to evoke emotions in people has been well documented in previous research, as well as the differences in the emotional perception of images when viewed on different-sized screens and device types. The ability of thermal stimuli to evoke emotions in people when used for media augmentation has also been examined. However little is known about how thermal stimuli can be used to enhance or reduce affect in images with varying emotional properties displayed in different sizes or on different devices. To the best of our knowledge no work has been conducted to investigate if there is any difference in the effect thermal augmentation has on images displayed in different sizes on different device types. This paper presents two user studies to address this research gap. Study 1 explored the effect thermal stimulation has on images displayed in different sizes. Images were displayed in sizes corresponding to the full screen display of a laptop, tablet and mobile phone. In study 2 we examined whether the actual presentation device (tablet and mobile) plays a role in the emotional perception of images displayed on mobile devices. Results showed that thermal augmentation was most effective in modulating emotions in small-sized pictures (427x240 pixels display size) and pictures displayed on a mobile phone. Thermal stimuli also reduced emotions in medium display sizes (corresponding to the full screen display of a tablet)

ThermalWear: Exploring wearable on-chest thermal displays to augment voice messages with affect

Voice is a rich modality for conveying emotions, however emotional prosody production can be situationally or medically impaired. Since thermal displays have been shown to evoke emotions, we explore how thermal stimulation can augment perception of neutrally-spoken voice messages with affect. We designed ThermalWear, a wearable on-chest thermal display, then tested in a controlled study (N=12) the effects of fabric, thermal intensity, and direction of change. Thereafter, we synthesized 12 neutrally-spoken voice messages, validated (N=7) them, then tested (N=12) if thermal stimuli can augment their perception with affect. We found warm and cool stimuli (a) can be perceived on the chest, and quickly without fabric (4.7-5s) (b) do not incur discomfort (c) generally increase arousal of voice messages and (d) increase / decrease message valence, respectively. We discuss how thermal displays can augment voice perception, which can enhance voice assistants and support individuals with emotional prosody impairments

Augmenting communication technologies with non-primary sensory modalities

Humans combine their senses to enhance the world around them. While computers have evolved to reflect these sensory demands, only the primary senses of vision and audition (and to an extent, touch) are used in modern communication. This thesis investigated how additional information, such as emotion and navigational assistance, might be communicated using technology-based implementations of sensory displays that output the non-primary modalities of smell, vibrotactile touch, and thermo-touch. This thesis explored using a portable atomiser sprayer to deliver emotional information via smell to mobile phone users, a ring-shaped device worn on the finger to display emotional information using vibration and colours, and an array of thermoelectric coolers worn on the arm to create temperature sensations. Additionally, this thesis explored two methods of signalling temperature using the thermal implementation, and finally, used it in a controlled study to augment the perceived emotion of text messages using temperature. There were challenges with using some of these implementations to display information. Smells produced with the scent technology were ambiguous and highly cognitive, and poor delivery to the user produced undesirable cross-adaption effects when smells lingered and mixed in the environment. The device used to communicate vibrotactile and colour lighting cues neutralized emotions in text messages. Furthermore, temperature pattern discrimination using the thermal implementation was difficult due to non-linear interaction effects that occurred on the skin’s surface, as well as latency resulting from the thermal neurological pathway and the technology used to heat and cool the skin. However, the thermal implementation enabled more accurate user discrimination between thermal signals than what a single stimulator design provided. Furthermore, the utility of continuous thermal feedback, in the context of spatial navigation, was demonstrated, which improved user performance compared to when the user was not presented with any thermal information. Finally, temperature was demonstrated to elicit arousal reactions across subjects using the thermal implementation, and could augment the arousal of text messages, especially when the content of the message was strongly neutral. However, no similar statistical significance was observed with valence, demonstrating the complex implications of using thermal cues to convey emotional information

Augmenting media with thermal stimulation

Thermal interfaces are a new area of research in HCI, with one of their main benefits being the potential to influence emotion. To date, studies investigating thermal feedback for affective interaction have either provided concepts and prototypes, or looked at the affective element of thermal stimuli in isolation. This research is the first to look in-depth at how thermal stimuli can be used to influence the perception of different media. We conducted two studies which looked at the effect of thermal stimuli on subjective emotional responses to media. In the first we presented visual information designed to evoke emotional responses in conjunction with different thermal stimuli. In the second we used different methods to present thermal stimuli in conjunction with music. Our results highlight the possibility of using thermal stimuli to create more affective interactions in a variety of media interaction scenarios