Multi-Moji: Combining Thermal, Vibrotactile and Visual Stimuli to Expand the Affective Range of Feedback

This paper explores the combination of multiple concurrent modalities for conveying emotional information in HCI: temperature, vibration and abstract visual displays. Each modality has been studied individually, but can only convey a limited range of emotions within two-dimensional valencearousal space. This paper is the first to systematically combine multiple modalities to expand the available affective range. Three studies were conducted: Study 1 measured the emotionality of vibrotactile feedback by itself; Study 2 measured the perceived emotional content of three bimodal combinations: vibrotactile + thermal, vibrotactile + visual and visual + thermal. Study 3 then combined all three modalities. Results show that combining modalities increases the available range of emotional states, particularly in the problematic top-right and bottom-left quadrants of the dimensional model. We also provide a novel lookup resource for designers to identify stimuli to convey a range of emotions

Social touch in human–computer interaction

Touch is our primary non-verbal communication channel for conveying intimate emotions and as such essential for our physical and emotional wellbeing. In our digital age, human social interaction is often mediated. However, even though there is increasing evidence that mediated touch affords affective communication, current communication systems (such as videoconferencing) still do not support communication through the sense of touch. As a result, mediated communication does not provide the intense affective experience of co-located communication. The need for ICT mediated or generated touch as an intuitive way of social communication is even further emphasized by the growing interest in the use of touch-enabled agents and robots for healthcare, teaching, and telepresence applications. Here, we review the important role of social touch in our daily life and the available evidence that affective touch can be mediated reliably between humans and between humans and digital agents. We base our observations on evidence from psychology, computer science, sociology, and neuroscience with focus on the first two. Our review shows that mediated affective touch can modulate physiological responses, increase trust and affection, help to establish bonds between humans and avatars or robots, and initiate pro-social behavior. We argue that ICT mediated or generated social touch can (a) intensify the perceived social presence of remote communication partners and (b) enable computer systems to more effectively convey affective information. However, this research field on the crossroads of ICT and psychology is still embryonic and we identify several topics that can help to mature the field in the following areas: establishing an overarching theoretical framework, employing better research methodologies, developing basic social touch building blocks, and solving specific ICT challenges

Touch Technology in Affective Human, Robot, Virtual-Human Interactions: A Survey

Given the importance of affective touch in human interactions, technology designers are increasingly attempting to bring this modality to the core of interactive technology. Advances in haptics and touch-sensing technology have been critical to fostering interest in this area. In this survey, we review how affective touch is investigated to enhance and support the human experience with or through technology. We explore this question across three different research areas to highlight their epistemology, main findings, and the challenges that persist. First, we review affective touch technology through the human–computer interaction literature to understand how it has been applied to the mediation of human–human interaction and its roles in other human interactions particularly with oneself, augmented objects/media, and affect-aware devices. We further highlight the datasets and methods that have been investigated for automatic detection and interpretation of affective touch in this area. In addition, we discuss the modalities of affective touch expressions in both humans and technology in these interactions. Second, we separately review how affective touch has been explored in human–robot and real-human–virtual-human interactions where the technical challenges encountered and the types of experience aimed at are different. We conclude with a discussion of the gaps and challenges that emerge from the review to steer research in directions that are critical for advancing affective touch technology and recognition systems. In our discussion, we also raise ethical issues that should be considered for responsible innovation in this growing area

Comparing technologies for conveying emotions through realistic avatars in virtual reality-based metaverse experiences

With the development of metaverse(s), industry and academia are searching for the best ways to represent users' avatars in shared Virtual Environments (VEs), where real-time communication between users is required. The expressiveness of avatars is crucial for transmitting emotions that are key for social presence and user experience, and are conveyed via verbal and non-verbal facial and body signals. In this paper, two real-time modalities for conveying expressions in Virtual Reality (VR) via realistic, full-body avatars are compared by means of a user study. The first modality uses dedicated hardware (i.e., eye and facial trackers) to allow a mapping between the user’s facial expressions/eye movements and the avatar model. The second modality relies on an algorithm that, starting from an audio clip, approximates the facial motion by generating plausible lip and eye movements. The participants were requested to observe, for both the modalities, the avatar of an actor performing six scenes involving as many basic emotions. The evaluation considered mainly social presence and emotion conveyance. Results showed a clear superiority of facial tracking when compared to lip sync in conveying sadness and disgust. The same was less evident for happiness and fear. No differences were observed for anger and surprise

Affective Affordance of Message Balloon Animations: An Early Exploration of AniBalloons

We introduce the preliminary exploration of AniBalloons, a novel form of chat balloon animations aimed at enriching nonverbal affective expression in text-based communications. AniBalloons were designed using extracted motion patterns from affective animations and mapped to six commonly communicated emotions. An evaluation study with 40 participants assessed their effectiveness in conveying intended emotions and their perceived emotional properties. The results showed that 80% of the animations effectively conveyed the intended emotions. AniBalloons covered a broad range of emotional parameters, comparable to frequently used emojis, offering potential for a wide array of affective expressions in daily communication. The findings suggest AniBalloons' promise for enhancing emotional expressiveness in text-based communication and provide early insights for future affective design.Comment: Accepted by CSCW 2023 poste

Affect Conveying Instant Messaging

Instant messaging applications cannot convey non-verbal communication through text-based messages. That can lead to an unpleasant misunderstanding between dyads when the discussion is held on a computer or smartphone. This study aims to determine if the affect conveying instant messaging applications has any usage within users who are daily users of instant messaging applications. Furthermore, does the application benefit the test users in the real variant group compared with the control group users? The tests were conducted with an instant messaging prototype application developed just for this experiment. To test the affect conveying instant messaging prototype, we gathered a test group, which were randomly divided into two different groups, those that tested the correct version and the control group. Both test groups tested the same application but with different affect conveying module or variant. The real group tested the real variant, and the random or control group tested the variant, which randomly chooses the conveyed affect or emotion. The affect is conveyed with emojis in both of the variants. After the tests were done, testers had to answer nine different interview questions. Finally, for three interview questions, testers give a grade on how satisfied they were with that particular function. The grades were analyzed with descriptive statistical methods, and the verbal interview answers were analyzed by gathering recurring themes across the answers. The study results show that the real variant of the affect conveying instant messaging prototype performed better overall than the random variant. Test users also think that the prototype and its affect conveying functionality was fun. However, they did not see any exact situations where they would use the affect conveying functionality in an instant messaging application. Testers thought that they would use it with friends and family rather than in professional life. Generally, the way emotions were conveyed in the prototype was well-received. Test users did not see any significant issues in it or if the same functionality would be used in applications like games.Pikaviestintäsovellukset eivät oletusarvoisesti välitä sanatonta viestintää tekstimuodossa olevien viestien mukana. Se on omiaan aiheuttamaan epämiellyttäviä väärinymmärryksiä keskusteluosapuolien välillä, kun keskustelu käydään tietokoneiden tai älypuhelimien avulla. Tämän tutkielman tavoitteena on selvittää, onko tunteita automaattisesti välittävällä pikaviestintäsovelluksella käyttöä niiden käyttäjien keskuudessa, jotka käyttävät pikaviestintäsovelluksia päivittäin omassa elämässä. Lisäksi selvitämme, tuoko testisovellus hyötyä sovelluksen oikeaa versiota testanneille verrattuna kontrolliryhmään. Testit ovat suoritettu sille erikseen luodulla pikaviestintäsovelluksella. Testatakseen testiin erikseen tehtyä pikaviestintäsovellus prototyyppiä, kokosimme testiryhmän, joka satunnaisesti jaettiin kahteen eri ryhmään, oikeaa versiota testanneisiin ja kontrolliryhmään. Molemmat testiryhmät testasivat samaa sovellusta, mutta joissa olivat kuitenkin eri affektiivinen moduuli. Todellisen testin suorittaneet testasivat niin sanottua oikeaa versiota, jossa tunteiden välitys toimi kuten sen oli tarkoitus toimia ja kontrolliryhmä testasi sovelluksen versiota, jossa tunteet, joita välitettiin, olivat sovelluksen satunnaisesti valitsemia tunteita, eivätkä oikeasti sovelluksen analysoimia tunteita käyttäjästä. Sovellusprototyypin testin jälkeen testaajat vastasivat yhdeksään haastattelukysymykseen, joista kolme kysymystä olivat sellaisia, joihin testaajan tuli antaa numeerinen arvosana siten kuinka tyytyväisiä he olivat kyseiseen toiminnallisuuteen. Numeeriset arvosanat analysoitiin kuvailevia tilastointimenetelmiä käyttäen ja sanalliset haastatteluvastaukset analysointiin teemoittamalla haastattelut ja löytämällä sieltä toistuvia teemoja. Tutkimustulokset osoittavat, että pikaviestintäsovellus prototyypin niin sanottu oikea versio toimi yleisesti paremmin mitä satunnaistettu tunteiden välitys. Testaajat myös ajattelivat, että prototyyppi ja sen automaattinen tunteidenvälitys oli hauskaa ja mielekästä seurattavaa keskustelun aikana. Testaajat eivät kuitenkaan haastattelun aikana löytäneet mitään tarkkaa reaalielämän käyttöä sovellukselle. Testaajat käyttäisivät sovellusta mieluimmin läheisten ja jo tuntemien ihmisten kanssa kuin esimerkiksi työelämässä. Testaajat yleisesti pitivät pikaviestintäsovelluksen tunteiden välityksestä, ja he eivät nähneet mitään suurta ongelmaa sen toiminnan kanssa tai että samanlainen toiminto olisi vapaaehtoisesti käytössä joissain muissa sovelluksissa, kuten videopeleissä