Recently, the field of Human-Computer Interaction has begun to embrace the crucial role of our bodies across our cognitive and

social processes. The movement of embodied interaction has been followed by a somatic and material turn that strives to design

technology for richer multi-sensory somatic experiences. Despite the booming developments in tangible and wearable technologies,

there is strikingly limited research exploring the potential of our own physical and virtual bodies to become the material for the design

of interaction, in the same ways that tangible materials are. In this position paper, we propose a series of approaches for designing

technology for social interaction, that integrate human bodies as interaction materials to elicit a multisensory embodied experience.

We illustrate this approach with examples of several virtual reality experiences that we have designed for supporting social connection

Desnoyers-Stewart, J. (John)

El Ali, A. (Abdallah)

Huisman, G. (Gijs)

Riecke, B.E. (Bernhard)

Stepanova, E.R. (Ekaterina)

Recently, the field of Human-Computer Interaction has begun to embrace the crucial role of our bodies across our cognitive and
\nsocial processes. The movement of embodied interaction has been followed by a somatic and material turn that strives to design
\ntechnology for richer multi-sensory somatic experiences. Despite the booming developments in tangible and wearable technologies,
\nthere is strikingly limited research exploring the potential of our own physical and virtual bodies to become the material for the design
\nof interaction, in the same ways that tangible materials are. In this position paper, we propose a series of approaches for designing
\ntechnology for social interaction, that integrate human bodies as interaction materials to elicit a multisensory embodied experience.
\nWe illustrate this approach with examples of several virtual reality experiences that we have designed for supporting social connection

CWI's Institutional Repository

Human Bodies as Interaction Materials for Somatic, Social, and MultisensoryVirtual Reality ExperiencesEKATERINA R. STEPANOVA, JOHN DESNOYERS-STEWART, and BERNHARD E. RIECKE, SimonFraser University, CanadaGIJS HUISMAN, Delft University of Technology, The NetherlandsABDALLAH EL ALI, Centrum Wiskunde & Informatica, The NetherlandsRecently, the field of Human-Computer Interaction has begun to embrace the crucial role of our bodies across our cognitive andsocial processes. The movement of embodied interaction has been followed by a somatic and material turn that strives to designtechnology for richer multi-sensory somatic experiences. Despite the booming developments in tangible and wearable technologies,there is strikingly limited research exploring the potential of our own physical and virtual bodies to become the material for the designof interaction, in the same ways that tangible materials are. In this position paper, we propose a series of approaches for designingtechnology for social interaction, that integrate human bodies as interaction materials to elicit a multisensory embodied experience.We illustrate this approach with examples of several virtual reality experiences that we have designed for supporting social connection.Additional Key Words and Phrases: embodied interaction, social connection, virtual reality, pseudohaptics, biosignalsACM Reference Format:Ekaterina R. Stepanova, John Desnoyers-Stewart, Bernhard E. Riecke, Gijs Huisman, and Abdallah El Ali . 2023. Human Bodies asInteraction Materials for Somatic, Social, and Multisensory Virtual Reality Experiences. In CHI’23: Workshop on Body x Materials: Aworkshop exploring the role of material-enabled body-based multisensory experiences, April 23, 2023, Hamburg, Germany. ACM, NewYork, NY, USA, 4 pages. https://doi.org/10.1145/1122445.11224561 BACKGROUNDThe fundamental role of our bodies in our experiences is becoming more readily recognized in the development ofthe technologies that shape our lives. Considerable progress has been made since Dourish argued for a shift towardsEmbodied Interaction [5] to capitalize on the full potential of engagement of our active bodies beyond the screen andmouse-based interfaces. More recently, a deeper consideration that is focused not only on interaction, but considersthe entire rich sensual experiences of the feelings in our bodies began to enter the technology design landscapewith the somatic [15] and material turns [6]. These approaches often draw from somaesthetics [11] and granularphenomenological methods [18] to gain a deeper appreciation of our bodily experiences in an effort to augment andcultivate rich multisensory experiences through design of technologies.The technology used to cultivate these rich experiences often involves some form of tangible material-basedinteraction—engaging our senses through fabrics, wearables, interactive fashion, thermal stimulation, etc. This technol-ogy can be used to support embodied genuine feelings of connection between people [20] by, for example, mediatinginterpersonal touch or supporting interaffectivity through biosignal sharing. In most of these developments, the body isPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.© 2023 Association for Computing Machinery.Manuscript submitted to ACM1CHI ’23 Workshop on Body x Materials, April 23, 2023, Hamburg, Germany Stepanova, et al.seen as the receiving end of interaction, a site where the rich multisensory experiences are embodied, and upon whichthe technology or material acts to elicit these sensations. The dimension that is less discussed is the opportunity toengage users’ bodies as the material itself, as part of the interface with which another user can interact. Our bodies arenot only potent in terms of the richness of our own somatic experiences, but they also offer a considerably complexpotential as a highly expressive interaction material. Moreover, our bodies are deeply embedded with social meaninginherent in engaging with the live bodies of other people and situated within each individual’s social and physicalcontext. Grounded in enactive cognition, this reconceptualization of social interaction emphasizes the integral role ofall the interaction partners and the environment they are situated in as a dynamic interactive loop, providing a morecompelling phenomenological account that can inspire novel design approaches beyond the sender-receiver model [12].From a phenomenological perspective, our bodies are central to all lived experiences. From self-consciousness, toperceiving our relation to the world we share and those we share it with, the interaction between our bodies is at thecore of all we know [9]. Specifically, touch is fundamental to our connection with others due to its key role in thedevelopmental process [3, 8]. Even non-tactile experiences of social interaction can be understood as an extrapolation ofearly experiences of tactile social engagement with others [16]. This rooting of our social interactions in the experiencesof touch inspires us to consider how bodies themselves can constitute a tactile material we can design interaction for.Beyond touch, the internal processes of our bodies, provide another intriguing material source for social interaction.2 APPROACHES FOR ENGAGING THE HUMAN BODY AS INTERACTION MATERIALWhile embodied approaches to computing are often concerned with the design of tangible materials, Virtual Reality(VR), despite commonly only including audio-visual design, nonetheless offers a promising avenue to explore materialdesign. The tracking technology integrated in most VR headsets, when used to its full potential, can engage immersants’bodies fully in an interactive experience. When creating social experiences for multiple users, we can now utilizethose tracked bodies as interaction materials. Arguably, the potential for granularity of the experience of the embodiedinteraction with another live person’s body is much more potent than what other non-human materials may offer.While there are endless opportunities for engaging others’ bodies as interaction material, here we propose to considerthe three following approaches:(1) Augmenting physical social touch: Interpersonal touch is deeply expressive with many factors affectingits perceptions, such as quality of touch, location, social relationship, and context [1]. Affective interpersonal touchprovides a very rich form of interaction, supporting affective exchange and fostering intimacy. Despite social touchhaving a strong potential for fostering intimacy and connection, opportunities to engage in social touch are restricted bycultural and social norms [1], even leading to “touch hunger” experienced by some western cultures [7]. Here technologycan act as a medium that can transform social norms between participants to support socially acceptable touch betweenstrangers that might otherwise feel awkward or inappropriate [13]. This often takes shape through playful interactionsthat turn the experience into a game that establishes a new set of social norms [10]. Alternately this can be accomplishedby altering visual perception and concealing direct eye-contact, for example by putting participants in VR [2, 22].(2) Audio-visual experience to evoke tactile sensations: Recognizing the typical lack of touch in VR, there hasbeen much development into the integration of haptic devices [8]. However, haptic devices are far from supportingthe rich experience of touch that we seek when touching another human. The vibrating motors typically used forhaptics are distant from the qualities of inter-human touch: softness, warmth, dynamics, complex muscular composition,and responsiveness. Here we propose to consider an alternative approach to haptics by stimulating illusionary tactilesensations through pseudohaptics. Pseudohaptics refers to the illusion of perceived haptic sensation that is evoked from2Human Bodies as Interaction Materials CHI ’23 Workshop on Body x Materials, April 23, 2023, Hamburg, Germanyother sensory channels, e.g. audio-visual [19]. There is considerable promising research on pseudohaptics exploringthe experience of touch of objects and surfaces [21]. However, strikingly little research has looked at its potential forsocial touch. We propose that social touch might be an even more fruitful avenue for application of psuedohapticsgiven the additional social significance of the experience of inter-human touch that may heighten the sensation. Whilepseudohaptic sensations are often very faint, we argue that for telepresent social touch, pseudohaptics may be a betterfit than many haptic devices that distract through uncanny sensations of rumbling haptic device.(3) Social biosignals to peak into our own inner bodily experience and share it with others: Biosensingtechnology (e.g. heartrate, breathing, skin conductance, etc.) allows designers to engage inner bodily experiences ofusers. While most bio-responsive technology development has focused on individual health and self-improvementapplications, there is a growing interest in using biosignals to augment social interaction [17]. Biosignals providea window ‘inside’ others’ bodies, providing potential cues to previously unnoticed affective or cognitive states. Byrepresenting biosignals in a social context we afford a novel form of embodied interaction between users as they gainaccess to an ordinarily concealed and intimate dimension of others’ somatic experiences [20]. This approach takesphysiological processes of one user as an interaction material for the perception and engagement by other users, withthe aim to respond to, mirror or impact others physiological state.3 EXEMPLARY DESIGN ARTEFACTSNow we would like to briefly introduce a series of examples of our work in Virtual Reality (VR), that illustrates theabove approaches to using others’ bodies as interaction material.Body RemiXer1 [4] is a multi-user co-located installation that transforms immersants’ bodies into ethereal aurasof light particles. As immersants look around they see other’s bodies as abstract enticing auras, and as they reach tomake contact, their bodies connect and begin to exchange particles, now floating between their virtual bodies. Here thetactility of the interaction comes from touch between the physical bodies of the participants. This often comes as asurprise, as immersants don’t expect these magical ethereal auras to have a physical counterpart that they can feel.This tactile experience encourages further exploration through touch and movement.Embodied Telepresent Connection (ETC)2 is an artistic project exploring a series of interaction prototypes that canelicit the feeling of embodied connection across distance. Two immersants interact remotely using a VR headset. Theysee both their own and the other person’s bodies as a cloud of particles. With no other form of communication availableto them except for gesture and touch, they begin to approach this mysterious other to explore what the interactionmay afford. As they approach the other person’s body, they see the colour of both bodies become warmer, evoking afaint sensation of warmth. When they try to make contact through touch, they notice resistance in the visuals whichprevents their hands from passing through the other body as if they were physically present, evoking a subtle sensationof pressure. They also see and hear others’ heart beating inside their chest, detected by their partner’s smartwatch. Theyare tempted to approach the other and touch their virtual heart, feeling the preciousness of this life source. Immersantscan notice how their partner’s heartrate changes as they approach them for such an intimate engagement.The Social VR Avatar Biosignal Visualization project [14] explores how to design and visualize biosignals in the contextof social VR entertainment. Through this project we asked: which are the most effective visualizations for heart rateand breathing rate in an immersive, virtual music event scenario? We explored several visualizations (Skeuomorphic,Particles, Creature, Environment) across different signal rate levels, and found that skeuomorphic visualizations for1See a video here: http://ispace.iat.sfu.ca/project/body-remixer/2See a video here: http://ispace.iat.sfu.ca/project/etc/3CHI ’23 Workshop on Body x Materials, April 23, 2023, Hamburg, Germany Stepanova, et al.both biosignals allow differentiable arousal inference, in part due to their visual familiarity. We also found that biosignalperceptions often depend on the type of relationship with an avatar (friend, stranger), entertainment type (movie orconcert), and whether making inferences about emotion pertained to the other avatar or to the virtual space. We founda strong link between visualizing such biodata and their capacity to strengthen social presence. These visualizationsbecame an extension of the avatars affording additional social cues to infer user’s arousal state, their availability to beapproached, or to simply have a tell-tale sign about what kind of virtual environment it is that they are in.4 CONCLUSIONThe recent material turn in HCI has evolved hand and hand with the embodied and somatic turns. These developmentscapitalize on the richness of our embodied, often tactile experience, to evoke rich and nuanced somatic experiences inusers. However, there is a surprising lack of exploration of how our own actual and virtual bodies can be integrated asinteractive materials for mediated social interaction. We propose seeing the body as more than the medium throughwhich we experience technology, but as a material that can itself be central to the design. Considering how our bodiescan be integrated as a material will allow for the design of new, rich, social somatic experiences.REFERENCES[1] Peter Andersen, Jillian Gannon, and Jessica Kalchik. 2013. Proxemic and haptic interaction: The closeness continuum. (2013).[2] Margherita Bergamo Meneghini. 2019. In IEEE GEM 2019. New Haven, CT, 6.[3] Carissa J Cascio, David Moore, and Francis McGlone. 2019. Social touch and human development. Developmental cognitive neuroscience 35 (2019),5–11.[4] John Desnoyers-Stewart, Ekaterina R Stepanova, Bernhard E Riecke, and Patrick Pennefather. 2020. Body RemiXer: extending bodies to stimulatesocial connection in an immersive installation. In ACM SIGGRAPH 2020 Art Gallery. 394–400.[5] Paul Dourish. 1999. Embodied interaction: Exploring the foundations of a new approach to HCI. Work (1999), 1–16.[6] Adam Drazin and Susanne Küchler. 2015. The social life of materials: studies in materials and society. Bloomsbury Publishing.[7] Tiffany Field. 2014. Touch Hunger. In Touch, Second Edition. MIT Press, Cambridge, UNITED STATES. http://ebookcentral.proquest.com/lib/sfu-ebooks/detail.action?docID=3339875[8] Alberto Gallace and Matteo Girondini. 2022. Social touch in virtual reality. Current Opinion in Behavioral Sciences 43 (2022), 249–254.[9] Shaun Gallagher and Dan Zahavi. 2020. The phenomenological mind. Routledge.[10] Mads Hobye and Jonas Löwgren. 2011. Touching a stranger: Designing for engaging experience in embodied interaction. International Journal ofDesign 5, 3 (2011), 31–48.[11] Kristina Hook. 2018. Designing with the body: Somaesthetic interaction design. Mit Press.[12] Gijs Huisman. 2022. An Interaction Theory Account of (Mediated) Social Touch. Frontiers in Psychology (2022), 1725.[13] Carey Jewitt, Sara Price, Jürgen Steimle, Gijs Huisman, Lili Golmohammadi, Narges Pourjafarian, William Frier, Thomas Howard, SimaIpakchian Askari, Michela Ornati, et al. 2021. Manifesto for digital social touch in crisis. Frontiers in Computer Science (2021), 97.[14] Sueyoon Lee, Abdallah El Ali, Maarten Wijntjes, and Pablo Cesar. 2022. Understanding and Designing Avatar Biosignal Visualizations for SocialVirtual Reality Entertainment. In CHI’22 (New Orleans, LA, USA). ACM, New York, NY, USA, Article 425, 15 pages.[15] Lian Loke and Thecla Schiphorst. 2018. The somatic turn in human-computer interaction. Interactions 25, 5 (2018), 54–5863.[16] Maurice Merleau-Ponty. 1964. Signs. Northwestern University Press.[17] Clara Moge, Katherine Wang, and Youngjun Cho. 2022. Shared User Interfaces of Physiological Data: Systematic Review of Social BiofeedbackSystems and Contexts in HCI. In CHI Conference on Human Factors in Computing Systems. 1–16.[18] Mirjana Prpa, Sarah Fdili-Alaoui, Thecla Schiphorst, and Philippe Pasquier. 2020. Articulating experience: Reflections from experts applyingmicro-phenomenology to design research in HCI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.[19] Andreas Pusch and Anatole Lécuyer. 2011. Pseudo-haptics: from the theoretical foundations to practical system design guidelines. In Proceedings ofthe 13th international conference on multimodal interfaces - ICMI ’11. ACM Press, Alicante, Spain, 57. https://doi.org/10.1145/2070481.2070494[20] Ekaterina R Stepanova, John Desnoyers-Stewart, Kristina Höök, and Bernhard E Riecke. 2022. Strategies for Fostering a Genuine Feeling ofConnection in Technologically Mediated Systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–26.[21] Yusuke Ujitoko and Yuki Ban. 2021. Survey of pseudo-haptics: Haptic feedback design and application proposals. IEEE Transactions on Haptics 14, 4(2021), 699–711.[22] Harry Robert Wilson. 2020. New ways of seeing, feeling, being: intimate encounters in virtual reality performance. International Journal ofPerformance Arts and Digital Media 16, 2 (2020), 114–133.4

Human bodies as interaction materials for somatic, social, and multisensory Virtual Reality experiences

Abstract

Similar works

Full text

Available Versions

CWI's Institutional Repository