Creating and evaluating embodied interactive experiences: case studies of full-body, sonic and tactile enaction.

This thesis contributes to the field of embodied and multimodal interaction by presenting the development of different original interactive systems. Using a constructive approach, a variety of real-time user interaction situations were designed and tested, two cases of human-virtual character bodily interaction, two interactive sonifications of trampoline jumping, collaborative interaction in mobile music performance and tangible and tactile interaction with virtual sounds. While diverse in terms of application, all the explored interaction techniques belong to the context of augmentation and are grounded in the theory of embodiment and strategies for natural human-computer interaction (HCI). The cases have been contextualized within the umbrella of enaction, a paradigm of cognitive science that addresses the user as an embodied agent situated in an environment and coupled to it through sensorimotor activity. This activity of sensing and action is studied through different modalities: auditory, tactile and visual and combinations of these. The designed applications aim at a natural interaction with the system, being full-body, tangible and spatially aware. Particularly sonic interaction has been explored in the context of music creation, sports and auditory display. These technology-mediated scenarios are evaluated in order to understand what the adopted interaction techniques can bring to the user experience, how they modify impressions and enjoyment. The publications also discuss the enabling technologies used for the development, including motion tracking and programmed hardware for the tactile-sonic interaction and sonic and tangible interaction. Results show that combining full-body interaction with auditory augmentation and sonic interaction can modify the perception, observed behavior and emotion during the experience. Using spatial interaction together with tangible interaction or tactile feedback provides for a multimodal experience of exploring a mixed reality environment where audio can be accessed and manipulated with natural interaction. Embodied and spatial interaction brings playfulness to a mobile music improvisation, shifting the focus of the experience from music-making towards movement-based gaming. Finally, two novel implementations of full-body interaction based on the enactive paradigm are presented. In these designed scenarios of enaction the participant is motion tracked and a virtual character rendered as a stick figure is displayed in front of her on a screen. Results from the user studies show how the involvement of the body is crucial in understanding the behavior of a virtual character or a digital representation of the self in a gaming scenario

In Gameplay : the invariant structures and varieties of the video game gameplay experience

This dissertation is a multidisciplinary study on video game gameplay as an autonomous form of vernacular experience. Plays and games are traditional research subjects in folkloristics, but commercial video games have not been studied yet. For this reason, methods and concepts of the folkloristic research tradition have remained unknown in contemporary games studies. This thesis combines folkloristics, game studies and phenomenological enactive cognitive science in its investigations into player–game interaction and the video game gameplay experience at large. In this dissertation, three representative survey samples (N=2,594, N=845, N=1,053) on “Rewarding gameplay experience” are analyzed using statistical analysis methods. The samples were collected in 2014–2017 from Finnish and Danish adult populations. This dissertation also analyzes data from 32 interviews, through which the survey respondents’ gameplay preferences, gaming memories, and motivations to play were further investigated. By combining statistical and qualitative data analyses, this work puts forward a mixed-methods research strategy and discusses how the findings relate to prior game research from several disciplines and schools of thought. Based on theoretical discussions, this dissertation argues that the video game gameplay experience as a cultural phenomenon consists of eight invariants in relation to which each individual gameplay experience can be interpreted: The player must demonstrate a lusory attitude (i), and a motivation to play (ii). The gameplay experience consists of explorative and coordinative practices (iii), which engender a change in the player’s self-experience (iv). This change renders the gameplay experience inherently emotional (v) and performative (vi) in relation to the gameworld (vii). The gameplay experience has the dramatic structure of a prototypical narrative (viii) although a game as an object cannot be regarded a narrative in itself. As a key result of factor analytical studies and qualitative interview analyses, a novel approach to understanding player–game interaction is put forward. An original gameplay preference research tool and a player typology are introduced. This work argues, that, although video games as commercial products would not be intuitive research subjects for folkloristics, video game gameplay, player–game interaction, and the traditions in experiencing and narrating gameplay do not differ drastically from those of traditional social games. In contrast to this, all forms of gameplay are argued to be manifestations of the same vernacular phenomenon. Indeed, folkloristic research could pay more attention to how culture is experienced, modified, varied and expressed, regardless of whether the research subject is a commercial product or not.Käsillä oleva väitöskirja on monitieteellinen tutkimus videopelien pelaamisesta itsenäisenä kansanomaisen kokemuksen muotona. Pelien ja leikkien tutkimus on perinteikäs tutkimusaihe folkloristiikassa, mutta kaupallisten videopelien tutkimusta ei ole juuri tehty. Tästä syystä folkloristiikan tutkimusmenetelmät ja -käsitteet ovat jääneet tuntemattomaksi nykyaikaisessa pelitutkimuksessa. Tutkimus yhdistää folkloristiikan ja pelitutkimuksen näkökulmien lisäksi enaktiivisen kognition fenomenologista teoriaa pelaaja–peli-vuorovaikutuksen tutkimukseen sekä pelikokemuksen analyysiin. Tutkimuksessa analysoidaan tilastotieteellisin menetelmin kolmea aikuisväestöä edustavaa ”Palkitseva pelikokemus” -kyselytutkimusaineistoa (N=2,594, N=845, N=1,053), jotka kerättiin Suomesta ja Tanskasta vuosina 2014–2017. Kyselytutkimusaineiston rinnalla analysoidaan 32 teemahaastattelun aineistoa. Haastatteluilla tuotettiin syvempää ymmärrystä kyselyyn vastanneiden henkilöiden pelimieltymyksistä, pelimuistoista ja pelimotivaatioista. Tilastoaineiston ja haastatteluaineiston analyysi tuodaan yhteen monimenetelmällisellä ja dialogisella tutkimusotteella, joka yhdistää havainnot usealla eri tutkimusalalla tehtyyn pelitutkimukseen. Teoreettisen analyysin tuloksena argumentoidaan, että videopelien pelikokemusta ilmiönä määrittää kahdeksan muuttumatonta ominaisuutta, joiden suhteen kunkin yksittäisen pelikokemuksen ainutlaatuisuutta voidaan tarkastella: Pelaajalla tulee olla leikkisä asenne (i) ja motivaatio pelaamiseen (ii). Pelaamisen kokemus rakentuu tutkivista ja suorittavista käyntänteistä (iii), jotka tuovat väliaikaisen muutoksen pelaavan henkilön minäkokemukseen (iv). Tämän muutoksen myötä pelaajuudesta muodostuu emotionaalinen (v) ja performatiivinen (vi) positio suhteessa pelimaailmaan (vii). Näin syntyvän omakohtaisen pelikokemuksen rakenne vastaa kertomuksen dramaattista perusrakennetta (viii), vaikka peliä itsessään ei voida pitää kertomuksena. Tutkimuksen empiirisenä tuloksena esitellään faktorianalyyttisiin tapaustutkimuksiin ja laadullisten aineistojen analyysiin perustuva uudenlainen näkökulma ja menetelmä pelaaja–peli-vuorovaikutuksen ja pelimieltymyksen tutkimukseen, sekä edelliseen perustuva pelaajatyyppiluokittelu. Samalla väitetään, että vaikka videopelit kaupallisina esineinä eivät olisi folkloristiikan tutkimuskohteita, videopelien pelaaminen, pelaaja–peli-vuorovaikutus ja pelien kokemisen tavat eivät eroa ratkaisevasti pihaleikeistä vaan ovat saman kansanomaisen ilmiön esiintymiä. Folkloristisen tutkimuksen soisikin kiinnittävän nykyistä painokkaampaa huomiota kulttuurin kokemisen, muokkaamisen ja ilmaisun tapoihin riippumatta siitä, onko tarkastelun kohteena kaupallinen tuote vai ei

AN ENACTIVE APPROACH TO TECHNOLOGICALLY MEDIATED LEARNING THROUGH PLAY

This thesis investigated the application of enactive principles to the design of classroom technolo- gies for young children’s learning through play. This study identified the attributes of an enactive pedagogy, in order to develop a design framework to accommodate enactive learning processes. From an enactive perspective, the learner is defined as an autonomous agent, capable of adapta- tion via the recursive consumption of self generated meaning within the constraints of a social and material world. Adaptation is the parallel development of mind and body that occurs through inter- action, which renders knowledge contingent on the environment from which it emerged. Parallel development means that action and perception in learning are as critical as thinking. An enactive approach to design therefore aspires to make the physical and social interaction with technology meaningful to the learning objective, rather than an aside to cognitive tasks. The design framework considered in detail the necessary affordances in terms of interaction, activity and context. In a further interpretation of enactive principles, this thesis recognised play and pretence as vehicles for designing and evaluating enactive learning and the embodied use of technology. In answering the research question, the interpreted framework was applied as a novel approach to designing and analysing children’s engagement with technology for learning, and worked towards a paradigm where interaction is part of the learning experience. The aspiration for the framework was to inform the design of interaction modalities to allow users’ to exercise the inherent mechanisms they have for making sense of the world. However, before making the claim to support enactive learning processes, there was a question as to whether technologically mediated realities were suitable environments to apply this framework. Given the emphasis on the physical world and action, it was the intention of the research and design activities to explore whether digital artefacts and spaces were an impoverished reality for enactive learning; or if digital objects and spaces could afford sufficient ’reality’ to be referents in social play behaviours. The project embedded in this research was tasked with creating deployable technologies that could be used in the classroom. Consequently, this framework was applied in practice, whereby the design practice and deployed technologies served as pragmatic tools to investigate the potential for interactive technologies in children’s physical, social and cognitive learning. To understand the context, underpin the design framework, and evaluate the impact of any techno- logical interventions in school life, the design practice was informed by ethnographic methodologies. The design process responded to cascading findings from phased research activities. The initial fieldwork located meaning making activities within the classroom, with a view to to re-appropriating situated and familiar practices. In the next stage of the design practice, this formative analysis determined the objectives of the participatory sessions, which in turn contributed to the creation of technologies suitable for an inquiry of enactive learning. The final technologies used standard school equipment with bespoke software, enabling children to engage with real time compositing and tracking applications installed in the classrooms’ role play spaces. The evaluation of the play space technologies in the wild revealed under certain conditions, there was evidence of embodied presence in the children’s social, physical and affective behaviour - illustrating how mediated realities can extend physical spaces. These findings suggest that the attention to meaningful interaction, a presence in the environment as a result of an active role, and a social presence - as outlined in the design framework - can lead to the emergence of observable enactive learning processes. As the design framework was applied, these principles could be examined and revised. Two notable examples of revisions to the design framework, in light of the applied practice, related to: (1) a key affordance for meaningful action to emerge required opportunities for direct and immediate engagement; and (2) a situated awareness of the self and other inhabitants in the mediated space required support across the spectrum of social interaction. The application of the design framework enabled this investigation to move beyond a theoretical discourse

Haptic Media Scenes

The aim of this thesis is to apply new media phenomenological and enactive embodied cognition approaches to explain the role of haptic sensitivity and communication in personal computer environments for productivity. Prior theory has given little attention to the role of haptic senses in influencing cognitive processes, and do not frame the richness of haptic communication in interaction design—as haptic interactivity in HCI has historically tended to be designed and analyzed from a perspective on communication as transmissions, sending and receiving haptic signals. The haptic sense may not only mediate contact confirmation and affirmation, but also rich semiotic and affective messages—yet this is a strong contrast between this inherent ability of haptic perception, and current day support for such haptic communication interfaces. I therefore ask: How do the haptic senses (touch and proprioception) impact our cognitive faculty when mediated through digital and sensor technologies? How may these insights be employed in interface design to facilitate rich haptic communication? To answer these questions, I use theoretical close readings that embrace two research fields, new media phenomenology and enactive embodied cognition. The theoretical discussion is supported by neuroscientific evidence, and tested empirically through case studies centered on digital art. I use these insights to develop the concept of the haptic figura, an analytical tool to frame the communicative qualities of haptic media. The concept gauges rich machine- mediated haptic interactivity and communication in systems with a material solution supporting active haptic perception, and the mediation of semiotic and affective messages that are understood and felt. As such the concept may function as a design tool for developers, but also for media critics evaluating haptic media. The tool is used to frame a discussion on opportunities and shortcomings of haptic interfaces for productivity, differentiating between media systems for the hand and the full body. The significance of this investigation is demonstrating that haptic communication is an underutilized element in personal computer environments for productivity and providing an analytical framework for a more nuanced understanding of haptic communication as enabling the mediation of a range of semiotic and affective messages, beyond notification and confirmation interactivity

Designing Atmospheres: Theory and Science

Recent advances in science confirm many of the architects’ deep-rooted intuitions, improving knowledge about the perception of space and the meaning of architectural and urban design. This volume collects four essays: “Investigating Atmosphere in Architecture: An Overview of Phenomenological and Neuroscientific Methods” by Elisabetta Canepa; “Rhythms of the Brain, Body, and Environment: A Neuroscientific Perspective on Atmospheres” by Zakaria Djebbara; “A History of Tool-Atmospheres” by Kory Beighle; and “Atmospheric Histrionics” by Harry Francis Mallgrave. Bob Condia provided a critical introduction entitled “The Design of Atmospheres.

Designing Atmospheres: Theory and Science

This book was born as the legacy of the “Designing Atmospheres: Theory and Science” Symposium, an Interfaces event of the Academy of Neuroscience for Architecture (ANFA), sponsored by the EU’s Horizon 2020 MSCA Program — RESONANCES Project, the Perkins Eastman Studio, and the Architecture Department at Kansas State University. The event was hosted in the College of Architecture, Planning and Design (APDesign), Kansas State University (K-State), Manhattan, KS, on March 28, 2023. Recent advances in science confirm many of the architects’ deep-rooted intuitions, improving knowledge about the perception of space and the meaning of architectural and urban design. This volume collects four essays: “Investigating Atmosphere in Architecture: An Overview of Phenomenological and Neuroscientific Methods” by Elisabetta Canepa; “Rhythms of the Brain, Body, and Environment: A Neuroscientific Perspective on Atmospheres” by Zakaria Djebbara; “A History of Tool-Atmospheres” by Kory Beighle; and “Atmospheric Histrionics” by Harry Francis Mallgrave. Bob Condia provided a critical introduction entitled “The Design of Atmospheres.”https://newprairiepress.org/ebooks/1050/thumbnail.jp

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments