Does embodied training improve the recognition of mid-level expressive movement qualities sonification?

This research is a part of a broader project exploring how movement qualities can be recognized by means of the auditory channel: can we perceive an expressive full-body movement quality by means of its interactive sonification? The paper presents a sonification framework and an experiment to evaluate if embodied sonic training (i.e., experiencing interactive sonification of your own body movements) increases the recognition of such qualities through the auditory channel only, compared to a non-embodied sonic training condition. We focus on the sonification of two mid-level movement qualities: fragility and lightness. We base our sonification models, described in the first part, on the assumption that specific compounds of spectral features of a sound can contribute to the cross-modal perception of a specific movement quality. The experiment, described in the second part, involved 40 participants divided into two groups (embodied sonic training vs. no training). Participants were asked to report the level of lightness and fragility they perceived in 20 audio stimuli generated using the proposed sonification models. Results show that (1) both expressive qualities were correctly recognized from the audio stimuli, (2) a positive effect of embodied sonic training was observed for fragility but not for lightness. The paper is concluded by the description of the artistic performance that took place in 2017 in Genoa (Italy), in which the outcomes of the presented experiment were exploited

INTERACTIVE SONIFICATION STRATEGIES FOR THE MOTION AND EMOTION OF DANCE PERFORMANCES

The Immersive Interactive SOnification Platform, or iISoP for short, is a research platform for the creation of novel multimedia art, as well as exploratory research in the fields of sonification, affective computing, and gesture-based user interfaces. The goal of the iISoP’s dancer sonification system is to “sonify the motion and emotion” of a dance performance via musical auditory display. An additional goal of this dissertation is to develop and evaluate musical strategies for adding layer of emotional mappings to data sonification. The result of the series of dancer sonification design exercises led to the development of a novel musical sonification framework. The overall design process is divided into three main iterative phases: requirement gathering, prototype generation, and system evaluation. For the first phase help was provided from dancers and musicians in a participatory design fashion as domain experts in the field of non-verbal affective communication. Knowledge extraction procedures took the form of semi-structured interviews, stimuli feature evaluation, workshops, and think aloud protocols. For phase two, the expert dancers and musicians helped create test-able stimuli for prototype evaluation. In phase three, system evaluation, experts (dancers, musicians, etc.) and novice participants were recruited to provide subjective feedback from the perspectives of both performer and audience. Based on the results of the iterative design process, a novel sonification framework that translates motion and emotion data into descriptive music is proposed and described

User-Defined Gestural Interactions Through Multi-Modal Feedback

Rapid advancements are being made in the field of HCI. However, most commercial designs using these technologies employ limited control schema and privilege interactions borrowed from pre-existing technologies. This paper suggests considering the whole body to create rich environments using coadaptive technologies

Automated Analysis of Synchronization in Human Full-body Expressive Movement

The research presented in this thesis is focused on the creation of computational models for the study of human full-body movement in order to investigate human behavior and non-verbal communication. In particular, the research concerns the analysis of synchronization of expressive movements and gestures. Synchronization can be computed both on a single user (intra-personal), e.g., to measure the degree of coordination between the joints\u2019 velocities of a dancer, and on multiple users (inter-personal), e.g., to detect the level of coordination between multiple users in a group. The thesis, through a set of experiments and results, contributes to the investigation of both intra-personal and inter-personal synchronization applied to support the study of movement expressivity, and improve the state-of-art of the available methods by presenting a new algorithm to perform the analysis of synchronization

Musical Cities

Musical Cities represents an innovative approach to scholarly research and dissemination. A digital and interactive 'book', it explores the rhythms of our cities, and the role they play in our everyday urban lives, through the use of sound and music. Sara Adhitya first discusses why we should listen to urban rhythms in order to design more liveable and sustainable cities, before demonstrating how we can do so through various acoustic communication techniques. Using audio-visual examples, Musical Cities takes the ‘listener’ on an interactive journey, revealing how sound and music can be used to represent, compose, perform and interact with the city. Through case studies of urban projects developed in Paris, Perth, Venice and London, Adhitya demonstrates how the power of music, and the practice of listening, can help us to compose more accessible, inclusive, engaging, enjoyable, and ultimately more sustainable cities

Sonifying Urban Rhythms: Towards the spatio-temporal composition of the urban environment

This thesis is concerned with the composition of the urban rhythms generated by urban design and planning. It recognises the temporal limitations of the graphic urban masterplan, with its tendency of being static and singular in the composition of urban experience. Thus it proposes the integration of rhythm into the urban design and planning process, with the aim to improve the temporal quality of urban design. In order to represent these urban rhythms, as designed in the graphic masterplan, we propose their sonification. A Sonified Urban Masterplan (SUM) tool was developed, allowing the sonification of multiple layers of maps (raster or vector images) along a number of paths of interest. An urban sonic code was then developed in order to map the relevant graphic urban parameters into sound parameters. This sonification strategy was applied to the city of Paris as a case study, producing a sonified set of maps whose composition could be ‘listened’ to over time. Temporal issues concerning human movement, transport infrastructure, activity distribution, and the structuring of urban form and design elements could be represented and heard. We then investigated the potential of the SUM tool as a design and planning tool. We explored how sound could be used to inform the composition of urban form in both time and space, in order to generate the urban rhythms we may desire to experience. Thus through the integration of sonification in urban design and planning, this thesis permits the spatio-­‐temporal representation and composition of urban form. It allows urban designers and planners to compose future urban rhythms and improve the temporal quality of our urban environments. Furthermore, the potential of this tool in other fields has also be recognized, for example in music and the composition of multi-­‐layered open graphic scores

Musical Cities

Musical Cities represents an innovative approach to scholarly research and dissemination. A digital and interactive 'book', it explores the rhythms of our cities, and the role they play in our everyday urban lives, through the use of sound and music. Sara Adhitya first discusses why we should listen to urban rhythms in order to design more liveable and sustainable cities, before demonstrating how we can do so through various acoustic communication techniques. Using audio-visual examples, Musical Cities takes the ‘listener’ on an interactive journey, revealing how sound and music can be used to represent, compose, perform and interact with the city. Through case studies of urban projects developed in Paris, Perth, Venice and London, Adhitya demonstrates how the power of music, and the practice of listening, can help us to compose more accessible, inclusive, engaging, enjoyable, and ultimately more sustainable cities

Deep Flow: a tentacular worlding of dance, biosensor technology, lived experience and embodied materials of the human and non-humankind

How to find relations between lived experience and biosensor technology in dance practice? To do this, current dance practices using biosensor and interactive technologies are studied. These practices visualise invisible bodily events such as heart rate, in external media environments, illustrating certain aspects of a dancer’s experience, to which they respond. This neglects the storehouse of felt experience that technologies, used instrumentally, cannot capture. The research challenges these strategies of looking outwardly to looking inwardly (Höök, 2018), to understand dance as experience (Fraleigh, 2018) more fully. It explores phenomenological methods of dance practice and biosensor technology to; explore the felt senses (Gendlin, 2003) through whole body experiences; investigate experiential interactions in differing environments; and explore human relations with technologies and materials that are both human and non-human. A practice as research methodology is used to explore this strategy. It uses a relational theoretical approach, an interdisciplinary study of the human body in dance practice, phenomenology, technology and feminist posthumanism. Phenomenological methods are explored in methods of dance practice using biosensor technologies. Multimodal qualitative and quantitative methods of data collection are used to interpret the experiences of these practices. Methods of analysis generate an understanding of the relations between these experiences, the body and biometric data. The methods and research methodology have given birth to the practice of Deep Flow, that reveals relations between states of flow (Csikszentmihalyi, 1990; 2004), movement meditation, fascia release and heart rate variability. This replaces practices with biosensor technologies that quantifies and visualises human experience. By looking inwardly, within an ecology of lived experience, biometric data, tangible and intangible materials, Deep Flow collapses notions of inside and outside. It proposes a return to subjective experience, to understand our relations with our bodies, the world and technology, through states of flow and well-being

Deep Flow: a tentacular worlding of dance, biosensor technology, lived experience and embodied materials of the human and non-humankind

How to find relations between lived experience and biosensor technology in dance practice? This PaR presents a novel methodology, tentacular worlding, to explore Embodied Dance practice as lived experience, using phenomenological methods and biosensor technologies to better understand experiential aspects of dance more fully, by looking inwardly. It challenges dance practice intersecting with biosensors that visualise invisible physiological events such as heart rate, in external mediated environments, to which dancer’s respond. These ocularcentric practices illustrate only certain aspects of a dancer’s bodily engagement with technology thereby privileging vision over bodily experience. Looking outwardly neglects the vast storehouse of lived experiences that technologies used instrumentally, cannot capture. To explore the strategy of looking inwardly, a relational methodological approach tentacular worlding is applied. This inspires an interdisciplinary study of the human body in dance practice, phenomenology, technology, and ecofeminist posthumanism. Phenomenological dance methods are used to; explore whole bodily experiences; investigate bodily interactions with differing environments; and discover human relations with biosensor technologies and differing materials. It challenges ocularcentrism by blindfolding the practitioner to augment bodily sensing in the absence of visual information. Multimodal qualitative and quantitative methods are used to interpret these experiences and methods of analysis emphasise tentacular relations between lived experience, the heart, and biometric data. Tentacular worlding gave birth to the Embodied Dance practice Deep Flow, to foreground relations between lived and bodily experiencing, meditation, fascia release and heart rate variability. By looking inwardly, within an ecology of embodied experience, visible and invisible, tangible, and intangible materials, Deep Flow collapses binary notions of inside and outside, subject and object, an embodied materiality. It proposes; a return to bodily experience and embodied states of flow, to construct knowledge from a first-person perspective and to explore the complexity of relations between the heart, the human and nonhuman