Does embodied training improve the recognition of mid-level expressive movement qualities sonification?

This research is a part of a broader project exploring how movement qualities can be recognized by means of the auditory channel: can we perceive an expressive full-body movement quality by means of its interactive sonification? The paper presents a sonification framework and an experiment to evaluate if embodied sonic training (i.e., experiencing interactive sonification of your own body movements) increases the recognition of such qualities through the auditory channel only, compared to a non-embodied sonic training condition. We focus on the sonification of two mid-level movement qualities: fragility and lightness. We base our sonification models, described in the first part, on the assumption that specific compounds of spectral features of a sound can contribute to the cross-modal perception of a specific movement quality. The experiment, described in the second part, involved 40 participants divided into two groups (embodied sonic training vs. no training). Participants were asked to report the level of lightness and fragility they perceived in 20 audio stimuli generated using the proposed sonification models. Results show that (1) both expressive qualities were correctly recognized from the audio stimuli, (2) a positive effect of embodied sonic training was observed for fragility but not for lightness. The paper is concluded by the description of the artistic performance that took place in 2017 in Genoa (Italy), in which the outcomes of the presented experiment were exploited

A Perspective on Implementing Movement Sonification to Influence Movement (and Eventually Cognitive) Creativity

Creativity represents an important feature in a variety of daily-life and domain-specific contexts. Recent evidence indicates that physical movement serves as a key resource for exploring and generating task-relevant creative ideas, supporting the embodied perspective on creative cognition. An intuitive link between movement and creative cognition is movement creativity. The process of exploring the movement solutions an environment offers (i.e., affordances) and exploiting novel, functional, and creative movements may translate to and improve how individuals explore and generate novel ideas. Opening perception to the variety of affordances (“conventional” and novel) an environment offers drives creative movement. Teachers and coaches can promote this process by designing a learning environment that invites performers to consider and utilize novel movement solutions. In this article, we present a rationale for using movement sonification to promote creative movement. Movement sonification consists of mapping a movement parameter into sound, with a sound being triggered or changing according to how movement unfolds. We argue that movement sonification can facilitate the emergence of creative movement via enhancing perception of currently performed movements and invite performers to utilize novel affordances, and emphasizing information for regulating subsequent creative actions. We exemplify this concept in a creative dance intervention for children during physical education classes. In conclusion, we contend that learning to explore original dance sequences using movement sonification may provide a meaningful link between creative movement and creative cognition. Children may use their minds and bodies as tools for creative thinking and exploration, such as shaping letters with their bodies

Bodily Expression Support for Creative Dance Education by Grasping-Type Musical Interface with Embedded Motion and Grasp Sensors

Dance has been made mandatory as one of the physical education courses in Japan because it can cultivate capacities for expression and communication. Among several types of dance education, creative dance especially contributes to the cultivation of these capacities. However, creative dance requires some level of particular skills, as well as creativity, and it is difficult to presuppose these pre-requisites in beginner-level dancers without experience. We propose a novel supporting device for dance beginners to encourage creative dance performance by continuously generating musical sounds in real-time in accordance with their bodily movements. It has embedded sensors developed for this purpose. Experiments to evaluate the effectiveness of the device were conducted with ten beginner-level dancers. Using the proposed device, the subjects demonstrated enhanced creative dance movements with greater variety, evaluated in terms of Laban dance movement description. Also, using the device, they performed with better accuracy and repeatability in a task where they produced an imagined circular trajectory by hand. The proposed interface is effective in terms of creative dance activity and accuracy of motion generation for beginner-level dancers

As Light as You Aspire to Be: Changing body perception with sound to support physical activity

Supporting exercise adherence through technology remains an important HCI challenge. Recent works showed that altering walking sounds leads people perceiving themselves as thinner/lighter, happier and walking more dynamically. While this novel approach shows potential for physical activity, it raises critical questions impacting technology design. We ran two studies in the context of exertion (gym-step, stairs-climbing) to investigate how individual factors impact the effect of sound and the duration of the after-effects. The results confirm that the effects of sound in body-perception occur even in physically demanding situations and through ubiquitous wearable devices. We also show that the effect of sound interacted with participants’ body weight and masculinity/femininity aspirations, but not with gender. Additionally, changes in body-perceptions did not hold once the feedback stopped; however, body-feelings or behavioural changes appeared to persist for longer. We discuss the results in terms of malleability of body-perception and highlight opportunities for supporting exercise adherence

From a technology that replaces human perception action to one that expands it: some critiques of current technology use in sport

Information technology has been integrated into most areas of sport, providing new insights, improving the efficiency of operational processes, and offering unique opportunities for exploration and inquiry. While acknowledging this positive impact, this paper explores whether sufficient consideration has been directed towards what technology risks detracting from the learning and developmental experiences of its users. Specifically, viewed through the philosophical lens of the device paradigm, and considering a more ecological account of technological implementation, we discuss how technology in sport could subtly disengage educators and applied sports scientists from performance environments. Insights gleaned from such an ecological account of technology implementation could lead sports science and educational teams to ask and reflect on tough questions of current practice: i.e. has too much control been given to technological devices to ‘solve’ problems and communicate knowledge (about) in sport? Has technology improved the skills of players and performance staff? Or are performance staff at risk of becoming over-reliant on technology, and as a result, reducing the value of experiential knowledge (of) and intuition? Questions like these should be asked if technological devices, purported to support aspects of practice, are continually integrated into the sporting landscape

Interactive Sound in Performance Ecologies: Studying Connections among Actors and Artifacts

This thesis’s primary goal is to investigate performance ecologies, that is the compound of humans, artifacts and environmental elements that contribute to the result of a per- formance. In particular, this thesis focuses on designing new interactive technologies for sound and music. The goal of this thesis leads to the following Research Questions (RQs): • RQ1 How can the design of interactive sonic artifacts support a joint expression across different actors (composers, choreographers, and performers, musicians, and dancers) in a given performance ecology? • RQ2 How does each different actor influence the design of different artifacts, and what impact does this have on the overall artwork? • RQ3 How do the different actors in the same ecology interact, and appropriate an interactive artifact? To reply to these questions, a new framework named ARCAA has been created. In this framework, all the Actors of a given ecology are connected to all the Artifacts throughout three layers: Role, Context and Activity. This framework is then applied to one systematic literature review, two case studies on music performance and one case study in dance performance. The studies help to better understand the shaded roles of composers, per- formers, instrumentalists, dancers, and choreographers, which is relevant to better design interactive technologies for performances. Finally, this thesis proposes a new reflection on the blurred distinction between composing and designing a new instrument in a context that involves a multitude of actors. Overall, this work introduces the following contributions to the field of interaction design applied to music technology: 1) ARCAA, a framework to analyse the set of inter- connected relationship in interactive (music) performances, validated through 2 music studies, 1 dance study and 1 systematic literature analysis; 2) Recommendations for de- signing music interactive system for performance (music or dance), accounting for the needs of the various actors and for the overlapping on music composition and design of in- teractive technology; 3) A taxonomy of how scores have shaped performance ecologies in NIME, based on a systematic analysis of the literature on score in the NIME proceedings; 4) Proposal of a methodological approach combining autobiographical and idiographical design approaches in interactive performances.O objetivo principal desta tese é investigar as ecologias performativas, conjunto formado pelos participantes humanos, artefatos e elementos ambientais que contribuem para o resultado de uma performance. Em particular, esta tese foca-se na conceção de novas tecnologias interativas para som e música. O objetivo desta tese originou as seguintes questões de investigação (Research Questions RQs): • RQ1 Como o design de artefatos sonoros interativos pode apoiar a expressão con- junta entre diferentes atores (compositores, coreógrafos e performers, músicos e dançarinos) numa determinada ecologia performativa? • RQ2 Como cada ator influencia o design de diferentes artefatos e que impacto isso tem no trabalho artístico global? • RQ3 Como os diferentes atores de uma mesma ecologia interagem e se apropriam de um artefato interativo? Para responder a essas perguntas, foi criado uma nova framework chamada ARCAA. Nesta framework, todos os atores (Actores) de uma dada ecologia estão conectados a todos os artefatos (Artefacts) através de três camadas: Role, Context e Activity. Esta framework foi então aplicada a uma revisão sistemática da literatura, a dois estudos de caso sobre performance musical e a um estudo de caso em performance de dança. Estes estudos aju- daram a comprender melhor os papéis desempenhados pelos compositores, intérpretes, instrumentistas, dançarinos e coreógrafos, o que é relevante para melhor projetar as tec- nologias interativas para performances. Por fim, esta tese propõe uma nova reflexão sobre a distinção entre compor e projetar um novo instrumento num contexto que envolve uma multiplicidade de atores. Este trabalho apresenta as seguintes contribuições principais para o campo do design de interação aplicado à tecnologia musical: 1) ARCAA, uma framework para analisar o conjunto de relações interconectadas em performances interativas, validado através de dois estudos de caso relacionados com a música, um estudo de caso relacionado com a dança e uma análise sistemática da literatura; 2) Recomendações para o design de sistemas interativos musicais para performance (música ou dança), tendo em conta as necessidades dos vários atores e a sobreposição entre a composição musical e o design de tecnologia interactiva; 3) Uma taxonomia sobre como as partituras musicais moldaram as ecologias performativas no NIME, com base numa análise sistemática da literatura dos artigos apresentados e publicados nestas conferência; 4) Proposta de uma aborda- gem metodológica combinando abordagens de design autobiográfico e idiográfico em performances interativas

Using movement sonification to alter body perception and promote physical activity in physically inactive people

Mención Internacional en el título de doctorWorldwide, one out of four adults are not physically active enough. Supporting people to be physically active through technology remains thus an important challenge in the field of Human-Computer Interaction (HCI). Some technologies have tried to tackle this challenge of increasing physical activity (PA) by using sensing devices for monitoring the amount and quality of PA and providing motivational feedback on it. However, such technologies provide very limited support to physically inactive users: while users are aware of their physical inactivity level, they are frequently incapable of acting on these problems by themselves. Among the reasons for it are negative perceptions about one’s body (e.g., feelings of body tiredness or weakness in self-esteem) which may act as psychological barriers to PA. This research project aims to address this limitation by employing an approach that, through movement sonification (i.e., real-time auditory feedback on body movement), exploits bottom-up multisensory mechanisms related to BPs to ultimately support PA. This thesis presents the design, development, and evaluation of SoniShoes and SoniBand, two wearable technological devices with a gesture-sound palette that allows for a range of body movement sonifications aimed to alter BPs. These prototypes aim at changing BPs, and in turn emotional state and movement behavior, to address psychological barriers related to the perception of one’s body, and ultimately impact positively on people’s adherence to PA. First, this work proposes to organize knowledge through a taxonomy of the barriers to PA related to body perception (BP), which follows a process of four steps to inform the design of the movement-sound palette: (1) Identification, (2) Extraction and clustering of attributes, (3) Definition of instructions or considerations, and (4) Strategies. The first two steps allowed the identification and grouping of barriers to PA that are related to BPs, with inputs from a literature review, a survey, and a focus group with HCI experts. The third and fourth steps allowed defining the body features and dimensions to act upon, to finally propose movement sonification strategies that have the potential to tackle the barriers. Second, several movement-sound mappings, based on metaphors, are presented. Movements were selected from exercises included in guidelines for becoming more physically active (e.g., walking). The mappings of these movements into sounds were implemented in SoniShoes and SoniBand prototypes. They were evaluated through an iterative process, starting with an exploratory study that tested for the first time the potential of the proposed mappings to change BPs. In this first study, participants were asked to think aloud about their experiences using the first prototype of SoniShoes (from MagicShoes project), by describing their body sensations and sound characteristics during the exercise. Results suggested the potential of movement sonification to alter BP through movement sonification and informed the design of the subsequent studies and prototypes. This exploratory study was followed by quantitative and qualitative studies aimed to understand how to design movement sonifications and wearable devices integrating them to facilitate PA by tackling barriers related to BP. The quantitative studies were controlled laboratory studies, in which different versions of SoniShoes and SoniBand prototypes were evaluated, and which results led to further iterations of the prototypes. The results of these quantitative evaluations revealed movement-sound mappings that can lead to changes in feelings about the body (e.g., feeling lighter or less tired), feelings about the movement (e.g., having more movement control over the movement), and emotional feelings (e.g., having more comfort, motivation to complete the exercise, or feeling happier) during PA. Results also showed effects of sound on movement behavior, such as effects in movement deceleration/acceleration and stance time, and proprioceptive awareness. Furthermore, two qualitative studies were carried out, which involved using the SoniBand prototype for several days and in two different contexts of use, laboratory and home. The aim of these studies was two-fold. First, elucidating the effects that particular metaphorical sonifications’ qualities and characteristics have on people’s perception of their own body and their PA. Second, understanding how the observed effects may be specific to physically inactive (vs. active) populations. The results revealed specific connections between properties of the movement sonifications (e.g., gradual or frequency changes) on the one hand, and particular body feelings (e.g., feeling strong) and aspects of PA (e.g., repetitions) on the other hand, but effects seem to vary according to the PA-level of the populations. Finally, the findings, contributions, and principles for the design of movement sonifications and wearable technology to promote PA through acting upon BP are discussed, finishing by considering implications for potential interventions and applications supporting PA, as well as opportunities opened for future research.En todo el mundo, uno de cada cuatro adultos no es lo suficientemente activo físicamente. Por ello, ayudar a las personas a ser físicamente activas a través de la tecnología sigue siendo un reto importante en el campo de “Human-Computer Interaction” (HCI). Algunas tecnologías han tratado de abordar el reto de aumentar la actividad física (PA) mediante el uso de dispositivos de detección para controlar la cantidad y la calidad de la PA y proporcionar retroalimentación motivacional al respecto. Sin embargo, estas tecnologías proporcionan una ayuda muy limitada a los usuarios físicamente inactivos: aunque los usuarios son conscientes de su nivel de inactividad física, a menudo son incapaces de actuar por sí mismos sobre estos problemas. Entre las razones están las percepciones negativas sobre el propio cuerpo (por ejemplo, la sensación de cansancio corporal o el no sentirse capaces) que pueden actuar como barreras psicológicas para la PA. Este proyecto de investigación pretende abordar esta limitación empleando un enfoque que, a través de la sonificación del movimiento (es decir, la retroalimentación auditiva en tiempo real sobre el movimiento del cuerpo), explota los mecanismos “bottom-up” multisensoriales relacionados con las percepciones del cuerpo (BPs) para apoyar la PA. Esta tesis presenta el diseño, el desarrollo y la evaluación de “SoniShoes” y “SoniBand”, dos dispositivos tecnológicos vestibles con una paleta de gestos y sonidos que permiten una serie de sonificaciones del movimiento corporal destinadas a modificar las BPs. Estos prototipos tienen como objetivo cambiar las BPs, y a su vez el estado emocional y el comportamiento de movimiento, para abordar las barreras psicológicas relacionadas con la BP, y en última instancia impactar positivamente en la adherencia de las personas a la PA. En primer lugar, este trabajo propone organizar el conocimiento a través de una taxonomía de las barreras a la PA relacionadas con la BP, que sigue un proceso de cuatro pasos para informar el diseño de la paleta de movimiento-sonido: (1) Identificación, (2) Extracción y agrupación de atributos, (3) Definición de instrucciones o consideraciones, y (4) Estrategias. Los dos primeros pasos permitieron identificar y agrupar las barreras a la PA relacionadas con los BP, con aportaciones de una revisión bibliográfica, una encuesta y un grupo de discusión con expertos en HCI. El tercero y cuarto paso permitió definir las características y dimensiones corporales sobre las que actuar, para finalmente proponer estrategias de sonificación del movimiento que tienen el potencial de abordar las barreras. En segundo lugar, se presentan varios mapeos de movimiento-sonido, basados en metáforas. Los movimientos se seleccionaron a partir de ejercicios incluidos en las guías para ser más activos físicamente (por ejemplo, caminar). Los mapeos de estos movimientos en sonidos se implementaron en los prototipos “SoniShoes” y “SoniBand”. Se evaluaron a través de un proceso iterativo, comenzando con un estudio exploratorio que probó por primera vez el potencial de los mapeos propuestos para cambiar los BP. En este primer estudio, se pidió a los participantes que pensaran en voz alta sobre sus experiencias utilizando el primer prototipo de “SoniShoes” (llamado “MagicShoes”), describiendo sus sensaciones corporales y las características del sonido durante el ejercicio. Los resultados mostraron el potencial de la sonificación del movimiento para alterar la BP a través de la sonificación del movimiento e informaron el diseño de los estudios y prototipos posteriores. A este estudio exploratorio le siguieron estudios cuantitativos y cualitativos destinados a comprender cómo diseñar sonificaciones del movimiento y dispositivos vestibles que las integren para facilitar la PA abordando las barreras relacionadas con la BP. Los estudios cuantitativos fueron estudios de laboratorio controlados, en los que se evaluaron diferentes versiones de los prototipos “SoniShoes” y “SoniBand”, y cuyos resultados condujeron a nuevas iteraciones de los prototipos. Los resultados de estas evaluaciones cuantitativas mostraron que existen mapeos de movimiento-sonido que pueden provocar cambios en las sensaciones sobre el cuerpo (por ejemplo, sentirse más ligero o menos cansado), en las sensaciones sobre el movimiento (por ejemplo, tener más control sobre el movimiento) y en las sensaciones emocionales (por ejemplo, tener más comodidad, motivación para completar el ejercicio o sentirse más feliz) durante la PA. Los resultados también mostraron los efectos del sonido en el comportamiento del movimiento, como los efectos en la desaceleración/aceleración del movimiento y el tiempo de postura, y la conciencia propioceptiva. Además, se llevaron a cabo dos estudios cualitativos, en los que se utilizó el prototipo “SoniBand” durante varios días y en dos contextos de uso diferentes, el laboratorio y el hogar. El objetivo de estos estudios era doble. En primer lugar, dilucidar los efectos que determinadas cualidades y características de las sonificaciones con metáforas tienen en la percepción que las personas tienen de su propio cuerpo y de su PA. En segundo lugar, comprender cómo los efectos observados pueden ser específicos de las poblaciones físicamente inactivas (vs. las activas). Los resultados revelaron conexiones específicas entre las propiedades de las sonificaciones de movimiento (por ejemplo, los cambios graduales o de frecuencia) por un lado, y las sensaciones corporales particulares (por ejemplo, sentirse fuerte) y los aspectos de la PA (por ejemplo, las repeticiones) por otro lado, pero los efectos parecen variar según el nivel de PA de las poblaciones. Por último, se discuten los hallazgos, las contribuciones y las guías de diseño de sonificación de movimiento y tecnología vestible para promover la PA a través de la actuación sobre la BP, para finalmente considerar las implicaciones para las posibles intervenciones y aplicaciones de apoyo a la PA, así como las oportunidades abiertas para futuras investigaciones.I owe thanks to “MAGIC SHOES” (PSI2016-79004-R and BES-2017-080471) and “CROSS-COLAB” (PGC2018-101884-B-I00) projects that funded my research. Thanks to “MAGIC OUTFIT” (PID2019-105579RB-I00) for letting me be part of the team and project.Programa de Doctorado en Ciencia y Tecnología Informática por la Universidad Carlos III de MadridPresidente: Paloma Martínez Fernández.- Secretario: Domna Banakou.- Vocal: Mar González Franc

3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil