Sonification of surface tapping changes behavior, surface perception, and emotion

The audio feedback resulting from object interaction provides information about the material of the surface and about one's own motor behavior. With the current developments in interactive sonification, it's now possible to digitally change this audio feedback, making the use of interactive sonification a compelling approach to shape tactile surface interactions. Here, the authors present a prototype for a sonic interactive surface, capable of delivering surface tapping sounds in real time when triggered by users' taps on a real surface or on an imagined 'virtual' surface. In this system, the delivered audio feedback can be varied so that the tapping sounds correspond to different applied strengths during tapping. The authors also propose a multidimensional measurement approach to evaluate user experiences of multimodal interactive systems. They evaluated their system by looking at the effect of the altered tapping sounds on emotional action-related responses, the users' interactions with the surface, and perceived surface hardness. Results show the influence of the sonification of tapping at all levels: emotional, behavioral, and perceptual. These results have implications on the design of interactive sonification displays and tangible auditory interfaces aiming to change perceived and subsequent motor behavior as well as perceived material properties

Exploring the Design Space for Body Transformation Wearables to Support Physical Activity through Sensitizing and Bodystorming

Negative or disturbed body perceptions are often interwoven with people's physical inactivity. While wearables can support body perception changes (body transformation), the design space of body transformation wearables supporting physical activity remains narrow. To expand this design space, we conducted an embodied co-design workshop with users. Using conceptual and tangible sensitizing tools, we explored/reflected on bodily sensations at three moments of movement execution (before/during/after). Conceptual tools were used to evoke/reflect/capture past lived experiences, while tangible tools were used as ideation probes for sensory bodystorming. Two design concepts emerged, reflecting diverging approaches to body transformation wearables: one focused on reminders and movement correction; the other on sensory augmentation and facilitation. We reflect on how each facilitates useful representations of body sensations during movement, and present methodological recommendations for designing technology for sensory augmentation in this area. Finally, we propose a preliminary prototype based on our design concepts and discuss future steps

Designing a gesture-sound wearable system to motivate physical activity by altering body perception

People, through their bodily actions, engage in sensorimotor loops that connect them to the world and to their own bodies. People's brains integrate the incoming sensory information to form mental representations of their body appearance and capabilities. Technology provides exceptional opportunities to tweak sensorimotor loops and provide people with different experiences of their bodies. We recently showed that real-time sound feedback on one's movement (sonic avatar) can be used for sensory alteration of people's body perception, and in turn provoke enhanced motor behaviour, confidence and motivation for physical activity (PA) in people while increasing their positive emotions towards their own bodies. Here we describe the design process of a wearable prototype that aims to investigate how we can overcome known body-perception-related psychological barriers to PA by employing action-sound loops. The prototype consists of sensors that capture people's bodily actions and a gesture-sound palette that allows different action-sound mappings. Grounded in neuroscientific, clinical and sports psychology studies on body perception and PA, the ultimate design aim is to enhance PA in inactive populations by provoking changes on their bodily experience

A transdisciplinary collaborative journey leading to sensorial clothing

Recent science funding initiatives have enabled participants from a diverse array of disciplines to engage in common spaces for developing solutions for new wearables. These initiatives include collaborations between the arts and sciences, fields which have traditionally contributed very different forms of knowledge, methodology, and results. However, many such collaborations often turn out as science communication and dissemination activities that make no concrete contribution to technological innovation. Magic Lining, a transdisciplinary collaborative project involving artistic and scientific partners working in the fields of e-textile design, cognitive neuroscience and human-computer interaction, creates a shared experiential knowledge space. This article focuses on the research question of how a transdisciplinary collaborative design processinvolving material explorations, prototyping, first-person-perspective and user studies, can lead to the creation of a garment that invites various perceptual and emotional responses in its wearer. The article reflects on the design journey, highlighting the transdisciplinary team's research through design experience and shared language for knowledge exchange. This process has revealed new research paths for an emerging field of 'sensorial clothing', combining the various team members' fields of expertise and resulting in a wearable prototype.This work was partially supported by the VERTIGO project as part of the STARTS program of the European Commission, based on technological elements from the project Magic Shoes (grant PSI2016-79004-R, Ministerio de Economía, Industria y Competitividad of Spain, AEI/FEDER). The work was also supported by the project Magic outFIT, funded by the Spanish Agencia Estatal de Investigación (PID2019-105579RB-I00/AEI/10.13039/501100011033). Aleksander Väljamäe’s work was supported by the Estonian Research Council grant PUT1518; and Ana Tajadura-Jiménez’s work was supported by RYC-2014–15421 grant, Ministerio de Economía, Industria y Competitividad of Spain

As Light as You Aspire to Be: Changing body perception with sound to support physical activity

Supporting exercise adherence through technology remains an important HCI challenge. Recent works showed that altering walking sounds leads people perceiving themselves as thinner/lighter, happier and walking more dynamically. While this novel approach shows potential for physical activity, it raises critical questions impacting technology design. We ran two studies in the context of exertion (gym-step, stairs-climbing) to investigate how individual factors impact the effect of sound and the duration of the after-effects. The results confirm that the effects of sound in body-perception occur even in physically demanding situations and through ubiquitous wearable devices. We also show that the effect of sound interacted with participants’ body weight and masculinity/femininity aspirations, but not with gender. Additionally, changes in body-perceptions did not hold once the feedback stopped; however, body-feelings or behavioural changes appeared to persist for longer. We discuss the results in terms of malleability of body-perception and highlight opportunities for supporting exercise adherence

Using movement sonification to alter body perception and promote physical activity in physically inactive people

Mención Internacional en el título de doctorWorldwide, one out of four adults are not physically active enough. Supporting people to be physically active through technology remains thus an important challenge in the field of Human-Computer Interaction (HCI). Some technologies have tried to tackle this challenge of increasing physical activity (PA) by using sensing devices for monitoring the amount and quality of PA and providing motivational feedback on it. However, such technologies provide very limited support to physically inactive users: while users are aware of their physical inactivity level, they are frequently incapable of acting on these problems by themselves. Among the reasons for it are negative perceptions about one’s body (e.g., feelings of body tiredness or weakness in self-esteem) which may act as psychological barriers to PA. This research project aims to address this limitation by employing an approach that, through movement sonification (i.e., real-time auditory feedback on body movement), exploits bottom-up multisensory mechanisms related to BPs to ultimately support PA. This thesis presents the design, development, and evaluation of SoniShoes and SoniBand, two wearable technological devices with a gesture-sound palette that allows for a range of body movement sonifications aimed to alter BPs. These prototypes aim at changing BPs, and in turn emotional state and movement behavior, to address psychological barriers related to the perception of one’s body, and ultimately impact positively on people’s adherence to PA. First, this work proposes to organize knowledge through a taxonomy of the barriers to PA related to body perception (BP), which follows a process of four steps to inform the design of the movement-sound palette: (1) Identification, (2) Extraction and clustering of attributes, (3) Definition of instructions or considerations, and (4) Strategies. The first two steps allowed the identification and grouping of barriers to PA that are related to BPs, with inputs from a literature review, a survey, and a focus group with HCI experts. The third and fourth steps allowed defining the body features and dimensions to act upon, to finally propose movement sonification strategies that have the potential to tackle the barriers. Second, several movement-sound mappings, based on metaphors, are presented. Movements were selected from exercises included in guidelines for becoming more physically active (e.g., walking). The mappings of these movements into sounds were implemented in SoniShoes and SoniBand prototypes. They were evaluated through an iterative process, starting with an exploratory study that tested for the first time the potential of the proposed mappings to change BPs. In this first study, participants were asked to think aloud about their experiences using the first prototype of SoniShoes (from MagicShoes project), by describing their body sensations and sound characteristics during the exercise. Results suggested the potential of movement sonification to alter BP through movement sonification and informed the design of the subsequent studies and prototypes. This exploratory study was followed by quantitative and qualitative studies aimed to understand how to design movement sonifications and wearable devices integrating them to facilitate PA by tackling barriers related to BP. The quantitative studies were controlled laboratory studies, in which different versions of SoniShoes and SoniBand prototypes were evaluated, and which results led to further iterations of the prototypes. The results of these quantitative evaluations revealed movement-sound mappings that can lead to changes in feelings about the body (e.g., feeling lighter or less tired), feelings about the movement (e.g., having more movement control over the movement), and emotional feelings (e.g., having more comfort, motivation to complete the exercise, or feeling happier) during PA. Results also showed effects of sound on movement behavior, such as effects in movement deceleration/acceleration and stance time, and proprioceptive awareness. Furthermore, two qualitative studies were carried out, which involved using the SoniBand prototype for several days and in two different contexts of use, laboratory and home. The aim of these studies was two-fold. First, elucidating the effects that particular metaphorical sonifications’ qualities and characteristics have on people’s perception of their own body and their PA. Second, understanding how the observed effects may be specific to physically inactive (vs. active) populations. The results revealed specific connections between properties of the movement sonifications (e.g., gradual or frequency changes) on the one hand, and particular body feelings (e.g., feeling strong) and aspects of PA (e.g., repetitions) on the other hand, but effects seem to vary according to the PA-level of the populations. Finally, the findings, contributions, and principles for the design of movement sonifications and wearable technology to promote PA through acting upon BP are discussed, finishing by considering implications for potential interventions and applications supporting PA, as well as opportunities opened for future research.En todo el mundo, uno de cada cuatro adultos no es lo suficientemente activo físicamente. Por ello, ayudar a las personas a ser físicamente activas a través de la tecnología sigue siendo un reto importante en el campo de “Human-Computer Interaction” (HCI). Algunas tecnologías han tratado de abordar el reto de aumentar la actividad física (PA) mediante el uso de dispositivos de detección para controlar la cantidad y la calidad de la PA y proporcionar retroalimentación motivacional al respecto. Sin embargo, estas tecnologías proporcionan una ayuda muy limitada a los usuarios físicamente inactivos: aunque los usuarios son conscientes de su nivel de inactividad física, a menudo son incapaces de actuar por sí mismos sobre estos problemas. Entre las razones están las percepciones negativas sobre el propio cuerpo (por ejemplo, la sensación de cansancio corporal o el no sentirse capaces) que pueden actuar como barreras psicológicas para la PA. Este proyecto de investigación pretende abordar esta limitación empleando un enfoque que, a través de la sonificación del movimiento (es decir, la retroalimentación auditiva en tiempo real sobre el movimiento del cuerpo), explota los mecanismos “bottom-up” multisensoriales relacionados con las percepciones del cuerpo (BPs) para apoyar la PA. Esta tesis presenta el diseño, el desarrollo y la evaluación de “SoniShoes” y “SoniBand”, dos dispositivos tecnológicos vestibles con una paleta de gestos y sonidos que permiten una serie de sonificaciones del movimiento corporal destinadas a modificar las BPs. Estos prototipos tienen como objetivo cambiar las BPs, y a su vez el estado emocional y el comportamiento de movimiento, para abordar las barreras psicológicas relacionadas con la BP, y en última instancia impactar positivamente en la adherencia de las personas a la PA. En primer lugar, este trabajo propone organizar el conocimiento a través de una taxonomía de las barreras a la PA relacionadas con la BP, que sigue un proceso de cuatro pasos para informar el diseño de la paleta de movimiento-sonido: (1) Identificación, (2) Extracción y agrupación de atributos, (3) Definición de instrucciones o consideraciones, y (4) Estrategias. Los dos primeros pasos permitieron identificar y agrupar las barreras a la PA relacionadas con los BP, con aportaciones de una revisión bibliográfica, una encuesta y un grupo de discusión con expertos en HCI. El tercero y cuarto paso permitió definir las características y dimensiones corporales sobre las que actuar, para finalmente proponer estrategias de sonificación del movimiento que tienen el potencial de abordar las barreras. En segundo lugar, se presentan varios mapeos de movimiento-sonido, basados en metáforas. Los movimientos se seleccionaron a partir de ejercicios incluidos en las guías para ser más activos físicamente (por ejemplo, caminar). Los mapeos de estos movimientos en sonidos se implementaron en los prototipos “SoniShoes” y “SoniBand”. Se evaluaron a través de un proceso iterativo, comenzando con un estudio exploratorio que probó por primera vez el potencial de los mapeos propuestos para cambiar los BP. En este primer estudio, se pidió a los participantes que pensaran en voz alta sobre sus experiencias utilizando el primer prototipo de “SoniShoes” (llamado “MagicShoes”), describiendo sus sensaciones corporales y las características del sonido durante el ejercicio. Los resultados mostraron el potencial de la sonificación del movimiento para alterar la BP a través de la sonificación del movimiento e informaron el diseño de los estudios y prototipos posteriores. A este estudio exploratorio le siguieron estudios cuantitativos y cualitativos destinados a comprender cómo diseñar sonificaciones del movimiento y dispositivos vestibles que las integren para facilitar la PA abordando las barreras relacionadas con la BP. Los estudios cuantitativos fueron estudios de laboratorio controlados, en los que se evaluaron diferentes versiones de los prototipos “SoniShoes” y “SoniBand”, y cuyos resultados condujeron a nuevas iteraciones de los prototipos. Los resultados de estas evaluaciones cuantitativas mostraron que existen mapeos de movimiento-sonido que pueden provocar cambios en las sensaciones sobre el cuerpo (por ejemplo, sentirse más ligero o menos cansado), en las sensaciones sobre el movimiento (por ejemplo, tener más control sobre el movimiento) y en las sensaciones emocionales (por ejemplo, tener más comodidad, motivación para completar el ejercicio o sentirse más feliz) durante la PA. Los resultados también mostraron los efectos del sonido en el comportamiento del movimiento, como los efectos en la desaceleración/aceleración del movimiento y el tiempo de postura, y la conciencia propioceptiva. Además, se llevaron a cabo dos estudios cualitativos, en los que se utilizó el prototipo “SoniBand” durante varios días y en dos contextos de uso diferentes, el laboratorio y el hogar. El objetivo de estos estudios era doble. En primer lugar, dilucidar los efectos que determinadas cualidades y características de las sonificaciones con metáforas tienen en la percepción que las personas tienen de su propio cuerpo y de su PA. En segundo lugar, comprender cómo los efectos observados pueden ser específicos de las poblaciones físicamente inactivas (vs. las activas). Los resultados revelaron conexiones específicas entre las propiedades de las sonificaciones de movimiento (por ejemplo, los cambios graduales o de frecuencia) por un lado, y las sensaciones corporales particulares (por ejemplo, sentirse fuerte) y los aspectos de la PA (por ejemplo, las repeticiones) por otro lado, pero los efectos parecen variar según el nivel de PA de las poblaciones. Por último, se discuten los hallazgos, las contribuciones y las guías de diseño de sonificación de movimiento y tecnología vestible para promover la PA a través de la actuación sobre la BP, para finalmente considerar las implicaciones para las posibles intervenciones y aplicaciones de apoyo a la PA, así como las oportunidades abiertas para futuras investigaciones.I owe thanks to “MAGIC SHOES” (PSI2016-79004-R and BES-2017-080471) and “CROSS-COLAB” (PGC2018-101884-B-I00) projects that funded my research. Thanks to “MAGIC OUTFIT” (PID2019-105579RB-I00) for letting me be part of the team and project.Programa de Doctorado en Ciencia y Tecnología Informática por la Universidad Carlos III de MadridPresidente: Paloma Martínez Fernández.- Secretario: Domna Banakou.- Vocal: Mar González Franc