Go-with-the-flow: Tracking, Analysis and Sonification of Movement and Breathing to Build Confidence in Activity Despite Chronic Pain

Chronic (persistent) pain (CP) affects one in ten adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This paper proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement and relaxation with sound feedback. Home studies, a focus group and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation

Supporting Everyday Function in Chronic Pain Using Wearable Technology

While most rehabilitation technologies target situated exercise sessions and associated performance metrics, physiotherapists recommend physical activities that are integrated with everyday functioning. We conducted a 1-2 week home study to explore how people with chronic pain use wearable technology that senses and sonifies movement (i.e., movement mapped to sound in real-time) to do functional activity (e.g., loading the dishwasher). Our results show that real-time movement sonification led to an increased sense of control during challenging everyday tasks. Sonification calibrated to functional activity facilitated application of pain management techniques such as pacing. When calibrated to individual psychological needs, sonification enabled serendipitous discovery of physical capabilities otherwise obscured by a focus on pain or a dysfunctional proprioceptive system. A physiotherapist was invited to comment on the implications of our findings. We conclude by discussing opportunities provided by wearable sensing technology to enable better functioning, the ultimate goal of physical rehabilitation

Gait sonification for rehabilitation: adjusting gait patterns by acoustic transformation of kinematic data

To enhance motor learning in both sport and rehabilitation, auditory feedback has emerged as an effective tool. Since it requires less attention than visual feedback and hardly affects the visually dominated orientation in space, it can be used safely and effectively in natural locomotion such as walking. One method for generating acoustic movement feedback is the direct mapping of kinematic data to sound (movement sonification). Using this method in orthopedic gait rehabilitation could make an important contribution to the prevention of falls and secondary diseases. This would not only reduce the individual suffering of the patients, but also medical treatment costs. To determine the possible applications of movement sonification in gait rehabilitation in the context of this work, a new gait sonification method based on inertial sensor technology was developed. Against the background of current scientific findings on sensorimotor function, feedback methods, and gait analysis, three studies published in scientific journals are presented in this thesis: The first study shows the applicability and acceptance of the feedback method in patients undergoing inpatient rehabilitation after unilateral total hip arthroplasty. In addition, the direct effect of gait sonification during ten gait training sessions on the patients’ gait pattern was revealed. In the second study, the immediate follow-up effect of gait sonification on the kinematics of the same patient group is examined at four measurement points after gait training. In this context, a significant influence of sonification on the gait pattern of the patients was shown, which, however, did not meet the previously expected effects. In view of this finding, the effect of the specific sound parameter loudness of gait sonification on the gait of healthy persons was analyzed in a third study. Thus, an impact of asymmetric loudness of gait sonification on the ground contact time could be detected. Considering this cause-effect relationship can be a component in improving gait sonfication in rehabilitation. Overall, the feasibility and effectiveness of movement sonification in gait rehabilitation of patients after unilateral hip arthroplasty becomes evident. The findings thus illustrate the potential of the method to efficiently support orthopedic gait rehabilitation in the future. On the basis of the results presented, this potential can be exploited in particular by an adequate mapping of movement to sound, a systematic modification of selected sound parameters, and a target-group-specific selection of the gait sonification mode. In addition to a detailed investigation of the three factors mentioned above, an optimization and refinement of gait analysis in patients after arthroplasty using inertial sensor technology will be beneficial in the future.Akustisches Feedback kann wirkungsvoll eingesetzt werden, um das Bewegungslernen sowohl im Sport als auch in der Rehabilitation zu erleichtern. Da es weniger Aufmerksamkeit als visuelles Feedback erfordert und die visuell dominierte Orientierung im Raum kaum beeinträchtigt, kann es während einer natürlichen Fortbewegung wie dem Gehen sicher und effektiv genutzt werden. Eine Methode zur Generierung akustischen Bewegungsfeedbacks ist die direkte Abbildung kinematischer Daten auf Sound (Bewegungssonifikation). Ein Einsatz dieser Methode in der orthopädischen Gangrehabilitation könnte einen wichtigen Beitrag zur Prävention von Stürzen und Folgeerkrankungen leisten. Neben dem individuellen Leid der Patienten ließen sich so auch medizinische Behandlungskosten erheblich reduzieren. Um im Rahmen dieser Arbeit die Einsatzmöglichkeiten der Bewegungssonifikation in der Gangrehabilitation zu bestimmen, wurde eine neue Gangsonifikationsmethodik auf Basis von Inertialsensorik entwickelt. Zu der entwickelten Methodik werden, vor dem Hintergrund aktueller wissenschaftlicher Erkenntnisse zur Sensomotorik, zu Feedbackmethoden und zur Ganganalyse, in dieser Thesis drei in Fachzeitschriften publizierte Studien vorgestellt. Die erste Studie beschreibt die Anwendbarkeit und Akzeptanz der Feedbackmethode bei Patienten in stationärer Rehabilitation nach unilateraler Hüftendoprothetik. Darüber hinaus wird der direkte Effekt der Gangsonifikation während eines zehnmaligen Gangtrainings auf das Gangmuster der Patienten deutlich. In der zweiten Studie wird der unmittelbare Nacheffekt der Gangsonifikation auf die Kinematik der gleichen Patientengruppe zu vier Messzeitpunkten nach dem Gangtraining untersucht. In diesem Zusammenhang zeigte sich ein signifikanter Einfluss der Sonifikation auf das Gangbild der Patienten, der allerdings nicht den zuvor erwarteten Effekten entsprach. Aufgrund dieses Ergebnisses wurde in einer dritten Studie die Wirkung des spezifischen Klangparameters Lautstärke der Gangsonifikation auf das Gangbild von gesunden Personen analysiert. Dabei konnte ein Einfluss von asymmetrischer Lautstärke der Gangsonifikation auf die Bodenkontaktzeit nachgewiesen werden. Die Berücksichtigung dieses Ursache-Wirkungs-Zusammenhangs kann einen Baustein bei der Verbesserung der Gangsonifikation in der Rehabilitation darstellen. Insgesamt wird die Anwendbarkeit und Wirksamkeit von Bewegungssonifikation in der Gangrehabilitation bei Patienten nach unilateraler Hüftendoprothetik evident. Die gewonnenen Erkenntnisse verdeutlichen das Potential der Methode, die orthopädische Gangrehabilitation zukünftig effizient zu unterstützen. Ausschöpfen lässt sich dieses Potential auf Grundlage der vorgestellten Ergebnisse insbesondere anhand einer adäquaten Zuordnung von Bewegung zu Sound, einer systematischen Modifikation ausgewählter Soundparameter sowie einer zielgruppenspezifischen Wahl des Modus der Sonifikation. Neben einer differenzierten Untersuchung der genannten Faktoren, erscheint zukünftig eine Optimierung und Verfeinerung der Ganganalyse bei Patienten nach Endoprothetik unter Einsatz von Inertialsensorik notwendig

Enriching footsteps sounds in gait rehabilitation in chronic stroke patients: a pilot study

In the context of neurorehabilitation, sound is being increasingly applied for facilitating sensorimotor learning. In this study, we aimed to test the potential value of auditory stimulation for improving gait in chronic stroke patients by inducing alterations of the frequency spectra of walking sounds via a sound system that selectively amplifies and equalizes the signal in order to produce distorted auditory feedback. Twenty‐two patients with lower extremity paresis were exposed to real‐time alterations of their footstep sounds while walking. Changes in body perception, emotion, and gait were quantified. Our results suggest that by altering footsteps sounds, several gait parameters can be modified in terms of left–right foot asymmetry. We observed that augmenting low‐frequency bands or amplifying the natural walking sounds led to a reduction in the asymmetry index of stance and stride times, whereas it inverted the asymmetry pattern in heel–ground exerted force. By contrast, augmenting high‐frequency bands led to opposite results. These gait changes might be related to updating of internal forward models, signaling the need for adjustment of the motor system to reduce the perceived discrepancies between predicted–actual sensory feedbacks. Our findings may have the potential to enhance gait awareness in stroke patients and other clinical conditions, supporting gait rehabilitation

Staying active despite pain: Investigating feedback mechanisms to support physical activity in people with chronic musculoskeletal pain

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Physical activity is important for improving function and quality of life in people with chronic pain, but psychological factors such as fear of increased pain and damage due to activity, lack of confidence or support, make it difficult to build and maintain physical activity towards long-term goals. There is insufficient research to guide the design of interactive technology to support people with CP in self-managing physical activity. This thesis aims to bridge this gap through five contributions: first, a detailed analysis from a plethora of qualitative studies with people with CP and physiotherapists was done to identify factors to be incorporated into technology to promote physical activity despite pain. Second, we rethink the role of technology in improving uptake of physical activity in people with CP by proposing a novel sonification framework (Go-with-the-flow) that addresses psychological and physical needs raised by our studies; through an iterative approach, we designed a wearable device to implement and evaluate the framework. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. A focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the framework and device facilitate self-directed rehabilitation. Third, we understand the role of sensing technology and real-time feedback in supporting functional activity, using the Go-with-the-flow framework and wearable device; we conducted evaluations including contextual interviews, diary studies and a 7-14 days study of self-directed home-based use of the device by people with CP. Fourth, building on the understanding from all our studies and literature from other conditions where physical rehabilitation is critical, we propose a framework for designing technology for physical rehabilitation (RaFT). Fifth, we reflect on our studies with people with CP and physiotherapists and provide practical insights for HCI research in sensitive settings

An Embodied Sonification Model for Sit-to-Stand Transfers

Interactive sonification of biomechanical quantities is gaining relevance as a motor learning aid in movement rehabilitation, as well as a monitoring tool. However, existing gaps in sonification research (issues related to meaning, aesthetics, and clinical effects) have prevented its widespread recognition and adoption in such applications. The incorporation of embodied principles and musical structures in sonification design has gradually become popular, particularly in applications related to human movement. In this study, we propose a general sonification model for the sit-to-stand (STS) transfer, an important activity of daily living. The model contains a fixed component independent of the use-case, which represents the rising motion of the body as an ascending melody using the physical model of a flute. In addition, a flexible component concurrently sonifies STS features of clinical interest in a particular rehabilitative/monitoring situation. Here, we chose to represent shank angular jerk and movement stoppages (freezes), through perceptually salient pitch modulations and bell sounds. We outline the details of our technical implementation of the model. We evaluated the model by means of a listening test experiment with 25 healthy participants, who were asked to identify six normal and simulated impaired STS patterns from sonified versions containing various combinations of the constituent mappings of the model. Overall, we found that the participants were able to classify the patterns accurately (86.67 ± 14.69% correct responses with the full model, 71.56% overall), confidently (64.95 ± 16.52% self-reported rating), and in a timely manner (response time: 4.28 ± 1.52 s). The amount of sonified kinematic information significantly impacted classification accuracy. The six STS patterns were also classified with significantly different accuracy depending on their kinematic characteristics. Learning effects were seen in the form of increased accuracy and confidence with repeated exposure to the sound sequences. We found no significant accuracy differences based on the participants' level of music training. Overall, we see our model as a concrete conceptual and technical starting point for STS sonification design catering to rehabilitative and clinical monitoring applications

Roles for Personal Informatics in Chronic Pain

Self-management of chronic pain is a complex and demanding activity. Multidisciplinary pain management programs are designed to provide patients with the skills to improve, maintain functioning and self-manage their pain but gains diminish in the long-term due to lack of support from clinicians. Sensing technology can be a cost-effective way to extend support for self-management outside clinical settings but they are currently under-explored. In this paper, we report studies carried out to investigate how Personal Informatics Systems (PIS) based on wearable body sensing technology could facilitate pain self-management and functioning. Five roles for PIS emerged from a qualitative study with people with chronic pain and physiotherapists: (i) assessment, planning and prevention (ii) a direct supervisory and co-management role, (iii) facilitating deeper understanding, (iv) managing emotional states, and (v) sharing for social acceptability. A web-based survey was conducted to understand the parameters that should be tracked to support self-management and what tracked information should be shared with others. Finally, we suggest an extension to previous PIS models and propose design implications to address immediate, short-term and long-term information needs for personal use of people with chronic pain and for sharing with others. / Note: As originally published there is an error in the document. The following information was omitted by the authors: "The project was funded by the EPSRC grant Emotion & Pain Project EP/H017178/1 rather than the EPSRC grant EP/G043507/1: Pain rehabilitation: E/Motion-based automated coaching.." The article PDF remains unchanged

Using movement sonification to alter body perception and promote physical activity in physically inactive people

Mención Internacional en el título de doctorWorldwide, one out of four adults are not physically active enough. Supporting people to be physically active through technology remains thus an important challenge in the field of Human-Computer Interaction (HCI). Some technologies have tried to tackle this challenge of increasing physical activity (PA) by using sensing devices for monitoring the amount and quality of PA and providing motivational feedback on it. However, such technologies provide very limited support to physically inactive users: while users are aware of their physical inactivity level, they are frequently incapable of acting on these problems by themselves. Among the reasons for it are negative perceptions about one’s body (e.g., feelings of body tiredness or weakness in self-esteem) which may act as psychological barriers to PA. This research project aims to address this limitation by employing an approach that, through movement sonification (i.e., real-time auditory feedback on body movement), exploits bottom-up multisensory mechanisms related to BPs to ultimately support PA. This thesis presents the design, development, and evaluation of SoniShoes and SoniBand, two wearable technological devices with a gesture-sound palette that allows for a range of body movement sonifications aimed to alter BPs. These prototypes aim at changing BPs, and in turn emotional state and movement behavior, to address psychological barriers related to the perception of one’s body, and ultimately impact positively on people’s adherence to PA. First, this work proposes to organize knowledge through a taxonomy of the barriers to PA related to body perception (BP), which follows a process of four steps to inform the design of the movement-sound palette: (1) Identification, (2) Extraction and clustering of attributes, (3) Definition of instructions or considerations, and (4) Strategies. The first two steps allowed the identification and grouping of barriers to PA that are related to BPs, with inputs from a literature review, a survey, and a focus group with HCI experts. The third and fourth steps allowed defining the body features and dimensions to act upon, to finally propose movement sonification strategies that have the potential to tackle the barriers. Second, several movement-sound mappings, based on metaphors, are presented. Movements were selected from exercises included in guidelines for becoming more physically active (e.g., walking). The mappings of these movements into sounds were implemented in SoniShoes and SoniBand prototypes. They were evaluated through an iterative process, starting with an exploratory study that tested for the first time the potential of the proposed mappings to change BPs. In this first study, participants were asked to think aloud about their experiences using the first prototype of SoniShoes (from MagicShoes project), by describing their body sensations and sound characteristics during the exercise. Results suggested the potential of movement sonification to alter BP through movement sonification and informed the design of the subsequent studies and prototypes. This exploratory study was followed by quantitative and qualitative studies aimed to understand how to design movement sonifications and wearable devices integrating them to facilitate PA by tackling barriers related to BP. The quantitative studies were controlled laboratory studies, in which different versions of SoniShoes and SoniBand prototypes were evaluated, and which results led to further iterations of the prototypes. The results of these quantitative evaluations revealed movement-sound mappings that can lead to changes in feelings about the body (e.g., feeling lighter or less tired), feelings about the movement (e.g., having more movement control over the movement), and emotional feelings (e.g., having more comfort, motivation to complete the exercise, or feeling happier) during PA. Results also showed effects of sound on movement behavior, such as effects in movement deceleration/acceleration and stance time, and proprioceptive awareness. Furthermore, two qualitative studies were carried out, which involved using the SoniBand prototype for several days and in two different contexts of use, laboratory and home. The aim of these studies was two-fold. First, elucidating the effects that particular metaphorical sonifications’ qualities and characteristics have on people’s perception of their own body and their PA. Second, understanding how the observed effects may be specific to physically inactive (vs. active) populations. The results revealed specific connections between properties of the movement sonifications (e.g., gradual or frequency changes) on the one hand, and particular body feelings (e.g., feeling strong) and aspects of PA (e.g., repetitions) on the other hand, but effects seem to vary according to the PA-level of the populations. Finally, the findings, contributions, and principles for the design of movement sonifications and wearable technology to promote PA through acting upon BP are discussed, finishing by considering implications for potential interventions and applications supporting PA, as well as opportunities opened for future research.En todo el mundo, uno de cada cuatro adultos no es lo suficientemente activo físicamente. Por ello, ayudar a las personas a ser físicamente activas a través de la tecnología sigue siendo un reto importante en el campo de “Human-Computer Interaction” (HCI). Algunas tecnologías han tratado de abordar el reto de aumentar la actividad física (PA) mediante el uso de dispositivos de detección para controlar la cantidad y la calidad de la PA y proporcionar retroalimentación motivacional al respecto. Sin embargo, estas tecnologías proporcionan una ayuda muy limitada a los usuarios físicamente inactivos: aunque los usuarios son conscientes de su nivel de inactividad física, a menudo son incapaces de actuar por sí mismos sobre estos problemas. Entre las razones están las percepciones negativas sobre el propio cuerpo (por ejemplo, la sensación de cansancio corporal o el no sentirse capaces) que pueden actuar como barreras psicológicas para la PA. Este proyecto de investigación pretende abordar esta limitación empleando un enfoque que, a través de la sonificación del movimiento (es decir, la retroalimentación auditiva en tiempo real sobre el movimiento del cuerpo), explota los mecanismos “bottom-up” multisensoriales relacionados con las percepciones del cuerpo (BPs) para apoyar la PA. Esta tesis presenta el diseño, el desarrollo y la evaluación de “SoniShoes” y “SoniBand”, dos dispositivos tecnológicos vestibles con una paleta de gestos y sonidos que permiten una serie de sonificaciones del movimiento corporal destinadas a modificar las BPs. Estos prototipos tienen como objetivo cambiar las BPs, y a su vez el estado emocional y el comportamiento de movimiento, para abordar las barreras psicológicas relacionadas con la BP, y en última instancia impactar positivamente en la adherencia de las personas a la PA. En primer lugar, este trabajo propone organizar el conocimiento a través de una taxonomía de las barreras a la PA relacionadas con la BP, que sigue un proceso de cuatro pasos para informar el diseño de la paleta de movimiento-sonido: (1) Identificación, (2) Extracción y agrupación de atributos, (3) Definición de instrucciones o consideraciones, y (4) Estrategias. Los dos primeros pasos permitieron identificar y agrupar las barreras a la PA relacionadas con los BP, con aportaciones de una revisión bibliográfica, una encuesta y un grupo de discusión con expertos en HCI. El tercero y cuarto paso permitió definir las características y dimensiones corporales sobre las que actuar, para finalmente proponer estrategias de sonificación del movimiento que tienen el potencial de abordar las barreras. En segundo lugar, se presentan varios mapeos de movimiento-sonido, basados en metáforas. Los movimientos se seleccionaron a partir de ejercicios incluidos en las guías para ser más activos físicamente (por ejemplo, caminar). Los mapeos de estos movimientos en sonidos se implementaron en los prototipos “SoniShoes” y “SoniBand”. Se evaluaron a través de un proceso iterativo, comenzando con un estudio exploratorio que probó por primera vez el potencial de los mapeos propuestos para cambiar los BP. En este primer estudio, se pidió a los participantes que pensaran en voz alta sobre sus experiencias utilizando el primer prototipo de “SoniShoes” (llamado “MagicShoes”), describiendo sus sensaciones corporales y las características del sonido durante el ejercicio. Los resultados mostraron el potencial de la sonificación del movimiento para alterar la BP a través de la sonificación del movimiento e informaron el diseño de los estudios y prototipos posteriores. A este estudio exploratorio le siguieron estudios cuantitativos y cualitativos destinados a comprender cómo diseñar sonificaciones del movimiento y dispositivos vestibles que las integren para facilitar la PA abordando las barreras relacionadas con la BP. Los estudios cuantitativos fueron estudios de laboratorio controlados, en los que se evaluaron diferentes versiones de los prototipos “SoniShoes” y “SoniBand”, y cuyos resultados condujeron a nuevas iteraciones de los prototipos. Los resultados de estas evaluaciones cuantitativas mostraron que existen mapeos de movimiento-sonido que pueden provocar cambios en las sensaciones sobre el cuerpo (por ejemplo, sentirse más ligero o menos cansado), en las sensaciones sobre el movimiento (por ejemplo, tener más control sobre el movimiento) y en las sensaciones emocionales (por ejemplo, tener más comodidad, motivación para completar el ejercicio o sentirse más feliz) durante la PA. Los resultados también mostraron los efectos del sonido en el comportamiento del movimiento, como los efectos en la desaceleración/aceleración del movimiento y el tiempo de postura, y la conciencia propioceptiva. Además, se llevaron a cabo dos estudios cualitativos, en los que se utilizó el prototipo “SoniBand” durante varios días y en dos contextos de uso diferentes, el laboratorio y el hogar. El objetivo de estos estudios era doble. En primer lugar, dilucidar los efectos que determinadas cualidades y características de las sonificaciones con metáforas tienen en la percepción que las personas tienen de su propio cuerpo y de su PA. En segundo lugar, comprender cómo los efectos observados pueden ser específicos de las poblaciones físicamente inactivas (vs. las activas). Los resultados revelaron conexiones específicas entre las propiedades de las sonificaciones de movimiento (por ejemplo, los cambios graduales o de frecuencia) por un lado, y las sensaciones corporales particulares (por ejemplo, sentirse fuerte) y los aspectos de la PA (por ejemplo, las repeticiones) por otro lado, pero los efectos parecen variar según el nivel de PA de las poblaciones. Por último, se discuten los hallazgos, las contribuciones y las guías de diseño de sonificación de movimiento y tecnología vestible para promover la PA a través de la actuación sobre la BP, para finalmente considerar las implicaciones para las posibles intervenciones y aplicaciones de apoyo a la PA, así como las oportunidades abiertas para futuras investigaciones.I owe thanks to “MAGIC SHOES” (PSI2016-79004-R and BES-2017-080471) and “CROSS-COLAB” (PGC2018-101884-B-I00) projects that funded my research. Thanks to “MAGIC OUTFIT” (PID2019-105579RB-I00) for letting me be part of the team and project.Programa de Doctorado en Ciencia y Tecnología Informática por la Universidad Carlos III de MadridPresidente: Paloma Martínez Fernández.- Secretario: Domna Banakou.- Vocal: Mar González Franc