A Goals-Based review of physical Modelling

International audiencePhysical modelling (PM) encounters a growing success. Though, in the context of computer music, il covers numerous goals, needs and challenges. The article aims at contributing to their understanding. It provides a bibliographic overview of the various goals that researchers and musicians may pursue. Il introduces a set of features that an ideal PM technique should offer

Physically-based modelling techniques for sound and synthesis

Among the many physically-based modelling techniques, various have been designed in computer music for sound synthesis. This text reviews some of the most important

Towards a multi-layer architecture for multi-modal rendering of expressive actions

International audienceExpressive content has multiple facets that can be conveyed by music, gesture, actions. Different application scenarios can require different metaphors for expressiveness control. In order to meet the requirements for flexible representation, we propose a multi-layer architecture structured into three main levels of abstraction. At the top (user level) there is a semantic description, which is adapted to specific user requirements and conceptualization. At the other end are low-level features that describe parameters strictly related to the rendering model. In between these two extremes, we propose an intermediate layer that provides a description shared by the various high-level representations on one side, and that can be instantiated to the various low-level rendering models on the other side. In order to provide a common representation of different expressive semantics and different modalities, we propose a physically-inspired description specifically suited for expressive actions

Go-with-the-Flow: Tracking, Analysis and Sonification of Movement and Breathing to Build Confidence in Activity Despite Chronic Pain

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This article proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. Home studies, a focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation

Go-with-the-flow: Tracking, Analysis and Sonification of Movement and Breathing to Build Confidence in Activity Despite Chronic Pain

Chronic (persistent) pain (CP) affects one in ten adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This paper proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement and relaxation with sound feedback. Home studies, a focus group and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation

Effort in gestural interactions with imaginary objects in Hindustani Dhrupad vocal music

Physical effort has often been regarded as a key factor of expressivity in music performance. Nevertheless, systematic experimental approaches to the subject have been rare. In North Indian classical (Hindustani) vocal music, singers often engage with melodic ideas during improvisation by manipulating intangible, imaginary objects with their hands, such as through stretching, pulling, pushing, throwing etc. The above observation suggests that some patterns of change in acoustic features allude to interactions that real objects through their physical properties can afford. The present study reports on the exploration of the relationships between movement and sound by accounting for the physical effort that such interactions require in the Dhrupad genre of Hindustani vocal improvisation. The work follows a mixed methodological approach, combining qualitative and quantitative methods to analyse interviews, audio-visual material and movement data. Findings indicate that despite the flexibility in the way a Dhrupad vocalist might use his/her hands while singing, there is a certain degree of consistency by which performers associate effort levels with melody and types of gestural interactions with imaginary objects. However, different schemes of cross-modal associations are revealed for the vocalists analysed, that depend on the pitch space organisation of each particular melodic mode (rāga), the mechanical requirements of voice production, the macro-structure of the ālāp improvisation and morphological cross-domain analogies. Results further suggest that a good part of the variance in both physical effort and gesture type can be explained through a small set of sound and movement features. Based on the findings, I argue that gesturing in Dhrupad singing is guided by: the know-how of humans in interacting with and exerting effort on real objects of the environment, the movement–sound relationships transmitted from teacher to student in the oral music training context and the mechanical demands of vocalisation

Technologies logicielles de la modélisation, de la simulation et de l'interaction multisensorielles instrumentales pour la création

The presented works have firstly questioned the possible inter-relations between physics-based modelling and simulation on the one hand, and creation and creativity on the other. More specifically, the possibility and fundamental interests of rooting creative activities onto physics-based modelling have been considered.This theme was approached from a Computer Sciences perspective, through the invention and the design of a new family of software environments aimed at creation with masses-interactions physics-based networks. In particular: GENESIS for musical creation, and MIMESIS for movement and computer graphics animation.These software environments empower the artist with innovative and complete creation processes, centred on physical modelling activities.Each and all of these environments are characterized by a specific “software environment paradigm”: dedicated features, software architecture specificities, adequate and necessary human-computer interaction means (interfaces and languages), and particular usage protocols (in our case: modelling and creative processes).These elements have been situated in the scientific and technological history of computer creation tools, and in the state of the art of the various concerned scientific domains. This work has in particular led to analysing the properties of masses-interactions physical networks (modularity, generic potential, calculability, usability, …), in relation with other categories of algorithms, models and formalisms used nowadays to address human sensori-motor skills. It also accompanied the demonstration that the building of physical models by an artist can be deeply relevant. This relevance lies in the qualities of the phenomena that are finally generated, but also, which is more original, and probably more essential, in the modelling/creation processes themselves.Finally, the works have been extended to various other axes: contributions to synchronous real-time simulation architecture and software for multisensory interaction, including force-feedback, sound, and image, by means of physics-based simulation; experimental modelling works; contributions to the evolutions of the formal basis of the physics networks paradigm; contributions to graphical representation of punctual movements, etc.These past works and the achieved results altogether lead to introduce new research perspectives, which we name software technologies of multisensory instrumental interaction, with, as a far end view, the foundation of a new “science of the multisensory computer” and the possibility of a “multisensory art”.Les travaux présentés ont été pour l’essentiel traversés par un questionnement sur les relations liant modélisation et simulation physiques et création ou, plus spécifiquement, sur la possibilité et l’intérêt fondamental d’appuyer une activité de création sur la modélisation. Ce thème a été abordé sous l’angle de l’informatique appliquée (en anglais, les « computer sciences »), au moyen de l’invention et de la réalisation d’une nouvelle famille d’environnements logiciels pour la création avec les réseaux physiques masses-interactions, dont en particulier les modeleurs-simulateurs GENESIS pour la création musicale, puis MIMESIS pour la création du mouvement et l’image animée. Ces environnements permettent à l’artiste d’organiser des processus de création complets et novateurs, fondés sur l’activité de modélisation physique.Ils sont chacun, et dans leur ensemble, caractérisés par un paradigme d’environnement logiciel nouveau : des fonctionnalités, des caractéristiques en matière d’architecture logicielle, des ergonomies nécessaires, adéquates et spécifiques, ainsi que des protocoles d’utilisation – ici des processus de modélisation et de création – qui leur sont propres. Ces propriétés ont été situées dans l’histoire scientifique et technologique des outils de création et dans les états de l’art des communautés scientifiques concernées. Ce travail a notamment donné lieu à une étude des propriétés des réseaux physiques masses-interactions (généricité, modularité, calculabilité, utilisabilité, généralité...) en vis-à-vis des catégories d’algorithmes, modèles et formalismes utilisés pour adresser la sensori-motricité humaine. Il s’est de plus accompagné d’une analyse du fait que l’activité de construction de modèles physiques peut présenter une pertinence profonde pour l’artiste – pertinence qui se joue non seulement dans les qualités des phénomènes générés, mais encore, de façon plus originale et peut-être plus essentielle, dans le processus de modélisation/création lui-même.Enfin, les travaux se sont étendus à plusieurs autres axes d’études : contributions aux logiciels de simulation temps réel pour l’interaction multisensorielle incluant le retour d’effort, le son et l’image, au moyen de la simulation physique ; travaux expérimentaux en modélisation ; contributions à l’amélioration du cadre formel des réseaux physiques masses-interactions ; contributions aux procédés de mise en forme visuelle de mouvements ponctuels ; travaux de positionnement théorique ; etc.L’expérience et les résultats acquis conduisent à introduire de nouvelles perspectives de recherche, sous le fil conducteur des technologies logicielles de l’interaction multisensorielle instrumentale, avec en ligne de mire la fondation d’une « science de l’ordinateur multisensoriel » et la possibilité d’un « art multisensoriel »

Staying active despite pain: Investigating feedback mechanisms to support physical activity in people with chronic musculoskeletal pain

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Physical activity is important for improving function and quality of life in people with chronic pain, but psychological factors such as fear of increased pain and damage due to activity, lack of confidence or support, make it difficult to build and maintain physical activity towards long-term goals. There is insufficient research to guide the design of interactive technology to support people with CP in self-managing physical activity. This thesis aims to bridge this gap through five contributions: first, a detailed analysis from a plethora of qualitative studies with people with CP and physiotherapists was done to identify factors to be incorporated into technology to promote physical activity despite pain. Second, we rethink the role of technology in improving uptake of physical activity in people with CP by proposing a novel sonification framework (Go-with-the-flow) that addresses psychological and physical needs raised by our studies; through an iterative approach, we designed a wearable device to implement and evaluate the framework. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. A focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the framework and device facilitate self-directed rehabilitation. Third, we understand the role of sensing technology and real-time feedback in supporting functional activity, using the Go-with-the-flow framework and wearable device; we conducted evaluations including contextual interviews, diary studies and a 7-14 days study of self-directed home-based use of the device by people with CP. Fourth, building on the understanding from all our studies and literature from other conditions where physical rehabilitation is critical, we propose a framework for designing technology for physical rehabilitation (RaFT). Fifth, we reflect on our studies with people with CP and physiotherapists and provide practical insights for HCI research in sensitive settings