Linking embodied coordination dynamics and subjective experiences in musical interactions : a renewed methodological paradigm

Embodied music cognition provides a valuable and comprehensive research paradigm within systematic musicology to describe and explain musical sense-making. The basic claim underlying musical embodiment is that subjective meaning, in its broadest sense, is actively constructed within humans’ bodily interaction with music. As such, the empirical study of bodily coordination may provide insights into the subjective aspects of musical experiences. In the present paper, we advocate for a dynamical systems approach to human music interaction, focusing on the time-varying principles, and the relational aspects of the musical interaction process. We propose a model that integrates these focus points, to investigate the link between embodied coordination dynamics, subjective experience and sense-making. We then discuss possible quantitative and qualitative techniques that allow to operationalise the model into concrete empirical music research. Finally, we conclude by presenting some illustrative research cases conducted at IPEM, Ghent University institute for systematic musicology

INTERACTIVE SONIFICATION STRATEGIES FOR THE MOTION AND EMOTION OF DANCE PERFORMANCES

The Immersive Interactive SOnification Platform, or iISoP for short, is a research platform for the creation of novel multimedia art, as well as exploratory research in the fields of sonification, affective computing, and gesture-based user interfaces. The goal of the iISoP’s dancer sonification system is to “sonify the motion and emotion” of a dance performance via musical auditory display. An additional goal of this dissertation is to develop and evaluate musical strategies for adding layer of emotional mappings to data sonification. The result of the series of dancer sonification design exercises led to the development of a novel musical sonification framework. The overall design process is divided into three main iterative phases: requirement gathering, prototype generation, and system evaluation. For the first phase help was provided from dancers and musicians in a participatory design fashion as domain experts in the field of non-verbal affective communication. Knowledge extraction procedures took the form of semi-structured interviews, stimuli feature evaluation, workshops, and think aloud protocols. For phase two, the expert dancers and musicians helped create test-able stimuli for prototype evaluation. In phase three, system evaluation, experts (dancers, musicians, etc.) and novice participants were recruited to provide subjective feedback from the perspectives of both performer and audience. Based on the results of the iterative design process, a novel sonification framework that translates motion and emotion data into descriptive music is proposed and described

Musical Cities

Musical Cities represents an innovative approach to scholarly research and dissemination. A digital and interactive 'book', it explores the rhythms of our cities, and the role they play in our everyday urban lives, through the use of sound and music. Sara Adhitya first discusses why we should listen to urban rhythms in order to design more liveable and sustainable cities, before demonstrating how we can do so through various acoustic communication techniques. Using audio-visual examples, Musical Cities takes the ‘listener’ on an interactive journey, revealing how sound and music can be used to represent, compose, perform and interact with the city. Through case studies of urban projects developed in Paris, Perth, Venice and London, Adhitya demonstrates how the power of music, and the practice of listening, can help us to compose more accessible, inclusive, engaging, enjoyable, and ultimately more sustainable cities

Musical Cities

Musical Cities represents an innovative approach to scholarly research and dissemination. A digital and interactive 'book', it explores the rhythms of our cities, and the role they play in our everyday urban lives, through the use of sound and music. Sara Adhitya first discusses why we should listen to urban rhythms in order to design more liveable and sustainable cities, before demonstrating how we can do so through various acoustic communication techniques. Using audio-visual examples, Musical Cities takes the ‘listener’ on an interactive journey, revealing how sound and music can be used to represent, compose, perform and interact with the city. Through case studies of urban projects developed in Paris, Perth, Venice and London, Adhitya demonstrates how the power of music, and the practice of listening, can help us to compose more accessible, inclusive, engaging, enjoyable, and ultimately more sustainable cities

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Musical Cities

Musical Cities represents an innovative approach to scholarly research and dissemination. A digital and interactive 'book', it explores the rhythms of our cities, and the role they play in our everyday urban lives, through the use of sound and music. Sara Adhitya first discusses why we should listen to urban rhythms in order to design more liveable and sustainable cities, before demonstrating how we can do so through various acoustic communication techniques. Using audio-visual examples, Musical Cities takes the ‘listener’ on an interactive journey, revealing how sound and music can be used to represent, compose, perform and interact with the city. Through case studies of urban projects developed in Paris, Perth, Venice and London, Adhitya demonstrates how the power of music, and the practice of listening, can help us to compose more accessible, inclusive, engaging, enjoyable, and ultimately more sustainable cities

Interactive Spaces: Model for Motion-based Music Applications

With the extensive utilization of touch screens, smartphones and various reactive surfaces, reality- based and intuitive interaction styles have now become customary. The employment of larger interactive areas, like floors or peripersonal three-dimensional spaces, further increase the reality- based interaction affordances, allowing full-body involvement and the development of a co- located, shared user experience. Embodied and spatial cognition play a fundamental role for the interaction in this kind of spaces, where users act in the reality with no device in the hands and obtain an audio and graphical output depending on their movements. Starting from the early experiments of Myron Krueger in 1971, responsive floors have been developed through various technologies including sensorized tiles and computer vision systems, to be employed in learn- ing environments, entertainment, games and rehabilitation. Responsive floors allow the spatial representation of concepts and for this reason are suitable for immediate communication and engagement. As many musical features have meaningful spatial representations, they can easily be reproduced in the physical space through a conceptual blending approach and be made available to a great number of users. This is the key idea for the design of the original music applications presented in this thesis. The applications, devoted to music learning, production and active listening, introduce a novel creative approach to music, which can be further assumed as a general paradigm for the design of motion-based learning environments. Application assessment with upper elementary and high school students has proved that users engagement and bodily inter- action have a high learning power, which can be a valid resource for deeper music knowledge and more creative learning processes. Although further interface tests showed that touch screen interaction performs better than full-body interaction, some important guidelines for the design of reactive floors applications have been obtained on the basis of these test results. Moreover, the conceptual framework developed for the design of music applications can represent a valid paradigm also in the general field of human-computer interaction