A Statistical Approach to Analyzing Sound Tracings

This paper presents an experiment on sound tracing, meaning an experiment on how people relate motion to sound. 38 participants were presented with 18 short sounds, and instructed to move their hands in the air while acting as though the sound was created by their hand motion. The hand motion of the participants was recorded, and has been analyzed using statistical tests, comparing results between different sounds, between different subjects, and between different sound classes. We have identified several relationships between sound and motion which are present in the majority of the subjects. A clear distinction was found in onset acceleration for motion to sounds with an impulsive dynamic envelope compared to non-impulsive sounds. Furthermore, vertical movement has been shown to be related to sound frequency, both in terms of spectral centroid and pitch. Moreover, a significantly higher amount of overall acceleration was observed for non-pitched sounds as compared to pitched sounds. The original publication is available at www.springerlink.com. Lecture Notes in Computer Science, 2012, Volume 7172/2012, 120-145

Towards a gesture-sound cross-modal analysis

Abstract. This article reports on the exploration of a method based on canonical correlation analysis (CCA) for the analysis of the relationship between gesture and sound in the context of music performance and listening. This method is a first step in the design of an analysis tool for gesture-sound relationships. In this exploration we used motion capture data recorded from subjects performing free hand movements while listening to short sound examples. We assume that even though the relationship between gesture and sound might be more complex, at least part of it can be revealed and quantified by linear multivariate regression applied to the motion capture data and audio descriptors extracted from the sound examples. After outlining the theoretical background, the article shows how the method allows for pertinent reasoning about the relationship between gesture and sound by analysing the data sets recorded from multiple and individual subjects

A Biosemiotic and Ecological Approach to Music Cognition: Event Perception Between Auditory Listening and Cognitive Economy.

This paper addresses the question whether we can conceive of music cognition in ecosemiotic terms. It claims that music knowledge must be generated as a tool for adaptation to the sonic world and calls forth a shift from a structural description of music as an artifact to a process-like approach to dealing with music. As listeners, we are observers who construct and organize our knowledge and bring with us our observational tools. What matters is not merely the sonic world in its objective qualities, but the world as perceived. In order to make these claims operational we can rely on the ecological concept of coping with the sonic world and the cybernetic concepts of artificial and adaptive devices. Listeners, on this view, are able to change their semantic relations with the sonic world through functional adaptations at the level of sensing, acting and coordinating between action and perception. This allows us to understand music in functional terms of what it affords to us and not merely in terms of its acoustic qualities. There are, however, degrees of freedom and constraints which shape the semiotization of the sonic world. As such we must consider the role of event perception and cognitive economy: listeners do not perceive the acoustical environment in terms of phenomenological descriptions but as ecological events