Seeing Sounds, Hearing Shapes: a gamified study to evaluate sound-sketches

Sound-shape associations, a subset of cross-modal associations between the auditory and visual domain, have been studied mainly in the context of matching a set of purposefully crafted shapes to sounds. Recent studies have explored how humans represent sound through free-form sketching and how a graphical sketch input could be used for sound production. In this paper, the potential of communicating sound characteristics through these free-form sketches is investigated in a gamified study that was conducted with eighty-two participants at two online exhibition events. The results show that participants managed to recognise sounds at a higher rate than the random baseline would suggest, however it appeared difficult to visually encode nuanced timbral differences

Sketching sounds: an exploratory study on sound-shape associations

Sound synthesiser controls typically correspond to technical parameters of signal processing algorithms rather than intuitive sound descriptors that relate to human perception of sound. This makes it difficult to realise sound ideas in a straightforward way. Cross-modal mappings, for example between gestures and sound, have been suggested as a more intuitive control mechanism. A large body of research shows consistency in human associations between sounds and shapes. However, the use of drawings to drive sound synthesis has not been explored to its full extent. This pa- per presents an exploratory study that asked participants to sketch visual imagery of sounds with a monochromatic digital drawing interface, with the aim to identify different representational approaches and determine whether timbral sound characteristics can be communicated reliably through visual sketches. Results imply that the development of a synthesiser exploiting sound-shape associations is feasible, but a larger and more focused dataset is needed in followup studies

Hands where we can see them! Investigating the impact of gesture size on audience perception

This paper explores the relative effect of gesture size on audience perception of digital musical instrument (DMI) performance. In a study involving a total audience of 28 people (split into 2 groups of 13 and 15), we used a small and large version of a DMI to examine how the size of performers' gestures might differ, and how this affects post-hoc audience ratings of enjoyment, interest and understanding, as well as their indications of `enjoyment' and `error' in real time. For each audience we held two 5-minute performances, the first on a custom-designed percussion DMI, and the second on a laptop. The DMI used in each performance was made up of three elements identical in shape, materiality, interaction and sound, but the physical size was different: For one each element was approx 12x10x5cm, and the other was about 3.5 times bigger (approx. 40x30x20cm). Data was collected both during and after the performance via post-hoc and real-time methods. We found that beyond a performance simply involving physical gesture, the size of gesture has an impact on audience ratings. In this paper we detail this study and its results, and present the implications that this finding has for DMI design

A new approach for constraint programming in music using relation domains

Constraint programming (CP) has been used for several decades in music composition and analysis. It has served as the underlying technology of different tools that allow composers to compute with musical abstractions (e.g.,notes, scores). However, the traditional domains used in musical CP, namely finite domains (integers) and finite sets (integer sets), are not well suited to represent and express properties on structured information such as a score in a compact and efficient way. This paper introduces a new domain for musical CP, namely relations, where a relation is a set of integer n-tuples. It proposes new constraints on relations and shows how to use them for musical composition. A single relation variable can represent a score of any size and any transformation between scores. The result is a system that directly supports computing with musical abstractions at a high abstraction level more pleasant to composers. The relation domain and its constraints are implemented using Binary Decision Diagrams and are provided as a library in the Gecode platform