Using music and motion analysis to construct 3D animations and visualisations

This paper presents a study into music analysis, motion analysis and the integration of music and motion to form creative natural human motion in a virtual environment. Motion capture data is extracted to generate a motion library, this places the digital motion model at a fixed posture. The first step in this process is to configure the motion path curve for the database and calculate the possibility that two motions were sequential through the use of a computational algorithm. Every motion is then analysed for the next possible smooth movement to connect to, and at the same time, an interpolation method is used to create the transitions between motions to enable the digital motion models to move fluently. Lastly, a searching algorithm sifts for possible successive motions from the motion path curve according to the music tempo. It was concluded that the higher ratio of rescaling a transition, the lower the degree of natural motio

Holophonor: On the Future Technology of Visual Music

This chapter discusses the progression of visual music and related audio-visual artworks through the 20th Century and considers the next steps for this field of research. The principles of visual music are described, with reference to the films of early pioneers such as John Whitney. A further exploration of the wider spectrum of subsequent work in various audio-visual art forms is then given. These include visualisations, light synthesizers, VJ performances, digital audio-visual artworks, projection mapping artworks, and interactive visual music artworks. Through consideration of visual music as a continuum of related work, the authors consider the Holophonor, a fictional audio-visual instrument, as an example of the ideal visual music instrument of the future. They conclude by proposing that a device such as the Holophonor could be constructed in the near future by utilising inter-disciplinary approaches from the fields of HCI and affective computing

Designing an augmented reality exhibition: Leonardo's Impossible Machines

This paper discusses the origins, development and results of the animated and augmented reality aspects of the exhibition ‘Leonardo’s Impossible Machines’ that was developed at Ravensbourne University London and Birkbeck, University of London, with support from the Museo Galileo. The exhibition included novel reconstructions and visualisations of Leonardo’s perpetual motion machines from the Codex Forster, and the process is explained here, along with the challenges of mounting a combined physical and AR show

System upgrade: realising the vision for UK education

A report summarising the findings of the TEL programme in the wider context of technology-enhanced learning and offering recommendations for future strategy in the area was launched on 13th June at the House of Lords to a group of policymakers, technologists and practitioners chaired by Lord Knight. The report – a major outcome of the programme – is written by TEL director Professor Richard Noss and a team of experts in various fields of technology-enhanced learning. The report features the programme’s 12 recommendations for using technology-enhanced learning to upgrade UK education

A Compositional Exploration of Auditory-Visual Synaesthesia

This thesis is an autoethnographic exploration of how my music composition practice is influenced by my auditory-visual synaesthesia. I perceive music as coloured and textured shapes – ‘photisms’ – in my mind’s eye. There are few first-person accounts of synaesthesia’s impact on music composition processes and outputs. The aim of the study is to investigate how my synaesthetic experiences affect the music I make, through the creation and analysis of an album of electronic music titled Photisms. The accompanying textual document employs an autoethnographic method proposed by Chang (2008), providing a first-person account of my own synaesthesia and analysing how it influences my composition processes and outputs. Firstly, I consider the impact timbre has on the shape and colour of my photisms. I discuss how various timbral transformation techniques can result in photism multiplication and emergence, and composite photisms (terms I have coined to define previously uninvestigated phenomena). Secondly, I explain how my photisms also appear to have texture and weight, and materialise as solid, liquid, gaseous, or plasma substances. The perceived weight and state of matter of musically-induced photisms is a formerly unexplored topic in academic literature. Thirdly, the spatiality and temporality of my synaesthetic visualisations of music are examined. I explain how the perceived location of my photisms can be defined using the Cartesian coordinate system, and how tempo and speed manipulation affect my synaesthetic experiences. Finally, I explain how, through a process of audiation, I can translate visual images into music by reverse-engineering my synaesthesia. This innovative approach to music-making has not previously been investigated. Through examination of these four areas, this study demonstrates how auditoryvisual synaesthetic experiences can affect, and be exploited in, the composition of electronic music

Digital Fabrication Approaches for the Design and Development of Shape-Changing Displays

Interactive shape-changing displays enable dynamic representations of data and information through physically reconfigurable geometry. The actuated physical deformations of these displays can be utilised in a wide range of new application areas, such as dynamic landscape and topographical modelling, architectural design, physical telepresence and object manipulation. Traditionally, shape-changing displays have a high development cost in mechanical complexity, technical skills and time/finances required for fabrication. There is still a limited number of robust shape-changing displays that go beyond one-off prototypes. Specifically, there is limited focus on low-cost/accessible design and development approaches involving digital fabrication (e.g. 3D printing). To address this challenge, this thesis presents accessible digital fabrication approaches that support the development of shape-changing displays with a range of application examples – such as physical terrain modelling and interior design artefacts. Both laser cutting and 3D printing methods have been explored to ensure generalisability and accessibility for a range of potential users. The first design-led content generation explorations show that novice users, from the general public, can successfully design and present their own application ideas using the physical animation features of the display. By engaging with domain experts in designing shape-changing content to represent data specific to their work domains the thesis was able to demonstrate the utility of shape-changing displays beyond novel systems and describe practical use-case scenarios and applications through rapid prototyping methods. This thesis then demonstrates new ways of designing and building shape-changing displays that goes beyond current implementation examples available (e.g. pin arrays and continuous surface shape-changing displays). To achieve this, the thesis demonstrates how laser cutting and 3D printing can be utilised to rapidly fabricate deformable surfaces for shape-changing displays with embedded electronics. This thesis is concluded with a discussion of research implications and future direction for this work

A perceptually motivated approach to timbre representation and visualisation.

Musical timbre is a complex phenomenon and is often understood in relation to the separation and comparison of different sound categories. The representation of musical timbre has traditionally consisted of instrumentation category (e.g. violin, piano) and articulation technique (e.g. pizzicato, staccato). Electroacoustic music places more emphasis on timbre variation as musical structure, and has highlighted the need for better, more in-depth forms of representation of musical timbre. Similarly, research from experimental psychology and audio signal analysis has deepened our understanding of the perception, description, and measurement of musical timbre, suggesting the possibility of more exact forms of representation that directly reference low-level descriptors of the audio signal (rather than high-level categories of sound or instrumentation). Research into the perception of timbre has shown that ratings of similarity between sounds can be used to arrange sounds in an N-dimensional perceptual timbre space, where each dimension relates to a particular axis of differentiation between sounds. Similarly, research into the description of timbre has shown that verbal descriptors can often be clustered into a number of categories, resulting in an N-dimensional semantic timbre space. Importantly, these semantic descriptors are often physical, material, and textural in nature. Audio signal processing techniques can be used to extract numeric descriptors of the spectral and dynamic content of an audio signal. Research has suggested correlations between these audio descriptors and different semantic descriptors and perceptual dimensions in perceptual timbre spaces. This thesis aims to develop a perceptually motivated approach to timbre representation by making use of correlations between semantic and acoustic descriptors of timbre. User studies are discussed that explored participant preferences for different visual mappings of acoustic timbre features. The results of these studies, together with results from existing research, have been used in the design and development of novel systems for timbre representation. These systems were developed both in the context of digital interfaces for sound design and music production, and in the context of real-time performance and generative audio-reactive visualisation. A generalised approach to perceptual timbre representation is presented and discussed with reference to the experimentation and resulting systems. The use of semantic visual mappings for low-level audio descriptors in the representation of timbre suggests that timbre would be better defined with reference to individual audio features and their variation over time. The experimental user studies and research-led development have highlighted specific techniques and audio-visual mappings that would be very useful to practitioners and researchers in the area of audio analysis and representation