Narrative music: towards an understanding of musical narrative functions in multimedia

As the computer screen is replacing the book as the dominant medium for communication (Kress, 2003), questions about how meaning is constituted by the multimodal interaction of different media (including music) is becoming increasingly important in contemporary research of pedagogy, sociology and media studies. The overall aim with this licentiate thesis is to explore musical narrative functions as they appear in multimedia such as film and computer games. The thesis is based on three publications. Publication 1 proposes a classification of musical narrative functions, with 6 narrative classes(the Emotive, Informative, Descriptive, Guiding, Temporal and Rhetorical classes) and 11 categories. The relational interplay of music with contextual factors is emphasized. Publication 2 describes the design of a software tool, REMUPP (Relations Between Musical Parameters and Perceived Properties), to be used for experimental studies of musical expression. REMUPP is used for real time alteration of musical expression, by the manipulation of musical parameters such as tempo, harmony, rhythm, articulation, etc. Publication 3 describes a quasi-experiment using REMUPP, where a group of young participants (12-13 years old) were given the task of adapting musical expression – by manipulating 7 parameters – to make it fit 3 visual scenes shown on a computer screen. They also answered a questionnaire asking about their musical backgrounds and habits of listening to music, watching movies and playing computer games. Numerical data from the manipulations were analyzed statistically with regards to the preferred values of the musical parameters in relation to the different visual scenes. The results indicated awareness and knowledge about codes and conventions of musical narrative functions, and were to some degree affected by the participants’ gender, musical backgrounds and media habits

Understanding Music Interaction, and Why It Matters

This is the introductory chapter of a book dedicated to new research in, and emerging new understandings of, music and human-computer interaction—known for short as music interaction. Music interaction research plays a key role in innovative approaches to diverse musical activities, including performance, composition, education, analysis, production and collaborative music making. Music interaction is pivotal in new research directions in a range of activities, including audience participation, interaction between music and dancers, tools for algorithmic music, music video games, audio games, turntablism and live coding. More generally, music provides a powerful source of challenges and new ideas for human-computer interaction (HCI). This introductory chapter reviews the relationship between music and human-computer interaction and outlines research themes and issues that emerge from the collected work of researchers and practitioners in this book

BitBox!:A case study interface for teaching real-time adaptive music composition for video games

Real-time adaptive music is now well-established as a popular medium, largely through its use in video game soundtracks. Commercial packages, such as fmod, make freely available the underlying technical methods for use in educational contexts, making adaptive music technologies accessible to students. Writing adaptive music, however, presents a significant learning challenge, not least because it requires a different mode of thought, and tutor and learner may have few mutual points of connection in discovering and understanding the musical drivers, relationships and structures in these works. This article discusses the creation of ‘BitBox!’, a gestural music interface designed to deconstruct and explain the component elements of adaptive composition through interactive play. The interface was displayed at the Dare Protoplay games exposition in Dundee in August 2014. The initial proof-of- concept study proved successful, suggesting possible refinements in design and a broader range of applications

Sound for Fantasy and Freedom

Sound is an integral part of our everyday lives. Sound tells us about physical events in the environ- ment, and we use our voices to share ideas and emotions through sound. When navigating the world on a day-to-day basis, most of us use a balanced mix of stimuli from our eyes, ears and other senses to get along. We do this totally naturally and without effort. In the design of computer game experiences, traditionally, most attention has been given to vision rather than the balanced mix of stimuli from our eyes, ears and other senses most of us use to navigate the world on a day to day basis. The risk is that this emphasis neglects types of interaction with the game needed to create an immersive experience. This chapter summarizes the relationship between sound properties, GameFlow and immersive experience and discusses two projects in which Interactive Institute, Sonic Studio has balanced perceptual stimuli and game mechanics to inspire and create new game concepts that liberate users and their imagination

DigiWall - an audio mostly game

DigiWall is a hybrid between a climbing wall and a computer game. The climbing grips are equipped with touch sensors and lights. The interface has no computer screen. Instead sound and music are principle drivers of DigiWall interaction models. The gaming experience combines sound and music with physical movement and the sparse visuals of the climbing grips. The DigiWall soundscape carries both verbal and nonverbal information. Verbal information includes instructions on how to play a game, scores, level numbers etc. Non-verbal information is about speed, position, direction, events etc. Many different types of interaction models are possible: competitions, collaboration exercises and aesthetic experiences

Experimental Approaches to the Composition of Interactive Video Game Music

This project explores experimental approaches and strategies to the composition of interactive music for the medium of video games. Whilst music in video games has not enjoyed the technological progress that other aspects of the software have received, budgets expand and incomes from releases grow. Music is now arguably less interactive than it was in the 1990’s, and whilst graphics occupy large amounts of resources and development time, audio does not garner the same attention. This portfolio develops strategies and audio engines, creating music using the techniques of aleatoric composition, real-time remixing of existing work, and generative synthesisers. The project created music for three ‘open-form’ games : an example of the racing genre (Kart Racing Pro); an arena-based first-person shooter (Counter-Strike : Source); and a real-time strategy title (0 A.D.). These games represent a cross-section of ‘sandbox’- type games on the market, as well as all being examples of games with open-ended or open-source code

Whole body interaction in abstract domains

Whole Body Interaction appears to be a good fit of interaction style for some categories of application domain, such as the motion capture of gestures for computer games and virtual physical sports. However, the suitability of whole body interaction for more abstract application domains is less apparent, and the creation of appropriate whole body interaction designs for complex abstract areas such as mathematics, programming and musical harmony remains challenging. We argue, illustrated by a detailed case study, that conceptual metaphor theory and sensory motor contingency theory offer analytic and synthetic tools whereby whole body interaction can in principle be applied usefully to arbitrary abstract application domains. We present the case study of a whole body interaction system for a highly abstract application area, tonal harmony in music. We demonstrate ways in which whole body interaction offers strong affordances for action and insight in this domain when appropriate conceptual metaphors are harnessed in the design. We outline how this approach can be applied to abstract domains in general, and discuss its limitations.Published versio

An investigation of the role of background music in IVWs for learning

Empirical evidence is needed to corroborate the intuitions of gamers and game developers in understanding the benefits of Immersive Virtual Worlds (IVWs) as a learning environment and the role that music plays within these environments. We report an investigation to determine if background music of the genre typically found in computer‐based role‐playing games has an effect on learning in a computer‐animated history lesson about the Macquarie Lighthouse within an IVW. In Experiment 1, music stimuli were created from four different computer game soundtracks. Seventy‐two undergraduate students watched the presentation and completed a survey including biographical details, questions on the historical material presented and questions relating to their perceived level of immersion. While the tempo and pitch of the music was unrelated to learning, music conditions resulted in a higher number of accurately remembered facts than the no music condition. One soundtrack showed a statistically significant improvement in memorisation of facts over other music conditions. Also an interaction between the levels of perceived immersion and ability to accurately remember facts was observed. Experiment 2, involving 48 undergraduate students, further investigated the effect of music, sense of immersion and how different display systems affect memory for facts

Multi-touch interaction principles for collaborative real-time music activities: towards a pattern language

In this paper we give an analysis of the literature on a set of problems that can arise when undertaking the interaction design of multi-touch applications for collaborative real-time music activities, which are designed for multitouch technologies (e.g. smartphones, tablets, interactive tabletops, among others). Each problem is described, and a candidate design pattern (CDP) is suggested in the form of a short sentence and a diagram—an approach inspired by Christopher Alexander’s A Pattern Language. These solutions relate to the fundamental collaborative principles of democratic relationships, identities and collective interplay. We believe that this approach might disseminate forms of best design practice for collaborative music applications, in order to produce real-time musical systems which are collaborative and expressive