It's not what you play, it's how you play it: timbre affects perception of emotion in music.

Salient sensory experiences often have a strong emotional tone, but the neuropsychological relations between perceptual characteristics of sensory objects and the affective information they convey remain poorly defined. Here we addressed the relationship between sound identity and emotional information using music. In two experiments, we investigated whether perception of emotions is influenced by altering the musical instrument on which the music is played, independently of other musical features. In the first experiment, 40 novel melodies each representing one of four emotions (happiness, sadness, fear, or anger) were each recorded on four different instruments (an electronic synthesizer, a piano, a violin, and a trumpet), controlling for melody, tempo, and loudness between instruments. Healthy participants (23 young adults aged 18-30 years, 24 older adults aged 58-75 years) were asked to select which emotion they thought each musical stimulus represented in a four-alternative forced-choice task. Using a generalized linear mixed model we found a significant interaction between instrument and emotion judgement with a similar pattern in young and older adults (p < .0001 for each age group). The effect was not attributable to musical expertise. In the second experiment using the same melodies and experimental design, the interaction between timbre and perceived emotion was replicated (p < .05) in another group of young adults for novel synthetic timbres designed to incorporate timbral cues to particular emotions. Our findings show that timbre (instrument identity) independently affects the perception of emotions in music after controlling for other acoustic, cognitive, and performance factors

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance

PerformanceNet: Score-to-Audio Music Generation with Multi-Band Convolutional Residual Network

Music creation is typically composed of two parts: composing the musical score, and then performing the score with instruments to make sounds. While recent work has made much progress in automatic music generation in the symbolic domain, few attempts have been made to build an AI model that can render realistic music audio from musical scores. Directly synthesizing audio with sound sample libraries often leads to mechanical and deadpan results, since musical scores do not contain performance-level information, such as subtle changes in timing and dynamics. Moreover, while the task may sound like a text-to-speech synthesis problem, there are fundamental differences since music audio has rich polyphonic sounds. To build such an AI performer, we propose in this paper a deep convolutional model that learns in an end-to-end manner the score-to-audio mapping between a symbolic representation of music called the piano rolls and an audio representation of music called the spectrograms. The model consists of two subnets: the ContourNet, which uses a U-Net structure to learn the correspondence between piano rolls and spectrograms and to give an initial result; and the TextureNet, which further uses a multi-band residual network to refine the result by adding the spectral texture of overtones and timbre. We train the model to generate music clips of the violin, cello, and flute, with a dataset of moderate size. We also present the result of a user study that shows our model achieves higher mean opinion score (MOS) in naturalness and emotional expressivity than a WaveNet-based model and two commercial sound libraries. We open our source code at https://github.com/bwang514/PerformanceNetComment: 8 pages, 6 figures, AAAI 2019 camera-ready versio

Why We Experience Musical Emotions: William Gardiner’s “The Music of Nature” Revisited

This paper focuses and expands on the ideas of William Gardiner, an amateur musician who was the first to propose that human emotions experienced in music listening might be inspired by “the sounds of nature.” His book has been ignored for almost two centuries. We revisit his hypothesis from an evolutionary psychology approach. This contribution reviews environmental psychology and musical studies which focus on emotional reactions to basic musical cues such as pitch, timbre, and loudness, and also, on animal communication studies. Reported literature confirms the hypothesis that our ancestral soundscape might have shaped, at least in part, the basic emotions that organize our behavioral responses to environmental stimuli

Virtual orchestration: a film composer's creative practice

The advent of digital technologies has led to a major change in the process of film music composition; consequent developments in music technology have forced film composers to adapt to this change. Technological innovations such as digital audio workstations (DAWs) and virtual musical instruments have made possible the creation of virtual orchestras that are technologically capable of simulating the sound and behaviour of a traditional acoustic orchestra. This has had an effect on film music production and on the creative process of the professional film composer in a way that today, creating orchestral simulations or 'mock-ups' that imitate live orchestras (or smaller ensembles) has become a requirement in the film industry and thus an essential part of the film-scoring process. In the context of contemporary film music production, this thesis investigates how orchestral simulations are composed and created using computer music technology and virtual sample-based instruments. In asking 'how', the focus is on the film composer's activities and thought processes during this creative cycle, along with the nature of the interactive relationship between composer and music materials. This study aims to show the complexity of the film composer's creative practice and to advance understanding of how the use of computer music technology and orchestral sample libraries is influencing the compositional process and compositional outcome. To address these questions, a qualitative multiple case study methodology approach was chosen that included examination of the practice of seven professional film composers working in the field of feature film as the primary valid source of data. The exploration involved semi-structured interviews with composers, observations and analysis of their studio practice and inspection of their compositional tools. Taken as a whole, the evidence provided by this study is that the process of creating orchestral simulations is a process of film music composition during which professional film composers are creating orchestral sounds through the use of computers, digital sequencing, samplers and sample-based virtual acoustic instruments for the realisation of musical works. It is a process of using and manipulating recorded samples of real acoustic instruments to generate an expressive and convincing musical performance through sample-based orchestral simulation. A characteristic of this compositional practice is that it is a continuous process that proceeds in stages over time where all procedures can be applied repeatedly between stages. The process of creating orchestral simulations for the purpose of the film score is a multifaceted compositional activity involving a complex set of relationships among different compositional states of mind and compositional activities in which film composers experience music and interact with musical materials and media in various ways. This creative activity is a process involving a single person and a mixture of various compositional tools, the composer's skills and abilities brought into existence through a creative process that requires a thorough blend of art and craft to be demonstrated at all times

Getting Under Your Skin Until You Jump Out of It: The Psychological Effects of Music on The Experience of Film

Music is like magic. It can sweep you off your feet and spirit you away to places you never thought possible: it can serve as a teleportation device, achieve time travel, and let us read minds. Some pieces of music exist for their own sake, like Rachmaninoff’s Isle of the Dead, while others accompany different forms of media: ballets such as The Nutcracker and operas like La Bohème are instantly recognizable for their grandiose and immersive scores. For a moment in time, audiences can really believe that they are traveling to a magical world with Clara, and even without the stage one can see in their mind’s eye a looming and grave island of mortality… and it’s thanks to the music. This paper looks to examine the influential effects of music from a psychological perspective through the lens of film. Looking at three classic horror movies, Psycho, Halloween, and Scream, I aim to illustrate how music plays with our expectations to influence our perceptions of the screen and beyond

Timbre in the Musical Performance as a Result of Audio-Mental Operations

The aim of this paper is to reveal the relationship between the timbre i e sound quality obtained by an instrumentalist when playing a music piece and the role of the hearing listening and other musical specific mental operations that take place in the brain of the instrumentalist during the performance with the focus on the classical music which depends in the most cases on the score The timbre is a characteristic of every instrument or voice that makes their tone unique It is given by the different sound components partials with different frequencies and amplitudes The number and the amplitude of the partials are different from instrument to instrument they depend on the construction of the instrument and on the art of playing the last being the point in this article It is known that timbre has an emotional impact on the perception Its semantic features are represented through descriptors as dark bright round dull dry harsh etc that have to be created by the performers in order to affect the auditory Thus they have to possess a good technique able to get different timbres Although the technique has to be subordinated to the capacity of hearing in advance the sound with all its features pitch duration intensity and timbre The excessive attention of the performers from the pitch and duration that are exactly notated in the score has to be turned to the intensity and timbre that are less notated Therefore the audio-mental operations the term belongs to piano teachers Ana Piti and Ioana Minei both of the performers and listeners as part of music cognition are discussed here The decoding of the music information from the score and the sending of it to the auditory together with the regarding of the music as a language were be also approache