DDSP-Piano: A Neural Sound Synthesizer Informed by Instrument Knowledge

Instrument sound synthesis using deep neural networks has received numerous improvements over the last couple of years. Among them, the Differentiable Digital Signal Processing (DDSP) framework has modernized the spectral modeling paradigm by including signal-based synthesizers and effects into fully differentiable architectures. The present work extends the applications of DDSP to the task of polyphonic sound synthesis, with the proposal of a differentiable piano synthesizer conditioned on MIDI inputs. The model architecture is motivated by high-level acoustic modeling knowledge of the instrument, which, along with the sound structure priors inherent to the DDSP components, makes for a lightweight, interpretable, and realistic-sounding piano model. A subjective listening test has revealed that the proposed approach achieves better sound quality than a state-of-the-art neural-based piano synthesizer, but physical-modeling-based models still hold the best quality. Leveraging its interpretability and modularity, a qualitative analysis of the model behavior was also conducted: it highlights where additional modeling knowledge and optimization procedures could be inserted in order to improve the synthesis quality and the manipulation of sound properties. Eventually, the proposed differentiable synthesizer can be further used with other deep learning models for alternative musical tasks handling polyphonic audio and symbolic data

Model-based digital pianos: from physics to sound synthesis

International audiencePiano is arguably one of the most important instruments in Western music due to its complexity and versatility. The size, weight, and price of grand pianos, and the relatively simple control surface (keyboard) have lead to the development of digital counterparts aiming to mimic the sound of the acoustic piano as closely as possible. While most commercial digital pianos are based on sample playback, it is also possible to reproduce the sound of the piano by modeling the physics of the instrument. The process of physical modeling starts with first understanding the physical principles, then creating accurate numerical models, and finally finding numerically optimized signal processing models that allow sound synthesis in real time by neglecting inaudible phenomena, and adding some perceptually important features by signal processing tricks. Accurate numerical models can be used by physicists and engineers to understand the functioning of the instrument, or to help piano makers in instrument development. On the other hand, efficient real-time models are aimed at composers and musicians performing at home or at stage. This paper will overview physics-based piano synthesis starting from the computationally heavy, physically accurate approaches and then discusses the ones that are aimed at best possible sound quality in real-time synthesis

Action-based effects on music perception

The classical, disembodied approach to music cognition conceptualizes action and perception as separate, peripheral processes. In contrast, embodied accounts of music cognition emphasize the central role of the close coupling of action and perception. It is a commonly established fact that perception spurs action tendencies. We present a theoretical framework that captures the ways in which the human motor system and its actions can reciprocally influence the perception of music. The cornerstone of this framework is the common coding theory, postulating a representational overlap in the brain between the planning, the execution, and the perception of movement. The integration of action and perception in so-called internal models is explained as a result of associative learning processes. Characteristic of internal models is that they allow intended or perceived sensory states to be transferred into corresponding motor commands (inverse modeling), and vice versa, to predict the sensory outcomes of planned actions (forward modeling). Embodied accounts typically refer to inverse modeling to explain action effects on music perception (Leman, 2007). We extend this account by pinpointing forward modeling as an alternative mechanism by which action can modulate perception. We provide an extensive overview of recent empirical evidence in support of this idea. Additionally, we demonstrate that motor dysfunctions can cause perceptual disabilities, supporting the main idea of the paper that the human motor system plays a functional role in auditory perception. The finding that music perception is shaped by the human motor system and its actions suggests that the musical mind is highly embodied. However, we advocate for a more radical approach to embodied (music) cognition in the sense that it needs to be considered as a dynamical process, in which aspects of action, perception, introspection, and social interaction are of crucial importance