A graphical tool for the timbre space exploration and interactive
design of complex sounds by physical modeling synthesis
is presented. It is built around an auditory representation
of sounds based on spike functions and provides the
designer with both a graphical and an auditory insight. The
auditory representation of a number of reference sounds, located
as landmarks in a 2D sound design space, provides the
designer with an effective aid to direct his search along the
paths that lie in the proximity of the most inspiring landmarks

Adilo\u1e7lu, Kamil

Anni\ue9s, Robert

Delle Monache, Stefano

Drioli, Carlo

Obermayer, Klaus

Polotti, Pietro

Rocchesso, Davide

A graphical tool for the timbre space exploration and interactive
design of complex sounds by physical modeling synthesis
is presented. It is built around an auditory representation
of sounds based on spike functions and provides the
designer with both a graphical and an auditory insight. The
auditory representation of a number of reference sounds, located
as landmarks in a 2D sound design space, provides the
designer with an effective aid to direct his search along the
paths that lie in the proximity of the most inspiring landmarks

DRIOLI, Carlo

Adiloǧlu, Kamil

Anniés, Robert

Archivio istituzionale della ricerca - Università degli Studi di Udine

AUDITORY REPRESENTATIONS AS LANDMARKS IN THE SOUND DESIGNSPACECarlo DrioliPietro PolottiUniversity of Verona{carlo.drioli,pietro.polotti}@univr.itDavide Rocchesso,Stefano Delle MonacheIUAV University of Veniceroc@iuav.itstefano.dellemonache@gmail.comKamil Adilog˘lu,Robert, Annie´s,Klaus ObermayerBerlin Institute of Technology{kamil, robokopp, oby}@cs.tu-berlin.deABSTRACTA graphical tool for the timbre space exploration and inter-active design of complex sounds by physical modeling syn-thesis is presented. It is built around an auditory represen-tation of sounds based on spike functions and provides thedesigner with both a graphical and an auditory insight. Theauditory representation of a number of reference sounds, lo-cated as landmarks in a 2D sound design space, provides thedesigner with an effective aid to direct his search along thepaths that lie in the proximity of the most inspiring land-marks.1 INTRODUCTIONSound synthesis techniques are nowadays available whichoffer a high degree of naturalness and expressiveness. Some-times, however, this comes at a price of a consistent com-plexity in both the control of real time performance, andthe parametric tuning required in the sound design process.One such case is physical modeling, in which the processthat produces the sound is represented by means of moreor less simpliﬁed equations of the underlying physical laws.As a result, the parameters involved in the modeling shouldin principle be easy to understand because they have a realcounterpart, and should be easy to control because our sen-sorial experience should mediate the action-reaction patternsto which they relate[1]. Nonetheless, the most accurate andexpressive models available today are often described in termsof detailed physical relations and parameters that are notalways accessible or understandable to the non specialists.Moreover, the nonlinear nature of the phenomena under ob-servation may sometimes lead to numerical schemes that donot always reﬂect the behavior of the real systems in thewhole parameters space. These considerations motivate thesearch for new tools to aid the synthesis parametric tuning,SMC 2009, July 23-25, Porto, PortugalCopyrights remain with the authorsand are of particular interest in our opinion in the case ofphysical modeling audio synthesis.In this paper we propose a graphical tool for the timbrespace exploration and interactive design of complex soundsby physical modeling synthesis that is intended to assist thesound designer. It is aimed at exploiting the auditory repre-sentation of sounds based on spike functions and providesthe designer with both a graphical and an auditory insightthat may be used in place of, or combined with, the set oflow-level physical parameters of the models. The graphicaltool adopts a sonic landscape paradigm, in which the audi-tory representation of a number of reference sounds can belocated as landmarks in a 2D sound design space, and pro-vides an effective aid to direct the search along the paths thatlie in the proximity of the most inspiring sonic landmarks.The use of terminology and metaphors referring to theenvironment and landscapes has a rather old tradition in theﬁeld of sounds perception, especially when referred to theperception of ecological and everyday sounds. In the late70’s, the Canadian composer R. Murray Schafer introducedthe term “soundscape”, deﬁned as the auditory equivalentto landscape[2], and Barry Truax published his Handbookfor Acoustic Ecology[3]. The term soundscape perceptionis also used in a scientiﬁc context to characterize how inhab-itants perceive, experience and appraise their sonic environ-ment.Our use of the terms “sonic landscape” and “sonic land-mark” however refers to a particular organization of soundsin a 2D sonic space. The sound synthesis framework werely on is based on a class of physical models for everydaysounds, which includes low level events (impacts and fric-tion) and high level ones (bouncing, breaking, rolling, crum-pling). This Physically-based Sound Design Tools (SDTfrom now on) is being developed and supported within anumber of EU funded research projects on audio synthe-sis and sound design (Sounding Objetcs (SOB), Closing theLoop of Sound Evaluation and Design (CLOSED), Naturalinteractive walking (NIW))[4, 5]. An example of graphicalinterface for the parametric tuning of friction sounds, in-Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - PortugalPage 315cluded in the SDT and implemented in Max/Msp, is shownin Figure 1. The aim of the tool proposed here is to addto this class of tuning interfaces a perceptual representationoutput layer (spike functions) and a tool to perform paramet-ric interpolations using reference auditory representations inthe perceptual space. The representation of sounds based onthe spike functions provides the designer with both a graph-ical and an auditory insight. The latter point is of extremeinterest since it is based on a sort of ”auditory representa-tion of sound”. The peculiarity of the spike representationis, in fact, to be lossy. This aspect turns to be an advan-tage, whereas the reconstructed sounds maintain the tempo-ral articulation of the original ones but they are decoloredfrom their original timbre characteristics. Any reconstruc-tion of a sound has a “watery character”, so that all thesounds become timbrically similar, even if distinguishableaccording to their temporal structure. In this way, the sounddesigner has at disposal a kind of “auditory-radiography” ofthe sounds that can be compared as sonic archetypes, help-ing the designer in her/his sound space exploration.The paper is organized as follows: ﬁrst, a description ofthe sound synthesis engine and of the spike-based auditoryrepresentation is given in Section 2; Section 3 describes theinterpolation sound space and the graphical tool proposed toassist the sound designer; in Section 4, same sound designexamples obtained with the tool are illustrated; Section 5contains the conclusions and future issues.2 DESIGN AND IMPLEMENTATION OF THESPIKE-BASED PARAMETER INTERPOLATIONTOOLThe interactive interpolation tool presented is organized asa client-server distributed application, in which the clientside hosts the user graphical front-end based on the audi-tory spike sound representation, and the server side hosts theSDT audio synthesis engine. In the following we providesome details on the sound synthesis algorithm used to gen-erate the reference and the new sounds, on the spike analysisframework used to graphically represent the sounds, and theproperties of the interpolation in the 2D space.2.1 The sound synthesis engineThe sound synthesis chosen to illustrate our design tool isa physical modeling implementation of friction sounds syn-thesis, included in the aforementioned SDT package.The scope of the SDT is to provide a platform of soundsynthesis tools that interaction designers can easily exploitin their sketching activities and that can be run on commonreal time software such as Max/MSP and Pd. The aim isalso to provide the patches with a set of side tools to easilymanage projects and that can be of help when working withFigure 1. The friction MAX/Pd patch, contained in the SDTsynthesis tools package. The user can tune up to 27 synthe-sis parameters representing the driving forces, the character-istics of the modal resonators, and the characteristics of thenonlinear interaction.acquisition boards and sensors. Most of the models con-tained in the SDT, aimed at reproducing sounds from solidobjects interaction, are structured as two resonating objectsinteracting by means of a contact model. The friction modelspeciﬁcally referred to here, relies on a description of theaverage behavior of a multitude of micro-contacts made byhypothetical bristles extending from each of two sliding sur-faces. A modal decomposition is adopted for both interact-ing objects, leading to a ﬁrst parametric subset includingmodes frequency, decay factors and gain. The remaininglow-level parameters are related to the interaction mecha-nisms and to the interaction force speciﬁcation. To gain aninsight of the phenomenological role of the low-level phys-ical parameters of the friction model, and of what a sounddesigner can be asked to deal with, a description is given inTable 1, and are visible in the SDT tuning interface of Fig-ure 1 . Further details on the friction model can be found in[4], Chap. 8.Sym. Physical Description Phenomenological Descriptionσ0 bristle stiffness affects the evolution of mode lock-inσ1 bristle dissipation affects the sound bandwidthσ2 viscous friction affects the speed of timbre evolution and pitchσ3 noise coefﬁcient affects the perceived surface roughnessμd dynamic friction coeff. high values reduce the sound bandwidthμs static friction coeff. affects the smoothness of sound attackvs Stribeck velocity affects the smoothness of sound attackfN normal force high values give rougher and louder soundsTable 1. A phenomenological guide to the friction modelparameters.Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - PortugalPage 3162.2 Spike representationIn a classical signal representation approach, overlappingdiscrete blocks are used. However, in this method, in par-ticular for non-stationary signals, a small shift of a blockcan cause a large change in the representation, dependingon where an acoustic event falls within the block. A sparse,shiftable representation method based on atom-like ﬁlter func-tions can solve the problem. Hence, following [6], a soundsignal x(t) can be approximated as a linear combination ofK ﬁlter functions, the so-called spikes. In this study, we em-ployed the Gammatone functions γfk,tk(t) with amplitudesak and residual K+1(t):x(t) =K∑k=1akγfk,tk(t) + K+1(t). (1)A Gammatone function is deﬁned by its center frequencyfk and its ﬁlter order. In a Gammatone ﬁlter bank, the ﬁltercenter frequencies are distributed across the frequency axisin proportion to their bandwidth. The shape of the magni-tude characteristics of the fourth order Gammatone ﬁlter ap-proximates the magnitude characteristics of the human au-ditory ﬁlter in a proper way [7]. Hence, these ﬁlter functionshave a biological background.Each spike sk = (tk, fk, ak) is composed of the tempo-ral offset tk, the center frequency fk of the correspondingGammatone ﬁlter and the amplitude ak.For a pre-determined number of spikes K, the optimalcoefﬁcients ak, fk, tk, in terms of a minimal residual signal(t) have to be found. We employ matching pursuit [8, 6]to iteratively determine the spikes sk and to minimize theresidual. The algorithm is initialized by setting the residual1(t) := x(t). (2)Then for 1 ≤ k ≤ K, the optimal time offset tk and cen-ter frequency fk are determined iteratively so that the Gam-matone ﬁlter γfk,tk(t) maximally correlates with the signalk(t) :(fk, tk) = argmaxf,t∗ < k(t), γmf,t∗ (t) > . (3)In order to perform the scalar product shown in Equa-tion 3 between the ﬁlters and the signal, the ﬁlters shouldhave certain window lengths in time domain. In his audi-tory toolbox, Slaney [9] used ﬁxed window lengths for eachﬁlter in the ﬁlterbank. However, the energy levels decreasemuch slower for the low frequency ﬁlters than for the highfrequency ﬁlters. Therefore, we adapted the window lengthsconsidering the center frequencies. We calculated the posi-tions within each ﬁlter, where the total energy in the time en-velope falls to its thousandth. We used these position valuesin time as the window lengths to calculate the scalar prod-ucts. Hence, the amplitude of the kth spike is deﬁned to bethe scalar product between the corresponding ﬁlter, with itswindow length, and the residual signal, as follows:ak :=< k(t), γfk,tk(t) > . (4)Then we update the residual byk+1(t) = k(t)− akγfk,tk(t) (5)Finally, for k = K we yield Equation 1.By varying K within Equation 1, the SNR values of thespike code corresponding to the sparsity of the representa-tion can be changed. Increasing K increases the SNR of thespike code. Small K (high sparsity) decreases the SNR.Figure 2. For an impact sound, we display the sound waveand the spike code. This picture clearly indicates how thespike code captures the skeleton of the sound.Figure 2 shows the wave form and the spike code rep-resentation of an impact sound. In this example, a Gamma-tone ﬁlter bank of M = 256 ﬁlters is used for generating thespike code consisting of K = 32 spikes. Note that salientareas in the wave of this sound is coded with more spikesthan other areas. This indicates that the spike representationcaptures the signal characteristics properly.3 THE SONIC SPACE, SONIC LANDMARKS ANDGUIThe sonic landscape is organized as a 2D space in whichreference sounds (sonic landmarks) are located. The orga-nization of the reference sounds may rely on perceptual cri-teria, derived on a statistical basis (e.g., clustering), or maybe arbitrarily decided by the user.The sonic landmarks in the sonic space form a 2D scatterpoints set. The sound designer may chose to generate a newProceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - PortugalPage 317Figure 3. Scatter points interpolation: Delaunay triangula-tion (dashed lines) on a set of data points (circles), and fourinterpolated points (stars).sound in the vicinity of a set of sonic landmarks inspired bytheir sonic properties. Depending on the coordinates of thenew position indicated by the user, the new set of synthe-sis parameters is generated through an interpolation schemeaccounting for the neighboring sonic landmarks. Consider aset of landmark points{(x1, y1, z1), (x2, y2, z2), . . . , (xn, yn, zn)}where (xi, yi) is the position in the 2D space, and zi is thevalue of the synthesis parameters vector. A convenient in-terpolation scheme for this class of problems is based on thefollowing steps (for sake of simplicity, we consider a scalarparameter, z):• Compute a grid of triangles connecting the scatter pointstogether. One possibility, which is used here, is theDelaunay triangulation.• For a new interpolated point, ﬁnd the three vertices(xi1, yi1, zi1), (xi2, yi2, zi2) and (xi3, yi3, zi3) deﬁn-ing the triangle in which the point is conﬁned andcompute the coefﬁcients A,B,C,D for the equationof the plane deﬁned by the triangle Ax+By +Cz +D = 0.• Finally, compute the interpolated value through linearinterpolation: f(x, y) = −AC x− BC y − DCFigure 3 gives an example of such interpolation for asmall set of data points (circles) and a scalar z parameter.The interpolated data are depicted as stars.A graphical user front-end (GUI) was implemented inMatlab and is shown in Figures 4 and 5. Its main framerepresents the 2D sonic space with the sonic landmarks lo-cated in pre-deﬁned spots (the ﬁgures show a conﬁgurationsin which the landmarks were disposed as vertices of a rect-angle). The location of the reference sounds can be arbitrar-ily decided and is speciﬁed in a conﬁguration ﬁle loaded adthe beginning of each session. Once the landmarks soundshave been loaded and the auditory representation computed,the user can perform a number of actions, including:• Creating a new sound by choosing with the mouse aposition in the proximity of the sonic landmarks whitdesirable characteristics• Listening to landmarks and new sounds by clickingon the spike plot• Deleting newly generated sounds which are not of in-terest for the user• Saving the parametric setting of a new sound as pre-sets in the XML-based format used by the SDT GUI• Checking network connectivity with the Max/Msp orPd server running the SDT sound synthesis engineFigure 4. Matlab GUI. Spike representation of four sounds(sonic landmarks) representative of the entire sound spacegenerated by means of the SDT friction model. By click-ing in the graphical area around a spike representation it ispossible to listen to the corresponding sound (see the areahighlighted by circle 1). Circle 2 shows the control panelfor the communication with the SDT model in Max/MSPThe spike-based GUI is connected through an OSC net-work layer to a Max/Msp or Pd server running the SDTsound synthesis engine. When the user conﬁrms the positionof a new sound to be generated, the GUI starts the communi-cation with the SDT server, proceeding through the follow-ing steps: ﬁrst, the new interpolated parameters are sent tothe server, which updates the controller values; when done,the SDT synthesis engine is started and an auidio ﬁle is gen-erated; ﬁnally, the GUI loads the audio ﬁle, generates theProceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - PortugalPage 318new auditory representation, and plot the result in the 2Dsonic space.Figure 5. Creation of new sounds. By clicking in any partof the Matlab GUI it is possible to start the computation of anew sound, whose SDT synthesis parameters are calculatedaccording to the graphical position with respect to the foursonic landmarks (see circle 1). The interpolation of the pa-rameters of the four sonic landmarks for the creation of anew sound is linear. The new parameter set is stored as anSDT preset in xml format and acquired by the SDT modelin Max/MSP. Circle 2 highlights the Spike representation ofa new sound ready generated to be played in Max/MSP.4 INTERPOLATION EXAMPLESThis section presents the result of a sound design experi-ment in which six reference sounds (the sonic landmarks)where organized in the 2D space as shown in Figure 6 (thespike representation of the reference sounds, and the De-launay triangulation generated with this conﬁguration, areshown). Six new sounds were generated by choosing sixinterpolation positions (depicted as red stars). The synthe-sis sounds resulting from the interpolated parameters, con-verted into the spike-based representation, are representedin Figure 7.Perceptually, the results are in good agreement with theuser’s expectations, and the dynamical and spectral charac-teristics are recognized as actually deriving from the char-acteristics of the neighboring landmarks. The audio ﬁlescorresponding to the sound landmarks and to the interpo-lated sounds are available for download at the link providedin the footnote 1 .1 http://mordente.sci.univr.it/∼carlodrioli/SoMuCo09/Experiments.htmFigure 7. Interpolation example: new sounds ( a), b) and c) )are generated by interpolation of points in the same triangle;the new sounds d), e) and f) are located in three differenttriangles5 CONCLUSIONSThese results show that the visualization platform helps thesound designer to easily navigate through the soundscapeand to ﬁnd the desired sound fastly. In a following step, thisvisualization platform will be extended to perform a similarnavigation within the soundscape towards a pre-deﬁned (eg.recorded) sound with certain characteristics automatically.A probabilistic method will be deﬁned to model the distri-bution of the input parameters of the sound model. Giventhe pre-deﬁned sound, this method will automatically ﬁndthe optimal input parameters for the sound model, whichwill produce the closest possible sound to the given sound,in terms of a spike distance. Hence, a distance measure willbe deﬁned to measure the distance between the spike codeof the pre-deﬁned sound and the spike code of the currentsound produced by the sound model.6 ACKNOWLEDGEMENTSThis work is part of the research carried out within two EUfunded project: CLOSED (Closing the Loop of Sound Eval-uation and Design) and NIW (Natural Interactive Walking).Proceedings of the SMC 2009 - 6th Sound and Music Computing Conference, 23-25 July 2009, Porto - PortugalPage 319

Auditory representations as landmarks in the sound design space

https://air.uniud.it/bitstream/11390/1103910/1/SMC09paper.pdf

Auditory representations as landmarks in the sound design space

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università degli Studi di Udine