Timbre space is a cognitive model useful to address the problem of structuring timbre in electronic music. The recent concept of corpus-based concatenative sound synthesis is proposed as an approach to timbral control in both real- and deferred-time applications. Using CataRT and related tools in the FTM and Gabor libraries for Max/MSP we describe a technique for real-time analysis of a live signal to pilot corpus-based synthesis, along with examples of compositional realizations in works for instruments, electronics, and sound installation. To extend this technique to computer-assisted composition for acoustic instruments, we develop tools using the Sound Description Interchange Format (SDIF) to export sonic descriptors to OpenMusic where they may be further manipulated and transcribed into an instrumental score. This presents a flexible technique for the compositional organization of noise-based instrumental sounds

Bresson, J.

Einbond, A.

Schwarz, D.

English

cote interne IRCAM: Einbond09aNone / NoneNational audienceTimbre space is a cognitive model useful to address the problem of structuring timbre in electronic music. The recent concept of corpus-based concatenative sound synthesis is proposed as an approach to timbral control in both real- and deferred-time applications. Using CataRT and related tools in the FTM and Gabor libraries for Max/MSP we describe a technique for real-time analysis of a live signal to pilot corpus-based synthesis, along with examples of compositional realizations in works for instruments, electronics, and sound installation. To extend this technique to computer-assisted composition for acoustic instruments, we develop tools using the Sound Description Interchange Format (SDIF) to export sonic descriptors to OpenMusic where they may be further manipulated and transcribed into an instrumental score. This presents a flexible technique for the compositional organization of noise-based instrumental sounds

Einbond, Aaron

Schwarz, Diemo

Bresson, Jean

Name not available

Corpus-Based Transcription as an Approach to the Compositional Control of Timbre

University of Huddersfield Repository

University of Huddersfield RepositoryEinbond, A., Schwarz, D. and Bresson, J.Corpus-Based Transcription as an Approach to the Compositional Control of TimbreOriginal CitationEinbond, A., Schwarz, D. and Bresson, J. (2009) Corpus-Based Transcription as an Approach to the Compositional Control of Timbre. Proceedings of the International Computer Music Conference (ICMC). pp. 223-226.This version is available at http://eprints.hud.ac.uk/15447/The University Repository is a digital collection of the research output of theUniversity, available on Open Access. Copyright and Moral Rights for the itemson this site are retained by the individual author and/or other copyright owners.Users may access full items free of charge; copies of full text items generallycan be reproduced, displayed or performed and given to third parties in anyformat or medium for personal research or study, educational or not-for-profitpurposes without prior permission or charge, provided:• The authors, title and full bibliographic details is credited in any copy;• A hyperlink and/or URL is included for the original metadata page; and• The content is not changed in any way.For more information, including our policy and submission procedure, pleasecontact the Repository Team at: E.mailbox@hud.ac.uk.http://eprints.hud.ac.uk/CORPUS-BASED TRANSCRIPTION AS AN APPROACH TO THECOMPOSITIONAL CONTROL OF TIMBREAaron EinbondCenter for New Music and Audio Technologies (CNMAT)Department of Music, University of California, BerkeleyDiemo Schwarz, Jean BressonIrcam–Centre Pompidou CNRS-STMS1 Place Igor Stravinsky, 75004 ParisABSTRACTTimbre space is a cognitive model useful to address theproblem of structuring timbre in electronic music. The re-cent concept of corpus-based concatenative sound synthe-sis is proposed as an approach to timbral control in bothreal- and deferred-time applications. Using CataRT and re-lated tools in the FTM and Gabor libraries for Max/MSPwe describe a technique for real-time analysis of a live sig-nal to pilot corpus-based synthesis, along with examples ofcompositional realizations in works for instruments, elec-tronics, and sound installation. To extend this technique tocomputer-assisted composition for acoustic instruments, wedevelop tools using the Sound Description Interchange For-mat (SDIF) to export sonic descriptors to OpenMusic wherethey may be further manipulated and transcribed into aninstrumental score. This presents a flexible technique forthe compositional organization of noise-based instrumentalsounds.1. BACKGROUNDThe manipulation of timbre as a structural musical elementhas been a challenge for composers for at least the lastcentury. Pierre Boulez observes that compared to pitch orrhythm, “it is often difficult to find codified theories for dy-namics or timbre” [1]. Trevor Wishart proposed that “pitch-free materials” could be organized based on their timbresand that computers would provide an invaluable resourcefor understanding the “topology” of timbre space [11]. Yeta decade later there is still a predominance of tools for orga-nizing pitch and rhythm as compared to non-pitched mate-rials.Wishart’s observations were informed by research inmusic perception that suggested timbre could be organizedby listeners into a multi-dimensional spatial representa-tion. Wessel and Grey both used multi-dimensional scal-ing to model listeners’ perceptions of timbre into a 2- or3-dimensional space [10, 6]. Momeni and Wessel have usedsuch low-dimensional models to control computer synthesisbased on spatial representations subjectively chosen by theuser [7].We propose an approach to structuring timbre that isbased on perceptually-relevant descriptors and controllablein real-time. Using corpus-based concatenative synthesis(CBCS), a target sound is analyzed and matched to soundsin a pre-recorded database. While this technique can be usedfor more traditional sounds, it is especially effective for or-ganizing non-pitched sounds based on their timbral charac-teristics. We present the implementation of this techniquein CataRT and OpenMusic (OM) and its application in tworecent compositions for instruments and electronics.2. CORPUS-BASED CONCATENATIVE SYNTHESISThe recent concept of corpus-based concatenative soundsynthesis[8] makes it possible to create music by select-ing snippets of a large database of pre-recorded sound bynavigating through a space where each snippet is placed ac-cording to its sonic character in terms of sound descriptors,which are characteristics extracted from the source soundssuch as pitch, loudness, and brilliance, or higher level meta-data attributed to them. This allows one to explore a corpusof sounds interactively or by composing paths in the space,and to create novel harmonic, melodic, and timbral struc-tures while always keeping the richness and nuances of theoriginal sound.The database of source sounds is segmented into shortunits, and a unit selection algorithm finds the sequence ofunits that best match the sound or phrase to be synthesised,called the target. The selected units are then concatenatedand played, possibly after some transformations.2.1. Real-Time Interactive CBCSCBCS can be advantageously applied interactively using animmediate selection of a target given in real-time as is im-plemented in the CataRTsystem [9] for Max/MSP with theextension libraries FTM and Gabor,1 making it possible tonavigate through a two- or more-dimensional projection ofthe descriptor space of a sound corpus in real-time, effec-tively extending granular synthesis by content-based directaccess to specific sound characteristics.See Figure 1 for an example of CataRT’s sound brows-ing interface, where grains are played according to prox-1http://imtr.ircam.fr/index.php/CataRT, http://ftm.ircam.frProceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009223imity to the mouse- or controller-driven target position in auser-selected 2-descriptor plane.Figure 1. Screenshot of CataRT’s 2D navigation interface.2.1.1. Segmentation and descriptor analysisThe segmentation of the source sound files into units canbe imported from external files or calculated internally, ei-ther by arbitrary grain segmentation or by splitting accord-ing to silence or pitch change. Descriptors are either im-ported or calculated in the patch. The descriptors cur-rently implemented are the fundamental frequency, period-icity, loudness, and a number of spectral descriptors: spec-tral centroid, sharpness, flatness, high- and mid-frequencyenergy, high-frequency content, first-order autocorrelationcoefficient (expressing spectral tilt), and energy. For eachsegment, the mean value of each time-varying descriptor isstored in the corpus. Note that descriptors are also storeddescribing the unit segments themselves, such as each unit’sunique id, its start time and duration, and the soundfile andgroup from which it originated.2.1.2. SelectionCataRT’s model is a multi-dimensional space of descriptors,populated by the sound units. They are selected by calculat-ing the target distance Ct , which is a weighted Euclideandistance function that expresses the match between the tar-get x and a database unit uiCt(ui,x) =K∑k=1wtkCtk(ui,x) (1)based on the individual squared distance functionsCtk for de-scriptor k between target descriptor value x(k) and databasedescriptor value ui(k), normalised by the standard deviationof this descriptor over the corpus σk:Ctk(ui,x) =(x(k)−ui(k)σk)2(2)A weight wtk of zero means that descriptor k is not taken intoaccount for selection.Either the unit with minimal Ct is selected, or one israndomly chosen from the set of units with Ct < r2 when aselection radius r is specified, or one is chosen from the setof the k closest units to the target.2.2. Compositional ApplicationWe use CataRT as a source for real-time electronic treat-ment of a live signal, as well as a resource in deferred timefor computer-assisted composition. In the real-time case thetarget for synthesis is the live audio signal. In deferred timethe target may be either an audio signal or an abstract trajec-tory in descriptor space which may be drawn with a mouseor tablet. While the two techniques share similar tools andmechanisms, they yield contrasting results.3. REAL-TIME PERFORMANCE3.1. Signal AnalysisTo pilot CataRT synthesis with a live instrument, the audiosignal is analyzed in real time with tools from the Gabor li-brary according to the same descriptors and parameters usedby CataRT for its analysis of pre-recorded corpora. The listof calculated audio descriptor values, ten currently but withmore possible in future versions of CataRT, are sent to theselection module to output a unit. The relative weights ofeach of the descriptors used in the selection can be adjustedgraphically. At this point the rich possibilities of CataRT’striggering methods are available. We have had particularlyattractive results with the fence mode, where a new unitis chosen whenever the unit closest to the target changes.The selected unit index is then sent to the synthesis mod-ule to output the result taking into account CataRT’s granu-lar synthesis parameters. Otherwise the unit indices may berecorded to an SDIF file to be further processed (see below).The rate at which units are synthesized is affected by theanalysis window size, the triggering method, and the seg-mentation method of the corpus. Units need not be output ina regular rhythm: for example, in the fence mode not everynew target window triggers selection. The analysis framesmay be filtered by rate or descriptor value before being sentto CataRT. In particular, loudness and periodicity descrip-tors may be used to gate signal frames with values belowdesired thresholds.3.2. Musical Realizations3.2.1. Beside OneselfCataRT has been used by several composers to synthe-size fixed electronics, and has been used in improvisedperformance. However the first work to use it as a toolfor pre-composed real-time treatment is Aaron Einbond’sBeside Oneself for viola and live electronics, written asProceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009224part of the Cursus in music composition and technologiesat IRCAM. Controlled by a Max/MSP patch incorporatingCataRT and other Gabor and FTM objects, the computeranalyzes sounds from the live violist and synthesizes match-ing grains from a corpus of close-miked viola samples. Thegoal is a smooth melding of live and recorded sound, a gran-ular trail that follows the acoustic instrument. CataRT isalso used indirectly in this work to stimulate resonant fil-ters. Target pitches played by the viola are matched to acorpus containing samples of similar pitch. The synthesizedresult is then passed through resonance models, avoiding thepossibility of feedback from the live signal.3.2.2. What the Blind SeeA similar technique of real-time analysis and synthesis canbe used when the microphone is turned toward the public in-stead of an instrumentalist. In Einbond’s interactive soundinstallation What the Blind See, presented as part of the ex-hibition “Notation: Kalku¨l und Form in den Ku¨nsten” at theAkademie der Ku¨nste, Berlin, the sounds of the public aswell as the installation’s outdoor setting are analyzed as atarget for CataRT synthesis from a corpus of filtered fieldrecordings of insects and plants.4. CORPUS-BASED TRANSCRIPTIONRather than using the synthesized audio output directly,CataRT’s analysis and selection algorithms of can be usedas a tool for computer-assisted composition. In this case, acorpus of audio files is chosen corresponding to samples ofa desired instrumentation. The CataRT selection algorithmis called on to match units from this corpus to a given target.Instead of triggering audio synthesis, the descriptors corre-sponding to the selected units and the times at which theyare selected are stored and can be imported into a composi-tional environment such as OM where they can be convertedsymbolicaly into a notated score. The goal can be for the in-strumentalist reading the score to approximate the target inlive performance. The target used to pilot this process couldbe an audio file, analyzed as above, or it could be symbolic:an abstract gesture in descriptor space and time, designedby hand with a controller such as a tablet or mouse. Thisprocess is summarized in Figure 2.Soundfile Gabor analysis Instrumental sample corpusCatRT selection SDIF fileOM:  extracting descriptor data / processing / converting to symbolicScoreControllerFigure 2. Flowchart for corpus-based transcription.4.1. Exporting Data with SDIFThe results of the CataRT selection algorithm are recordedto an SDIF (Sound Description Interchange Format) file us-ing a specially-created recording module. This file can beread by other programs such as OM, or by Max/MSP usingFTM data structures and externals. SDIF is an establishedstandard for the well-defined and extensible interchange ofa variety of sound representations and descriptors [4, 12]. Itconsists of a basic data format framework and an extensibleset of standard sound descriptions. This flexible and expand-able format suggests a wide range of future applications forexporting audio descriptor data.The SDIF representation of the selection data is as fol-lows: first, three types of information about the corpus arewritten to the file header. The list of descriptor names forthe data matrix columns are encoded as a custom matrixtype definition. The list of sound files with their associatedindices, referenced in the data matrices, is stored in a name–value table, as are the symbol lists for textual descriptorssuch as SoundSet, which can be assigned from the soundfile folder name.Then, for each selected unit, a row matrix containing allits descriptor data is written to a frame at the time of theselection since recording started. Both frame and matrix areof the custom extended type defined in the file header.4.2. Processing Descriptor Data in OpenMusicOpenMusic (OM) provides a library of functions and datastructures for the processing of SDIF sound description data,which make it possible to link the results of the corpus-basedanalysis data to further compositional processing [2]. Thisintegration in the computer-aided composition environmentalso allows the composer to take advantage of OM objectsfor displaying data lists in music notation.Once imported, the descriptors of choice can be ex-tracted and displayed along with their SDIF time stamps.The SDIF frames can be filtered by descriptor value: for ex-ample to exclude units with loudness below a given thresh-old. They can also be segmented into multiple streams,which can be useful to group units to be played by differ-ent instruments. Finally, om-quantify can be used totranscribe the frame time values into traditional rhythmicnotation and display the results in a voice or poly object.Some descriptors lend themselves to a more intuitiverepresentation as MIDI notes than others: for example pitchor spectral centroid. However other descriptors can be tran-scribed as a more abstract representation of required infor-mation: for example sound file numbers can be assignedarbitrarily to MIDI note numbers, allowing the user to re-call the sound file source of each unit. This could then be auseful tool for the user to interpret the results of the CataRTselection in a conventional score with verbal performancedirections, articulation symbols, or noteheads. This stageof “manual transcription” could become better-automatedProceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009225with the development of OM tools to display extra infor-mation on a score page, for example with the sheet object[3]. However, a final stage of subjective refinement may al-ways be useful before the score is presented to an interpreter.Figure 3 shows an OM patch to transcribe a score with thepoly object, which can then be exported in ETF format andsubjectively-edited in Finale.2Figure 3. OM patch including raw score output.5. DISCUSSION5.1. Mapping ParadigmsThe applications presented proceed from a direct mapping,where parameters of the target are associated with param-eters of synthesis. However other mappings can be pro-posed both to correct for sources of noise, and to allow novelsources of compositional control. Mathematical transfor-mation of a descriptor of an input signal could be used tonormalize it to the range of that descriptor in the corpus.This could compensate, for example, for systematic offsetsin loudness or other parameters. A more interesting map-ping could be created to “transpose” or “invert” a target be-fore mapping it to a corpus, through an appropriate trans-lation or reflection in descriptor space. Wessel’s researchsuggests that listeners could be sensitive to such a timbraltransposition [10]. A further remove could be achieved bymapping one descriptor of the target analysis to a differentdescriptor in the corpus output. Rather than a transposition,this would be a kind of gestural “analogy.”5.2. Playability ConstraintsIn the transcription stage of our algorithm there are no con-straints based in instrumental playability. Samples of liveinstruments are selected by the CataRT algorithm accord-ing to spectral characteristics alone, and then transcribedto notation in OM. It would be interesting to incorporateconstraints based on speed, register, and playing techniquein this process such that the final score would require lessmanual editing. An example of such constraints on pitch2ETF may be replaced by the more flexible MusicXML format as thebridge between future versions of Finale and OM.and rhythm are implemented in OM by Elvio Cipollone [5];however a more sophisticated system would be necessary toincorporate extended playing techniques.6. ACKNOWLEDGEMENTSThis work is partially funded by the French National Agencyof Research ANR within the project SampleOrchestrator.We thank Alexis Baskind, Eric Daubresse, John MacCal-lum, David Wessel, and Adrian Freed for their assistanceimplementing the software and their invaluable feedback onthe manuscript.7. REFERENCES[1] P. Boulez, On Music Today. Cambridge, MA: Har-vard University Press, 1971.[2] J. Bresson and C. Agon, “SDIF sound description datarepresentation and manipulation in computer assistedcomposition,” in Proc. ICMC, Miami, USA, 2004.[3] ——, “Scores, programs and time representation: Thesheet object in openmusic,” Computer Music Journal,vol. 32, no. 4, 2008.[4] J. J. Burred, C. E. Cella, G. Peeters, A. Ro¨bel, andD. Schwarz, “Using the SDIF sound description inter-change format for audio features,” in ISMIR, 2008.[5] E. Cipollone, “CAC as maieutics: OM-Virtuoso andConcerto,” in OM Composer’s Book, vol. 2. Paris:Editions Delatour France / Ircam, 2008.[6] J. M. Grey, “Multidimensional perceptual scaling ofmusical timbres,” J. Acoust. Soc. Am., vol. 61, 1977.[7] A. Momeni and D. Wessel, “Characterizing and con-trolling musical material intuitively with geometricmodels,” in Proc. NIME, Montreal, Canada, 2003.[8] D. Schwarz, “Corpus-based concatenative synthesis,”IEEE Sig. Proc. Mag., vol. 24, no. 2, Mar. 2007.[9] D. Schwarz, R. Cahen, and S. Britton, “Principles andapplications of interactive corpus-based concatenativesynthesis,” in JIM, GMEA, Albi, France, Mar. 2008.[10] D. Wessel, “Timbre space as a musical control struc-ture,” Computer Music Journal, vol. 3, no. 2, 1979.[11] T. Wishart, On Sonic Art. London: Harwood Aca-demic Publishers, 1996.[12] M. Wright et al., “Audio applications of the sound de-scription interchange format standard,” in Audio Engi-neering Society 107th Convention, New York, 1999.Proceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009226

Audio applications of the sound description interchange format standard,”

CAC as maieutics: OM-Virtuoso and Concerto,”

Characterizing and controlling musical material intuitively with geometric models,” in

Corpus-based concatenative synthesis,”

Corpus-Based Transcription as an Approach to the Compositional Control of Timbre Original Citation Einbond,

Multidimensional perceptual scaling of musical timbres,”

On Music Today.

On Sonic Art.

Principles and applications of interactive corpus-based concatenative synthesis,”

programs and time representation: The sheet object in openmusic,”

SDIF sound description data representation and manipulation in computer assisted composition,” in

Timbre space as a musical control structure,”

Using the SDIF sound description interchange format for audio features,” in ISMIR,

City Research Online

cote interne IRCAM: Einbond09aInternational audienceTimbre space is a cognitive model useful to address the problem of structuring timbre in electronic music. The recent concept of corpus-based concatenative sound synthesis is proposed as an approach to timbral control in both real- and deferred-time applications. Using CataRT and related tools in the FTM and Gabor libraries for Max/MSP we describe a technique for real-time analysis of a live signal to pilot corpus-based synthesis, along with examples of compositional realizations in works for instruments, electronics, and sound installation. To extend this technique to computer-assisted composition for acoustic instruments, we develop tools using the Sound Description Interchange Format (SDIF) to export sonic descriptors to OpenMusic where they may be further manipulated and transcribed into an instrumental score. This presents a flexible technique for the compositional organization of noise-based instrumental sounds

Archive Ouverte a LUniversite Lyon 2

Aaron Einbond

Diemo Schwarz

Jean Bresson

University of Michigan Library Repository

http://hdl.handle.net/2027/spo.bbp2372.2009.051

Corpus-Based Transcription as an Approach to the Compositional Control of Timbre

Abstract

Similar works

Full text

Available Versions

Name not available

University of Huddersfield Repository

City Research Online

Archive Ouverte a LUniversite Lyon 2

University of Michigan Library Repository