Ambiances are background recordings of places used in audiovisual productions to make listeners feel they are in places like a pub or a farm. Accessing to commercially available atmosphere libraries is a convenient alternative to sending teams to record ambiances yet they limit the creation in different ways. First, they are already mixed, which reduces the flexibility to add, remove sounds or change the panning. Secondly the number of ambient libraries is limited. We propose a semi-automatic system for ambient generation. The system creates ambiances on demand given textual queries by fetching relevant sounds from a big sound effect database and delivering them into a sequencer multitrack project. Ambiances of diverse nature can be created easily. Controls are offered to the user to further specify its needs

Barbosa, Álvaro

Cano, Pedro

Fabig, Lars

Gouyon, Fabien

Koppenberger, Marcos

Loscos, Alex

English

Ambiances are background recordings used in audiovisual productions to make listeners feel they are in places like a pub or a farm. Accessing to commercially available atmosphere libraries is a convenient alternative to sending teams to record ambiances yet they limit the creation in different ways. First, they are already mixed, which reduces the flexibility to add, remove individual sounds or change its panning. Secondly, the number of ambient libraries is limited. We propose a semi-automatic system for ambiance generation. The system creates ambiances on demand given text queries by fetching relevant sounds from a large sound effect database and importing them into a sequencer multitrack project. Ambiances of diverse nature can be created easily. Several controls are provided to the users to refine the type of samples and the sound arrangement

Cano, P.

Fabig, L.

Gouyon, F.

Koppenberger, M.

Loscos, A.

Barbosa, A.

Repositório Institucional da Universidade Católica Portuguesa

Proc. of the 7th Int. Conference on Digital Audio Effects (DAFx’04), Naples, Italy, October 5-8, 2004SEMI-AUTOMATIC AMBIANCE GENERATIONP. Cano, L. Fabig, F. Gouyon, M. Koppenberger, A. Loscos Institut Universitari de l’AudiovisualUniversitat Pompeu Fabra, Barcelona, Spainpcano@iua.upf.esA. Barbosa Dept. Som e ImagemUniversidade Católica, Porto, Portugalabarbosa@porto.ucp.ptABSTRACTAmbiances are background recordings of places used in au-diovisual productions to make listeners feel they are in placeslike a pub or a farm. Accessing to commercially availableatmosphere libraries is a convenient alternative to sendingteams to record ambiances yet they limit the creation in dif-ferent ways. First, they are already mixed, which reducesthe flexibility to add, remove sounds or change the pan-ning. Secondly the number of ambient libraries is limited.We propose a semi-automatic system for ambient genera-tion. The system creates ambiances on demand given tex-tual queries by fetching relevant sounds from a big soundeffect database and delivering them into a sequencer multi-track project. Ambiances of diverse nature can be createdeasily. Controls are offered to the user to further specify itsneeds.1. INTRODUCTIONThe audio component of audiovisual productions has longbeen regarded as of minor importance. Nevertheless, in thelast years and especially after productions such as Apoca-lypse Now (1979), its importance has been acknowledged.Audio is gaining interest for its evocative and overall im-mersive experience of the audiences. Audio has a immensepower, even when accompanying coarsely drawn cartoons,for creating the illusion of reality.Traditionally, from the film production process point ofview, sound is broken into a series of layers: dialog, musicand sound effects—from now SFX [1]. SFX (synchronic)can be broken further into hard SFX—car doors openingand closing, and other foreground sound material—and foley—sound made by humans, e.g: footsteps—from one side, andambiances from the other side. Ambiances—also known asatmospheres—are the background recordings of scenes andidentify them aurally. They make the listener really feel likethey are in places like an airport, a church, a subway station,or the jungle. Ambiances have two components: The ambi-ent loop, which is a long, streaming, stereo recording, andspecifics or stingers, which are separate, short elements—e.g: dog barks,car horns, etc—that trigger randomly to breakup repetition [2].Sound engineers need to access sound libraries for theirvideo and film productions, multimedia and audio-visualpresentations, Web sites, computer games and music. Ac-cess to libraries is a convenient alternative to sending a teamto record a particular ambiances :“Rain forest”, “a cave”.However, the approach has some drawbacks:1. Accessing the right ambiances is not easy due to theinformation retrieval models, currently based mainlyon keyword search [3].2. The number of libraries is big but limited. Everybodyhas access to the same content although sound design-ers can use them as starting point and yield them un-recognizable and unique.3. The ambiances offered by SFX library providers arealready mixed. There may be SFX in the mix that thesound engineer does not want in that position of maybe does not want at all. It is a hassle to fix it.In this context, we present a system for the automaticgeneration of ambiances. Shortly, the system works as fol-lows: the user specifies his need with a standard textualquery, e.g: “farm ambiance”. The ambiance is created on-the-fly combining SFX related to the query. For example,thequery “farm ambiance” may return “chicken”, “tractors”,“footsteps on mud” or “cowbells” sounds. A subset of re-trieved sounds is randomly chosen. After listening to theambiance, the user may decide to refine the query—e.g: toremove the “cowbells” and add more “chickens”—, ask an-other random ambiance—with a “shuffle-type” option—ordecide that the ambiance is good enough to start workingwith. The system outputs the individual SCCS in a multi-track project (see Section 2).The intended goals of the approach can be summarizedas follows:Enhance creativity: The sound engineers have access to ahuge ever-changing variety of ambiances instead ofa fix set of ambiances. The combination of individ-ual SFX provides an substantially bigger number ofambiances.DAFX-1Proc. of the 7th Int. Conference on Digital Audio Effects (DAFx’04), Naples, Italy, October 5-8, 2004Enhance productivity: Engineers can have several possiblesonifications in short time.Enhance flexibility: Having the different SFX of the am-biance separately in a MultiTrack gives more flex-ibility to the ambiance specification process, somesounds–a bird singing in a forest ambiance–can beremoved or their location in the time line changed. Italso allow for spatialization using 5.1. In 5.1-channelaudio, different aspects of the sound, such as record-ing levels, program equalization, panning have beenstandardized.Enhance quality: By being able to have several ambiancetemplates with very low overhead, the production cy-cle reduces. The producers can give their feedbackfaster and their opinions be incorporated earlier in theproduction improving the overall quality.2. SYSTEM DESCRIPTIONThe system is based on a concept-based SFX search enginedeveloped within the AudioClas project (www.audioclas.org).The objectives of the project were to go beyond current pro-fessional SFX provider information retrieval model, basedon keyword-matching, mainly through two approaches [4]:Semantically-enhanced management of SFX using a gen-eral ontology, WordNet [5]1.Content-based audio technologies which allow automaticgeneration of perceptual meta data (such as promi-nent pitch, dynamics, beat, noisiness).These two approaches are the building blocks of thesemi-automatic ambiance generation. Current prototype uses80.000 sounds from a major on-line SFX provider2. Soundscome with textual descriptions which have been disambiguatedwith the augmented WordNet ontology [3]. WordNet is alexical database that, unlike standard dictionaries, insteadof indexing terms alphabetically, indexes concepts with re-lations among them.thrush (songbirds characteristically having ...)=> oscine, oscine bird=> passerine, passeriform bird=> bird=> vertebrate, craniate=> chordate=> animal, animate being...=> organism, being=> living thing, animate thing=> object, physical object=> entity, physical thing1http://www.cogsci.princeton.edu/˜wn/2http://www.sound-effects-library.comAccordingly, the sound “Thrush And Nightingale VariousCalls” becomes labeled with the following set of concepts:01234719%n thrush -- (songbirds havingbrownish upper plumage with a spotted breast)01237641%n nightingale, Luscinia megarhynchos-- (European songbird noted for its melodiousnocturnal song)05680983%n birdcall, call, birdsong, song --(the characteristic sound produced by a bird)The numbers before the definitions correspond to theunique identifiers, offsets, of the concepts, or synonym sets,synsets as referred in the WordNet literature [5].There are two main functional blocks in the system. Thefirst one retrieves the relevant sounds of the SFX Databaseand a second one organizes the sounds in a multitrack ac-cording to some heuristic rules (see figure 1).3. SOUND SELECTION AND RETRIEVALThe first step has been mining ambiance sounds to learn thetype of sources used. We use a database of SFX that hasbeen labeled with concepts rather than with words (see [3]for details). We are therefore able to study the co-occurrenceof concepts in sounds. For example, the ambiance “FarmAmbiance Of Rooster And Hen With Wagtail In Background”has been converted to:01466271%n hen, biddy -- (adult female chicken)01206115%n wagtail -- (Old World bird having a verylong tail that jerks up and down as it walks)02893950%n farm -- (workplace consisting of farmbuildings and cultivated land as a unit)By mining this information we learn that farm is relatedto the concept hen and the concept wagtail. Moreover, thereare relations encoded in WordNet, which knows that henand chicken are related. Whenever a user asks for farmsounds we can retrieve a set of sounds where farm appears.Besides we can also search for the sounds of the relatedconcepts, such as chicken. A random subset of the relevantsounds is forwarded to the subsequent block, the sound se-quencing.4. SOUND SEQUENCINGBesides the definition and selection of the suitable SFX, asignificant part of the work of the sound designer is settingup parameters and time lines in a multitrack project, such asvolumes or panoramic envelopes. This section detail someof the rules used to mix all fetched tracks and compoundthe synthetic atmosphere. The SFX retrieval module returnsDAFX-2Proc. of the 7th Int. Conference on Digital Audio Effects (DAFx’04), Naples, Italy, October 5-8, 2004Match User QUERYwith Knowledge BaseBrokerMatch Knowledge Base ManagerResults Ambience Sound Data BasePriority ManagerAgentGenerate User Interface withPreliminary Ambience proposalAmbienceSoundData BaseManagerKnowledgeBaseManagerSatisfactory result?Match Users requiredmodifications with AmbienceSound Data Base Entries?Generated Ambience Multi-Track SoundYesNoShuffleFigure 1: Flow diagram of the system.mono and dry (no effect has been applied) tracks. When-ever available, the module differentiates between two typesof tracks: long ambient tracks and several short isolated ef-fects. One long track is selected to serve as a ambient loopon which the short sounds, or specifics, are added. Withsuch picture of the workspace we hint some rules on how toplace the tracks in the mix, how to adjust channel controls(gain, panning and equalization), and which effects (echo,reverb) can be applied to each track.The systems automatically distributes the tracks alongthe mix, placing first the ambient loop and inserting sequen-tially the specifics, with a probabilistic criterion. This prob-abilistic criterion is based on the inverse of a frame-basedenergy computation. This means the more energetic regionsof the mix will have less probability to receive the followingeffect track. This process is depicted in figure 1.It is a cunning feature to keep a certain degree of ran-domness. Again, a shuffle button can remix the atmosphereas many times as desired. Also, further probabilistic modelscan take into account other parameters such as energy varia-tion (in order to avoid two transients happening at the sametime), such as spectrum centroid (in order to avoid as muchas possible the frequency content overlap), or others.Another important feature is the automatic adjustmentof channel controls: gain, panning and equalization. Re-garding the levels, these are set so that the track maximumFigure 2: Mix example. a. long ambient sound and thecorresponding probability density function. b and c SFXadded and the corresponding recalculated probability den-sity functionlevels are 3 dB above the long ambient mean level and thatno saturation / clipping problem appears. Regarding thestereo panning the ambient sound is centered and the iso-lated tracks are panned one left one right along time in or-der to minimize time overlap. The amount of panning de-pends on how close are two consecutive tracks, the closer,the more panned. Equalizing is only applied to those tracksDAFX-3Proc. of the 7th Int. Conference on Digital Audio Effects (DAFx’04), Naples, Italy, October 5-8, 2004that overlap significantly in frequency domain with the ad-jacent tracks or with the ambient loop sound. In these casesthe effect track is 6-band equalized to flatten down to -12dB the overlapping frequency region.Finally, the strategy for the automation of the effects wepropose is based on rules. These rules are mainly relatedwith the context of the ambiance. Say we are reconstructingan office atmosphere, we will apply a medium room reverbto whatever effect track we drop to the mix; if we are recon-structing a mountain atmosphere, we can apply some echoto the tracks.4.1. Integration in professional environmentsThe advent of high quality audio and spatialization surroundsetups (e.g: 5.1), first in the film industry, and more recentlyin home entertainment with DVD, arouse the possibilitiesto create more engaging and immersive ambient sound. Itis now possible to have ambient loops that take advantageof very low pitch sound (with the subwoofers) and flow ina tri-dimensional space or specific sound elements that panin every direction we wish. On the other hand the complex-ity of setting up a multitrack project for a surround scenarioincreased a lot. It would be extremely useful for a sound de-signer to specify at a higher level which surround character-istics are desirable for the ambiance, so that the system canprovide him a multitrack project file, and respective soundfiles, already configured to be integrated in his main project.5. EXAMPLESLet us comment some typical examples on ambiance gener-ation:Some of the ambiances created had too many events init. The jungle ambiance had plenty of tropical birds,elephants and monkeys and sounded more like a zoothan a jungle.Some of the ambiances need greater detail in the specifica-tion. Asking for a war ambiance yielded war soundsof different epochs, e.g: bombs, machine guns andswords.The sex ambiance retrieved sounds produced by too manypeople to be realistic.These experiences lead us to the conception of a refine-ment control to add/remove specific sources or another con-trol for the density of specifics.As a multitrack application, we have used the free editorAudacity3. In addition to common sound editing function-alities, Audacity allows to mix several tracks together and3http://audacity.sourceforge.net/apply effects to tracks. Audacity allows to save multitracksessions yet it does not read sessions created by externalprograms. We have therefore tweaked the application in or-der to load our automatically generated ambiance multitracksessions. The use of Audacity helped us in the design ofsome of the mixing rules of section 4.During the conference we will demonstrate the genera-tion of ambiances.6. CONCLUSIONSWe have presented a system for semi-automatic ambiancegeneration. The ambiances generated by textual query canbe further refined by the user. The user controls the numberof sounds that should be returned and can add and removetypes of sounds, e.g: “more penguin sounds”. Furthermorethe ambiance is delivered to the user as a multitrack project,providing thus flexibility to fine tune the results. We planto extend this work to semi-automatic sonifications of au-diovisual productions given scripts (or briefings) and someinformation of the timing.7. ACKNOWLEDGMENTSWe thank the staff from the Tape Gallery for all the support,discussion and feedback.This work is partially funded by the AUDIOCLAS ProjectE! 2668 Eureka (http://www.audioclas.org). We thank thecollaboration from Sylvain Le Groux, Julien Ricard and Nico-las Wack. We thank the review and feedback from PerfectoHerrera and José Lozano.8. REFERENCES[1] Robert L.Mott, Sound Effects: Radio, TV, and Film,Focal Press, 1990.[2] Nick Peck, “Beyond the library: Applying film post-production techniques to game sound design,” in Proc.of Game Developers Conference, San Jose CA, USA,2001.[3] Pedro Cano, Markus Koppenberger, Perfecto Herrera,and Oscar Celma, “Sound effects taxonomy manage-ment in production environments,” in Proc. AES 25thInt. Conf., London, UK, 2004.[4] Pedro Cano, Markus Koppenberger, Sylvain Le Groux,Perfecto Herrera, and Nicolas Wack, “Perceptual andsemantic management of sound effects,” in Submittedfor publication, 2003.[5] G. A. Miller, “WordNet: A lexical database forenglish,” Communications of the ACM, pp. 39–45,November 1995.DAFX-4

Semi-automatic ambiance genereation

https://repositorio.ucp.pt/bitstream/10400.14/5011/1/2004-DAFX.pdf

Semi-Automatic Ambiance Genereation

Semi-Automatic Ambiance Genereation

Abstract

Similar works

Full text

Available Versions

Repositório Institucional da Universidade Católica Portuguesa