Array, the journal of the International Computer Music Association

Hagan, Kerry

array2022Spatial Audio in Online Presenta-tionsBy Kerry HaganThe pandemic forced conferences toevaluate hybrid and online alterna-tives for academic and artistic pre-sentations. Although many confer-ence activities adapted quickly toonline formats, e.g., paper presenta-tions, some elements suffered moresevere losses when moved online. Inparticular, online formats signifi-cantly impacted spatial music con-certs.Considering binaural recordings arecurrently the only method for repro-ducing spatial music in two chan-nels, it made sense that conferencesfor spatial music relied on deliveringbinaural audio in various ways.Previous to ICMC 2022, I attendedseveral online concerts that tried toprovide spatial audio. In some cases,artists created VR simulations oftheir works. VR was especially inter-esting for spatial compositions withvisual elements, but it was a convo-luted method for creating binauralrecordings for audio-only pieces.Additionally, composers without ac-cess to VR engines could notpresent with this method.spaceIn other cases, the conferences re-lied on the artists to provide two-channel recordings of their work,and the organizers hosted thoserecordings. Several participants pro-vided their works binaurally insteadof mixing down to stereo. Given thatmost electronic and computer mu-sic audiences would attend onlineconcerts with headphones, the re-sults were relatively acceptable. Theonus to encode binaural recordingsfell on the composers.In the case of the Klingt Gut festivalof spatial music, the organizers ac-quired the multi-channel worksfrom the composers and used adummy head for binaural capture intheir sound dome. This stream led toa more realistic experience, partlybecause the dummy head picked upthe environment. Adding roomnoise may be counterintuitive sincecomposers often try to reduce thephysical space to create a virtualspace in their work. But for an audi-ence member, the experience feltmore live, present, and engaging.Due to this experience, ICMC 2022organizers chose to stream audiofrom a dummy head for its spatialaudio concerts. The feedback from5array2022online delegates matched the expe-rience of Klingt Gut. One audiencemember also mentioned that therewas more intimacy because we po-sitioned the dummy head inside theaudience. Therefore, the stream con-tained environmental noises andapplause from the audience’s per-spective, giving the online delegatea realistic concert experience.The trade-offs of these differentways of presenting spatial audio on-line may be apparent. If the host re-quires composers to mix their bin-aural versions, some artists may findthis restrictive depending on theirsoftware, hardware and laboratoryaccess. Also, composers with moreknowledge of virtual systems mayfind it easier than early career re-searchers or composers who workentirely in audio-only media.The dummy head solution requiresnothing more than what a com-poser would supply in any othermulti-channel presentation.However, despite the positives ofaudience integration, composersmay wish to have more control overthe mix than the dummy head pro-vides. Additionally, good dummyhead microphones can be prohibi-tively expensive for concert orspaceconference organizers.There is one more factor that noneof these solutions address. Spatialconcerts delivered online also facethe same problem as regular onlineconcerts: streaming audio requirescompression, and most streamingservices compress audio as MP3 orcomparable. Although some onlineplatforms may enable ‘original au-dio’, other common platforms com-press audio significantly. When com-bined with the simulation HRTF thatbinaural audio employs, compres-sion can hurt spatial images. Manycomposers are vocal in their disap-pointment with compressionthrough standard channels.Ultimately, conferences and festivalsattempt to provide the best approxi-mation of the in-person experience.Audiences, composers and organiz-ers seem to have different needs re-garding online spatial audio con-certs, and there currently isn’t anideal solution that satisfies all stake-holders.ICMC 2022 chose the dummy headsolution. The compression thatcame from streaming didn’t impacttoo severely, though there weresome instances where the formatwas not ideal. Though the pandemicis slowly ending, permitting more6array2022 spacetravel, other issues, such as climatechange, motivate organizers to con-tinue employing online or hybridoptions for delegates. It stands toreason that future research shouldaddress the composer and audienceexperience and the technologies forstreaming audio, especially for elec-tronic and computer music and spa-tial audio.7

English

Spatial Audio in Online Presentations

von Coler, Henrik

array2022Chaos in the Garden. Human-assisted AI Composition in Experi-mental Spatial MusicBy Henrik von ColerAI and SpatializationApplications of Artificial Intelligence(AI) have been manifold in the pastdecades. However, since the launchof the ChatGPT1 chatbot in Novem-ber 2022, the use of AI-based tech-nologies has been growing expo-nentially in any field of human pro-ductivity and creativity. Our societyis currently debating the future ofsuch technologies, regarding theeffects they might have on thestructure of our working world andprivate lives.There is consensus that these sys-tems can and will perform repetitiveand automatable tasks, even of highcomplexity. How and whether theycan – or rather should – be used increative processes is more contro-versial. This has already caused is-sues with copyright and intellectualproperty for AI-generated images(Hristov, 2016).Large Language Models (LLMs) suchas ChatGPT can be integrated intothe conceptualization phase in anyartistic discipline, by answeringspaceprompts and thus influencing deci-sions in the design process. This arti-cle illustrates and discusses how thistool can be used to create a text-based score for experimental spatialmusic. Related work in AI-basedcomposition and algorithmic meth-ods for spatial audio practices willbe presented in the introduction.Afterwards, the chat and the fullscore are presented. This is followedby an analysis and a report on a firstperformance based on the composi-tion.AI and ML in Music CreationThe use of tools and auxiliary agentsin music composition is older thancomputers or mechanical devices.Dice were used to introduce chanceinto classical compositions (Hedges,1978; Ruttkay, 1997). This can beconsidered an early example oftechnology-aided or algorithmiccomposition.In computer music, algorithmiccomposition and computer-aidedcomposition have been an integralpart of its history (Supper, 2001).Early examples of computer-aidedcomposition produced symbolicscores, based on algorithms andheuristics. These include LejarenHiller’s work from 1955, Moorer’s33array2022rule-based systems and Xenakis’ap-proach, who considered algorithmiccomposition a ‘tool for constructingcomplex structures with minimummeans’ (Roads, 1980). Lopez-Rinconet al. (2018) present a taxonomy ofAI based music composition, divid-ing the field into soft computingbased music composition methodsand symbolic AI based music compo-sition methods, both with additionalsubareas.With increasing capabilities, artificialintelligence and machine learning(ML) can be incorporated into musiccomposition and performance atmany levels, including not onlyscore generation, but also interac-tive performance and improvisationsystems or complete sound and mu-sic synthesis methods.Modern generative music systems,such as WaveNet (Oord et al., 2016)and OpenAI’s Jukebox (Dhariwal etal., 2020) create complete musicalpieces in the audio domain, in thestyle of artists included in the train-ing data. Noise2Music (Huang et al.,2023) is a diffusion model, that gen-erates audio from text prompts, sim-ilar to the well-known diffusionmodels for image generation.Composition is not the only applica-tion scenario for AI and machinespacelearning in music. The integration ofML algorithms into human-com-puter interfaces for the control ofdigital musical instruments leads toa form of human-machine partner-ship (Fiebrink, 2017).Algorithmic SpatializationIn the context of electroacousticmusic, spatialization refers to thedynamic distribution of sound incomposition or performance. Oftenthis involves algorithms for control-ling the spatial distribution, or moreprecisely the virtual positions ofmultiple sound sources in object-based spatialization (Schacher,2018). Such algorithms can providemeta-parameters and semi-auto-mated workflows, needed to controlthe many parameters involved inthese processes.Tools like OM-Spat (Garcia et al.,2015) and OM-Prisma (Schumacherand Bresson, 2010) allow algorith-mic control over spatial audio pro-cesses in offline computer-aidedcomposition. Applications also in-clude spatial sound synthesis, basedon algorithmically generated con-trol trajectories.Spatialization algorithms can alsobe derived from natural phenom-ena. For example, swarm behavior34array2022has been used for various musicalparameters in compositions (Kali-akatsos-Papakostas et al., 2020).The inherent spatial qualities ofswarm motion predestine it forspatial aspects, as for example inswarm-based spatial granularsynthesis (Wilson, 2008). A similarexample uses Boids algorithm forspectral and granular spatialization(Kim-Boyle, 2006).Other examples have more prag-matic applications. In a self-super-vised framework for audio spatial-ization Lu et al., 2019, video data isused to generate 3D spatial audiofrom monophonic recordings. Thisprocedure can be used to create im-mersive audio from simple record-ings.Spatial Performance ConceptsThe performance of experimentalmusic on spatial sound reproduc-tion systems opens up a wealth ofpossibilities. Especially in the case oflive electronics, the usual musicalgrid of time and pitch is not applica-ble. Complex synthesis systems,such as modular synthesizers orcustom digital musical instruments(DMI) offer a wide range of parame-ters that cannot be expressed inmusical notation. Additional spatialspaceparameters for each instrument re-quire further structuring to allow forintentional control. Concepts for aguided improvisation are a meansfor adding this structure. Such con-cepts can be text-based instruc-tions, graphical notations, or anytype of conceptual guideline.The Electronic Orchestra Charlotten-burg (EOC)2 is an ensemble thatfocuses on live electronic music forlarge loudspeaker systems. It hasexplored various approaches toguided improvisation, some ofwhich include explicit treatment ofthe spatial aspect. These includetheir own compositions, adapta-tions of existing pieces and andcommissioned works.Mark Applebaum’s S-Tog (Apple-baum, 1996), for example, is agraphical score, based on the trainmap of Copenhagen. Musicianstravel fictitious routes on the map,realizing playing instructions thatare connected to train stations onthe map. For the EOC adaptation,automatic source movements wereprogrammed, according to the mu-sicians’ routes on the map.For Robert Stokowy’s Coffee Rings(Stokowy and von Coler 2018; p. 34),the virtual source positions of eachmusician are controlled by signal35array2022analysis. The detected pitches of theinstruments determine the azimuthangle of a virtual sound source on afixed radius. Thus, the source distri-bution is modulated according tothe pitch distribution of all instru-ments.spaceplanned but happened during thechat on April 14, 2023:Chaos in the Garden3ConceptThe goal of this experiment is tohave the AI chatbot create a scorefor five arbitrary instruments, includ-ing instructions for the spatializa-tion. Since the score is intended tobe as open as possible, no tonal orinstrument-specific instructionsshould be included. The system wasinstructed to remove or replacesuch content and asked to refinecertain aspects, until a useful ver-sion was reached. Although theprocess is influenced by all prompts,the aesthetic concept was intention-ally left to the chatbot as much aspossible.The ChatThe following text is the originalconversation with the Large Lan-guage Model trained by OpenAI,based on the GPT-3.5 architecture.None of the steps were previously36array2022 space37array2022 space38array2022 spaceICMA awardsBy PerMagnus Lindborg and Christo-pher Haworth39array2022 spaceICMA awards40array2022ICMA awardsBy PerMagnus Lindborg and Christo-pher Haworthspaceplaced throughout the space’. Thesystem was then prompted to refinethese passages, resulting in the fol-lowing list of spatial instructions forthe sound director:The above chat was followed by anattempt to let the chatbot create ananimated graphical score. Althoughit delivered short clips using Pythonand the pygame library4, it did notcreate a useful result.AnalysisIt took nine prompts from the hu-man composer to reach the finalscore. With more precise user input,it could have been less. The struc-ture of the piece, as well as thetopos, were not influenced byprompts, as far as possible. How-ever, the basic concept of an experi-mental spatial music score for un-specified instruments was kept asthe boundary condition.The first version of the score withspatial instructions had a frequentuse of the vague expression ’sounds1. panning from left to right andfront to back2. panning from front to back andmoving around the space in arandom pattern3. sounds placed in different areasof the space and movingaround in a circular pattern (2occurrences)4. sounds placed in different areasof the space and movingaround in a random pattern (2occurrences)5. sounds placed in different areasof the space and movingaround in a random, chaoticpattern6. panning from front to back andmoving around the space in acircular pattern7. sounds swirling around thespace in different directions8. sounds placed in different areasof the space and movingaround in a random, unpre-dictable pattern9. sounds placed in different areasof the space and moving41array2022around in a circular or randompattern (3 occurrences)10. sounds placed in different areasof the space and movingaround in a random, gentle pat-tern11. panning from right to left andmoving around the space in arandom, celebratory pattern12. sounds placed in different areasof the space and movingaround in a random, energeticpatternThe instructions listed above corre-spond to typical methods used inspatial music practice. Circular, ran-dom and chaotic patterns are oftenused for source movements, includ-ing performances by the EOC.Although attributes such as swirling,unpredictable, gentle,energetic and celebratory are rathervague, they are interpretable andconclusive. Some instructions eveninclude specific directions, like fromleft to right and front to back.In most cases, the spatial instruc-tions are coherent with the overallcharacter of the sections and the in-structions for the instrumental per-formers. Hectic movements such asswirling, unpredictable and chaoticappear in the more dramatic Sec-spacetions 2 and 3, combined with disso-nant sounds, aggressive sounds andirregular rhythms. The releasing ten-sion of Section 4 is supported bygentle patterns, while the upliftingcharacter of Section 5 is emphasizedby celebratory and energetic move-ments.Rehearsal and PerformanceThe composition was premiered onMay 22 2023 by the Electronic Or-chestra Charlottenburg (EOC)2 aspart of the Intersonanzen festival5 inPotsdam, Germany. During re-hearsal and performance, the en-semble’s main goal was to do do jus-tice to the AI’s ’compositional inten-tions’, in order to bring out the arc oftension and the individual characterof the five sections.Since some aspects of the score areleft open to a certain extent, the en-semble made the according deci-sions. Each phrase in the sections’instructions was assigned to one ormore of the four musicians on stage.The sound director used a combina-tion of Python6, Supercollider7 andhardware input devices to imple-ment the spatial instructions duringthe performance.42array2022DiscussionChaos in the Garden is the result ofusing an AI-chatbot as a tool for cre-ating an experimental score. Re-gardless of the instruments used ina performance, any interpretationof the piece can be considered ‘com-puter music,’since it would not havebeen possible without the help of aspecific software.The cornerstones of the conceptwere specified by the user, whereasthe overarching structure and thetopos have been chosen by the sys-tem. Being trained on existing textsand data found in the internet, thequestion arises whether the compo-sition is original or a combination ofrelated material and publications.Even this simple example raises thequestion of authorship and copy-right. A subsequent Google searchfound several pieces of music withthe same title from different musicalgenres, none of them related to ex-perimental spatial music.Bown (2021) presents a model forhuman-computer interaction in cre-ative processes. It starts with a hu-man intention and then enters aniterative cycle of machine outputand user evaluation, until the de-sired result is achieved. The creationspaceof Chaos in the Garden follows a sim-ilar pattern. However, ideally themachine does not assist the com-poser but the composer assists themachine in reaching a form that canbe used as a score. In this case, theabort criterion is in the hands of theuser, who decides whether to con-tinue altering the score throughadditional prompts. With furtherprompts, the instructions mightbecome even more specific. This,however, would further bias thesystem and it would lose it’s claimon authorship with every step.The creative process involves directguidance, assistance or input of hu-man beings (Hristov, 2016) and cannot be considered an autonomouslygenerated AI creation. Consideringthe concept of CREATIVITY byMoruzzi (2018), the system did notact autonomously in the reception,selection or elaboration of stimuli.Hence, it did not create a musicalwork, but was merely used as a toolin a creative process. According toChatGPT 4.0, however, the ’interac-tion in creating this piece can be con-sidered a collaboration’. Ultimately,Chaos in the Garden is considered ahuman-assisted AI composition.43array2022Notes[1] https://chat.openai.com[2] https://eo-charlottenburg.de[3] Project website with recordingsand further information https://eo-charlottenburg.de/repertoire/chaos_in_the_garden/[4] https://www.pygame.org[5] https://www.potsdam-museum.de/veranstaltung/intersonanzen-2023[6] https://www.python.org/[7] https://supercollider.github.io/ReferencesApplebaum, M. (1996). S-Tog. http://www.markapplebaum.com/stog.html (last access 27 July 2023).Bown, O. (2021). “Sociocultural anddesign perspectives on AI-basedmusic production: why do we makemusic and what changes if ai makesit for us?” In: Handbook of ArtificialIntelligence for Music: Foundations,Advanced Approaches, and Develop-ments for Creativity: 1–20.Dhariwal, P. et al. (2020). “Jukebox: Agenerative model for music”. arXivpreprint, arXiv:2005.00341.Fiebrink, R. (2017). “Machinelearning as meta-instrument:Human-machine partnershipsshaping expressive instrumentalspacecreation”. Musical Instruments in the21stCentury: Identities, Configurations,Practices: 137–151.Garcia, J. et al. (2015). “Tools andapplications for interactive-algorithmic control of soundspatialization in OpenMusic”.inSONIC2015, Aesthetics of SpatialAudio in Sound, Music and Sound Art.Hedges, S. A. (1978). “Dice music inthe eighteenth century”. Music &Letters 59(2): 180– 187.Hristov, K. (2016). “Artificialintelligence and the copyrightdilemma”. Idea 57: 431.Huang, Q. et al. (2023). “Noise2music:Text-conditioned music generationwith diffusion models”. arXivpreprint arXiv:2302.03917.Kaliakatsos-Papakostas, M. et al.(2020). “Artificial intelligencemethods for music generation: areview and future perspectives”.Nature-Inspired Computation andSwarm Intelligence: 217–245.Kim-Boyle, D. (2006). “Spectral andgranular spatialization with boids”.Proceedings of the InternationalComputer Music Conference ICMC.Lopez-Rincon, O. et al. (2018).“Algorithmic music compositionbased on artificial intelligence: A44array2022survey”. International Conference onElectronics, Communications andComputers: 187–193.Lu, Y.-D. et al. (2019). “Self-super-vised audio spatialization withcorrespondence classifier”. IEEEInternational Conference on ImageProcessing ICIP: 3347–3351.Moruzzi, c. (2018). “Creative AI:Music composition programs as anextension of the composer’s mind”.Philosophy and Theory of ArtificialIntelligence 2017: 69–72.van den Oord, A. et al. (2016).“Wavenet: A generative model forraw audio”. arXiv preprintarXiv:1609.03499.Roads, C. (1980). “Artificial intelli-gence and music”. Computer MusicJournal 4(2): 13–25.Ruttkay, Z. (1997). “ComposingMozart variations with dice”.Teaching Statistics 19(1): 18–19.Schacher, J. (2018). “AlgorithmicSpatialization”. In: The OxfordHandbook of Algorithmic Music.edited by R. T. Dean and A. McLean.Oxford University Press. doi:10.5281/zenodo.3921793.Schumacher, M. and J. Bresson(2010). “Compositional control ofperiphonic sound spatialization”.space2nd International Symposium onAmbisonics and Spherical Acoustics.Stokowy, R. and H. von Coler (2018).Coffee Rings (EOC Version). https://eo-charlottenburg.de/repertoire/coffee_rings/ (last access 27 May2023).Supper, M. (2001). “A few remarks onalgorithmic composition”. ComputerMusic Journal 25(1): 48–53.Wilson, S. (2008). “Spatial SwarmGranulation”. Proceedings of theInternational Computer MusicConference ICMC. Belfast, Ireland.45

Chaos in the Garden. Human-assisted AI Composition in Experimental Spatial Music

Margarethe, Maierhofer-Lischka

array2022Vertigo of the ears and eyes:Embodied experiences of hybridspatiality in Brice Pauset’s Ver-tigo/Infinite Screen (2021)By Margarethe Maierhofer-LischkaRecent research on musical perfor-mance highlights music listening asan embodied activity strongly influ-enced by and connected to mentalrepresentations of music-relatedmovement schemata (Cox 2016).Listening thus can be considered aninner “dance on memorizedschemata” (Stahnke 2001, 25) thatcombines inputs from hearing,touch, vision and bodily motion(Schroedter 2017). Especially com-posers of contemporary music chal-lenge their listeners by creatingcomplex tonal, temporal and spatialimpressions (Utz 2016, 602-609),a phenomenon which is broughtfurther by the increasing use ofelectronic audiovisual media. Beingimmersed into an audiovisual per-formance can be pleasurable, but itcan also make listeners feel painfulor sick. This article examines theperceptual phenomenon of vertigoin regard to musical intermedia per-formance, demonstrated by a briefanalysis of Vertigo/Infinite Screen(2021) by French composer BricespacePauset, an intermedia piece inhomage to Alfred Hitchcock’s iconicfilm Vertigo (1958). Many currentintermedia compositions call intoquestion the distinction between“live”and “mediatized”, physical andvirtual, resulting in a growing bodyof scholarship on musical multime-dia or intermedia performance (cf.Auslander 2008, Chapple & Katten-belt 2006). The following analysisshows how Vertigo/Infinite Screenplays with sensations of vertigo byblurring the boundaries betweenexperiences of real and virtual spacethrough sound and moving images.Vertigo describes disorders of thegravitational sense located in the in-ner ear, resulting in partial loss ofspatial orientation and movementcoordination. Those can be causedby turning motions of the body(motion-induced vertigo) or theycan appear as somatoformic vertigorelated to mental disorders. Somato-formic vertigo can result from pho-bias, depression and schizophrenia,as well as from the consumption ofmedia (Brandt et al. 2013, 146ff.).Vertigo as media sickness resultsfrom incoherent and excessive sen-sory stimuli that create sensory trau-mata resulting in somatic reactions8array2022of the nervous system. Recent stud-ies show that vertigo appears in asignificant number of people in cor-relation with experiences of virtual(VR) environments. It can be alsocaused by wearing noise-cancellingheadphones (Dan-Goor & Samra2012, Chattha et al. 2020). Virtualreality can be defined generally asa medially created environment thatemploys several modalities (vision,hearing, touch) to imitate or imple-ment sensory features, behaviorsand qualities of known physical en-vironments, thus allowing for under-standing and purposeful interaction(Doerner et al. 2014, 3ff.). FollowingEric Clarke (2013), a musical piececan also be considered a virtualspace, making listening an effortof balancing between the real andthe virtual. Sociologist ShuheiHoso-kawa had already describednegative effects when listeners’attention navigates between virtualand physical sonic environments,which he dubbed as the “walkmaneffect” (Hosokawa 1984). WhileHosokawa had referred to urbancontexts, Nicholas Cook later onsuggested three categories to de-scribe intermedial relations in per-formance: conformance, comple-mentation and contest (Cook 2001).spaceMedia-induced vertigo can be un-derstood as a result of states of con-test between hearing and vision aswell as between virtual and real-world experience.Hitchcock’s Vertigo is considered alandmark of suspense cinema. Thefilm tells the story of a policemanfalling in love with a woman whocommits suicide. In an attempt torelive his failed relationship, hestarts an affair with a young girlwhich also ends tragically. Vertigo isconsidered one of Hitchcock’s mainworks and has received consider-able attention in film studies for itsintricate visual language. The lossof perspective and orientation capti-vating the characters throughoutthe film is presented as visual forcethat destroys their attempts to re-gain individual agency (Cavalletti2022). Hitchcock used a special cam-era technique, the Dolly-Zoom (lateralso called Vertigo effect), to repre-sent vertigo by distortion of per-spective and visual space (Hasche &Ingwer 2016, 148f ). The iconic filmscore by Bernhard Herrmannpresents an auditory maze made ofconstantly looping motives, avoid-ing a tonal centre or formal direc-tion (Gelly 2021).9array2022 spacenotated as an own part in the score.The performance happens in a blackspace, rendering the orchestrabarely visible against the brightFig. 1: Vertigo/Infinite Screen, performance by Klangforum Wien 20211Vertigo/Infinite Screen screen waspremiered 2021 at the Wittener Tagefür Neue Kammermusik. It featuresan orchestra in six groups, electronicsounds and a visual installation.The composition follows closely theform and structure of the film, incor-porating elements of Herrmann’ssoundtrack as well as sound snip-pets from the film’s dialogue andfoley. The visual part of Vertigo/Infinite Screen is a video installationpresented on a giant projection wallmade up of a grid of 9x2 LED panels.The visuals are performed live onscreen. In the beginning, swirlingchords cross all instrumentalgroups. Sudden appearances offilm-stills create moments of incar-nated perception (Kaltenecker2008), followed by darkness andcolor flashes disturbing the sense oftime, body and space. Unlike in thefilm, visuals and music do not createa common narrative but two parallelsensory inputs (Auslander 2013).While Pauset’s music has nuanceddynamics, the visuals retain highcontrast, strong colours and highspeed.10array2022One scene in Hitchcock’s Vertigo fea-tures recorded music playing from agramophone. Hitchcock wanted therecord to sound fragile and blurred,almost hallucinogenic. In Vertigo/Infinite Screen, Pauset picks this upand lets fragments of the gramo-phone music appear in the elec-tronics (Pauset 2021, bar 42ff., 288ff.)These inserts are a key element forcreating sound-related impressionsof vertigo: they are clearly tonaland melodic, marking a shift fromthe harmonic and temporal disori-entation happening in the begin-ning section towards a momentof recognition of known elements(Schroedter 2017, 225). Throughoutthe piece, the inserts becomelonger, dislocating and warping theorchestral sounds until the spatial-ized electronics sometimes takelead over the acoustic instruments.The recognizable acoustic and vis-ual features of the physical room areconstantly blurred and overwrittenby the fast-changing flashes of lightand dark and the changing sonicspace created by the electronics.When entering the performance,the listeners arrive with a perceptionof the physical space that gets sub-sequently confused by the virtualaudiovisual environment whichspaceremains inconsistent and fluid, in-troducing slips of perceptual disori-entation, a “Tasten und Zögern”(Waldenfels 2007, 19). Like the“walkman effect”, a subtle estrange-ment happens between the listen-ers’proprioception and the impres-sions of the virtual space. Through-out the piece, electronic sound andvisuals are used in conformance – tospeak with Cook – to create a situa-tion of sensory overload, competingfor attention. The fast-changing vis-ual and auditory spatialities chal-lenge the perceptual capacity of theaudience, stimulating embodiedexperiences similar to VR-inducedmotion sickness. Unlike artists whointended their piece to pose a cri-tique against contemporary mediaaccesses, the actual use of audiovi-sual media in Vertigo/Infinite Screenappears like a “psychic apparatus”, aterm Friedrich Kittler (1995, 352ff.)introduced to describe electronicmedia’s potential to modulate thepsychic states of an audience.Vertigo/Infinite Screen is not only abare retelling of Hitchcock’s movie.It uses the unsettling potential ofexcessive visible and audible move-ment dynamics to transfer the film’svisual effects into the sonic and spa-tial domain. It draws from embodied11array2022listening to place motion sickness,or vertigo, at the heart of an intensespatial and audio-visual experience.Notes[1] Brice Pauset / Arotin & Serghei:Vertigo/Infinite Screen (2020-21),intermedial composition based onAlfred Hitchock‘s “Vertigo” for en-semble in 6 groups, 18 pictures andelectronics. Foto from a perfor-mance played by Klangforum Wien,Centre Pompidou, Paris 2021; Photo:AROTIN & SERGHEI ContemporaryArt 2021.ReferencesAuslander, P. (2008). Liveness:Performance in a Mediatized Culture,Routledge.-ibid. (2013). “Sound and Vision: Theaudiovisual economy of musicalperformance”, in: Richardson, J. et al.(eds.). The Oxford Handbook of NewAudiovisual Aesthetics, Oxford Univ.Press: 605–621.Pauset, B. (2021), Vertigo/InfiniteScreen, Edition Gravis.Cavalletti, A. (2022). Vertigo: TheTemptation of Identity, FordhamUniversity Press.spaceChapple, F. & Kattenbelt, C. (2006).Intermediality in Theatre andPerformance, Rodopi: 11-25.Chattha, Umer Asghar a.o. (2020)“Motion Sickness in Virtual Reality:An Empirical Evaluation”, IEEE Access8: 130486–130499.Clarke, E. F. (2013). “Music, space andsubjectivity”, in: Born, G. (ed.). Music,Sound and Space. Transformations ofPublic and Private Experience,Cambridge Univ. Press: 91–110.Cook, N. (2001). Analysing MusicalMultimedia, Oxford University Press.Cox, A. (2016). Music and embodiedcognition: listening, moving, feeling,and thinking, Musical meaning andinterpretation, Indiana Univ. Press.Dan-Goor, E. and Samra, M. (2012).“Benign paroxysmal positionalvertigo after use of noise-cancelingheadphones”, American Journal ofOtolaryngology 33/3: 364–366.Gelly, C. (2021). “Music, Memory andRepression in Hitchcock’s Vertigo(1958)”, Miranda. Revue Pluridisci-plinaire Du Monde Anglophone 22,https://journals.openedition.org/miranda/36484 (last access Feb. 16,2022).12array2022Hasche, E. and Ingwer, P. (2016).Game of Colors: Moderne Bewegt-bildproduktion: Theorie und Praxis fürFilm, Video und Fernsehen, Springer.Hosokawa, S. (1984). “The WalkmanEffect”, Popular Music 4: 165–180.Kaltenecker, M. (2008). “Subtraktionund Inkarnation. Hören und Sehenin der Klangkunst und der musiqueconcréte instrumentale”, in: Musik alsWahrnehmungskunst. Untersuchun-gen zur Kompositionsmethodik undHörästhetik bei Helmut Lachenmann,Pfau: 101–126.Kittler, F. (1995). Aufschreibesysteme1800-1900, Fink.Stahnke, M. (2001). “InfiniteMeloharmonik: Vermutungen überden Wind”, in: Stahmer, K. (ed.). NeueMusik 2000, Königshausen & Neu-mann: 19-26.Schroedter, S. (2017). “Musik erlebenund verstehen durch Bewegung.Zur Körperlichkeit des Klanglichenin Choreografie und Performance”,in: Musik und Körper. InterdisziplinäreDialoge zum körperlichen Erleben undVerstehen von Musik, transcript:221–243.Utz, C. (2016). “Wahrnehmung”, in:Hiekel, J. P. and Utz, C. (eds.). Lexikonneue Musik, Metzler: 602–609.spaceWaldenfels, B. (2007). “Sichbewegen”, in: Brandstetter, G. andWulf, C. (eds.). Tanz alsAnthropologie, Fink: 14–30.13

Vertigo of the ears and eyes

Akkermann, Miriam

array2022EditorialThe concept of space appears has al-ways been a pivotal aspect to music.The emergence of spatial thinkingcan already be seen in the Venetianpolychoral style (cori spezzati) of the16th century when multiple choirswhere spatially displaced at St.Mark’s Cathedral.Today, we have expanded andmaybe even removed the notion ofspatial confinement with the aid oftechnology. Physical and architec-tural space as well as displacement(seemingly) has no longer the samemeaning as it did to church com-posers of 16th century Venice.However, with the complex combi-nation of real and digital spacesarise new challenges which consoli-date in a huge variety of aspectstaking influence on a musical piece.Some of these concepts of space arerather inherent such as the (studio)space music is created in, the con-ceptual space in the composer’smind, the space audible in a soundsample, and the audible but virtualspace within a musical work createdby a composer as part of an artisticvision; other aspects are moreexplicit such as the sonic and archi-spacetectural features of the space a mu-sical work is performed in, as wellas the design of the presentationwithin a performance space witheventually specific seating of theaudience and (technically adjusted)spatialization of appearing soundswithin this space. An interplay ofthese aspects influences then howa musical piece is created, how itscompositional structure and therequired performance environmentand technical set-up looks like,which strategies for documentationand preservation apply, and how wetalk about a composition.This issue of Array presents howcomposers and sound artist consoli-date these aspects in their works,how concepts of space can beframed from an analytical perspec-tive, and what it means to createartificial (virtual) spaces to presentcurrent practices and technicaldiscourses in hybrid settings suchas an ICMC.A special thank goes to MartinRitter, who initiated and outlinedthis wonderful topic.Miriam Akkermann (Editor)4

Editorial

Giuseppe, Pisano

array2022Stereo and Ambisonics: A reflec-tion over parallel spatializationtechniques in the piece Scie Lumi-nosissimeBy Giuseppe PisanoAs a composer of acousmatic music,I have often presented my work on avariety of different loudspeaker sys-tems whese can be subdivided intwo macro-families: those intendedfor the playback of multichannelpieces – often symmetric systems,in the shape of a circle, a dome, ora regular polygon – and those in-tended for acousmatic diffusions –the loudspeaker orchestras, mainlyemployed for the diffusion of stereopieces, where the performer spatial-izes sound in real time. In this prac-tice, it is not uncommon to listen tore-adaptations of pieces, presentedin different formats to suit differenttypes of systems, and I have alwaysfound the Ambisonics workflow avery successful approach to scalemy pieces on systems belonging tothe first macro-family. However,when I first faced the need of pre-senting my multichannel pieces onsystems meant for stereo diffusion, Iwas very unhappy with the results Iachieved through the typical meth-ods employed to adapt Ambisonicsspaceto stereo, such as rendering mypieces in Ambisonics UHJ format, orusing virtual microphones to recordthe sound field in stereo. Based onthis experience, I decided to com-pose a piece in two versions in par-allel, for stereo-setting and Am-bisonics.This process initiated a reflectionprocess on how I would managesonic materials differently accordingto the format to emphasize theircharacteristics and has become anacquired strategy in my composi-tional practice.The first piece I composed in parallelas a stereo- and Ambisonics-versionwas Scie Luminosissime (2018-2020)1.In this paper I outline the composi-tional process of this piece, andhighlight some conceptual differ-ences that determined my choicesin the use of the materials and theapproach to spatialization.Scie Luminosissime was inspired bythe album Fratello Mare by MikeCooper and the poem Sii dolce conme. Sii gentile by MariangelaGualtieri. This inspiration is rather in-direct but provides the setting inwhich I wanted to compose.The materials used in Scie Luminosis-sime include field recordings ob-14array2022tained during the summers of 2016and 2017 – mostly fragments of hu-man agencies in different reverber-ating spaces – in dialogue with con-crete and synthesized sounds – thescie, usually distorted and/or convo-luted trails of synthesized soundtones – and a few bits of sound sam-pled from different guitar pieces byMike Cooper. My goal was to gener-ate momentary and elusive visual-izations of human interactions in anotherwise diffused blur of gesturesand fragments.The time-structure of the piece canbe divided in two parts, each subdi-vided in two sections.spaceThe first section is mainly character-ized by simple tones (1) to whichother concrete elements are gradu-ally added, occupying wider por-tions of the spectrum (2). The cumu-lation ends with a gesture (3) thatopens a new section and reiterateslater (4) with a different organiza-tion of the materials. The idea was tocreate a new perspective, as if thesame picture was seen from a differ-ent angle. After this, the space dissi-pates fading back into luminoustrails again.In his article “Space-form and theAcousmatic image”, Denis SmalleyFigure 1: spectrogram of Scie Luminosissime, the numbers highlight the foursections of the piece.15array2022(2007) describes a soundscape helistens to sitting by his window. Heis inside, and the whole world is out-side, beyond the fourth wall. Theidea of being a spectator of soundfrom a vantage point reinforcesonce again the parallel betweenacousmatic music and cinemawhich has been structural in the dis-course originated in the French liter-ary and musical school (Chion 1990;Normandeau 1992). This connectionis even more important now thatthe expansion of sound from stereoto more immersive formats is be-coming a widely available technol-ogy, especially through the filmindustry and other audio-visualmedia, and demonstrates howSmalley’s front-based approach tothe description of the soundscapein the article is in reality much moretechno-centric than openly in-tended.In Smalley’s narration, the window isthe point from which the soundradiates in the listening space, it isboth an access to the external sonicenvironment and a filter thatfacilitates the localization of thedifferent sound sources.When we listen to music in stereo,loudspeakers behave in a very simi-spacelar way as this window. They definea front from which sounds radiatesin our listening space, a two-dimen-sional surface on which a three-dimensional illusion is representedas in a trompe-l’oeil (more a trompe-l’oreille in our case) that uses per-spective to generate depth. Thischanges radically when we are sur-rounded by loudspeakers. We donot sit by a window, we sit under a“glass dome”, we can still have afrontal perspective, but our percep-tion responds to stimuli comingfrom every direction. In this situa-tion, music is then not merely pro-jected in the listening space, but itbecomes the listening space. Thelistener is in the sound field, im-mersed, and the sound is an archi-tecture.Materials in an immersive sound-field context are hierarchically orga-nized in a figure/ground relation-ship. This gestalt notion, that hasbeen applied before to many otherfields such as soundscape analysis(Porteous and Mastin 1985) andacousmatic music (Santarcangeloand Wanke 2020), is of course stillrelevant for immersive formatswhere music-space and listeningspace overlap. A good examplemight be the nomenclature used to16array2022distinguish between the two typesof sound sources in Dolby Atmos:Beds and Objects. Beds are multi-channel sources, meant to establisha scene, while Objects are monosources that would benefit from amore precise positioning and frompanning.2 This distinction is not justpractical or operational, but reflectsa difference in the perceptual andmusical roles of the sound materials,which can also be applied to otherimmersive formats. When compos-ing in Ambisonics, authors canadopt a similar approach. Somesounds can be treated as structuralpillars of a scene, establishing theenvironment. Other materials canhave more prominent roles, not onlyin terms of levels and clarity, butalso in their spatial behavior withfaster trajectories, swift movements,dramatic panning. However, the ab-sence of a structural distinction be-tween sources and their roles allowsfor more nuanced musical solutionsand alterations in the roles of thematerials.An example of how I apply this para-digm is at the beginning of Scie Lu-minosissime, where, in the begin-ning, every single tone is a fixed po-sition, like in a constellation, wherespaceevery star has its own place. Thesebecome spatial references, describ-ing and defining the sonic environ-ment with their presence.As the piece develops, more reso-nances add up together with similarelements of different timbres, mov-ing slightly or appearing in oppositepositions to the sound they origi-nate from. These new materials gen-erate a movement around the origi-nal structure, and every new tone isintroduced with similar copies of it-self, each presenting slightly differ-ent characteristics. The original in-stances are still fixed in space, assolid statements of presence. Theother ones emerge from the back-ground, creating a shapeshifting en-vironment.In the stereo version of the piece,this effect was achieved by deter-mining a position for each originalsource on the horizontal plane,choosing to leave low frequencysounds in the middle and to enlargethe picture progressively at higherfrequencies. Resonances were ob-tained with a stereo reverb, invert-ing channels after a low pass filter.In the Ambisonics version, I used adifferent technique encoding theoriginal mono sources with different17array2022instances of IEM MultiEncoder,3 cre-ating new copies of some of themto fill the entire space.For the additional sounds emerging,I decided to create small alterationsin their position modulating the val-ues for the coordinates with a ran-dom LFO at a very small amplitude,which generated an organic liveli-ness.Lower frequency sounds and reso-nances which I wanted to soundbigger and further away were en-coded using the AmbiX suite byMatthias Kronlachner (2013). Thissoftware presents an additionalparameter called size that can beused to adjust spatial directivity, ad-justing the ratio between the levelsof the omnidirectional signal andthose of the other spherical har-monic components. At higher val-ues of size, the sound sources arespatialized as more omnidirectional,equally distributing the sound to allloudspeaker, making them appearbigger, but also less directional. Ad-justing this parameter, I was able tospatialize sounds with a weakersense of directionality which playeda different role.spaceGestural sounds which, in the stereoversion of the piece, were spatial-ized using traditional linear pan-ning, in Ambisonics expand theirmovement to the second and thirddimensions, swooshing from oneside to the other of the sphere.These movements are achieved us-ing crossfades between two en-coded monophonic sources, and arepresented over multiple diameters,to increase the sense of tridimen-sionality.Another important difference be-tween 3D Ambisonics and stereo isthe introduction of verticality. How-ever, while real height can only befound in three-dimensional soundformats, different techniques can beemployed in stereo to generate aperceptual illusion of height. DenisSmalley provides some good exam-ples for this when introducing theconcept of gravitation, referring tothe work of French composers suchas François Bayle, François Dela-lande, and in particular to BernardParmegiani and his piece GeologieSonore (Kronlachner 2013). In hiswritings, Smalley addresses heightfrom a spectromorphological stand-point, claiming that the illusion ofvertical movement is based on the18array2022quality and the behavior of thesonic materials. A psychoacousticapproach to the production of verti-cal panning illusions is provided byCurtis Roads in the book ComputerMusic Tutorial where he stresses therole of the pinnae in sound localiza-tion on the vertical plane. Showingthe Head Related Transfer Functionspectra of sounds coming fromabove and below ear level, he de-scribes the functioning of verticalpanning (Roads 1996).In Ambisonics, I found it very usefulto spatialize sounds over the verticalplane trying to imitate or to referthe behavior and spectral character-istics of sounds that I am most likelyto perceive above or below ear levelin real life. For example, there aresounds of birds or sounds that “be-have like”birds, there is the sound ofrain and a sound that “behaves like”rain. Exploiting the vertical plain tospatialize sounds can provide an ex-tra layer of realism to unreal sounds.Similar effects are much more diffi-cult to achieve in stereo. As pointedout by Roads (1996, 10), realistic ver-tical panning illusions depend ontoo many factors such as speakerplacement and listener’s position,and are therefore extremely situa-spacetional and very fragile. But Roadsalso describes that sound localiza-tion on the vertical plane is strongerfor sounds above 6 kHz. Keepingthis in mind, I found it useful to splitsounds with very complex spectraover different frequency bands us-ing a set of EQs, and spatialize thesebands individually. Increasing thewidth of high frequency compo-nents did not provide any clear illu-sion of a vertical panning move-ment, but it widened the stereo im-age, giving more air and a strongersense of openness to the sound im-age. I found it also useful to com-bine this frequency-based approachto the material- and spectromor-phology-based approach describedby Smalley, and to see them as twosides of the same coin whichtogether contribute to create anillusion of height in a stereo mix.Spatializing filtered components ofsound sources is a technique thatwas used extensively in Scie Lumi-nosissime, especially for granularmaterials, where multiple mono-phonic sources are processed usingthe GRM Space Grain4 a multi-chan-nel granulator whose resulting sig-nals are then filtered and encodedusing multiple IEM Multi-Encoderwith some personal presets.19array2022Scie Luminosissime was the firstpiece I composed using this parallelmethod, which I later adopted forother works, too. I found this com-positional process of creating twoversions of the same piece very use-ful to thoroughly exploit thestrengths of both formats. The ver-sions share the same materials, thesame structure and the same dura-tion, but manifest themselves ani-mated by very different spirits.While the stereo version appearsmore compact, perhaps with astronger body, more punch and aconcrete foundation, the Ambison-ics version is more enveloping andrelaxed: gestures are perhaps lessdramatic, but more realistic.When working in Ambisonics, myfeeling was that certain sounds didnot require the same dynamic pres-ence that I was using in stereo. In-stead, they could interpret their rolein the narrative development of thepiece much more successfully withjust a different organization of thespace.Another relevant aspect is how therelationship between full and emptyspace was managed differently inthe two versions of the piece. Fillingspace in Ambisonics simply requires“more sound”, and adding multiplespacevoices of a sound material, if cor-rectly positioned, will rarely resultsin the space feeling crowded. It wastherefore necessary to double somesound sources in the Ambisonicsversion or to cut out few others inthe stereo version to balance thesense of fullness and emptiness indifferent moments of the piece tomake sure that the same impressionwas achieved.In conclusion I think that my ap-proach to the composition of thispiece has perfectly filled the pur-pose of making the two versionsperfectly functional to the two dif-ferent types of loudspeaker systemsand performance styles. Expandingthis workflow to other pieces hasmade possible to present the en-tirety of my work in all possible con-texts.Notes[1] www.giuseppepisano.com/fixed-media/ (last access Feb. 16, 2023).[2] Dolby Atmos Renderer v3.0.Dolby Lab. Inc, 2 August 2018, 116.[3] IEM – Plug-in description“multiencoder”, 2022.https://plugins.iem.at/docs/plugindescriptions/#multiencoder(last access Feb. 16, 2023).20array2022 space[4] InaGRM – Spaces, 2022. https://inagrm.com/en/store/product/15/spaces (last access Feb. 16, 2023).ReferencesSmalley, D. (2007). "Space-Form andthe Acousmatic Image." Organisedsound 12(1): 35-58.Chion, M. (1990). L’audio-Vision. SonEt Image Au Cinéma. Nathan, Paris.Normandeau, R. (1992). "Un CinemaPour L’oreille." Doctoral dissertation,Montreal: Universite de Montreal.Porteous, J. D., and Mastin, J. F.(1985). "Soundscape", Journal ofArchitectural and Planning Research:169-86.Santarcangelo, V. and Wanke, R.(2020). "The Early Stage ofPerception of Contemporary ArtMusic: A Matter of Time." OrganisedSound 25 (2): 130-41.Kronlachner, M. (2013). "AmbisonicsPlug-in Suite for Production andPerformance Usage." Paperpresented at the Linux AudioConference.Roads, C. (1996). “SoundSpatialization and Reverberation”. Inidem, The Computer Music Tutorial.MIT press: 466-469.21

Stereo and Ambisonics: A reflection over parallel spatialization techniques

Huang, Chieh

Indigeneity and Computer Music in the Anthropocene

Jarema, Andy

Currents: AWeather Data Sonifica-tion ProjectBy Andy JaremaCommunicating the existentialthreat posed by climate change isessential to achieving a courageousleadership and strong policychanges needed to avoid the mostcatastrophic possible futures instore for our planet. Composers andother artists have much to offer inthe conversation about our shiftingplanet through the power of artisticstorytelling mediums. Making anaudience feel scientific data cankeep the conversation afloat aboutthe urgency of climate change andthe ominous shifts in weather pat-terns on our planet.Currents is a multifaceted andongoing musical work that centersaround two guiding questions: Howcan we use sonification to musicallydepict weather and climate data,and how can this musical narrativeaffect how we feel about the data?Specifically, Currents sonifies a data-base comprising forty years of datacreated by the National Oceanic andAtmospheric Administration (NOAA)that tracks Billion-DollarWeatherand Climate Disasters in the UnitedStates. In this article, I discuss myfirst iteration of the piece, providinginsight into the sonification processof Currents, including technical chal-lenges in the sonification process,how the data elements were soni-fied, and important questions thatemerged when translating big datainto a creative musical work.CurrentsInformation about climate changetraditionally relies on communicat-ing rigorous evidence from the sci-entific community: facts, figures,graphs, and data. But truthfully, Ihave only personally been moved tochange by artistic works. It was afilm that first lit my passion to fightclimate change (An InconvenientTruth, a documentary by DavisGuggenheim from 2006), and an au-thor that convinced me to take mymost potent individual climate ac-tion by eating no animal products(Jonathan Safran Foer’sWe Are TheWeather: Saving the Planet Begins atBreakfast). As a composer, I find my-self wondering what we artists haveto offer in terms of making peoplefeel something different about thealarming climate data presented tous by the scientific community.52array2023 fluxCurrents is a fixed audio-visual workcreated in Max/MSP that covers allforty years of data (1980-2019) inroughly eight minutes.1 It utilizesvisual graphs, field recordings, andsynthesized sounds to provide alofty overview of the entire datasetand gives the numbers a more com-pelling artistic narrative.I want to begin by outlining theinitial challenging steps of workingwith the data used for creatingCurrents. I hope that this will helpilluminate the issues with big datasonification other composers mightface in the future. Max/MSP was thesoftware I chose for the data sonifi-cation process due to my familiaritywith the programming language,however my technological abilitiesquickly had to expand to accommo-date the multiple moving partsneeded to access and sort such alarge web-based data set. The com-position’s Max patch utilizes a fea-ture introduced in Max 8 known as“Node.js,”which is a Javascript-based method of communicatingwith web-based resources. “Node.js”assisted with accessing a JSON datafile of the Billion-DollarWeather andClimate Disasters database throughthe NOAA website, including sortingand parsing the data into three col-umns of information: number of bil-lion-dollar weather events per year,the cost of each individual billion-dollar weather event, and the num-ber of deaths that occurred witheach event. From there, a “coll”ob-ject in Max was used to store andexport the three columns of data asthree separate .txt files to my hard-drive. By reimporting the .txt filesback into three separate “coll”ob-jects, Max was able to travelthrough the data and create the ba-sis for translating these numbersinto sonic information.I want to stress that accessing andparsing this data was non-intuitive,oblique, and challenging. It was afrustrating monthslong endeavorfilled with failed experiments, des-perate shifting through YouTube tu-torials, and re-tooling existing Javascript code designed for other pur-poses entirely. Smaller data setsmight be able to be fed into an au-dio program like Max/MSP ratherquickly, but there seems to be nosingle method of sonifying largedata sets without the use of multi-ple programming techniques (Max/MSP, Node.js, Javascript, JSON files)which creates a high barrier of entryinto this process for the average53array2023 fluxcomposer.After accessing and parsing thedata, Max/MSP was used to sonifythe weather data using field record-ings and generative FM synthesizedsounds. The field recordings used inthis piece were recorded with myZoom H4 recorder in Great SmokyMountains National Park (crickets,birds) and Hawai’i Volcanoes Na-tional Park (hydrophone recordingof the ocean). The three sets of natu-ral sounds (crickets, birds, ocean)represent three different environ-ments (land, sky, and sea), function-ing as an abstract barometer that al-lows us to check in and take read-ings on the health of three distantbiomes on our planet.The top graph tracks the numberof billion dollar weather events thatoccur in each year (40 data points)and is sonically depicted with crick-ets (center pan). There is an inversepitch relationship between thesound and the data, where a highernumber of weather events corre-lates to a lower shift in the pitch ofthe cricket recording. This was pri-marily an artistic choice given thatthe cricket recording takes on analarm bell-like quality at lower pitchlevels. The middle graph tracks thecost of each individual weatherevent (258 data points) and is soni-cally depicted with a crackling un-derwater hydrophone recording ofthe ocean (right pan). A higher costtriggers a longer articulation (enve-lope release) of the sound, and sig-nificantly high-cost events over $20billion trigger the sound of loudocean waves. The bottom graphtracks the deaths caused by each in-dividual weather event (258 datapoints) and is sonically depictedwith a bird call (left pan). There isalso an inverse pitch relationshipsimilar to the cricket sounds (ahigher death toll triggers a ghostly,lower pitched bird call) and signifi-cantly fatal events over 100 deathstrigger the sound of chickadees(chickadees use an intricate lan-guage of bird calls as a way tobroadcast perceived threat levels inits environment).In addition to the three sonified ele-ments operating at a micro level inthe piece, there are sonic elementsthat function on a much larger timescale to convey a broader narrativein the data. Taken as a whole, theforty year data set outlines an over-all rising trend of billion dollarweather events. To artistically reflectthe urgency of this narrative, certain54array2023 fluxmusical elements such as tempo (agradual accelerando over time) andtimbre (stark FM synthesized soundslayering in over the field recordings)unfurl over the eight minutes timespan of the piece. Approximatelyhalf way through the piece, anotherhydrophone recording sneaks intothe right speaker. The recordingslowly begins creeping to fill the leftspeaker, dropping in pitch as it goesalong. The recording wildly in-creases in pitch and speed at theend of the piece (the year 2020), fur-ther signifying urgency as the soundburns and crackles out of control.One question that frequentlyemerged in the sonification processof Currentswas how to address thetension between data accuracy andcreative artistry. Being faithful to thedata in order to tell a scientificallysound narrative was a priority, buton the other hand I found myselftaking creative licenses in the sakeof creating a compelling musicalnarrative. One of these challengeswas figuring out how to scale andcalibrate the data in the sonic do-main. For example, the number ofevents per year was easy to sonifyinto the pitch domain due to its rela-tively limited range of data (therewere 0 to 16 events, which was soni-fied into a two octave range ofnotes from C2 to C4). But the cost55array2023 fluxFigure 1: still taken from Currents to demonstrate these visual graphs andsonified elementsprovided a unique challenge sincethis data mostly operated in a nar-row band (costs for each weatherevent hovered mostly between oneand 20 billion) with occasional out-liers stretching the upper bounds ofthe data (Hurricane Katrina’s 192 bil-lion being the highest). If this datawas sonified directly into the pitchdomain, it resulted in a flat pitchoutput with mostly imperceptiblepitch changes (semitones or tones)with occasional jumps to a highernote when an outlier occurred. Onthe other hand, adjusting my sonifi-cation to pick up the nuance insmaller values failed to sonicallycapture the magnitude of the out-liers when they occurred (in somecases, the outlier was so high it pro-duced a pitch outside the range ofhuman hearing). To fix this, I created“parameter bands” (Figure 2) thatwould sonify different ranges ofdata to different ranges of pitches:1-10 billion was sonified within aone octave range (C4 to C5), 10-100billion in another octave (C5 to C6),and the upper outliers between100-167 in a final octave (C6 to C7).These types of decisions throughoutthe sonification process of Currentsforced me to confront the bound-aries between scientific accuracyand creative artistry, ultimatelyhelping to shape my narrative inten-tions and clarify the messaging be-hind the piece.Another question that emerged inthe sonification process was how toutilize big data in an ethical way. Avalid critique of Currents is that thedata points scrolling across thescreen can feel objective, imper-sonal, and disconnected from indi-vidual experience. I myself was apart of one of these extremeweather events (2014 Detroit flood-ing) and seeing the data point slipso quickly across the screen some-what diminishes my harrowing rec-ollection of that day. It especiallymakes me uncomfortable to see thedata points for deaths, real humanlives being lost, ticking away acrossthe screen with an almost impas-sionate calculus. It reinforces the56array2023 fluxFigure 2:ParamterBandsnarrative that the problem of cli-mate change feels too massive, itscomplexities and horrors beyondany sort of action that can possiblyreverse its course. If the objective ofclimate-related art is to emotionallymove people to awareness (andtherefore- action), I sometimes won-der if using such a large data set isalways the most potent approach.This is where Currents blossoms intonot just one iteration, but an overar-ching weather data sonificationproject. Shortly after the premiere ofthis fixed audio-visual iteration atthe 2020 Earth Day Art Model Festi-val, I began working on the nextchapter of the Currents project. Asopposed to a fixed audio-visualpiece sweeping through mountainsof data, the next iteration of Currentszoomed in on seven specificweather events out of the 258 cata-logued in the database. I craftednotated sheet music shaped by datasonification techniques and enlistedseven wind/percussion players toperform these short solo works withlive electronic processing via Max/MSP. For example, the aforemen-tioned 2014 Detroit flooding wasoutlined by a percussion player us-ing five different instruments to de-pict the five inches that fell on De-troit in roughly five hours. Usinghourly precipitation amounts fromtheWeather Underground, Imapped out the rate of rainfall andprojected this timeline overtop the31 measures of the piece. The per-cussion player adds a new instru-ment into the mix after each newinch of rain falls, leading to an even-tual flood of five instrument soundsbolstered by Max’s live processinginto back-masked“waves”of percus-sion sound. This iteration of Currentsfelt so much more personal: the de-piction of individual weather events,a community of live musicians tan-gibly engaging with the music, theinclusion of a single weather eventthat personally affected me and mycommunity. Though, related to theprevious question of data accuracyversus creative artistry, this projecttook many more artistic libertiesand did not have the same rigorousadherence to the numbers that thefirst audio-visual iteration did.To me, Currents functions as a blue-print, an ever evolving set of worksthat strives to make artistic narra-tives out of scientific data. At itsheart it is a project concerned withcommunicating an artistic narrative57array2023 fluxabout climate change, and I hope tocontinue to adapt the project intoradically new mediums to reach asmany audiences as possible. Per-haps it could be adapted into sheetmusic for young beginning bandplayers and include a curricular ele-ment that could facilitate climatechange discussion in the classroom.Or perhaps it could function as avideo game, an interactive fictionpiece using the programming lan-guage Twine that combines writtenand sonic narratives in a way thatimplicates the audience into itsprocess (video games demand theplayer’s participation for the genesisof the story to occur). I believe we asartists have so much to offer in theconversation about our shiftingplanet through the power of artisticstorytelling mediums, and makingan audience feel something differ-ent then scientific data can show,keeping the conversation afloatabout the urgency of climatechange and the ominous shifts inweather patterns on our planet.Notes[1] Recording of Currentswww.youtube.com/watch?v=97J1CG88UV0 (last access Nov 11, 2023).58array2023 flux

Currents: A Weather Data Sonification Project

array2022Music is in the air. Sounding per-formances in hybrid and virtualspaceBy Miriam AkkermannWhile physical space is fundamentalto any sound's (physical) being –and thus being heard – advance-ments in technology and COVIDpandemic-related limitations tophysical travel and meeting in largercrowds prompted debate on how todesign hybrid and virtual spaces inwhich music and sound art can beperformed adequately. The questionof how to make music togetherwhile being located at distantplaces, as well as issues concerningthe integration of a wide-spread au-dience using telecommunicationtechnologies, is, however, neithercompletely new nor limited to digi-tal virtuality. Currently termed as‘telematic’and ‘networked’art worksand performances, there exists aquite long history of using distrib-uted sounds and sound related in-formation in order to create artisticsettings and performances. For ex-ample, listening to live music perfor-mances or entertainment programsfrom a distance was already possiblein the transition to the 20th century.Facilitated by Electrophone tele-spacephone broadcast services (which arebest documented in the UK andFrance), listeners could dial in andlisten to requested performances(theatre plays, music performances,etc.) via telephone by means ofheadphones. This was possible bothalone or in groups, the latter en-abled by multi-headset tables(“Electrophone tables”) at home andin public, for example in commercialElectrophone salons as a ‘pay as yougo’ service. This service was suc-ceeded by radio broadcasts whichwere less expensive for the audi-ence and more easy to receive.While telephone lines and most ofthe emerging television programsformats aimed at a single directed(sending) broadcast, Simon Emmer-son emphasizes the role of the radiowhich “could be turned into a bi- ormulti-directional instrument, a net-work of connections that could facil-itate a telematic exchange betweenall participants” (Emmerson 2000:181). As an example, Emmersonmentions Horizontal Radio (1995)and Rivers and Bridges (1996), twoprojects organized by ORF-Kunstra-dio which explore the radio’s capac-ity for global networking by bring-ing remotely situated sound artists,engineers and producers in contact46array2022and interaction.The idea of a real-time interactionbetween distant partitioners wasalso explored using the emergingdigital technologies towards theend of the 20th century. Startingwith single connection lines, it wasdue to Georg Hajdu in 1991/92 thatRichard Teitelbaum could partici-pate in the anniversary concert ofthe composers’collective MusicaElettronica Viva at Mills Collegewhile being stuck in his car due to ablizzard using a modem connection“that would enable him to remotelyplay a MIDI keyboard while listeningto the performance over the tele-phone” (Hajdu 2012: 2). The devel-opments of the early 1990s were fol-lowed by a decent number of wide-spread, artistically and technicallydiverse approaches which can besubsumed under the terms ‘Netz-musik‘ (Föllmer 2005), ‘internetmusic’or ‘internet-based musicnetworks’ (Manning 2004), and‘computer network music’(Gresham-Lancaster 2013; for anoverview until 2008 see also Akker-mann 2014), while other works ap-peared in the context of multi-mediaart and web-based (fine) arts, soundart, or simply as music perfor-mances (Wahl 2013). Not all of thesespaceartworks deliberately work on spaceas a fundamental, but all involve tosome extent the creative examina-tion and handling of musical mate-rial at geographically distant loca-tions which are meant to share ide-ally synchronous sound at differentlocations, whether by networkeddistribution or via a shared virtualspace.Together apart in the same(virtual) spaceWith the possibility of being inde-pendent from physical (or architec-tural) space, it becomes obvious –even though not explicitly ad-dressed – how strongly the aspectof space relates to the created musi-cal work. Being able to play or per-ceive the same sound togetherwithout the limitations of physicalspace opens up new perspectiveson artistic processes in sound cre-ation and its performance. Hence,the technologies that enable real-time and offline transmitting and re-ceiving of data including sound andvisuals seem to lead to a situationwhere temporal and geographicboundaries start to dissolve. Thisraises new questions for artistic cre-ations: How can or should the com-47array2022plex (technical and artistic) settingbe integrated in a composition’soutline? How can sounds, musicians,interfaces, audience, responses, etc.be (re)present(ed) at distant loca-tions? What does it mean to create ashared moment in a virtual perfor-mance? How do artistic works dealwith the relationship of physical andvirtual spaces? And how does thisinfluence the way an artwork can bedocumented and analyzed?A challenge that emerges for telem-atic performances is, following Em-merson in 2000, that it is difficult tounderstand what “happens”when‘just’ following a performance – theinteraction can seem incomprehen-sible for the audience (Emmerson2000, 186).Emmerson claims that“[t]elematic work is an offer for ex-change. Artists and businesses alikewill have to come to terms with thesenew technological and cultural facts[… ]  [suggesting] changes in soundart creation and perception that gobeyond modernist concepts of au-thorship, owner-ship, artistic self-ex-pression through the production ofunique works and ‘truthful’percep-tion. They advocate new ways of artmaking – not defined by work butprocess; not through the polarisationof creator and receiver, but ratherspacethrough communication and interac-tion between them.” (Emmerson2000, 187).In 2000, Randall Packer and SteveBradley presented Telemusic#1, acollaborative performance in whichPacker and Bradley mention sevendesigners and two studios as co-creators. In their introduction to thepremiere they explained that„the demarcation between physicaland virtual space, between on-lineand local proximity, between the selfand the network, converges andblurs into a shared, participatory ex-perience through sound and our at-tention to its spatial and transforma-tional qualities.” (Packer and Bradley2000)Now, in the early 2020s, technicaldevelopments such as real-time on-line platforms for concerts and mu-sic festivals, interactive virtual spa-ces in 2D and 3D including VirtualReality and Augmented Realitytechnologies, online platforms forcollaborative music making in real-time and offline as well as a varietyof streaming and receiving optionsfor audio-visual contents exist (seee.g. Cope 2001; Lazzaro 2001; Hugill2005; Duckworth 2013; Serafin et al.2016; 2017; Hamilton 2019). Mean-while, some of the basic questionsare still in negotiation; for example:48array2022How to deal with shared author-and ownerships in a multi-dimen-sional setting? What does a musicalor sound related work look like thatis genuinely made for virtual spaceand takes all the advantages of thissetting? How does virtual spacechange the roles of audience, com-posers and musicians when a per-formance is intended to be accessi-ble (also) via telecommunication de-vices – online and offline?Settings and spaces side by sideIt seems especially clear that newdigitally enabled virtual spacespresent major changes concerningartistic, technical, and social aspects,ranging from creating musicalworks dedicated to this very specificperformance environment, estab-lishing performances in virtualspace, and providing experiencesfor the audience that include inter-action and involvement as intendedin physical settings. This goes handin hand with new technological de-velopments including virtual spa-tialization of sounds and their repre-sentations, leading to another newchallenge: While invisible connec-tions enable the audience to ex-plore and eventually interact withspacesounds from all over the world, thelistening situation (as well as the au-dio-visual representation at the mo-ment of perceiving a musical work)becomes both a tool for creationand a requirement for the individualaudience who can not necessarilyexpect a prepared and accessible lis-tening room but who needs to facili-tate their own personal setting forexperiencing a performance. Thisagain leads to a choice for the audi-ence and to the need of a suitabletechnical set-up. What equipmentdoes someone need to experiencethe virtual space in the intendedway? Does everyone need speakersystems or is it created to be forheadphones? This, in some ways,would this lead us back to the ideaof the Electrophone table, maybenow as a ‘VR headset station’provid-ing public access to technology thatmay not be at each home, thus en-abling audiences to experience ondemand content together in virtualspace – side by side.NotesScience Museum Group.„Electro-phone table, by National TelephoneCompany, England, c.1895-1925“;https://collection.sciencemuseum-group.org.uk/objects/co8615954/49array2022electrophone-table-by-national-telephone-company-england-c-1895-1925-electrophone-table(last access 3 April 2023); NatashaKitcher.„Electrophone: the Victo-rian-era gadget that was a precursorto live-streaming“; The Conversation,12. Nov. 2020; https://theconversa-tion.com/electrophone-the-victo-rian-era-gadget-that-was-a-precur-sor-to-live-streaming-148944 (lastaccess 3 April 2023).ReferencesAkkermann, M. (2014), “ComputerNetwork Music – Approximation toa Far-Scattered History”, Proceedingsof the EMS 2014, www.ems-network.org/spip.php?article363.Cope, D. (2001), Virtual Music:Computer Synthesis of Musical Style,MIT Press: Cambridge.Duckworth, W. (2013), Virtual MusicHow the Web Got Wired for Sound,Taylor and Francis.Emmerson, S. (2000), Music, Elec-tronic Media and Culture, Ashgate:Adlershot.Föllmer, Golo (2005), Netzmusik,Wolke: Hofheim.Gresham-Lancaster, S. (2013),“Computer Music Network”,Leonardo, MIT Press.spaceHajdu, Georg (2012), “Composing forNetworks”, www.georghajdu.de/gh/fileadmin/material/articles/Composing_for_Networks.pdf (lastaccess 3 June 2023).Hamilton, Rob (2019), “Collaborativeand Competitive Futures for VirtualReality Music and Sound”, 2019 IEEEConference on Virtual Reality and 3DUser Interfaces VR: 1510–12, https://doi.org/10.1109/VR.2019.8798166.Hugill, Andrew (2005), “From OnlineSymphonies to MusiMOO: SomeCreative Uses of the Internet inMusic and Education”, ContemporaryMusic Review 24 (6): 527–40.Lazzaro, J. (2001), “A Case for Net-work Musical Performance”, Pro-ceedings of the 11th internationalworkshop on Network and operatingsystems support for digital audio andvideo, NOSSDAV, New York, UnitedStates: 157–66.Manning, P. (2004), Electronic &Computer Music, Oxford UniversityPress: Oxford.Packer, R. and S. Bradley (2000),“TELEMUSIC #1 – PremierePerformance”. Rhizome, 22 October2000. http://rhizome.org/discuss/view/29788/ (last access 3 June2023).50array2022 spaceSerafin, S. Et al. (2017), “Considera-tions on the Use of Virtual andAugmented Reality Technologies inMusic Education”, IEEE Virtual RealityWorkshop on K-12 Embodied Learn-ing through Virtual & AugmentedReality (KELVAR): 1–4, https://doi.org/10.1109/KELVAR.2017.7961562.Serafin, S. et al. (2016). “VirtualReality Musical Instruments: State ofthe Art, Design Principles, andFuture Directions”, Computer MusicJournal 40 (3): 22–40, https://doi.org/10.1162/COMJ_a_00372.Wahl, C. (2013), “Between ArtHistory and Media History: A BriefIntroduction to Media Art”, in:Challenges and Perspectives, editedby V. Hediger et al., Amsterdam:25–58, doi:10.1515/9789048513833-004.51

Music is in the air. Sounding performances in hybrid and virtual space

4array2023 fluxImprintGuest editor: Scott DealMain Editor Array: Miriam AkkermannPublished by the International Computer Music Association ICMA 20211819 Polk Street, San Francisco, California 94019, USADOI: https://doi.org/10.25370/array.v2023https://journals.qucosa.de/array/ www.computermusic.orgEditorialThe influence of humans on the earth is as evident as challenging.What is re-ferred to as the Anthropocene is shaping our current environment. This is notonly subject to research on geology, ecosystems, and biodiversity of earth,but also reflected and discussed in and through artistic works and work pro-cesses.This issue brings together explicitly artistic perspectives on topics such asglobal warming, the influence of climate change on our environment rangingfrommelting of glaciers to the effects on indigenous cultures, new reflectionsresulting from listening to the environment, as well as reflections on how ourown behavior in the field of music tech related arts may effect the ongoingenvironmental changes.Thanks to this issue’s guest editor, Scott Deal, we are able to present a varietyof perspectives and artistic approaches, with which we hope to broaden thediscussion of topics related to Anthropocene.Miriam Akkermann

Not a member yet

Spatial Audio in Online Presentations

Chaos in the Garden. Human-assisted AI Composition in Experimental Spatial Music

Vertigo of the ears and eyes

Editorial

Stereo and Ambisonics: A reflection over parallel spatialization techniques

Indigeneity and Computer Music in the Anthropocene

Currents: A Weather Data Sonification Project

Music is in the air. Sounding performances in hybrid and virtual space

Editorial

Patagonian Soundscapes: An acoustic survey of sub-antarctic Chile and generative soundscape composition

137

165