Presented at the 19th International Conference on Auditory Display (ICAD2013) on July 6-9, 2013 in Lodz, Poland.Visitors to aquariums typically rely on their vision to interact
with live exhibits that convey rich descriptive and aesthetic
visual information. However, some visitors may prefer or need to
have an alternative interpretation of the exhibitÕs visual scene to
improve their experience. Musical sonification has been explored
as an interpretive strategy for this purpose and related work
provides some guidance for sonification design, yet more
empirical work on developing and validating the music-to-visual
scene mappings needs to be completed. This paper discusses
work to validate mappings that were developed through an
investigation of musician performances for two specific live
animal exhibits at the Georgia Aquarium. In this proposed study,
participants will provide feedback on musical mapping examples
which will help inform design of a real-time sonification system
for aquarium exhibits. Here, we describe our motivation,
methods, and expected contributions

Bruce, Carrie M.

Henry, Ashley G.

Walker, Bruce N.

Winton, Riley J.

English

Scholarly Materials And Research @ Georgia Tech

2776–10 july, 2013, Łódź, Poland international Conference on auditory DisplayICaD 2013SONIFICATION MAPPING CONFIGURATIONS:  PAIRINGS OF REAL-TIME EXHIBITSAND SOUNDAshley G. Henry, Carrie M. Bruce, Riley J. Winton, and Bruce N. WalkerGeorgia Institute of Technology,Sonification Laboratory,654 Cherry Street, Atlanta, GA 30332{ashleyghenry, rjwinton, carrie.bruce, bruce.walker}@gatech.eduThe 19th International Conference on Auditory Display (ICAD-2013) July 6-10, 2013, Lodz, PolandSONIFICATION MAPPING CONFIGURATIONS: PAIRINGS OF REAL- IME EXHIBITS AND SOUND Ashley G. Henry, Carrie M. Bruce, Riley J. Winton, and Bruce N. WalkerGeorgia Institute of Technology,Sonification Laboratory,654 Cherry Street, Atlanta, GA 30332{ashleyghenry, rjwinton, carrie.bruce, bruce.walker}@gatech.eduABSTRACTVisitors to aquariums typically rely on their vision to interact with live exhibits that convey rich descriptive and aesthetic visual information. However, some visitors may prefer or need to have an alternative interpretation of the exhibitÕs visual scene to improve their experience. Musical sonification has been explored as an interpretive strategy for this purpose and related work provides some guidance for sonification design, yet more empirical work on developing and validating the music-to-visual scene mappings needs to be completed. This paper discusses work to validate mappings that were developed through an investigation of musician performances for two specific live animal exhibits at the Georgia Aquarium. In this proposed study, participants will provide feedback on musical mapping examples which will help inform design of a real-time sonification system for aquarium exhibits. Here, we describe our motivation, methods, and expected contributions. [1] INTRODUCTIONLive aquarium exhibits are complex visual scenes that are comprised of static and dynamic components such as rocky outcroppings and moving animals. Most visitors who are able to see typically focus on the characteristics (e.g., color, size, movement, etc.) of these components to make sense of what they are looking at and assign descriptive and aesthetic qualities. Descriptive qualities include information about an animalÕs size, brightness of a color, and speed of movement. A person can decide that one fish is bigger than another, the rocks are a dark gray against the white sandy bottom, and fish are moving very quickly. Aesthetic qualities are associated with the mood, emotion, and feeling evoked by the scene and can result in opinions such as the exhibit is calming or the fish are happy. The mapping from visual components and characteristics to descriptive and aesthetic qualities happens naturally for many visitors as part of innate and learned processes. However, what happens if a visitor isnÕt able to see the visual scene (e.g., has a vision impairment or canÕt look over the crowd), prefers an alternative interpretation of visual information, or wants to supplement the visual experience with something else?  Sonification through musical parameters (e.g., pitch, tempo, loudness) is one way to provide an auditory translation of the visual scene. For this form of sonification to convey live exhibit details, musical parameters must map to the static and dynamic visual components of the exhibit in a way that supports descriptive and aesthetic qualities. Further, these mappings must make sense and be of interest to listeners. There is a wide array of literature that is relevant to developing sonification for this context (see [1] for a fuller overview). For the purpose of this paper, we will briefly mention literature that discusses musical mappings to motion (descriptive) and emotion (aesthetics). Many researchers have investigated the relationship between music and motion (see [2] for a detailed review),often discussing listenersÕ mappings of pitch, tempo, and loudness. It is common for listeners to link tempo to speed or velocity; loudness to distance or level of energy; and pitch changes to spatial ascent/descent or distance fluctuations [2]. These typical mappings (as well as others) have been used by researchers in our group to develop a MIDI-based system that associates fish characteristics such as  swimming speed or direction with tempo and pitch changes [1, 3, 4]. That research demonstrates that it is feasible to sonify fish movement through an automated system. However, more formal studies still need to be completed to examine how music can be mapped to motion in this context and how music-motion mappings are perceived.Findings on the relationship between the structure of music and perceived emotions have shown that many factors including musical notation and the performerÕs expression of these notations effect the interpretation of the listener [5]. Additionally, performer variations in expressive strategies of tempo, timing, dynamics, articulation and vibrato have a marked effect on how basic emotions (i.e., ÒhappyÓ, ÒsadÓ, ÒangryÓ) are perceived [6]. Influencing the musical parameters of mode, tempo, loudness, articulation, pitch and harmony, listenersÕ perceived emotion was influenced towards the basic moods of happy, sad and dreamy but complex moods were not perceived [7]. While that work helps us know how to evoke an emotional response with music, it does not provide direct guidance on how to translate an observed aesthetics into musical parameters. As previously mentioned, we have developed an automated system for mapping fish characteristics to music [1, 3, 4]. However, the mappings used in that system were largely generated from research team knowledge and experience. As intended from the outset of our project, we are working towards empirically-driven mappings to ensure that our system is designed to deliver sonifications that effectively interpret live aquarium exhibits for all aquarium visitors, including those with vision impairments. We have identified mapping strategies used by musicians who provided real-time interpretations of two 278ICAD 2013SONIFICaTION MaPPING CONFIGURaTIONS: PaIRINGS OF REaL-TIME EXHIBITS aND SOUNDThe 19th International Conference on Auditory Display (ICAD-2013) July 6-10, 2013, Lodz, Polandexhibits at the Georgia Aquarium. This extended abstract describes a recently initialized study that will validate these mappings with non-expert listeners and further inform the design of our sonification system. [2] RESEARCH PLAN2.1. Leveraging Prior WorkThis proposed work seeks to validate the mappings produced from an investigation of musiciansÕ interpretive strategies. In that previous investigation, fourteen musicians were individually shown videos of real aquarium exhibits, and asked to discuss their performance strategies and give a live performance for the Beluga and Ocean Voyager exhibits at the Georgia Aquarium. Each musician played their instrument of choice which included guitars, pianos/keyboards, drums, and violins. A taxonomy of exhibit components and characteristics and a vocabulary for describing the musical parameters were created. The performance strategy discussions were coded by several raters according to the taxonomy and vocabulary to establish inter-rater reliability. This resulted in mappings of exhibit components to musical parameters including: animal size to pitch, loudness; animal height in habitat to pitch; proximity of animal to loudness, tempo; animal speed to tempo; overall activity level, quantity of animals to note density; mood. Specific examples of the mappings were identified in the musician performances and prepared as stimuli for this study.2.2. Participants in New StudyInitially, our participants will be undergraduate males and females (n= 30). They will report normal or corrected to normal vision and hearing, and will not be musical experts (to be defined in the research protocol). After this initial phase of 30 participants, this study will also be conducted with several specific populations including students with musical expertise, adults and minors with vision impairment, older adults, and individuals who represent a typical visitor to an aquarium without vision or hearing impairment.2.3. StimuliTwenty-five of the video and audio clips composed by the musicians in the previous study will be used as stimuli for this new study. All stimuli are between 20-30 seconds in length and feature moving animals and aesthetic qualities of the habitat from the two aquarium exhibits. These examples were chosen as they represent diversity in exhibit characteristics. Depending on the randomization procedures, some of the stimuli will be presented as video and audio pairings while others will only be presented in an audio format. 2.4. Data CollectionParticipants who consent to completing the research study will be asked to view and/or listen to a randomized series of video and/or audio clips and answer questions. During individual sessions, participants will be seated in front of a research computer running the E-Prime 2.0 software (Psychology Software Tools, Inc). Video will be presented on the computer screen and audio will be delivered through standard computer speakers (audio for the stimuli was not originally recorded in stereo or spatialized formats). Participants will view the 25 stimuli one at a time and be prompted through the E-Prime software to answer a question after the end of each one. The questions will vary in format (e.g., Likert-style scale, multiple choice, and open-ended) and be based on the descriptive and aesthetic nature of a specific stimulus such as: 1) how well do you think the music related to the arrival and departure of the animal?; and 2) describe the mood of the exhibit using one-word adjectives. For most of the questions, the participants will select or type their responses using the software enabling some automated data collection and management. Some open-ended questions requiring lengthier responses will be recorded and transcribed by a researcher. Participants will continue this procedure for all of the stimuli and finish with a debriefing.[3] EXPECTED CONTRIBUTIONSWe anticipate that this work will validate our mappings and lead to improved system design and effectiveness for the Georgia Tech Accessible Aquarium Project. This project aims to leverage sonification to enable all visitors, including those with vision impairments, to participate in aquariums and other informal learning environments. Moreover, our findings will help us better understand how listeners perceive mappings and contribute to the overall science of sonification.[4] REFERENCES[1] Walker, B.N., et al., "Aquarium sonification: Soundscapes for accessible dynamic informal learning environments," in International Conference on Auditory Display, London, UK, 2006, pp. 238-241.[2] Eitan, Z. and Granot, R. (2006). How music moves: Musical parameters and listeners' images of motion. Music Perception, 23(3), 221-247. [3] Walker, B.N., et al., "Musical soundscapes for an accessible aquarium: Bringing dynamic exhibits to the visually impaired," in Proceedings of ICMC, Copenhagen, Denmark, 2007. [4] Pendse, A., Pate, M., and Walker, B., The accessible aquarium: identifying and evaluating salient creature features for sonification. In Proceedings of the 10th international ACM SIGACCESS conference on Computers  and accessibility. (New York, NY, 2008), 297Ð298.[5] Juslin, P. N. (2000). Cue utilization in communication of emotion in music performance: Relating performance to perception. Journal of Experimental Psychology: Human Perception and Performance, 26(6), 1797-1813. [6] Gabrielsson, A., & Juslin, P. N. (1996). Emotional expression in music performance: Between the performerÕs intention and the listenerÕs experience. Psychology of Music, 24, 68Ð91.[7] Livingstone, S. R., and Brown, A. R., Dynamic response: A real-time adaptation for music emotion. In Proc. Australasian Conf. Interactive Entertainment, (Sydney, Australia, 2005), 105-111.

Sonification Mapping Configurations: Pairings Of Real-Time Exhibits And Sound

https://smartech.gatech.edu/bitstream/1853/51679/1/35_S9-01_Henry.pdf

Sonification Mapping Configurations: Pairings Of Real-Time Exhibits And Sound

Abstract

Similar works

Full text

Available Versions

Scholarly Materials And Research @ Georgia Tech