Presented at the 11th International Conference on Auditory Display (ICAD2005)Time is not just a parameter of auditory disply, but is rather the principle dimension within which all other auditory parameters are placed. While this is a feature of time ordered information, it poses a particular challenge to effective sonification of time-independent data, such as images. In this paper we present two concepts of time mapping, scanning and probing to provide a framework for con- ceptualizing mappings of static data to the time domain. We then consider the geometric characteristics of images to define mean- ingful references in time. Finally, we proceed to suggest a new approach of modeling human image perception for image sonifi- cation, which can be designed upon our framework

Berger, Jonathan

Seung Yeo, Woon

Scholarly Materials And Research @ Georgia Tech

Proceedings of ICAD 05-Eleventh Meeting of the International Conference on Auditory Display, Limerick, Ireland, July 6-9, 2005A FRAMEWORK FOR DESIGNING IMAGE SONIFICATIONMETHODSWoon Seung Yeo and Jonathan BergerCenter for Computer Research in Music and Acoustics,Department of Music, Stanford University,Stanford, California, 94305 USA.woony@ccrma.stanford.eduABSTRACTTime is not just a parameter of auditory disply, but is rather theprinciple dimension within which all other auditory parameters areplaced. While this is a feature of time ordered information, it posesa particular challenge to effective sonification of time-independentdata, such as images. In this paper we present two concepts of timemapping, scanning and probing to provide a framework for con-ceptualizing mappings of static data to the time domain. We thenconsider the geometric characteristics of images to define mean-ingful references in time. Finally, we proceed to suggest a newapproach of modeling human image perception for image sonifi-cation, which can be designed upon our framework.1. INTRODUCTIONDue to its inevitable time dependence, representation of data inthe auditory domain should involve the problem of organizationover time. In the area of auditory display, time is not just a param-eter, but the principle dimension within which all other auditoryparameters are placed. Although this hardly becomes an issue fortime ordered information, it plays a crucial role in the problem ofsonification of static data such as still images, which are neitherorganized in time nor containing any time-relevant information in-side. Therefore, it requires a mapping to time that is not arbitrarilyoriented towards a left-right scan. So far, however, the role of timeas the principle in the auditory display has not been paid enoughattention compared to its significance for designing and analyzingmethods of image sonification. Even the most widely used andeffective methods, such as inverse spectrogram mapping, have notbeen categorized in terms of time mapping. Another importantfactor for image sonification is its geometric characteristics. Sincea still image is defined on a two-dimensional space and each pixelcan have up to four different color values, it can be representedby a three-dimensional matrix. This multi-dimensional propertymakes it possible to define different types of reference pointers bywhich current dataset for sonification is located.In this paper we present two major concepts of time mappingfor image sonification, scanning and probing, to construct a mean-ingful mapping of static data to the time domain. Together withthis time concept, we also suggest the idea of pointers as an essen-tial components of time reference based on the geometric frame-work of images, and discuss the problem of defining their paths.Based on these concepts, we present and analyze examples of map-pings, and then proceed to suggest a new approach for image soni-fication.2. CONSTRUCTION OF TIME DOMAIN: SCANNING VS.PROBINGMethods for organizaing time-independent data for auditory dis-play can be generally classified into two major categories: whetherthey are pre-scheduled and fixed, or arbitrary and freely adjustable.2.1. ScanningThe term scanning refers to the case in which data is scheduled tobe sonified in a fixed, non-modifiable order. Figure 1 illustratesthe mapping of inverse spectrogram method, which is by far themost popular method for sonifying images. We refer to this asthe inverse spectrogram in that the sonification is analogous to thereconstruction of a sound from its spectrogram.Figure 1: Inverse spectrogram scanning.In this system x-axis is mapped to time, and each timestep thecorresponding vertical line is sonified. This, at the same time, isalso similar to the time vs. pitch structure of traditional musicalscores.The speed of scanning is usually fixed, and not allowed to bechanged arbitrarily throughout the process of sonification. Moredetailed description of inverse spectrogrammapping is in 4.1. Scan-ning, however, is not necessarily to be performed along a continu-ous path. Furthermore, it does not have to cover the whole imagearea. This issue of path for scanning is discussed in 3.3.2.2. ProbingIf the speed and the location of sonification “pointer” can be variedby an operator arbitrarily during sonification, we classify this asprobing. In this case, the time domain is sequential only insofar asone may arbitrarily probe anywhere and anytime within an image.ICAD05-323Proceedings of ICAD 05-Eleventh Meeting of the International Conference on Auditory Display, Limerick, Ireland, July 6-9, 2005Compared with the scanning process whose result is constructedin a pre-determined and fixed manner, probing delivers much moreflexibility for determining the location/path, as well as the speed,of sonification. Any pixel or collection of pixels (i.e., line, area,etc.) can be selected for sonification by navigating around the vi-sual image, with the time domain being sequential.One of the major characteristics of probing is that it usuallyrequires the image to be analyzed prior to its sonification. Thispreprocessing of images, together with the issue of combinationof scanning and probing processes, is discussed in 4.2.In case of scanning, an auditory display of the general periph-ery may be informative about context, while probing might be in-formative about features of specific points. Also in musical terms,probing is an improvisational action as opposed to the strict scorefollowing characteristic of scanning.Freedom in time and location is certainly the most importantcriterion for sonification methods. However, this by itself still fallsshort of providing sufficient background to analyze, categorize,and explain every type of sonifications. In the following sectionwe discuss the geometric charcteristics of images and introducepointers and paths, which provide us a new perspective.3. REFERENCE TO TIME: POINTERS AND PATHSIn terms of data sturcture, a still image can be respresented as athree-dimensional matrix. This multi-dimensional property makesit both essential and complicated to define different types of refer-ence by which current dataset for sonification is pointed.3.1. PointerA pointer means a data element, or a set of data elements, that ismapped to auditory domain at the same time. Pointers in differentgeometric shapes are illustrated in figure 2.Figure 2: Pointers in different shapes: (a) single point, (b)line/curve, (c) area, and (d) set of distributed points.A single point on image shown in figure 2 (a), however, doesnot necessarily correspond to one data element: in case of imageswith colors, up to three or four values could be contained withina point. This possible multi-dimensionality should also be consid-ered for other types of pointer shapes.Under our definition, a pointer can be mapped to more thanone auditory parameters at the same time. However, it is not sup-posed to be utilized for multiple events over time: instead, we sug-gest that this be considered as a path.3.2. PathPointers defined in the previous section can be maneuvered alonga number of different paths over time. They can be pre-defined(scanning), or freely improvised (probing). Figure 3 shows exam-ples of typical paths.Figure 3: Examples of paths: (a) stationary point (no geometricmovement), (b) straight lines along axes, (c) arbitrary curves, and(d) set of distributed points.Figure 3 (a) shows the example of a “single point” path: thisbecomes the case when• the same pixel value is repeated to be mapped, or• different color values (i.e., RGB or CYMK) are sent in or-der,over time.4. ANALYSIS OF SONIFICATIONMAPPINGS:EXAMPLESIn this section, we provide five mapping examples of image sonifi-cation and classify them according to the aforementioned conceptsto show the effectiveness of our model.4.1. Inverse Spectrogram MappingThe general mapping of inverse spectrogram is shown in figure 1.By the terms we defined, this can be classified asICAD05-324Proceedings of ICAD 05-Eleventh Meeting of the International Conference on Auditory Display, Limerick, Ireland, July 6-9, 2005• a scanning process,• with a line-shaped pointer, moving linearly along the x-axis.As mentioned before, this is mostly performed as a “scanning”process. However it can also become a probing, which meansthat its pointer does not have to move according to the pre-definedschedule; in fact, its position on the x-axis can be freely changedduring sonification process. This suggests us an important point:any set of pointer and path can be used both for scanning andprobing.Although convenient and familiar, this mapping is rather re-strictive in that it fails to account for attention grabbing featuresthat may demand prominence in effective auditory display.4.2. Raster Scanning MappingRaster scanning is a well-known technique for display devices(i.e., TV, CRT monitors) in which two dimensional information isrepresented as a serial data stream. Consequently, this could serveas a method for sonification in which a path of visual explorationis sonified as a melodic stream.Figure 4: Raster scanning method.And this is an example of• a scanning process,• with a point pointer, which scans through the whole imagehorizontally.One advantage of this model is its nature of repetition, whichcould serve as a guide to regular tempo; in musical terms, the con-tents (pixels) of image could be thought of note events, while bothends of image function as bar lines. Also, in a much shorter timescale, size of repetition can function as the length of a delay line,thereby determining the frequency of resulting sound.For sounds with more than one channel, we can apply the ideaof interlaced raster scanning.4.3. Virtual PathsTo show that paths for pointers can be taken in various ways -especially neither x nor y axis, we present two new mappings with“virtual” paths.4.3.1. Path on Color DepthIn this mapping, horizontal and vertical position of each pixel ismapped to spatial parameter and pitch, respectively. Path for timeFigure 5: Interlaced Raster scanning.pointer is not mapped to either axis: instead, the color/brightnessvalue of each pixel determines the duration of corresponding soundcomponent (i.e., brighter pixels get longer durations).Figure 6: 3D representation of color/brightness. (a) original im-age, and (b) its 3D representation.And it can be defined as• a scanning process,• with an areal pointer which takes up the whole image area,whose path lies on the third dimension of image data (thatis, color values of pixels)This mapping can provide an effective means for implemen-tation of various detailed decay patterns of sound not only overfrequency but also over panning space. For colored images, colorcomponents can be arranged in various sequences to produce dif-ferent distributions of sound particles over time.ICAD05-325Proceedings of ICAD 05-Eleventh Meeting of the International Conference on Auditory Display, Limerick, Ireland, July 6-9, 20054.3.2. Path on Imaginary AxisThis model was created under the inspiration of one of the illustra-tions in ’The Pedagogical Sketchbook’ by Paul Klee [2]. Althoughbeing a result of optical illusion, the perpendicular direction thatgoes into (or out of) the paper/screen provides us a new dimensionto which any sonic properties can be assigned. In this case, it isutilized as the dimension for pointer paths.Figure 7: Time on “perpendicular” axis.This is a model of sonic space in which we are listening tosounds from walls, floor, and ceiling while going “into” the pa-per/screen. And this can be categorized as• a scanning process,• with a rectangular pointer, moving along a perpendicularpath. This makes the size of the pointer vary.4.4. Pointers on Arbitrary PathsIn this section we present two of our sonification tools. Unlike theprevious examples, the followings can be identified as a probingprocess, and a combination of probing and local scanning, respec-tively.4.4.1. SonARTSonART [3] is a flexible, multi-purpose multimedia environmentthat allows for networked collaborative interaction with applica-tions for art, science and industry. Figure 8 is a screenshot ofSonART.It supports an arbitrary number of layered canvases, and eachcanvas can display a unlimited number of images in layers. Bydragging mouse over any canvas, the opacity and RGB values ofpointed pixel in every layer can be transmitted through network forsonification. Detailed description about SonART can be found in[3].• a probing process,• with a point pointer on an arbitrarily controllable path.4.4.2. Viewer.appViewer.app [4] is a data analysis application which works quitesimilar to SonART: a visual representation of complex dataset (oran image itself) is displayed on a canvas, and users can probe overthe image to send out the data values of visibly salient areas overnetwork. Figure 9 shows its screenshot, with a hyperspectral imageof a colon cell displayed on canvas.Figure 8: SonART: a Mac OS X application for multi-purpose,networked multimedia application.Figure 9: Viewer.app: a Mac OS X application for probing imagesof hyperspectral data.Unlike SonART, however, Viewer.app provides a local scan-ning feature: once a visually salient area is detected, user can ei-ther probe around that point, or set a linear path over the area toperform a scanning sonification. Therefore, Viewer.app can be de-scribed as:• a probing process, combined with local linear scanning• with a point pointer which could move both on an arbitrarypath (probing) and a linear path (scanning).This combination of scanning and probing methods, togetherwith the issue of preprocessing, will be discussed more in the fol-lowing section.ICAD05-326Proceedings of ICAD 05-Eleventh Meeting of the International Conference on Auditory Display, Limerick, Ireland, July 6-9, 20055. IMAGE ANALYSIS PRIOR TO SONIFICATION:PREPROCESSING5.1. Combination of Scanning and ProbingA variety of scanning methods and mappings provide numerousmeans of auditory represention of a static image. The primarydrawback of these approaches lies in the inability to sequentiallyfocus on particular details or features of an image. Meanwhile,probing method enables us to focus on points of interest and ex-amine its details. However, it lacks the feature of overviewing thewhole image in a contextual sense. In addition, unlike the scanningmethods which depend on pre-defined trajectory of timeline, pathsof probing will vary from user to user. Therefore, it is quite natu-ral to design an alternate approach as a combined form of scanningand probing. Viewer.app, previously discussed in 4.4.2., serves asan example of global probing supported by local scanning.In general, however, two methods are utilized together as acombination of global scanning followed by local probing. Inthose cases, the foregoing scanning process is referred to as pre-processing of the sonified image. By nature, most probing methodsinvolve preprocessing of data, which could be categorized into twogroups: preprocessing in visual domain, and in auditory domain.5.2. Preprocessing in Visual DomainAuditory probes can be initiated and guided by vision, and anattention-grabbing area may prompt placing a pointer over a par-ticular point to hear the sonified output. This process of locatingthe position to probe - whether being performed with enough “at-tention” or not - can be interpreted as a processing/analysis of data,which occurrs prior to the actual sonification of data.Pre-processing of images could also be done automatically byscanning the image with threshold rules, and probing any pointthat crossed the threshold.5.3. Preprocessing in Auditory DomainIn certain areas of auditory display, preprocessing in visual domainis not available as an option; For example, synthetic vision throughauditory representations such as the vOICe [5], and other types ofauditory aids for people already concentrating on different visualrepresentations, cannot utilize the image itself for presenting in-formation; instead, it must be scanned in the auditory (or somenon-visual) domain for further analysis on salient points.6. POSSIBLE EXTENSION: HUMAN IMAGEPERCEPTION AS A SONIFICATIONMODELWhen viewing a static image the brain engages in separate pro-cesses of object feature finding and spatial adjustment as the vi-sual system combines foveal vision with peripheral vision. Thescanpath theory [1] proposes that eye movement is driven in a top-down process in which a path of discovery is created between at-tention grabbing features.In fact, this scanpath combines fixation with saccadic eye move-ments during image viewing. Therefore, a simple model of theseprocesses can be thought of in terms of probing and scanning tointegrate identification with localization, and this fits well intoour framework. We believe this will result in a new meaningfulmethod providing comprehensive and informative auditory feed-back - both in global context and local feature - of a static image.7. CONCLUSIONIn this paper we presented a framework for image sonificationanalysis based on scanning and probing copcents. It provides us acomplete and clear description and definition of sonification map-pings. We believe this will help us develop a new, effective modelof human image perception as a method for image sonification.8. REFERENCES[1] L. W. Stark, and S. R. Ellis, Scanpaths Revisited: CognitiveModels Direct Active Looking, 1981.[2] P. Klee, Pedagogical Sketchbook, Faber and Faber, Boston,1968.[3] Woon Seung Yeo, Jonathan Berger, and Zune Lee, “SonART:A framework for data sonification, visualization and net-worked multimedia applications” in Proceedings of the In-ternaional Computer Music Conference, Miami, FL, USA,November 2004.[4] Woon Seung Yeo, Jonathan Berger, and R. Scott Wilson,“A Flexible Framework For Real-time Sonification withSonART” in Proceedings of the Internaional Conference onAuditory Display, Sydney, Australia, July 2004.[5] Peter Meijer, The vOICe: Visual Prosthesis for the TotallyBlind, http://www.visualprosthesis.com/voice.htm.ICAD05-327

Framework for designing image sonification methods

https://smartech.gatech.edu/bitstream/1853/50158/1/YeoBerger2005.pdf

Framework for designing image sonification methods

Abstract

Similar works

Full text

Available Versions

Scholarly Materials And Research @ Georgia Tech