Presented at the 12th International Conference on Auditory Display (ICAD), London, UK, June 20-23, 2006.This paper presents a ``think-aloud'' study investigating the ability of visually impaired participants to make comparisons between haptic and audio line graphs. Graphs with two data series were presented. One data series was explored with a PHANTOM haptic device, whilst the other was sonified using one of two data - sound mappings. The results show that participants can make comparisons between the two lines. However, there is some cross-modal interference which makes it difficult to extract detailed information about the data series presented in audio

Brewer, S.

McGookin, D.

English

Scholarly Materials And Research @ Georgia Tech

Proceedings of the 12th International Conference on Auditory Display, London, UK June 20 - 23, 2006CONTEXTUAL AUDIO IN HAPTIC GRAPH BROWSINGDavid K. McGookin Stephen A. BrewsterThe MultiVis ProjectDepartment of Computing ScienceUniversity of Glasgow, G12 8QQ{mcgookdk,stephen }@gla.ac.ukwww.multivis.orgABSTRACTThis paper presents a “think-aloud” study investigating the abil-ity of visually impaired participants to make comparisons betweenhaptic and audio line graphs. Graphs with two data series werepresented. One data series was explored with a PHANTOM hapticdevice, whilst the other was sonified using one of two data - soundmappings. The results show that participants can make compar-isons between the two lines. However, there is some cross-modalinterference which makes it difficult to extract detailed informationabout the data series presented in audio.1. INTRODUCTIONAccess to graphical data is a significant problem for visually im-paired people. Whilst textual data can be easily accessed usingscreen reader software such as JAWS (www.freedomscientific.com),data such as graphs and tables presents a significant problem. Sev-eral researchers have proposed and evaluated the use of sonifica-tions to provide access to this data with systems such as the Soni-fication Sandbox [1] allowing easy access to data for visually im-paired people.Interaction with such sonifications however loses the spatialrelationships that exist in the visual representations of graphs. Forexample, where a particular bar is in a bar graph, or where (ratherthan when) a turning point in a line graph occurs. This loss of acommon vocabulary to describe the graphs may make it difficultfor sighted and visually impaired colleagues to collaborate usingsuch information [2]. Additionally sonifications can be difficult todirectly manipulate in the way that is available with visual inter-faces. Cohen’s work in filtears [3] is a notable exception; howeverit is only recently that the interactive control of sonification hasreceived greater prominence [4].One solution to this lack of spatial awareness is to incorpo-rate the addition of haptic (or touch) interaction alongside the useof sonification. A haptic device such as SensAble technologie’sPHANTOM (see Figure 1), which uses motors and optical sensorsto provide dynamic information through the sense of touch, canprovide a control mechanism for interactive sonification control aswell as provide a fixed reference frame device to provide spatialinteraction with a sonification [5]. Work by Yu and Brewster [6]has shown that haptic line graphs can be successfully interpretedby visually impaired users. There remain problems however, sincedevices such as the PHANTOM only allow one point of contactwith the data. If a line graph were to contain two data series, onlyone could be browsed at a time making it difficult to determine therelationship between the two data series; a relationship that wouldFigure 1: A picture of a SensAble Technologies’ PHANTOMOmni Haptic Device. The use of motors and optical sensors al-lows the user to “feel” virtual objects.be visually obvious. Whilst it is possible to present multiple dataseries via sonification, it is difficult to combine multiple simulta-neously sonified data series without those data series interferingwith each other making understanding difficult [7].Since the communications bandwidth of the auditory and hap-tic systems is limited in comparison to the visual system, the mostappropriate approach to present information to visually impairedpeople would be to exploit both the haptic and auditory systems inconjunction to communicate data. Doing so allows the advantagesof each modality to be used to improve data access [6]. Usingsuch a system, one data series could be communicated throughthe haptic modality, whilst the other could be sonified. Howeverwhen browsing line graphs with multiple data series users wouldneed to make comparisons between the data communicated usingdifferent modalities. With such cross-modal presentation howeverthere are problems that may arise when data are presented to dif-ferent modalities [8]. In this paper we present the results of aninitial study investigating the cross-modal presentation of hapticline graphs to visually impaired people. In the following sectionwe review existing research on cross-modal presentation beforeintroducing our study and the results obtained.ICAD06-1ICAD06 - 91Proceedings of the 12th International Conference on Auditory Display, London, UK June 20 - 23, 20062. BACKGROUND AND CONTEXTThe investigation of cross-modal integration and equivalence is notnew. Flowerset al. [9] have found a high correlation between par-ticipants ability to make judgments on data which was presentedvia auditory and visual graphs, although the auditory and visualrepresentations were not simultaneously presented. McGee [8] hasperformed several experiments and identified that audio can influ-ence the perception of a haptic texture, making it appear to the useras more or less rough.Whilst work has been undertaken on cross-modal perception,our work is novel in that no previous work has attempted to presentdifferent data to different modalities and ask users to make com-parisons between them. This is important as if users can success-fully integrate information from the two modalities, such interpre-tation can be exploited to provide improved access to graphs. Thiswork is also important as prior research indicates that the interac-tion between modalities is not clear and must be investigated on acase by case basis [10]. As McGee [8] states“When a multi sen-sory percept is processed, the level at which the combined stimuliare integrated into the whole piece of information can vary. Sen-sory stimuli can be ignored or attended to; used alongside eachother to reinforce a mental representation generated by one of thestimuli; combined in an additive way to produce a greater effectthan either single stimuli; or processed against each other in com-petition to produce a cognitive conflict.”. Whilst it is reasonablethat cross-modal presentation can improve access to informationfor visually impaired people, it is unclear what the underlying per-ceptual and cognitive outcome will be. It is therefore important toinvestigate the issue of cross modal integration of line graph datato identify whether the two modalities interfere with each other, orcomplement each other providing information that would not beavailable by one modality alone.3. STUDY OUTLINEIn order to investigate the issues previously discussed, a study in-volving seven participants from the Royal National College for theBlind (RNCB) in Hereford, UK was carried out. All participantswere visually impaired and were paid£15 for their participation.Each participant was asked to browse and describe line graphs,each containing two data series (See Figure 2). In all cases onedata series was browsed using a PHANTOM haptic device“hapticline” , whilst the other was simultaneously sonified“audio line” .As the user moved the PHANTOM device along the haptic line, aMIDI note corresponding to the position on the audio line, at thatx position, was played to the user. This allowed the user to interac-tively control the speed of presentation of the audio line. The MIDInote presented depended on the data to sound mapping used. Twodifferent mappings were used, each of which provided contextualinformation about the audio line in relation to the haptic line, butproviding varying amounts of detail.Whilst the mappings do not represent all of the possible data- sound mappings that can be applied to the audio line, they dorepresent two points in the complexity of sonifications that couldbe presented in the scenario described above. As such they pro-vide a good foundation to start investigating cross-modal hapticand audio interaction in line graphs.Audio LineHaptic LinePosition of The PHANTOM CursorFigure 2: Illustration of the graphs that were browsed during thestudy, and how they were presented to the users.3.1. Two Pitch MappingIn this system the audio line was represented using two differentpitches. If the audio line wasbelow the haptic line, a low note(General MIDI note 35) was played on the general MIDI piano(MIDI patch number 1). If the audio line wasabovethe hapticline, a high note (General MIDI note 100) was played also onthe same instrument. These notes being at the extremities sug-gested by Brown and Brewster [11]. With this mapping the useris provided with only low pass information about the audio lineand its relationship with the haptic line. However, salient infor-mation about the relationship between the two lines can still bedetermined, such as intersections and relative positions of the twolines.3.2. Standard Pitch Mapping with Two TimbresThis sonification provides more detailed information about the au-dio line. It uses a standard mapping between y-axis value and pitchaccording to the equation derived in Brown and Brewster [11] (seeEquation 1), which maps the y-axis value on a uniform scale to theMIDI notes 35-100.note = (yAxisV alue(yAxisV aluemax − yAxisV aluemin) ∗65)+35 (1)To provide information about the relative positions of the linesto each other, two timbres were used. When the audio line wasbelow the haptic line a French horn timbre (general MIDI patch60) was used. When the audio line wasbovethe haptic line apiano timbre (General MIDI patch 1) was used. These patchesallowing all of the notes used to be played well.3.3. MethodThe study followed a “think aloud” methodology with participantsencouraged to verbally describe each of the line series in the graph,and the relationship between them, in as much detail as they could.Participants were firstly introduced to the PHANTOM haptic de-vice. At this stage the tasks that they would be carrying out andthe general relationship between what the user felt and what theyheard was explained with the aid of a raised paper diagram. OnceICAD06-2ICAD06 - 92Proceedings of the 12th International Conference on Auditory Display, London, UK June 20 - 23, 2006(B)(A)Figure 3: Illustration of a graph drawn by a participant in the two-pitch condition (A), and (B) a screenshot of the graph.this was explained participants proceeded to carrying out one ofthe conditions. Ideally, the order in which participants carried outconditions would be counterbalanced, however several of the par-ticipants found the standard pitch mapping with two timbres diffi-cult to grasp at the same time as the general relationship betweenthe haptic and audio lines. As such the two pitch condition wasgenerally performed first, followed by the standard pitch mappingwith two timbre condition. In all cases participant comments wererecorded for later transcription. Additionally, one participant wasable to provide drawings of the graphs on German film (a specialpaper that leaves tactile ridges when drawn on with a pen).4. RESULTSThe comments given by participants, and notes taken by the exper-imenter were transcribed and analysed. In this section we outlinethe main findings in terms of access to general, low-pass, infor-mation in the graph, detailed information about the audio line andother useful comments made by the participants.4.1. Low Pass InformationOne of the key features of a graph containing two data series isthe relationship between those data series. This can largely becategorised as the identification of crossing points (where one dataseries crosses the other), and the relative proportion of each dataseries that is higher than the other, such as might be important ona graph of one company’s sales in comparison to another.In both conditions participants were able to identify intersec-tion points. Additionally many participants were able to correctlyidentify the direction of some of the crossings, i.e. whether theaudio line moved from below the haptic line to above it, or viceversa. Figure 3 for example shows the sketch made by one of theparticipants in the two pitch mapping condition. The participanthas been able to correctly identify the crossings and the directionof the first crossing. Whilst participants were able to identify thisinformation they did not find it easy, many said that it was a dif-ficult concept to get their head around, with one noting“Havingsomething to hear and something to feel and trying to compare thetwo I think is quite strange”.4.2. Cross-Modal InteractionsWhilst participants can retrieve the low pass information from bothof the conditions, albeit with some effort, it was difficult for par-Figure 4: An example “predictable” graph that was explained indetail in the standard pitch mapping with two timbres condition.The graph is “predictable” since the haptic line is straight.ticipants to extract more information from the graph. Participantsfound it very difficult to describe the audio line in any detail, ob-viously the two pitch condition does not communicate such in-formation, but even when such information was available in thesonification participants found it difficult to extract. From com-ments participants made, it appears that the perception of the au-dio and haptic lines interfered with each other, with the haptic linetaking precedence. As one participant stated when carrying outthe standard pitch with two timbre condition“I cant tell what theother (audio) line is without getting distracted by the haptic line”.Some participants were able to discuss the shape of the audio linein cases where either the audio or haptic lines were “predictable”,i.e. they were largely straight. See Figure 4 for an example. How-ever even in such cases mistakes were still made. For example, oneparticipant stated that the audio line of the graph shown in Figure4, was “M” shaped. Thus in spite of getting the first part of theaudio line correct, the second part appears to be incorrect. Unfor-tunately the data recorded in this study does not provide reasonsfor this. Klatzky and Lederman [10] propose that the modality thatis most appropriate for the task dominates the other, however theirwork largely discusses real world phenomena, rather than the “ar-tificial” scenario discussed here. In some cases where participantscould not discuss the audio line in detail, the experimenter tookcontrol of the PHANTOM and moved along the haptic line, caus-ing the audio line to be played. In such instances, participants wereable discuss the shape of the audio line. This indicates that hapticinformation was interfering with the perception of the audio line.4.3. Other CommentsIn addition to the comments which directly relate to comparingthe data represented through the haptic and audio modalities, par-ticipants also discussed other features of the interaction betweensound and haptics that they used to help complete the task. Oneparticipant commented that although they found it easy to slip offof the haptic line, the absence of audio alerted them to this fact.The audio acting as a reassurance that the user was still connectedto the haptic line. In addition users discussed that they liked the useof the PHANTOM as a control mechanism, allowing them to movefaster or slower through the sonifications as they wished; such con-trol being useful if the user is trying to find the exact sport of anICAD06-3ICAD06 - 93Proceedings of the 12th International Conference on Auditory Display, London, UK June 20 - 23, 2006intersection (moving slowly), or trying to get a quick overview(moving more quickly).5. CONCLUSIONFrom the comments made by participants during the “think-aloud”study, we can consider that undesirable cross-modal interactionsoccurred when participants tried to retrieve detailed informationfrom the audio line when using the more detailed sonification (stan-dard pitch mapping with two timbre conditions). Even when tryingto concentrate solely on the audio line, participants still describedthat the haptic line was interfering. Participants could however de-termine where the crossing points were, as well as the direction ofthose crossing points, indicating that at least some contextual in-formation could be extracted from the audio line. As such the twopitch condition, which only provides such information, is prefer-able to the standard pitch mapping with two timbres.The information provided by the sonifications in this studymay be useful in haptic graph displays where only one data seriescan be browsed at a time. The use of contextual audio providinginformation about other data series in the graph, even if only lowpass information is available, may help the user to construct a moreeffective “mental picture” [12] of the relationship between the twodata series. Since users can gain low pass information via differ-ent modalities it would be useful to identify to what extent suchinformation assists in creating such a picture.6. ACKNOWLEDGEMENTSWe would like to thank all participants and staff at the RNCB Here-ford. This work is supported by EPSRC grant GR/S86150/01.7. REFERENCES[1] B. N. Walker and J. T. Cothran, “Sonification sandbox:A graphical toolkit for auditory graphs,” inProceedingsof ICAD 2003, Boston, Massachusetts, 2003, pp. 161–163,ICAD.[2] F. Winberg and S. O. Hellstrom, “Qualitative aspects of audi-tory direct manipulation a case study of the towers of hanoi,”in Proceedings of ICAD 2001, Espoo, Finland, 2001, vol. 1,pp. 16–20, ICAD.[3] M. Cohen and L. F. Ludwig, “Multidimensional audio win-dow management,”International Journal of Man-MachineStudies, vol. 34, pp. 319–336, 1991.[4] T. Herman and A. Hunt, “An introduction to interactive soni-fication,” IEEE Multimedia, vol. 12, no. 2, pp. 20–24, 2005.[5] C. Sjostrom, “Using haptics in computer interfaces for blindpeople,” inCHI 2001, Seattle,Washington, 2001, vol. 1, pp.245–246, ACM Press.[6] W. Yu and S. A. Brewster, “Evaluation of multimodal graphsfor blind people,”Universal Access in the Information Soci-ety, vol. 2, no. 2, pp. 105–124, 2003.[7] J. H. Flowers, “Thirteen years of reflection on audi-tory graphing: Promises, pitfalls, and potential new direc-tions,” in First International Symposium on Auditory Graphs(AGS2005), Limerick, Ireland, 2005, pp. 1–5.[8] M. R. McGee, Investigating a multimodal solution for im-proving force feedback generated textures, Ph.D. thesis, Uni-versity of Glasgow, 2002.[9] J. H. Flowers and T. A. Hauer, “Musical versus visual graphs:Cross-modal equivalence in perception of time series data,”Human Factors, vol. 37, no. 3, pp. 553–569, 1995.[10] R. J. Klatzky and S. J. Lederman, “Touch,” inHandbookof Psychology: Volume 4 Experimental Psychology, Alice FHealy and Robert W Proctor, Eds., vol. 4 ofHandbook ofPsychology, pp. 147–176. John Wiley & Sons, Hoboken,Hew Jersey, 2003.[11] L. M. Brown, S. A. Brewster, R. Ramloll, M. Burton, andB. Ridel, “Design guidelines for audio presentation of graphsand tables,” inProceedings of ICAD 2003, Boston, Massa-chusetts, 2003, pp. 284–287, ICAD.[12] S. Pinker, “A theory of graph comprehension,” inArtificialIntelligence and the Future of Testing, Roy Freedle, Ed., pp.73–126. Laurence Erlbaum Associates, Hillsdale, New Jer-sey, 1990.ICAD06-4ICAD06 - 94

Contextual audio in haptic graph browsing

https://smartech.gatech.edu/bitstream/1853/50492/1/McGookinBrewster2006b.pdf

Contextual audio in haptic graph browsing

Abstract

Similar works

Full text

Available Versions

Scholarly Materials And Research @ Georgia Tech

Scholarly Materials And Research @ Georgia Tech