Proceedings of the 9th International Conference on Auditory Display (ICAD), Boston, MA, July 7-9, 2003.Previous research on non-speech audio interfaces has demonstrated that they can enhance performance on menu navigation tasks. Most of this work has focused on tasks in which the menu is not spoken and visual representation of the menu is accessible throughout the task. In this paper we explore the potential benefits that earcons, a type of structured sound, might bring to spoken menu systems for which a visual representation is not available. Evaluation of two spoken menu systems that differ only in whether they also employ earcons, indicates that the use of earcons improves task performance by reducing the number of keystrokes required, while also increasing the time spent for each task

Anderson, Sven

Vargas, Maria L.M

Scholarly Materials And Research @ Georgia Tech

Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, July 6-9, 2003COMBINING SPEECH AND EARCONS TO ASSIST MENU NAVIGATIONMaria L.M. VargasUniversity of North DakotaDepartment of Computer ScienceGrand Forks, ND 58201vargas@cs.und.eduSven AndersonBard CollegeComputer Science ProgramAnnandale-on-Hudson, NY 12504sanderso@bard.eduABSTRACTPrevious research on non-speech audio interfaces has demon-strated that they can enhance performance on menu navigationtasks. Most of this work has focused on tasks in which the menuis not spoken and visual representation of the menu is accessiblethroughout the task. In this paper we explore the potential bene-fits that earcons, a type of structured sound, might bring to spo-ken menu systems for which a visual representation is not avail-able. Evaluation of two spoken menu systems that differ only inwhether they also employ earcons, indicates that the use of earconsimproves task performance by reducing the number of keystrokesrequired, while also increasing the time spent for each task.1. INTRODUCTIONThe need for efficient, auditory-based, navigation of informationstructures has grown with the ubiquity of telephone-based auto-mated information services. Increasingly, telephone users are pre-sented with a large number of choices that are structured and pre-sented as a hierarchical menu. Telephone-based interfaces usu-ally present information acoustically and are therefore inherentlyserial. Spoken entries in telephone-based menu systems includeshort words or phrases grouped together under related headings.Menu options at the current level are spoken in sequence and userspress buttons on their phone’s numeric keypad to select from asmall number of choices.In additional to their relatively slow serial presentation, audi-tory menus are difficult to navigate. Rosson [1] investigated hi-erarchical telephone-based interfaces and found that positional in-formation was often implicit in the speech token used to representa menu item. A user must, therefore, understand the hierarchy’scategory/sub-category relationships, since no feedback concerningchange in hierarchy level is provided.This paper describes a technique that uses earcons to conveyadditional locational cues during menu navigation. Earcons areshort, non-speech sounds, that can be constructed to impart addi-tional information. The use of earcons to represent hierarchicallyarranged auditory menus has been studied by several researchers[2, 3]. Results from a number of studies indicate that earcons cansignificantly increase user efficiency during navigation of a visualmenu system [3, 4, 5]. Leplâtre and Brewster (2000) found thatwhen navigating a cellular telephone interface participants made17% fewer keystrokes when earcons were included in the inter-face. Additionally, the sonified interface significantly decreasederrors and required no more time to navigate.The present research investigates whether the same advantagesexist when earcons are added to spoken menu systems like thoseencountered during telephone-mediated database access. The pro-cessing of speech in humans is known to rely on information pro-cessing mechanisms that differ from the processing mechanismsused to processes non-speech acoustic signals (see [6] for a re-view). If speech and non-speech sound can be processed in parallelwith limited interference, we hypothesize that earcons may furtherenhance a spoken interface, much as they are known to enhancegraphical menu systems. To test this hypothesis, we evaluated theusefulness of earcons in the context of a novel interface to automo-bile accessories, such as headlights and windshield wipers. Thesafe operation of an automobile demands significant attention tovisual cues, and therefore precludes use of extensive visual menus.An auditory interface for control of automobile accessories mightpermit the driver to change the state of various accessories withoutattending to a visual display.2. A SONIFIED AUTOMOBILE INTERFACEWe explored the sonification of a spoken interface using a novelautomobile simulation that would be completely unfamiliar to ex-periment participants [7]. The control of many existing automobileaccessories (e.g., the radio) requires a driver to redirect her atten-tion away from the road; for example, a driver may need divertattention from the road while tuning the current radio station. Di-rect visual feedback from such controls can divert visual attentionfrom driving and should be minimized. An auditory interface toautomobile accessories would permit drivers to maintain full vi-sual attention on the driving task, and may therefore be safer. Inaddition, an auditory interface may be practical for drivers whohave physical limitations that make it necessary to engage com-plex controls via a small set of buttons attached to the steeringwheel. In this instance, a driver might adjust accessories by usingthe set of buttons to navigate an auditory interface.We created a simulated interface for common automobile ac-cessories as shown in Figure 1. The simulated interface was im-plemented in Java 1.3 using the standard Application Program-mer’s Interface (API). None of the various graphical controls isactive; instead, the state of various accessories is changed by us-ing key presses to navigate an acoustically presented menu of sub-categories corresponding to the lights, the windshield wipers, theventilation system, and the radio.Figure 2 is a graphic representation of part of the menu thatwas used in this experiment. It depicts the top-level of the menuand all entries within the lights submenu. Due to the size of menu,submenus for the wipers, ventilation, and radio are not shown.Each top-level menu item has a depth ranging from three to five,yielding a total of 133 different menu items. Users traverse the treeICAD03-38Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, July 6-9, 2003Figure 1: The simulated automobile interface. Original interface is in color and occupies the entire 17-inch display.to locate the accessory they wish to select. Changes to the state ofan accessory only occur when a leaf node is selected. For example,the lights-cabin-rear-on/off node allows the user to toggle the stateof the rear cabin lights between off and on.Up-arrow and down-arrow keys are used to change level. Theright and left arrow keys are used to traverse the current level,which is linked to form a circular list. Menu items are acousti-cally presented immediately after navigation keys are pressed. Thecurrently spoken item corresponds to the node in the menu that isbeing visited. This control strategy permits users to hear the listof items, one-at-time, as many times as desired in the forward orreverse order. The home key returns the user to the root node ofthe tree, the top-level (i.e., lights-wipers-ventilation-radio). Non-terminal nodes are selected using the down-arrow. Entries at theterminal nodes are selected using the enter key. Selection of anitem results in a “click” sound followed by the indicated change ofstate on the simulated dashboard display.Limited visual feedback is provided on the simulated wind-shield and dashboard. For example, when the rear windshieldwipers are on they appear to move in the rear-view mirror.All spoken menu items were prerecorded tokens collected froman adult male speaker of American English. Structured earconswere created to represent each menu item using the guidelinespresented in [8]. Each top-level menu item is represented by aparticular simulated instrument (timbre) and motif (chord). Thelights family timbre is piano; the windshield wipers family uses achorus; the ventilation family uses bells; and the radio family useshorns. Arpeggios and chords were alternated at the top level toincrease distinctiveness of the earcons.All items beneath a top-level entry inherit the instrument andnotes of the top-level motif. Within each node, earcons sharetimbre and motif and are therefore differentiated on the basis ofmelody and rhythm as described in [5]. Each earcon begins with avery brief (10-25 msec) percussive sound that indicates the depthof the item in the tree. We experimented with simultaneous play-back of speech and earcon, but found that this made it difficult tohear the either the speech or earcon clearly. Therefore, the play-back of earcons precedes speech in these experiments. Both earconand speech playback can be interrupted simply by pressing any ofthe navigation keys. We found that once participants were familiarwith the menu, they did not wait to hear entire menu items, butquickly moved to the next item. The Java Sound API has play-back latency that is much greater than 100 msec, which wouldhave slowed the interface appreciably and compromised the exper-iments. Audio playback therefore made use of the Tritonus JavaSound API, which exhibits a playback latency of about 20 msec.3. METHODSThirty six participants were recruited and randomly placed in theSpeech Only Group or Earcon and Speech Group. During the ex-periment participants completed a list of tasks, one at a time (e.g.,“turn on the headlights”). Participants in the Speech Only Groupheard spoken menu items with no earcons. Participants in theEarcon and Speech Group heard an earcon preceding each spokenmenu item.Prior to each experiment, the simulated automobile interfaceand auditory menu were explained to participants by reading froma prepared script. Participants were shown a graphic representationof the menu and permitted to become familiar with the software forfive minutes during which time they performed five practice tasksand explored on their own initiative. The graphic menu was thenremoved and the participant began to perform the 43 tasks of theexperiment.All participants completed the same set of tasks by navigatingthe menu and making selections using a small subset of keys on theICAD03-39Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, July 6-9, 2003LightsHeadlights Parking Flashers CabinLow BeamHighBeamFront RearOn Off On OffOn Off On OffVentilationWipersOffRadio.   .   . .   .   . .   .   .Figure 2: Automobile interface options menu.Table 1: Sample of tasks used for the experiment.1. Turn the radio on.2. Tune in to FM - KKXL.3. Turn the AC on.4. Set the temperature to cold.5. Set FM - KKXL to preset3.6. Switch the parking lights on.7. Set the volume to 8.8. Turn the front intermittent wipers to slow.9. Switch the vent to mixture.keyboard as described in the previous section. After completingthe current task, participants moved to the next task by pressing alabeled key. If the user did not end the task on the correct menuitem, the software automatically made the necessary correctionsto the graphical state and currently selected item. This allowedeach user to start each task from the same item in the menu. Thesoftware automatically logged all user keystrokes as well as theirtime of occurrence. After completing the experiment, participantperceptions of task workload were measured using the computerbased version of the NASA Task Load Index (TELEX) [9].A total of 36 participants completed the experiment. One par-ticipant had significant difficulty using the interface and required50% longer to complete the tasks than any other participant fromeither group. This participant’s data was excluded from furtheranalysis. Therefore, the results reported in this paper are based on17 participants in the Speech Only Group and 18 participants inthe Earcon and Speech Group.4. RESULTS4.1. TimeOne measure of the efficiency of an interface is the time it re-quires. The average time spent to complete a task was 11.5 sec-onds for the Speech Only Group and 13.6 seconds for the Earconand Speech Group, an additional task time of 18%. The extra timespent by members of the Earcon and Speech Group was significant( 	 	 ). One explanation is that the auditoryitems (earcon plus speech) takes longer than the speech alone. Onaverage, the earcons plus speech take approximately 90% longerthan speech alone. However, the 18% increased time spent by theEarcon and Speech Group is substantially shorter than this increasein the duration of the items they heard, indicating that participantsdid not simply slow their responses to listen to the longer stimuli.4.2. Keystroke CountA second measure of efficiency is the number of keystrokes neededto complete a task. Total keystrokes also reflects participant famil-iarity with the organization of the menu, since extra keystrokesindicate the subject is searching for the correct item. The mini-mum number of keystrokes necessary to complete all of the tasksis approximately 250. The mean number of keystrokes for theSpeech Only Group was 496.8; it was 431.0 for the Earcon andSpeech Group. Thus, the Speech Only Group required 15% morekeystrokes ( 		 ), suggesting that the Earconand Speech Group participants benefited from information con-veyed by the earcons.The average number of keystrokes made for each task is plot-ted in Figure 3. Tasks are arranged in the order in which theywere presented to all subjects. Note that the Speech Only Grouprequires more keystrokes for nearly all tasks. We might expectthat the Earcon and Speech Group participants would also use theextra acoustic cues to learn their task more quickly. In this casewe would expect the between-groups difference in the number ofkeystrokes to increase as the experiment progressed. In fact, nosignificant change in keystrokes as a function of task number wasobserved for the two groups.4.3. Task Completion and ErrorsOnly three participants were able to complete all 43 tasks. Theaverage number of completed tasks was slightly greater for theEarcon and Speech Group (40.3 vs. 39.5), but this difference wasnot significant.Successful completion of a task does not necessarily mean thatsubjects were able to select the correct item with their first selec-ICAD03-40Proceedings of the 2003 International Conference on Auditory Display, Boston, MA, USA, July 6-9, 2003TaskNo. Keypresses0510152025300 5 10 15 20 25 30 35 40 45’Speech’’Earcon’Figure 3: Number of keystrokes to complete each task.Table 2: Average workload scores for the two groups. The finalcolumn lists the probability of a t-test indicating whether the dif-ferences of the means are significant.Workload Speech Earcon-Speech t-testCategory Group GroupMental Demand 41.5 52.8 0.153Physical Demand 11.2 11.7 0.833Temporal Demand 32.4 39.7 0.272Effort 43.2 36.9 0.382Performance 31.5 23.3 0.265Frustration Level 27.4 24.4 0.692Weighted Workload 36.9 35.4 0.761tion. Very often incorrect items are selected prior to selection ofthe correct item. The Speech Only Group made an average of 5.6errors on completed tasks, whereas the Earcon and Speech Groupmade only 1.7 errors. This difference did not reach significance(     	    	), since the mean difference was largelyattributed to one task.4.4. WorkloadThe NASA Task Load Index is a multi-dimensional rating proce-dure that provides an overall workload score based on a weightedaverage of ratings on six subscales: Mental Demands, PhysicalDemands, Temporal Demands, Own Performance, Effort, and Frus-tration. The mean scores for the TELEX are shown in Table 2.Temporal and mental demands were ranked higher by the Earconand Speech Group, though effort was lower. No differences at-tained significance.5. DISCUSSIONIt is interesting to compare our findings with those obtained byLeplâtre and Brewster (2000), who employed earcons in a visual,non-spoken cellular phone interface. Both experiments involvedmenus of similar size and used similar principles for earcon con-struction. The reduction in the number of keystrokes is in goodagreement (15% vs. 17%). Leplâtre and Brewster found no signif-icant difference in the time per task, whereas in the present exper-iment participants who heard earcons required 18% more time pertask. This difference may be attributed to the additional time re-quired to present earcons in conjunction with spoken words. Whencombining earcons with a visual menu, earcons and menu itemscan be displayed in a completely parallel manner.Subjects in the Speech Only Group took less time but mademore keystrokes. As a result, the rate of keystrokes for the SpeechOnly Group was 1.0 key per second, whereas it was only .75 keyper second for the Earcon and Speech group. Despite this differ-ence, the TELEX subjective workload assessment did not revealsignificant differences in mental demand for the two groups.Experiment participants spent approximately 15-20 minutesworking with the simulated automobile menu system. This corre-sponds to an interface that is familiar but certainly not well-known.We anticipate that after users become very familiar with such aninterface, the possibility of making earcons that are much shorterthan spoken words might lead to an interface that can be navigatedmore quickly than current spoken interfaces.The results of the current experiment suggest that earcons canbe added to spoken menu systems and thereby convey additionalinformation via the auditory modality. Their inclusion may de-crease the number of keystrokes and errors involved in the com-pletion of such tasks without making appreciable changes to theoverall perceived workload. The explanation for this finding isthat earcons provide additional cues about the the relative loca-tion of the current menu item for spoken and visual menus alike.Future research will focus on how to optimize and automate theconstruction of earcons for use with speech.6. REFERENCES[1] M.B. Rosson, “Using synthetic speech for remote access toinformation,” Behaviour Research Methods, Instruments andComputes, vol. 17, 1985.[2] M. Blattner, D. Sumikawa, and R. Greenberg, “Earcons andicons: Their structure and common design principles,” HumanComputer Interaction, vol. 4, pp. 11–44, 1989.[3] Stephen Brewster, V.-P Raty, and A. Kortekangas, “Earcons asa method of providing navigational cues in a menu hierarchy,”in Proceedings of BCS HCI. 1996, pp. 169–183, Springer.[4] Stephen Brewster, Adrian Capriotti, and Cordelia Hall, “Us-ing compound earcons to represent hierarchies,” HCI Letters,vol. 1, pp. 6–8, 1998.[5] Grégory Leplâtre and Stephen Brewster, “Designing non-speech sounds to support navigation in mobile phone menus,”in Proceedings of the International Conference on AuditoryDisplay, 2000, pp. 190–199.[6] A. M. Liberman, Ed., Speech: A Special Code, MIT Press,Cambridge, MA, 1996.[7] Tom E. O’Neil, “Adding some audio to the visual componentlibrary,” in Proceedings of the Midwest Instructionand Com-puting Symposium (MICS 2000), St. Paul, Minnesota, 2000.[8] Stephen A. Brewster, Peter C. Wright, and Alistair D.N. Ed-wards, “Guidelines for the creation of earcons,” in AdjunctProceedings of HCI, 1995.[9] NASA Human Performance Research Group, “Task load in-dex v1.0 (computerized version),” Tech. Rep., NASA AmesResearch Center, 1987.ICAD03-41

Combining speech and earcons to assist menu navigation

Previous research on non-speech audio interfaces has demon-strated that they can enhance performance on menu navigation tasks. Most of this work has focused on tasks in which the menu is not spoken and visual representation of the menu is accessible throughout the task. In this paper we explore the potential bene-fits that earcons, a type of structured sound, might bring to spo-ken menu systems for which a visual representation is not avail-able. Evaluation of two spoken menu systems that differ only in whether they also employ earcons, indicates that the use of earcons improves task performance by reducing the number of keystrokes required, while also increasing the time spent for each task. 1

Maria L. M. Vargas

Sven Anderson

CiteSeerX

http://www.icad.org/websiteV2.0/Conferences/ICAD2003/paper/09

Combining speech and earcons to assist menu navigation

Abstract

Similar works

Full text

Available Versions

Scholarly Materials And Research @ Georgia Tech

CiteSeerX