"Spindex" (speech index) enhances menu navigation user experience of touch screen devices in various input gestures: tapping, wheeling, and flicking

In a large number of electronic devices, users interact with the system by navigating through various menus. Auditory menus can complement or even replace visual menus, so research on auditory menus has recently increased with mobile devices as well as desktop computers. Despite the potential importance of auditory displays on touch screen devices, little research has been attempted to enhance the effectiveness of auditory menus for those devices. In the present study, I investigated how advanced auditory cues enhance auditory menu navigation on a touch screen smartphone, especially for new input gestures such as tapping, wheeling, and flicking methods for navigating a one-dimensional menu. Moreover, I examined if advanced auditory cues improve user experience, not only for visuals-off situations, but also for visuals-on contexts. To this end, I used a novel auditory menu enhancement called a "spindex" (i.e., speech index), in which brief audio cues inform the users of where they are in a long menu. In this study, each item in a menu was preceded by a sound based on the item's initial letter. One hundred and twenty two undergraduates navigated through an alphabetized list of 150 song titles. The study was a split-plot design with manipulated auditory cue type (text-to-speech (TTS) alone vs. TTS plus spindex), visual mode (on vs. off), and input gesture style (tapping, wheeling, and flicking). Target search time and subjective workload for the TTS + spindex were lower than those of the TTS alone in all input gesture types regardless of visual type. Also, on subjective ratings scales, participants rated the TTS + spindex condition higher than the plain TTS on being 'effective' and 'functionally helpful'. The interaction between input methods and output modes (i.e., auditory cue types) and its effects on navigation behaviors was also analyzed based on the two-stage navigation strategy model used in auditory menus. Results were discussed in analogy with visual search theory and in terms of practical applications of spindex cues.M.S.Committee Chair: Bruce N. Walker; Committee Member: Frank Durso; Committee Member: Gregory M. Cors

An exploration of semiotics of new auditory displays: A comparative analysis with visual displays

Communicability is an important factor of user interfaces. To address communicability, extensive research has been done on visual displays, whereas relatively little research has been done on auditory displays. The present paper attempts to analyze semiotics of novel auditory displays (spearcon, spindex, and lyricon) using Peirce’s classification of signs: icon, symbol, and index. After the aesthetic developmental patterns of the visual counterparts are presented, semiotics of auditory cues is discussed with future design directions

The effect of experience on the use of multimodal displays in a multitasking interaction

Theories and previous work suggest that performance while multitasking can benefit from the use of displays that employ multiple modalities. Studies often show benefits of these multimodal displays but not to the extent that theories of multimodal task-sharing might suggest. However, it is often the case that the studies investigating this effect give users at least one type of display that they are not accustomed to, often an auditory display, and compare their performance on these novel displays to a visual display, with which most people are familiar. This leaves a question open regarding the effects of longer-term experience with these multimodal displays. The current study investigated the effect of practice with multimodal displays, comparing two multimodal displays to a standard visuals-only display. Over the course of four sessions, participants practiced a list-searching secondary task on one of three display types (two auditory plus visual displays, and one visual-only display) while performing a visual-manual task. Measures of search-task and primary task performance along with workload, visual behaviors, and perceived performance were collected. Results of the study support previous work with regard to more visual time on the primary task for those using multimodal displays, and show that perceived helpfulness increased over time for those using the multimodal displays. However, the results also point to practice effects taking place almost equally across the conditions, which suggest that initial task-sharing behaviors seen with well-designed multimodal displays may not benefit as much from practice as hypothesized, or may require additional time to take hold. The results of the research are discussed regarding their use in research and applying multimodal displays in the real world as well as in how these results fit with theories of multimodal task-sharing.Ph.D

A survey on hardware and software solutions for multimodal wearable assistive devices targeting the visually impaired

The market penetration of user-centric assistive devices has rapidly increased in the past decades. Growth in computational power, accessibility, and cognitive device capabilities have been accompanied by significant reductions in weight, size, and price, as a result of which mobile and wearable equipment are becoming part of our everyday life. In this context, a key focus of development has been on rehabilitation engineering and on developing assistive technologies targeting people with various disabilities, including hearing loss, visual impairments and others. Applications range from simple health monitoring such as sport activity trackers, through medical applications including sensory (e.g. hearing) aids and real-time monitoring of life functions, to task-oriented tools such as navigational devices for the blind. This paper provides an overview of recent trends in software and hardware-based signal processing relevant to the development of wearable assistive solutions

The Role of Sonification as a Code Navigation Aid: Improving Programming Structure Readability and Understandability For Non-Visual Users

Integrated Development Environments (IDEs) play an important role in the workflow of many software developers, e.g. providing syntactic highlighting or other navigation aids to support the creation of lengthy codebases. Unfortunately, such complex visual information is difficult to convey with current screen-reader technologies, thereby creating barriers for programmers who are blind, who are nevertheless using IDEs. This dissertation is focused on utilizing audio-based techniques to assist non-visual programmers when navigating through large amounts of code. Recently, audio generation techniques have seen major improvements in their capabilities to covey visually-based information to both sighted and non-visual users – making them a potential candidate for providing useful information, especially in places where information is visually structured. However, there is little known about the usability of such techniques in software development. Therefore, we investigated whether audio-based techniques capable of providing useful information about the code structure to assist non-visual programmers. The major contributions in this dissertation are split into two major parts: The first part of this dissertation explains our prior work that investigates the major challenges in software development faced by non-visual programmers, specifically code navigation difficulties. It also discusses areas of improvement where additional features could be developed in order to make the programming environment more accessible to non-visual programmers. The second part of this dissertation focuses on studies aimed to evaluate the usability and efficacy of audio-based techniques for conveying the structure of the programming codebase, which was suggested by the stakeholders in Part I. Specifically, we investigated various sound effects, audio parameters, and different interaction techniques to determine whether these techniques could provide adequate support to assist non-visual programmers when navigating through lengthy codebases. In Part II, we discussed the methodological aspects of evaluating the above-mentioned techniques with the stakeholders and examine these techniques using an audio-based prototype that was designed to control audio timing, locations, and methods of interaction. A set of design guidelines are provided based on the evaluation described previously to suggest including an auditory-based feedback system in the programming environment in efforts to improve code structure readability and understandability for assisting non-visual programmers

Concurrency in auditory displays for connected television

Many television experiences depend on users being both willing and able to visually attend to screen-based information. Auditory displays offer an alternative method for presenting this information and could benefit all users. This thesis explores how this may be achieved through the design and evaluation of auditory displays involving varying degrees of concurrency for two television use cases: menu navigation and presenting related content alongside a television show. The first study, on the navigation of auditory menus, looked at onset asynchrony and word length in the presentation of spoken menus. The effects of these on task duration, accuracy and workload were considered. Onset asynchrony and word length both caused significant effects on task duration and accuracy, while workload was only affected by onset asynchrony. An optimum asynchrony was identified, which was the same for both long and short words, but better performance was obtained with the shorter words that no longer overlapped. The second experiment investigated how disruption, workload, and preference are affected when presenting additional content accompanying a television programme. The content took the form of sound from different spatial locations or as text on a smartphone and the programme's soundtrack was either modified or left unaltered. Leaving the soundtrack unaltered or muting it negatively impacted user experience. Removing the speech from the television programme and presenting the secondary content as sound from a smartphone was the best auditory approach. This was found to compare well with the textual presentation, resulting in less visual disruption and imposing a similar workload. Additionally, the thesis reviews the state-of-the-art in television experiences and auditory displays. The human auditory system is introduced and important factors in the concurrent presentation of speech are highlighted. Conclusions about the utility of concurrency within auditory displays for television are made and areas for further work are identified

Synthèse - Reproduction - Perception des Sons Instrumentaux et Environnementaux : Application au Design Sonore

This dissertation presents a composition of studies and research works articulated around three main topics : synthesis, reproduction and perception of sounds, considering both musical and environmental sounds. Moreover, it focuses on an application field, the sound design, that globally involves the conception of intentional everyday sounds. The document is based on a rather uniform structure and contains, for each part, a general presentation of the topic which brings theoretical elements together with an overview of the state-of-the-art, followed by more precise developments in order to focus on the specific matters related to each topic – in detail, modal formalism in sound synthesis by physical modeling, for the "Synthesis" section ; measurement and control of musical instruments directivity, for the "Reproduction" section ; timbre and sound sources identification, for the "Perception" section – and then followed by a detailed presentation of the personal works related to each matter, in some cases, in the form of published papers. Then, these several elements of knowledge and experience offer a personal and original contribution, deliberately put in a broad, multidisciplinary and applied framework.Ce mémoire présente une composition d’études et de travaux de recherche orientés autour de trois grandes thématiques : la synthèse, la reproduction et la perception des sons, en considérant à la fois les sons de nature musicale mais aussi environnementale. Il vise en outre un champ d’application, le design sonore, qui implique globalement la création intentionnelle de sons du quotidien. La structure du document est conçu selon un schéma relativement uniforme et comporte, pour chaque partie, une présentation générale de la thématique apportant des éléments théoriques et des données relatives à l’état de l’art, suivie de développements spécifiques permettant de converger vers les sujets d’étude propres à chaque thème – explicitement, formalisme modal dans la synthèse par modélisation physique, pour la partie "Synthèse" ; mesure et contrôle de la directivité des instruments de musique, pour la partie "Reproduction" ; timbre et identification des sources sonores, pour la partie "Perception"– puis d’une présentation détaillée des travaux personnels relatifs à chacun des sujets, le cas échéant, sous la forme d’un article publié. Ces divers éléments de connaissance et d’expérience propose donc une contribution personnelle et originale, volontairement inscrite dans un cadre de recherche élargi, pluridisciplinaire et appliqué

DESIGN FOUNDATIONS FOR CONTENT-RICH ACOUSTIC INTERFACES: INVESTIGATING AUDEMES AS REFERENTIAL NON-SPEECH AUDIO CUES

Indiana University-Purdue University Indianapolis (IUPUI)To access interactive systems, blind and visually impaired users can leverage their auditory senses by using non-speech sounds. The current structure of non-speech sounds, however, is geared toward conveying user interface operations (e.g., opening a file) rather than large theme-based information (e.g., a history passage) and, thus, is ill-suited to signify the complex meanings of primary learning material (e.g., books and websites). In order to address this problem, this dissertation introduces audemes, a new category of non-speech sounds, whose semiotic structure and flexibility open new horizons for facilitating the education of blind and visually impaired students. An experiment with 21 students from the Indiana School for the Blind and Visually Impaired (ISBVI) supports the hypothesis that audemes increase the retention of theme-based information. By acting as memory catalysts, audemes can play an important role in enhancing the aural interaction and navigation in future sound-based user interfaces. For this dissertation, I designed an Acoustic EDutainment INterface (AEDIN) that integrates audemes as a way by which to vividly anticipate text-to-speech theme-based information and, thus, act as innovative aural covers. The results of two iterative usability evaluations with total of 20 blind and visually impaired participants showed that AEDIN is a highly usable and enjoyable acoustic interface. Yet, designing well-formed audemes remains an ad hoc process because audeme creators can only rely on their intuition to generate meaningful and memorable sounds. In order to address this problem, this dissertation presents three experiments, each with 10 blind and visually impaired participants. The goal was to examine the optimal combination of audeme attributes, which can be used to facilitate accurate recognitions of audeme meanings. This work led to the creation of seven basic guidelines that can be used to design well-formed audemes. An interactive application tool (ASCOLTA: Advanced Support and Creation-Oriented Library Tool for Audemes) operationalized these guidelines to support individuals without an audio background in designing well-formed audemes. An informal evaluation conducted with three teachers from the ISBVI, supports the hypothesis that ASCOLTA is a useful tool by which to facilitate the integration of audemes into the teaching environment

The Effect of Enhanced Navigational Affordances on College Students' Comprehension of Informational Auditory Text, and the Role of Metacognitive and Motivational Factors

A proliferation of natural speech audio texts as well as improvements in synthetic text-to-speech technology have created new opportunities for learners. While many studies have examined factors affecting comprehension of print texts, few have examined factors affecting comprehension of audio texts and fewer still the effects of specific moderating variables. This study examines the effects of navigational affordance use on comprehension of informational audio texts. Factors of metacomprehension, including self-regulation and rehearsal, as well as motivational factors of interest, effort regulation, and test anxiety were studied for their relationship to the use of navigational affordances. The study utilized a mobile application distributed through the iTunes® store to administer the experimental procedure. Students enrolled in an introductory political science course at a large state university were solicited to participate. Participants were randomly assigned to either the experimental or control group. The experimental group (N = 74) had access to enhanced navigational affordances including pause and continue, forward by sentence, forward by paragraph, backward by sentence, and backward by paragraph. The control group (N = 11) only had access to pause and continue affordances. Results indicate that the presence of enhanced navigational affordances did not demonstrate a significant difference in comprehension between the experimental and control groups. However, there was a significant correlation between navigational affordance use and comprehension. The data indicate the relationship may be curvilinear meaning that affordance use is more frequent for learners with average comprehension, and less frequent for high and low comprehension learners. Metacomprehension and motivational factors were not significantly correlated with navigational affordance use. Motivational factors did positively correlate with comprehension for both groups with an F = 5.49 and α = 0.002. Beta weights for the three factors were 0.29 for interest, -0.35 for test anxiety, and 0.003 for motivation. Information on distractions during the study were also collected. Some participants demonstrated a pattern of skipping behavior when using navigational affordances in which they would quickly navigate through the audio text. The study platform could be used to administer other kinds of audio text comprehension experiments

Soundscape mapping: comparing listening experiences

The perceived auditory environment is an increasingly important part of people’s everyday interactive experiences. While sound design is an established discipline in media such as video games and cinema, this is not the case in Human-Computer Interaction (HCI). HCI designers are rarely trained in sound design, and may not make the most effective use of sound in the design of interactions. Even when sound is at the centre of a design it is rarely evaluated to compare the experiences of designers and listeners. This dissertation reports work conducted to develop a way of comparing sound designers’ intentions for a sound design with the experiences of listeners. Literature on methods of measuring, classifying and visualising sound was reviewed, as well as approaches to sound design in different forms of media and computing. A published method for representing auditory environments was selected for preliminary studies. The four studies addressed to the difficulties of describing auditory environments and how they might be visualised. Two surveys were conducted in order to identify attributes of sound that would be meaningful to 75 audio professionals and 40 listeners. A way of classifying and visualising sound events and their distribution in physical environments was developed and evaluated.The soundscape mapping tool (SMT) was trialled with sound designs from a range of fields within media and computing. The experiences of both the designer and listeners were captured for each of the designs using the SMT. This work demonstrated that the SMT was suitable for capturing the intentions of 10 sound designers and the experiences of 100 listeners. The trial also provided information about how the SMT could be developed further. The dissertation contributes evidence that auditory environments can be abstracted and visualised in a manner that allows designers to represent their designs, and listeners to record their experiences