The Integration Of Audio Into Multimodal Interfaces: Guidelines And Applications Of Integrating Speech, Earcons, Auditory Icons, and Spatial Audio (SEAS)

The current research is directed at providing validated guidelines to direct the integration of audio into human-system interfaces. This work first discusses the utility of integrating audio to support multimodal human-information processing. Next, an auditory interactive computing paradigm utilizing Speech, Earcons, Auditory icons, and Spatial audio (SEAS) cues is proposed and guidelines for the integration of SEAS cues into multimodal systems are presented. Finally, the results of two studies are presented that evaluate the utility of using SEAS cues, developed following the proposed guidelines, in relieving perceptual and attention processing bottlenecks when conducting Unmanned Air Vehicle (UAV) control tasks. The results demonstrate that SEAS cues significantly enhance human performance on UAV control tasks, particularly response accuracy and reaction time on a secondary monitoring task. The results suggest that SEAS cues may be effective in overcoming perceptual and attentional bottlenecks, with the advantages being most revealing during high workload conditions. The theories and principles provided in this paper should be of interest to audio system designers and anyone involved in the design of multimodal human-computer systems

Understanding concurrent earcons: applying auditory scene analysis principles to concurrent earcon recognition

Two investigations into the identification of concurrently presented, structured sounds, called earcons were carried out. One of the experiments investigated how varying the number of concurrently presented earcons affected their identification. It was found that varying the number had a significant effect on the proportion of earcons identified. Reducing the number of concurrently presented earcons lead to a general increase in the proportion of presented earcons successfully identified. The second experiment investigated how modifying the earcons and their presentation, using techniques influenced by auditory scene analysis, affected earcon identification. It was found that both modifying the earcons such that each was presented with a unique timbre, and altering their presentation such that there was a 300 ms onset-to-onset time delay between each earcon were found to significantly increase identification. Guidelines were drawn from this work to assist future interface designers when incorporating concurrently presented earcons

The effect of an active transcutaneous bone conduction device on spatial release from masking

Objective: The aim was to quantify the effect of the experimental active transcutaneous Bone Conduction Implant (BCI) on spatial release from masking (SRM) in subjects with bilateral or unilateral conductive and mixed hearing loss. Design: Measurements were performed in a sound booth with five loudspeakers at 0\ub0, +/−30\ub0 and +/−150\ub0 azimuth. Target speech was presented frontally, and interfering speech from either the front (co-located) or surrounding (separated) loudspeakers. SRM was calculated as the difference between the separated and the co-located speech recognition threshold (SRT). Study Sample: Twelve patients (aged 22–76 years) unilaterally implanted with the BCI were included. Results: A positive SRM, reflecting a benefit of spatially separating interferers from target speech, existed for all subjects in unaided condition, and for nine subjects (75%) in aided condition. Aided SRM was lower compared to unaided in nine of the subjects. There was no difference in SRM between patients with bilateral and unilateral hearing loss. In aided situation, SRT improved only for patients with bilateral hearing loss. Conclusions: The BCI fitted unilaterally in patients with bilateral or unilateral conductive/mixed hearing loss seems to reduce SRM. However, data indicates that SRT is improved or maintained for patients with bilateral and unilateral hearing loss, respectively

Factors affecting speech intelligibility improvement with exposure to reverberant room listening environments.

Speech intelligibility has been found to improve with prior exposure to a reverberant room environment. It is believed that perceptual mechanisms help maintain accurate speech perception under these adverse conditions. Potential factors underlying this speech enhancement effect were examined in three experiments. Experiment 1 studied the time course of speech intelligibility enhancement in multiple room environments. Carrier phrases of varying lengths were used to measure changes in speech intelligibility over time. Results showed an effect of speech enhancement with a time course that varied with the signal-to-noise ratio between the speech and a broad-band noise masker. Additionally, greater speech enhancement was found for reverberant environments compared to anechoic space, which suggests that a de-reverberation mechanism in the auditory system may enhance the temporal processing of speech. Experiment 2 examined the influence of the specific source and listener position within the room environment on speech enhancement. Source and listener configurations in three virtual room environments were altered to create a disparity between the position of a carrier phrase and a following speech target. Results showed robust effects of speech enhancement when the source and listener configuration were mismatched which suggests that speech enhancement relies on the general decay pattern of the room environment and not the specific temporal/spatial configuration of early reflections. Experiment 3 assessed the relationships between room-associated speech enhancement and single-reflection echo suppression by measuring echo thresholds for both a traditional click-based stimuli and with speech materials. Echo thresholds were found to be uncorrelated with the results of Experiment I. This suggests that early reflections have little impact on the de-reverberation aspect of speech enhancement, which is consistent with the results from Experiment II. A two-process hypothesis is proposed to account for the results of these experiments as well as previous research on this topic. Prior exposure to a speech pattern provided via carrier phrases is argued to elicit improved temporal processing of speech that results in speech enhancement. It is also argued that a process of de-reverberation effectively reduces the attenuation of temporal information in room environments

Exploring the use of speech in audiology: A mixed methods study

This thesis aims to advance the understanding of how speech testing is, and can be, used for hearing device users within the audiological test battery. To address this, I engaged with clinicians and patients to understand the current role that speech testing plays in audiological testing in the UK, and developed a new listening test, which combined speech testing with localisation judgments in a dual task design. Normal hearing listeners and hearing aid users were tested, and a series of technical measurements were made to understand how advanced hearing aid settings might determine task performance. A questionnaire was completed by public and private sector hearing healthcare professionals in the UK to explore the use of speech testing. Overall, results revealed this assessment tool was underutilised by UK clinicians, but there was a significantly greater use in the private sector. Through a focus group and semi structured interviews with hearing aid users I identified a mismatch between their common listening difficulties and the assessment tools used in audiology and highlighted a lack of deaf awareness in UK adult audiology. The Spatial Speech in Noise Test (SSiN) is a dual task paradigm to simultaneously assess relative localisation and word identification performance. Testing on normal hearing listeners to investigate the impact of the dual task design found the SSiN to increase cognitive load and therefore better reflect challenging listening situations. A comparison of relative localisation and word identification performance showed that hearing aid users benefitted less from spatially separating speech and noise in the SSiN than normal hearing listeners. To investigate how the SSiN could be used to assess advanced hearing aid features, a subset of hearing aid users were fitted with the same hearing aid type and completed the SSiN once with adaptive directionality and once with omnidirectionality. The SSiN results differed between conditions but a larger sample size is needed to confirm these effects. Hearing aid technical measurements were used to quantify how hearing aid output changed in response to the SSiN paradigm

Concurrency in auditory displays for connected television

Many television experiences depend on users being both willing and able to visually attend to screen-based information. Auditory displays offer an alternative method for presenting this information and could benefit all users. This thesis explores how this may be achieved through the design and evaluation of auditory displays involving varying degrees of concurrency for two television use cases: menu navigation and presenting related content alongside a television show. The first study, on the navigation of auditory menus, looked at onset asynchrony and word length in the presentation of spoken menus. The effects of these on task duration, accuracy and workload were considered. Onset asynchrony and word length both caused significant effects on task duration and accuracy, while workload was only affected by onset asynchrony. An optimum asynchrony was identified, which was the same for both long and short words, but better performance was obtained with the shorter words that no longer overlapped. The second experiment investigated how disruption, workload, and preference are affected when presenting additional content accompanying a television programme. The content took the form of sound from different spatial locations or as text on a smartphone and the programme's soundtrack was either modified or left unaltered. Leaving the soundtrack unaltered or muting it negatively impacted user experience. Removing the speech from the television programme and presenting the secondary content as sound from a smartphone was the best auditory approach. This was found to compare well with the textual presentation, resulting in less visual disruption and imposing a similar workload. Additionally, the thesis reviews the state-of-the-art in television experiences and auditory displays. The human auditory system is introduced and important factors in the concurrent presentation of speech are highlighted. Conclusions about the utility of concurrency within auditory displays for television are made and areas for further work are identified

Characterisation of disordered auditory processing in adults who present to audiology with hearing difficulties in presence of normal hearing thresholds: Correlation between auditory tests and symptoms

The diagnosis of auditory processing disorder (APD) remains controversial. Quantifying symptoms in individuals with APD by using validated questionnaires may help better understand the disorder and inform appropriate diagnostic evaluation. Aims: This study was aimed at characterising the symptoms in APD and correlating them with the results of auditory processing (AP) tests. Methods: Phase 1: Normative data of a speech-in-babble test, to be used as part of the APD test battery, were collected for 69 normal volunteers aged 20–57 years. Phase 2: Sixty adult subjects with hearing difficulties and normal audiogram and 38 healthy age-matched controls completed three validated questionnaires (Amsterdam Inventory for Auditory Disability; Speech, Spatial and Qualities of Hearing Scale; hyperacusis questionnaire) and underwent AP tests, including dichotic digits, frequency and duration pattern, gaps-in-noise, speech-in-babble and suppression of otoacoustic emissions by contralateral noise. The subjects were categorised into the clinical APD group or clinical non- APD group depending on whether they met the criterion of two failed tests. The questionnaire scores in the three groups were compared. Phase 3: The questionnaire scores were correlated with the APD test results in 58/60 clinical subjects and 38 of the normal subjects. Results: Phase 1: Normative data for the speech-in-babble test afforded an upper cut-off mean value of 4.4 dB for both ears Phase 2: Adults with APD presented with hearing difficulties in quiet and noise; difficulties in localising, recognising and detecting sounds and hyperacusis with significantly poorer scores compared to clinical non- APD subjects and normal controls. Phase 3: Weak to moderate correlations were noted among the scores of the three questionnaires and the APD tests. Correlations were the strongest for the gaps-in-noise, speech-in-babble, dichotic digit tests with all three questionnaires. Conclusions: The three validated questionnaires may help identify adults with normal hearing who need referral for APD assessment

Recognition and Comprehension of Speech in Noise in School-Aged Children with Unilateral Hearing Loss

Sentence recognition and auditory comprehension abilities of young adults with normal hearing (NH) and school-age children with NH and unilateral hearing loss (UHL) were tested in a mixed design. In Experiment 1, subjects’ sentence recognition abilities were measured in the presence of speech spectrum noise (SSN) and two-talker child babble (TTB) in co-located and spatially-separated target and masker configurations. In all conditions, reception thresholds for sentences (RTS) improved with age from six-to 12 years. Speech spectrum noise proved to be a more effective masker than TTB in all listening conditions, suggesting subjects were able to take advantage of temporal and spectral fluctuations in the masker. By 12 years of age, RTS appeared to be adult-like when children listened in the presence of SSN, but were still immature in TTB. Across all listening conditions, a majority of UHL subjects’ RTS fell outside ±1 standard deviation of the NH mean, indicating poorer performance for this group of listeners. Performance of UHL subjects heavily depended on spatial configuration and was poorest when the masker was directed towards their normal-hearing ear. In Experiment 2, subjects’ auditory comprehension abilities were measured in the presence of TTB at a variety of signal-to-noise ratios (SNRs). When averaged across age, NH subjects performed similarly across the different listening conditions. For most UHL subjects, performance was similar to NH subjects in all comprehension tasks suggesting like NH subjects they made use of story context to support understanding even when audibility was compromised and likely took advantage of gaps in the TTB and spatial separation of the target and masker to better glimpse/hear the target. The findings of the current study improve our understanding of both simple and complex auditory abilities of school-aged children with NH and UHL in classroom-like, noisy environments. Furthermore measurable auditory deficits were detected in the study’s sample of children with UHL