When eye meets ear : an investigation of audiovisual speech and non-speech perception in younger and older adults

This dissertation addressed important questions regarding audiovisual (AV) perception. Study 1 revealed that AV speech perception modulated auditory processes, whereas AV non-speech perception affected visual processes. Interestingly, stimulus identification improved, yet fewer neural resources, as reflected in smaller event-related potentials, were recruited, indicating that AV perception led to multisensory efficiency. Also, AV interaction effects were observed of early and late stages, demonstrating that multisensory integration involved a neural network. Study 1 showed that multisensory efficiency is a common principle in AV speech and non-speech stimulus recognition, yet it is reflected in different modalities, possibly due to sensory dominance of a given task. Study 2 extended our understanding of multisensory interaction by investigating electrophysiological processes of AV speech perception in noise and whether those differ between younger and older adults. Both groups revealed multisensory efficiency. Behavioural performance improved while the auditory N1 amplitude was reduced during AV relative to unisensory speech perception. This amplitude reduction could be due to visual speech cues providing complementary information, therefore reducing processing demands for the auditory system. AV speech stimuli also led to an N1 latency shift, suggesting that auditory processing was faster during AV than during unisensory trials. This shift was more pronounced in older than in younger adults, indicating that older adults made more effective use of visual speech. Finally, auditory functioning predicted the degree of the N1 latency shift, which is consistent with the inverse effectiveness hypothesis which argues that the less effective the unisensory perception was, the larger was the benefit derived from AV speech cues. These results suggest that older adults were better "lip/speech" integrators than younger adults, possibly to compensate for age-related sensory deficiencies. Multisensory efficiency was evident in younger and older adults but it might be particularly relevant for older adults. If visual speech cues could alleviate sensory perceptual loads, the remaining neural resources could be allocated to higher level cognitive functions. This dissertation adds further support to the notion of multisensory interaction modulating sensory-specific processes and it introduces the concept of multisensory efficiency as potential principle underlying AV speech and non-speech perceptio

Intelligibility and Listening Effort of Spanish Oesophageal Speech

Communication is a huge challenge for oesophageal speakers, be it for interactions with fellow humans or with digital voice assistants. We aim to quantify these communication challenges (both human-human and human-machine interactions) by measuring intelligibility and Listening Effort (LE) of Oesophageal Speech (OS) in comparison to Healthy Laryngeal Speech (HS). We conducted two listening tests (one web-based, the other in laboratory settings) to collect these measurements. Participants performed a sentence recognition and LE rating task in each test. Intelligibility, calculated as Word Error Rate, showed significant correlation with self-reported LE ratings. Speaker type (healthy or oesophageal) had a major effect on intelligibility and effort. More LE was reported for OS compared to HS even when OS intelligibility was close to HS. Listeners familiar with OS reported less effort when listening to OS compared to nonfamiliar listeners. However, such advantage of familiarity was not observed for intelligibility. Automatic speech recognition scores were higher for OS compared to HS.This project was supported by funding from the EUs H2020 research and innovation programme under the MSCA GA 67532*4 (the ENRICH network: www.enrich-etn.eu), the Spanish Ministry of Economy and Competitiveness with FEDER support (RESTORE project, TEC2015-67163-C2-1-R) and the Basque Government (DL4NLP KK-2019/00045, PIBA_2018_1_0035 and IT355-19)

Intelligibility and Listening Effort of Spanish Oesophageal Speech

Communication is a huge challenge for oesophageal speakers, be it for interactions with fellow humans or with digital voice assistants. We aim to quantify these communication challenges (both human-human and human-machine interactions) by measuring intelligibility and Listening Effort (LE) of Oesophageal Speech (OS) in comparison to Healthy Laryngeal Speech (HS). We conducted two listening tests (one web-based, the other in laboratory settings) to collect these measurements. Participants performed a sentence recognition and LE rating task in each test. Intelligibility, calculated as Word Error Rate, showed significant correlation with self-reported LE ratings. Speaker type (healthy or oesophageal) had a major effect on intelligibility and effort. More LE was reported for OS compared to HS even when OS intelligibility was close to HS. Listeners familiar with OS reported less effort when listening to OS compared to nonfamiliar listeners. However, such advantage of familiarity was not observed for intelligibility. Automatic speech recognition scores were higher for OS compared to HS.This project was supported by funding from the EUs H2020 research and innovation programme under the MSCA GA 67532*4 (the ENRICH network: www.enrich-etn.eu), the Spanish Ministry of Economy and Competitiveness with FEDER support (RESTORE project, TEC2015-67163-C2-1-R) and the Basque Government (DL4NLP KK-2019/00045, PIBA_2018_1_0035 and IT355-19)

Comunicación enriquecida a lo largo de la vida

Speech is a hugely efficient means of communication: a reduced capacity in listening or speaking creates a significant barrier to social inclusion at all points through the lifespan, in education, work and at home. Hearing devices and speech synthesis can help address this reduced capacity but their use imposes greater listener effort. The goal of the EU-funded ENRICH project is to modify or augment speech with additional information to make it easier to process. Enrichment reduces listening burden by minimising cognitive load, while maintaining or improving intelligibility. ENRICH investigates the relationship between cognitive effort and natural and synthetic speech. Non-intrusive metrics for listening effort will be developed and used to design modification techniques which result in low-burden speech. The value of various enrichment approaches will be evaluated with individuals and cohorts with typically sub-optimal communication ability, such as children, hearing-or speech-impaired adults, non-native listeners and individuals engaged in simultaneous tasks.El habla es un medio de comunicación sumamente eficiente: la capacidad reducida para oír o hablar crea barreras importantes para la inclusión social a lo largo de la vida en la educación, en el trabajo o en el hogar. Los audífonos y la síntesis del habla pueden ayudar a abordar esta capacidad reducida, pero su uso impone un mayor esfuerzo por parte del oyente. El objetivo del proyecto europeo ENRICH es modificar o aumentar el habla con información adicional resultando así más fácil de procesar. El enriquecimiento reduce el esfuerzo de escucha minimizando la carga cognitiva, mientras se mantiene o mejora la inteligibilidad. ENRICH investigará la relación entre el esfuerzo cognitivo y las diferentes formas de habla natural y sintética. Se desarrollarán métricas no intrusivas para el esfuerzo de escucha que se utilizarán para diseñar modificaciones que resulten en un habla de baja carga. El valor de los diversos enfoques de enriquecimiento se evaluará con individuos y cohortes con habilidades de comunicación típicamente subóptimas, como niños, adultos con problemas de audición o de habla, oyentes no nativos e individuos que realizan tareas simultáneas.ENRICH has received funding from the EU H2020 research and innovation programme under MSCA GA 675324

Assessing and reducing listening effort of listening to speech in adverse conditions

Background: Normally hearing listeners successfully compensate for speech distortions in everyday environments, but can become fatigued as a result. AdaptDRC is a near-end-listening-enhancement algorithm that alters speech signals for playback, dependent on environmental noise, for improved intelligibility (Schepker et al., 2013). Aim: In this electroencephalography (EEG) study I measured neurophysiological correlates of listening effort (LE) by comparing unprocessed speech to AdaptDRC-processed speech. Method: I recorded EEG while normally hearing participants (N=27) listened to unprocessed or AdaptDRC-processed sentences in noise, then rated the subjective listening effort. I also measured speech intelligibility, hearing and cognitive abilities. Results: For intelligible speech, subjective LE decreases with increasing SNR and is lower for AdaptDRC speech than unprocessed speech. Spectral alpha power (8-12Hz) analyses suggest that peak cognitive effort occurs at 0dB SNR. Spectral alpha also increases with task duration, indicating an association with fatigue. Conclusions: This experiment provides insight into the neurophysiological correlates of effortful listening in adverse conditions, and the benefits of near-end-listening-enhancement technology

EEG alpha power as a measure of listening effort reduction in adverse conditions

Noise levels in everyday environments are typically low enough to comprehend speech, but over extended periods of time, listeners may experience tiredness and even fatigue. We measured the neurophysiological compensation required to separate speech from noise listening effort using electroencephalography, for normal and AdaptDRC enhanced speech. AdaptDRC is a noise-dependent near-end-listening-enhancement algorithm which significantly improves the intelligibility of speech in noise and reduces subjectively rated listening effort (Rennies et al., 2018). Participants (N=27) completed a listening effort and intelligibility task using a categorical listening effort scale, in which sentences were presented at five SNRs and in two types of noise. Subjective listening effort was significantly lower, and speech intelligibility was significantly higher for AdaptDRC speech than unprocessed speech (p<.001). Analyses of the neurophysiological data show that there is a non-linear relationship between SNR and spectral alpha power with a peak between 0-5dB SNR, and that alpha power is sensitive to noise type and speech processing. These findings provide insight into the neurophysiological correlates of listening effort and aid the development of an objective measure of cognitive load for speech in noise

Intelligibility and Listening Effort of Spanish Oesophageal Speech

Communication is a huge challenge for oesophageal speakers, be it for interactions with fellow humans or with digital voice assistants. We aim to quantify these communication challenges (both human&ndash;human and human&ndash;machine interactions) by measuring intelligibility and Listening Effort (LE) of Oesophageal Speech (OS) in comparison to Healthy Laryngeal Speech (HS). We conducted two listening tests (one web-based, the other in laboratory settings) to collect these measurements. Participants performed a sentence recognition and LE rating task in each test. Intelligibility, calculated as Word Error Rate, showed significant correlation with self-reported LE ratings. Speaker type (healthy or oesophageal) had a major effect on intelligibility and effort. More LE was reported for OS compared to HS even when OS intelligibility was close to HS. Listeners familiar with OS reported less effort when listening to OS compared to nonfamiliar listeners. However, such advantage of familiarity was not observed for intelligibility. Automatic speech recognition scores were higher for OS compared to HS

Moderate Cardiovascular Exercise Speeds Up Neural Markers of Stimulus Evaluation During Attentional Control Processes

Moderate intensity cardiovascular exercise appears to provide a low-cost “intervention” on neurocognitive processes such as attentional control, yet the effects vary depending, for example, on cognitive task, time of testing, or exercise intensity. However, while a number of studies show that brief bouts of acute exercise can modulate behavioral indices of cognitive control, relatively few studies have attempted to identify the brain activity associated with these changes immediately following exercise. Here, we tested 11 young adults in a crossover design with a Flanker task at rest and immediately (within 2–3 min) following 20 min of acute exercise at 60% of the individual VO2max. In order to prevent delayed exercise effects that might confound or dilute immediate effects, a short version of the Flanker task (8 min) was chosen and an EEG was recorded simultaneously. The N2 and P3 ERP components were analyzed in addition to accuracy and response time. The N2 reflects conflict resolution, and the P3 has been linked to stimulus evaluation processes. No effect of exercise was found for behavioral data but P3 peak latencies were shorter following exercise as compared to rest. The N2 amplitude data suggest that exercise seems to prevent a decline in resources of attentional control over time. These data indicate that acute exercise, at a moderate intensity level, speeds up neural processing of attentional control by modulating stimulus evaluation processes immediately following exercise and that exercise helps maintain a steady level of neurocognitive resources