Cortical mechanisms of seeing and hearing speech

In face-to-face communication speech is perceived through eyes and ears. The talker's articulatory gestures are seen and the speech sounds are heard simultaneously. Whilst acoustic speech can be often understood without visual information, viewing articulatory gestures aids hearing substantially in noisy conditions. On the other hand, speech can be understood, to some extent, by solely viewing articulatory gestures (i.e., by speechreading). In this thesis, electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) were utilized to disclose cortical mechanisms of seeing and hearing speech. One of the major challenges of modern cognitive neuroscience is to find out how the brain integrates inputs from different senses. In this thesis, integration of seen and heard speech was investigated using EEG and MEG. Multisensory interactions were found in the sensory-specific cortices at early latencies and in the multisensory regions at late latencies. Viewing other person's actions activate regions belonging to the human mirror neuron system (MNS) which are also activated when subjects themselves perform actions. Possibly, the human MNS enables simulation of other person's actions, which might be important also for speech recognition. In this thesis, it was demonstrated with MEG that seeing speech modulates activity in the mouth region of the primary somatosensory cortex (SI), suggesting that also the SI cortex is involved in simulation of other person's articulatory gestures during speechreading. The question whether there are speech-specific mechanisms in the human brain has been under scientific debate for decades. In this thesis, evidence for the speech-specific neural substrate in the left posterior superior temporal sulcus (STS) was obtained using fMRI. Activity in this region was found to be greater when subjects heard acoustic sine wave speech stimuli as speech than when they heard the same stimuli as non-speech.reviewe

Audio-visual training effect on L2 perception and production of English /0/-/s/ and /d/-/z/ by Mandarin speakers

PhD ThesisResearch on L2 speech perception and production indicate that adult language learners are able to acquire L2 speech sounds that they initially have difficulty with (Best, 1994). Moreover, use of the audiovisual modality, which provides language learners with articulatory information for speech sounds, has been illustrated to be effective in L2 speech perception training (Hazan et al., 2005). Since auditory and visual skills are integrated with each other in speech perception, audiovisual perception training may enhance language learners’ auditory perception of L2 speech sounds (Bernstein, Auer Jr, Ebehardt, and Jiang, 2013). However, little research has been conducted on L1 Mandarin learners of English. Based on these hypotheses, this study investigated whether audiovisual perception training can improve learners’ auditory perception and production of L2 speech sounds. A pilot study was performed on 42 L1-Mandarin learners of English (L1-dialect: Chongqing Mandarin (CQd)) in which their perception and production of English consonants was tested. According to the results, 29 of the subjects had difficulty in the perception and production of /θ/-/s/ and /ð/-/z/. Therefore, these 29 subjects were selected as the experimental group to attend a 9-session audiovisual perception training programme, in which identification tasks for the minimal pairs /θ/-/s/ and /ð/-/z/ were conducted. The subjects’ perception and production performance was tested before, during and at the end of the training with an AXB task and “read aloud” task. In view of the threat to interval validity arising from a repeated testing effect, a control group was tested with the same AXB task and intervals as that of the experimental group. The results show that the experimental group’s perception and production accuracy improved substantially during and by the end of the training programme. Indeed, whilst the control group also showed perception improvement across the pre-test and post-test, their degree of improvement was significantly lower than that of the experimental group. These results therefore confirm the value of the audiovisual modality in L2 speech perception training

Windows into Sensory Integration and Rates in Language Processing: Insights from Signed and Spoken Languages

This dissertation explores the hypothesis that language processing proceeds in "windows" that correspond to representational units, where sensory signals are integrated according to time-scales that correspond to the rate of the input. To investigate universal mechanisms, a comparison of signed and spoken languages is necessary. Underlying the seemingly effortless process of language comprehension is the perceiver's knowledge about the rate at which linguistic form and meaning unfold in time and the ability to adapt to variations in the input. The vast body of work in this area has focused on speech perception, where the goal is to determine how linguistic information is recovered from acoustic signals. Testing some of these theories in the visual processing of American Sign Language (ASL) provides a unique opportunity to better understand how sign languages are processed and which aspects of speech perception models are in fact about language perception across modalities. The first part of the dissertation presents three psychophysical experiments investigating temporal integration windows in sign language perception by testing the intelligibility of locally time-reversed sentences. The findings demonstrate the contribution of modality for the time-scales of these windows, where signing is successively integrated over longer durations (~ 250-300 ms) than in speech (~ 50-60 ms), while also pointing to modality-independent mechanisms, where integration occurs in durations that correspond to the size of linguistic units. The second part of the dissertation focuses on production rates in sentences taken from natural conversations of English, Korean, and ASL. Data from word, sign, morpheme, and syllable rates suggest that while the rate of words and signs can vary from language to language, the relationship between the rate of syllables and morphemes is relatively consistent among these typologically diverse languages. The results from rates in ASL also complement the findings in perception experiments by confirming that time-scales at which phonological units fluctuate in production match the temporal integration windows in perception. These results are consistent with the hypothesis that there are modality-independent time pressures for language processing, and discussions provide a synthesis of converging findings from other domains of research and propose ideas for future investigations