On The Way To Linguistic Representation: Neuromagnetic Evidence of Early Auditory Abstraction in the Perception of Speech and Pitch

The goal of this dissertation is to show that even at the earliest (non-invasive) recordable stages of auditory cortical processing, we find evidence that cortex is calculating abstract representations from the acoustic signal. Looking across two distinct domains (inferential pitch perception and vowel normalization), I present evidence demonstrating that the M100, an automatic evoked neuromagnetic component that localizes to primary auditory cortex is sensitive to abstract computations. The M100 typically responds to physical properties of the stimulus in auditory and speech perception and integrates only over the first 25 to 40 ms of stimulus onset, providing a reliable dependent measure that allows us to tap into early stages of auditory cortical processing. In Chapter 2, I briefly present the episodicist position on speech perception and discuss research indicating that the strongest episodicist position is untenable. I then review findings from the mismatch negativity literature, where proposals have been made that the MMN allows access into linguistic representations supported by auditory cortex. Finally, I conclude the Chapter with a discussion of the previous findings on the M100/N1. In Chapter 3, I present neuromagnetic data showing that the re-sponse properties of the M100 are sensitive to the missing fundamental component using well-controlled stimuli. These findings suggest that listeners are reconstructing the inferred pitch by 100 ms after stimulus onset. In Chapter 4, I propose a novel formant ratio algorithm in which the third formant (F3) is the normalizing factor. The goal of formant ratio proposals is to provide an explicit algorithm that successfully "eliminates" speaker-dependent acoustic variation of auditory vowel tokens. Results from two MEG experiments suggest that auditory cortex is sensitive to formant ratios and that the perceptual system shows heightened sensitivity to tokens located in more densely populated regions of the vowel space. In Chapter 5, I report MEG results that suggest early auditory cortical processing is sensitive to violations of a phonological constraint on sound sequencing, suggesting that listeners make highly specific, knowledge-based predictions about rather abstract anticipated properties of the upcoming speech signal and violations of these predictions are evident in early cortical processing

Speaker-normalized sound representations in the human auditory cortex

The acoustic dimensions that distinguish speech sounds (like the vowel differences in “boot” and “boat”) also differentiate speakers’ voices. Therefore, listeners must normalize across speakers without losing linguistic information. Past behavioral work suggests an important role for auditory contrast enhancement in normalization: preceding context affects listeners’ perception of subsequent speech sounds. Here, using intracranial electrocorticography in humans, we investigate whether and how such context effects arise in auditory cortex. Participants identified speech sounds that were preceded by phrases from two different speakers whose voices differed along the same acoustic dimension as target words (the lowest resonance of the vocal tract). In every participant, target vowels evoke a speaker-dependent neural response that is consistent with the listener’s perception, and which follows from a contrast enhancement model. Auditory cortex processing thus displays a critical feature of normalization, allowing listeners to extract meaningful content from the voices of diverse speakers

Neural and Behavioral Responses to the Use of Auditory Feedback in Vocal Control

A large body of evidence suggests that the motor system maintains a forward model that predicts the sensory outcome of movements. When sensory feedback does not match the predicted consequences, a compensatory response corrects for the motor error and the forward model is updated to prevent future errors. Like other motor behaviours, vocalization relies on sensory feedback for the maintenance of forward models and to stabilize vocalizations. Experiment 1 used event-related potentials (ERPs) to examine sensory processing of short feedback perturbations during an ongoing utterance. In one session, participants produced a vowel at an FO of their own choosing. In another session, participants matched the FO of a cue voice. An FO perturbation of 0,25, 50,100, or 200 cents was introduced for 100 ms. A mismatch negativity (MMN) was observed. Differences between sessions were only found for 200 cents perturbations. Reduced compensation when speakers experienced the 200 cents perturbations suggests that this larger perturbation was perceived as externally generated. The presence of an MMN, and no earlier (N100) response suggests that the underlying sensory process used to identify and compensate for errors in mid-utterance may differ from feedback monitoring at utterance onset. In Experiment 2, we used a frequency altered feedback (FAF) paradigm to study the role of auditory feedback in the control of vocal pitch (F0). We adapted participants to a one semitone shift and induced a perturbation by briefly removing the altered feedback. This was compared to a control block in which a 1 semitone perturbation was introduced into an unshifted trial, or trials were randomly shifted up 1 semitone, and a perturbation was introduced by removing the feedback alteration. The compensation response to mid-utterance perturbations was identical in all conditions, and was always smaller than the compensation to a shift at utterance onset. These results are explained by a change in the control strategy at utterance onset and midutterance. At utterance onset, auditory feedback is compared to feedback predicted by a forward model to ensure the pitch goal is achieved. However, after utterance onset, the control strategy switches and stabilization is maintained by comparing feedback to previous FO production. Experiment 1 showed a MMN in response to a mid-utterance perturbation, which is distinct from the N100 found in previous studies that examined perturbations at utterance onset. This result suggests that there may be different underlying neurological mechanisms for the detection of perturbations at utterance onset and mid-utterance. Experiment 2 adds support for this idea by showing a difference in the compensation responses to mid-utterance and onset perturbations. We conclude that different mechanisms may be used to detect errors and compensate for these errors at utterance onset versus mid-utterance

Training of Working Memory Impacts Neural Processing of Vocal Pitch Regulation

Working memory training can improve the performance of tasks that were not trained. Whether auditory-motor integration for voice control can benefit from working memory training, however, remains unclear. The present event-related potential (ERP) study examined the impact of working memory training on the auditory-motor processing of vocal pitch. Trained participants underwent adaptive working memory training using a digit span backwards paradigm, while control participants did not receive any training. Before and after training, both trained and control participants were exposed to frequency-altered auditory feedback while producing vocalizations. After training, trained participants exhibited significantly decreased N1 amplitudes and increased P2 amplitudes in response to pitch errors in voice auditory feedback. In addition, there was a significant positive correlation between the degree of improvement in working memory capacity and the post-pre difference in P2 amplitudes. Training-related changes in the vocal compensation, however, were not observed. There was no systematic change in either vocal or cortical responses for control participants. These findings provide evidence that working memory training impacts the cortical processing of feedback errors in vocal pitch regulation. This enhanced cortical processing may be the result of increased neural efficiency in the detection of pitch errors between the intended and actual feedback

MEG, PSYCHOPHYSICAL AND COMPUTATIONAL STUDIES OF LOUDNESS, TIMBRE, AND AUDIOVISUAL INTEGRATION

Natural scenes and ecological signals are inherently complex and understanding of their perception and processing is incomplete. For example, a speech signal contains not only information at various frequencies, but is also not static; the signal is concurrently modulated temporally. In addition, an auditory signal may be paired with additional sensory information, as in the case of audiovisual speech. In order to make sense of the signal, a human observer must process the information provided by low-level sensory systems and integrate it across sensory modalities and with cognitive information (e.g., object identification information, phonetic information). The observer must then create functional relationships between the signals encountered to form a coherent percept. The neuronal and cognitive mechanisms underlying this integration can be quantified in several ways: by taking physiological measurements, assessing behavioral output for a given task and modeling signal relationships. While ecological tokens are complex in a way that exceeds our current understanding, progress can be made by utilizing synthetic signals that encompass specific essential features of ecological signals. The experiments presented here cover five aspects of complex signal processing using approximations of ecological signals : (i) auditory integration of complex tones comprised of different frequencies and component power levels; (ii) audiovisual integration approximating that of human speech; (iii) behavioral measurement of signal discrimination; (iv) signal classification via simple computational analyses and (v) neuronal processing of synthesized auditory signals approximating speech tokens. To investigate neuronal processing, magnetoencephalography (MEG) is employed to assess cortical processing non-invasively. Behavioral measures are employed to evaluate observer acuity in signal discrimination and to test the limits of perceptual resolution. Computational methods are used to examine the relationships in perceptual space and physiological processing between synthetic auditory signals, using features of the signals themselves as well as biologically-motivated models of auditory representation. Together, the various methodologies and experimental paradigms advance the understanding of ecological signal analytics concerning the complex interactions in ecological signal structure

Voice and speech perception in autism : a systematic review

Autism spectrum disorders (ASD) are characterized by persistent impairments in social communication and interaction, restricted and repetitive behavior. In the original description of autism by Kanner (1943) the presence of emotional impairments was already emphasized (self-absorbed, emotionally cold, distanced, and retracted). However, little research has been conducted focusing on auditory perception of vocal emotional cues, being the audio-visual comprehension most commonly explored instead. Similarly to faces, voices play an important role in social interaction contexts in which individuals with ASD show impairments. The aim of the current systematic review was to integrate evidence from behavioral and neurobiological studies for a more comprehensive understanding of voice processing abnormalities in ASD. Among different types of information that the human voice may provide, we hypothesize particular deficits with vocal affect information processing by individuals with ASD. The relationship between vocal stimuli impairments and disrupted Theory of Mind in Autism is discussed. Moreover, because ASD are characterized by deficits in social reciprocity, further discussion of the abnormal oxytocin system in individuals with ASD is performed as a possible biological marker for abnormal vocal affect information processing and social interaction skills in ASD population

Cortical mechanisms of seeing and hearing speech

In face-to-face communication speech is perceived through eyes and ears. The talker's articulatory gestures are seen and the speech sounds are heard simultaneously. Whilst acoustic speech can be often understood without visual information, viewing articulatory gestures aids hearing substantially in noisy conditions. On the other hand, speech can be understood, to some extent, by solely viewing articulatory gestures (i.e., by speechreading). In this thesis, electroencephalography (EEG), magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) were utilized to disclose cortical mechanisms of seeing and hearing speech. One of the major challenges of modern cognitive neuroscience is to find out how the brain integrates inputs from different senses. In this thesis, integration of seen and heard speech was investigated using EEG and MEG. Multisensory interactions were found in the sensory-specific cortices at early latencies and in the multisensory regions at late latencies. Viewing other person's actions activate regions belonging to the human mirror neuron system (MNS) which are also activated when subjects themselves perform actions. Possibly, the human MNS enables simulation of other person's actions, which might be important also for speech recognition. In this thesis, it was demonstrated with MEG that seeing speech modulates activity in the mouth region of the primary somatosensory cortex (SI), suggesting that also the SI cortex is involved in simulation of other person's articulatory gestures during speechreading. The question whether there are speech-specific mechanisms in the human brain has been under scientific debate for decades. In this thesis, evidence for the speech-specific neural substrate in the left posterior superior temporal sulcus (STS) was obtained using fMRI. Activity in this region was found to be greater when subjects heard acoustic sine wave speech stimuli as speech than when they heard the same stimuli as non-speech.reviewe

Neural mechanisms of foreign language phoneme acquisition in early adulthood : MEG study

Tämän tutkimuksen tavoitteena on selvittää omaan äidinkieleen kuulumattomien foneemikontrastien oppimisen mekanismeja nuorilla aikuisilla neurofysiologisten ja behavioraalisten menetelmien avulla. Perinteisesti kielen foneettisen avaruuden omaksumisen on ajateltu tapahtuvan ensisijaisesti varhaislapsuuden kielellisten herkkyyskausien aikana, jonka jälkeen uusien foneemien oppiminen on haastavaa. Myöhemmät tutkimukset ovat kuitenkin osoittaneet, että vieraiden foneemien omaksuminen on mahdollista myös aikuisiällä. Uusien foneemikategorioiden muodostuminen vaatii aivoissa solutason plastisia muutoksia. Aivojen kykyä erotella läheisesti toisiaan muistuttavia foneemikategorioita kielenprosessoinnin varhaisella tasolla on tutkittu neurofysiologisin menetelmin esimerkiksi tapahtumasidonnaisen poikkeavuusnegatiivisuusvasteen (eng. mismatch negativity, MMN) avulla. MMN-vaste, tai sen magneettinen vastine MMNm, syntyy seurauksena muutoksiin sensorisessa havaintoympäristössä. Tutkimuksissa lyhyenkin auditiivisen harjoittelujakson on havaittu vahvistavan aivojen kykyä erotella läheisesti toisiaan muistuttavia vieraita foneemeja ja voimistavan MMN- ja MMNm-vasteita. Tässä tutkimuksessa vieraan kielen foneettisen oppimisen neuraalista perustaa ja oppimisen aiheuttamia plastisia muutoksia aivoissa tutkittiin magnetoenkefalografialla (MEG) neuromagneettisten tapahtumasidonnaisten vasteiden (erityisesti MMNm) avulla. Tutkimuksessa mitattiin 20 suomalaista koehenkilöä, joiden tehtävänä oli oppia erottelemaan akustisesti toisiaan läheisesti muistuttavia venäjän kielen frikatiiveja Ш /ʂ/ ja Щ /ɕ(ː)/. Erottelukykyä mitattiin ensin behavioraalisella tehtävällä, jossa koehenkilöille toistettiin nauhoitettuja venäjänkielisiä epäsanaminimipareja, jossa sanan ensimmäistä foneemia varioitiin. Koehenkilöiden tehtävänä oli vastata, kuulivatko he sanoissa eroa. Samoja kuuloärsykkeitä toistettiin koehenkilöille sen jälkeen passiivisessa MEG-tehtävässä, jossa testattiin aivojen kykyä havaita ero ärsykkeissä ilman, että niihin kiinnitetään huomiota (koehenkilöt katselivat samalla äänetöntä elokuvaa). Mittauksen jälkeen koehenkilöt harjoittelivat foneemien erottelua kotona noin viikon ajan tietokoneavusteisen oppimispelin avulla, jonka jälkeen heidät mitattiin uudelleen. MEG-signaalien lähdemallinnusta varten koehenkilöiden aivoista otettiin myös rakenteelliset magneettikuvat. Tutkittavien foneemien behavioraalinen erottelukyky oli selvästi tuttuja kontrollifoneemeita heikompaa. Erottelukyky vaikutti paranevan harjoittelun seurauksena hieman, mutta ero ei ollut tilastollisesti merkitsevä. Hypoteesien vastaisesti tilastollisesti merkitseviä MMNm-vasteita ei löydetty ennen eikä jälkeen harjoittelun, eikä muissakaan auditorisissa MEG-vasteissa tai niiden neuraalisten lähdevirtojen voimakkuuksissa tai jakaumassa ollut tilastollisesti merkitsevää eroa mittauskertojen välillä. Yksilölliset erot oppimisessa olivat kuitenkin suuria. Koehenkilöillä, joilla behavioraalinen erottelukyky parani harjoittelun myötä, oli silmämääräisesti havaittavissa hypoteesien mukaista vahvistumista auditorisissa vasteissa. Vaikka efekti oli erittäin pieni eikä tilastollisesti merkitsevä, vastaavaa ei havaittu epäoppijoilla eikä kontrollitilanteessa. Tässä tutkimuksessa ei kyetty replikoimaan aiempien tutkimusten tuloksia foneemien omaksumisesta aikuisiällä. Vaikka on todennäköistä, että tietyt metodologiset heikkoudet (mm. vähäinen ärsykkeiden määrä MEG-tehtävässä, haastavat ärsykkeet) vaikuttivat tulosten merkitsevyyteen, voidaan tämän tutkimuksen valossa aiempien tutkimustulosten yleistettävyyttä kyseenalaistaa.The aim of this study is to examine the learning mechanisms and acquisition of non-native phoneme contrasts in young adults using neurophysiological and behavioral methods. According to the traditional view, acquiring novel phonemes after the sensitive periods in the early childhood is very difficult. However, later findings have shown that foreign phoneme contrasts can be learned at a later age, too. Acquiring new phonemic categories requires neuroplastic changes in the brain. Neurophysiological studies have examined the brain’s ability to differentiate between closely related phonemic categories at the early stage of spoken language processing by measuring, for example, event-related mismatch negativity responses (MMN). MMN, or its magnetic equivalent MMNm, is elicited when the brain registers a difference in a repetitive sensory stimulus. Studies have shown that even a moderate amount of auditory training with closely related foreign phonemes improves the brain’s ability to discriminate between them resulting in enhanced MMN or MMNm responses. In this experiment the neural mechanisms of foreign language phoneme acquisition and the learning-related neuroplastic changes were studied using magnetoencephalography (MEG) and neuromagnetic evoked responses (MMNm in particular). 20 Finnish subjects were measured in the experiment. Their task was to learn to differentiate between acoustically closely related Russian fricatives Ш /ʂ/ and Щ /ɕ(ː)/. The subjects’ differentiation skills were first tested in a behavioral task where Russian pseudoword minimal pairs were presented to them auditorily. The first phoneme in the word pairs was varied and the subjects had to report whether they heard a difference between the words or not. The same stimuli were then presented in a passive MEG task where the brain’s change detection responses were tested in an unattended situation as the subjects were watching a silent film. After the measurement the subjects practiced the phonemes at home for approximately one week by playing a learning game by computer. After training they were measured again. Structural magnetic resonance images of the subjects’ brain were also measured for MEG source localization purposes. Behavioral discrimination ability of the experimental phonemes was considerably worse than with familiar control phonemes. The discrimination skills seemed to improve by training, but the difference was not statistically significant. Contrary to the hypotheses, statistically significant MMNm responses were not found before or after training. No significant differences were found in other auditory MEG responses or their neural source current distributions between the measurements either. However, individual differences in learning were sizeable. For the subjects who improved their performance in the behavioral task a modest training-related boost in the auditory responses supporting the hypotheses could be observed. Although very small and statistically insignificant, the effect was opposite for control stimuli and did not exist in the non-learner group suggesting some sort of change in neural processing in the learner group. This study was not able to replicate the findings from various previous studies on phoneme acquisition in adulthood. Although it is likely that certain methodological limitations (e.g. small number of stimulus repetitions, challenging stimuli) affected the significance of the results, based on this study the generalizability of some of the previous findings can be called into question

Training of Working Memory Impacts Neural Processing of Vocal Pitch Regulation

Working memory training can improve the performance of tasks that were not trained. Whether auditory-motor integration for voice control can benefit from working memory training, however, remains unclear. The present event-related potential (ERP) study examined the impact of working memory training on the auditory-motor processing of vocal pitch. Trained participants underwent adaptive working memory training using a digit span backwards paradigm, while control participants did not receive any training. Before and after training, both trained and control participants were exposed to frequency-altered auditory feedback while producing vocalizations. After training, trained participants exhibited significantly decreased N1 amplitudes and increased P2 amplitudes in response to pitch errors in voice auditory feedback. In addition, there was a significant positive correlation between the degree of improvement in working memory capacity and the post-pre difference in P2 amplitudes. Training-related changes in the vocal compensation, however, were not observed. There was no systematic change in either vocal or cortical responses for control participants. These findings provide evidence that working memory training impacts the cortical processing of feedback errors in vocal pitch regulation. This enhanced cortical processing may be the result of increased neural efficiency in the detection of pitch errors between the intended and actual feedback