Cortical tracking of surprisal during continuous speech comprehension

Speech comprehension requires rapid online processing of a continuous acoustic signal to extract structure and meaning. Previous studies on sentence comprehension have found neural correlates of the predictability of a word given its context, as well as a of the precision of such a prediction. However, they have focussed on single sentences and on particular words in those sentences. Moreover, they compared neural responses to words with low and high predictability, as well as with low and high precision. However, in speech comprehension a listener hears many successive words whose predictability and precision vary over a large range. Here we show that cortical activity in different frequency bands tracks word surprisal in continuous natural speech, and that this tracking is modulated by precision. We obtain these results through quantifying surprisal and precision from naturalistic speech using a deep neural network, and through relating these speech features to electroencephalographic (EEG) responses of human volunteers acquired during auditory story comprehension. We find significant cortical tracking of surprisal at low frequencies including the delta band as well as in the higher-frequency beta and gamma bands, and observe that the tracking is modulated by the precision. Our results pave the way to further investigate the neurobiology of natural speech comprehension

Contributions of local speech encoding and functional connectivity to audio-visual speech perception

Seeing a speaker’s face enhances speech intelligibility in adverse environments. We investigated the underlying network mechanisms by quantifying local speech representations and directed connectivity in MEG data obtained while human participants listened to speech of varying acoustic SNR and visual context. During high acoustic SNR speech encoding by temporally entrained brain activity was strong in temporal and inferior frontal cortex, while during low SNR strong entrainment emerged in premotor and superior frontal cortex. These changes in local encoding were accompanied by changes in directed connectivity along the ventral stream and the auditory-premotor axis. Importantly, the behavioral benefit arising from seeing the speaker’s face was not predicted by changes in local encoding but rather by enhanced functional connectivity between temporal and inferior frontal cortex. Our results demonstrate a role of auditory-frontal interactions in visual speech representations and suggest that functional connectivity along the ventral pathway facilitates speech comprehension in multisensory environments

Stories (PhD thesis Kandylaki K.D.)

These stories were the material I created for my PhD thesis project and I would like to make available for further research. The stories are in German language, 40 stories in total, which consist of 20 original and 20 twin stories. Along with the audio files (wav format) of the experimental stories I have uploaded audio files of two practice stories and their transcription, the pdf transcription of the experimental stories and the files I used to construct and control the stories.<br><br>Please feel free to contact me for enquires on the architecture of the stories and on possible collaborations using this material. I am curious to see these stories from a new perspective.<br

Put it in context: the neurobiology of language explored with controlled stimuli in naturalistic auditory stories.

The current thesis presents a novel approach to studying the neurobiology of language. We embedded a variety of controlled stimuli into naturalistic stories thereby aiming for high ecological validity of the experiment and the results. The different manipulations in the stories ranged from phonology, syntax and semantics through to the interplay of language and social cognition. In order to simulate natural language comprehension, the stories were presented auditorily and included rich contextual information. In a functional magnetic resonance imaging (fMRI) experiment, participants listened to the stories and answered two comprehension questions after each story, in order to ensure they were listening attentively. The findings of each manipulation were inline with previous findings on the topics under consideration and offered novel insights into the neurobiological implementation of language. This research provides supporting evidence for the need of a new conceptualisation of language processing, rooted in neurobiologically plausible mechanisms of the human brain

Predicting "when" in discourse engages the human dorsal auditory stream: an fMRI study using naturalistic stories

The hierarchical organisation of human cortical circuits integrates information across different timescales via temporal receptive windows (TRWs), which increase in length from lower to higher levels of the cortical hierarchy (Hasson et al., 2015). A recent neurobiological model of higher-order language processing (Bornkessel-Schlesewsky et al., 2015) posits that TRWs in the dorsal auditory stream provide the basis for a hierarchically organised predictive coding architecture (Friston and Kiebel, 2009). In this stream, a nested set of internal models generates time-based (“when”) predictions for upcoming input at different linguistic levels (sounds, words, sentences, discourse). Here, we used naturalistic stories to test the hypothesis that multi-sentence, discourse-level predictions are processed in the dorsal auditory stream, yielding attenuated blood-oxygen-level-dependent (BOLD) responses for highly predicted versus less strongly predicted language input. The results were as hypothesised: discourse-related cues such as passive voice, which effect a higher predictability of remention for a character at a later point within a story, led to attenuated BOLD responses for auditory input of high versus low predictability within the dorsal auditory stream, specifically in the inferior parietal lobule (IPL), middle frontal gyrus (MFG) and dorsal parts of the inferior frontal gyrus (IFG) among other areas. Additionally, we found effects of content-related (“what”) predictions in ventral regions. These findings provide novel evidence that hierarchical predictive coding extends to discourse-level processing in natural language. Importantly, they ground language processing on a hierarchically organised predictive network, as a common underlying neurobiological basis shared with other brain functions

Neural representations of non-native speech reflect proficiency and interference from native language knowledge

Learning to process speech in a foreign language involves learning new representations for mapping the auditory signal to linguistic structure. Behavioral experiments suggest that even listeners that are highly proficient in a non-native language experience interference from representations of their native language. However, much of the evidence for such interference comes from tasks that may inadvertently increase the salience of native language competitors. Here we tested for neural evidence of proficiency and native language interference in a naturalistic story listening task. We studied electroencephalography responses of 39 native speakers of Dutch (14 male) to an English short story, spoken by a native speaker of either American English or Dutch. We modeled brain responses with multivariate temporal response functions, using acoustic and language models. We found evidence for activation of Dutch language statistics when listening to English, but only when it was spoken with a Dutch accent. This suggests that a naturalistic, monolingual setting decreases the interference from native language representations, whereas an accent in the listeners' own native language may increase native language interference, by increasing the salience of the native language and activating native language phonetic and lexical representations. Brain responses suggest that such interference stems from words from the native language competing with the foreign language in a single word recognition system, rather than being activated in a parallel lexicon. We further found that secondary acoustic representations of speech (after 200 ms latency) decreased with increasing proficiency. This may reflect improved acoustic-phonetic models in more proficient listeners. Behavioral experiments suggest that native language knowledge interferes with foreign language listening, but such effects may be sensitive to task manipulations, as tasks that increase metalinguistic awareness may also increase native language interference. This highlights the need for studying non-native speech processing using naturalistic tasks. We measured neural responses unobtrusively while participants listened for comprehension, and characterized the influence of proficiency at multiple levels of representation. We found that salience of the native language, as manipulated through speaker accent, affected activation of native language representations: significant evidence for activation of native language (Dutch) categories was only obtained when the speaker had a Dutch accent, whereas no significant interference was found to a speaker with a native (American) accent

Put it in context: the neurobiology of language explored with controlled stimuli in naturalistic auditory stories.

The current thesis presents a novel approach to studying the neurobiology of language. We embedded a variety of controlled stimuli into naturalistic stories thereby aiming for high ecological validity of the experiment and the results. The different manipulations in the stories ranged from phonology, syntax and semantics through to the interplay of language and social cognition. In order to simulate natural language comprehension, the stories were presented auditorily and included rich contextual information. In a functional magnetic resonance imaging (fMRI) experiment, participants listened to the stories and answered two comprehension questions after each story, in order to ensure they were listening attentively. The findings of each manipulation were inline with previous findings on the topics under consideration and offered novel insights into the neurobiological implementation of language. This research provides supporting evidence for the need of a new conceptualisation of language processing, rooted in neurobiologically plausible mechanisms of the human brain