Sentence context prevails over word association in aphasia patients with spared comprehension : evidence from N400 event-related potential

Behavioral and event-related potential (ERP) studies on aphasia patients showed that lexical information is not lost but rather its integration into the working context is hampered. Studies have been conducted on the processing of sentence-level information (meaningful versus meaningless) and of word-level information (related versus unrelated) in aphasia patients, but we are not aware of any study that assesses the relationship between the two. In healthy subjects the processing of a single word in a sentence context has been studied using the N400 ERP. It was shown that, even when there is only a weak expectation of a final word in a sentence, this expectation will dominate word relatedness. In order to study the effect of semantic relatedness between words in sentence processing in aphasia patients, we conducted a crossed design ERP study, crossing the factors of word relatedness and sentence congruity. We tested aphasia patients with mild to minimum comprehension deficit and healthy young and older (age-matched with our patients) controls on a semantic anomaly judgment task when simultaneously recording EEG. Our results show that our aphasia patient's N400 amplitudes in response to the sentences of our crossed-design study were similar to those of our age-matched healthy subjects. However, we detected an increase in the N400 ERP latency in those patients, indicating a delay in the integration of the new word into the working context. Additionally, we observed a positive correlation between comprehension level of those patients and N400 effect in response to meaningful sentences without word relatedness contrasted to meaningless sentences without word relatedness

Retrieval (N400) and integration (P600) in expectation-based comprehension

Expectation-based theories of language processing, such as Surprisal theory, are supported by evidence of anticipation effects in both behavioural and neurophysiological measures. Online measures of language processing, however, are known to be influenced by factors such as lexical association that are distinct from-but often confounded with-expectancy. An open question therefore is whether a specific locus of expectancy related effects can be established in neural and behavioral processing correlates. We address this question in an event-related potential experiment and a self-paced reading experiment that independently cross expectancy and lexical association in a context manipulation design. We find that event-related potentials reveal that the N400 is sensitive to both expectancy and lexical association, while the P600 is modulated only by expectancy. Reading times, in turn, reveal effects of both association and expectancy in the first spillover region, followed by effects of expectancy alone in the second spillover region. These findings are consistent with the Retrieval-Integration account of language comprehension, according to which lexical retrieval (N400) is facilitated for words that are both expected and associated, whereas integration difficulty (P600) will be greater for unexpected words alone. Further, an exploratory analysis suggests that the P600 is not merely sensitive to expectancy violations, but rather, that there is a continuous relation. Taken together, these results suggest that the P600, like reading times, may reflect a meaning-centric notion of Surprisal in language comprehension.SFB 1102, Deutsche Forschungsgemeinschaf

The electrophysiological reality of parafoveal processing: On the validity of language-related ERPs in natural reading

A central question in psycholinguistics is how the human brain processes language in real time. To answer this question, the differences between auditory and visual processing have to be considered. The present dissertation examines the extent to which event-related potentials (ERPs) in the human electroencephalogram (EEG) interact with different modes of presentation during sentence comprehension. Besides the two classical modalities, auditory and rapid serial visual presentation (RSVP), the monitoring of readers’ eye movements was chosen as a new mode of presentation. Here, the temporal paradox between neuronal ERP effects and behavioral effects in the eye movement record were of particular interest. Specifically, by concurrently measuring ERPs and eye movements in natural reading, the dissertation aimed to shed light on the counterintuitive fact that difficulties in sentence comprehension arise earlier in eye movement measures than in the corresponding neuronal ERP effects. In contrast to RSVP and the auditory modality, reading offers a parafoveal preview of upcoming words (Rayner 1998), which enables the brain to process information of words before these are fixated for the first time (in foveal vision). When the word Gegenteil in example (1) below is fixated and processed, the brain concurrently processes some information of the upcoming parafoveal words von and weiß. (1) Schwarz ist das Gegenteil von weiß. (2) Schwarz […] blau. (3) Schwarz […] nett. The parafoveal preview mostly provides orthographic (word form) information, while semantic information is not conveyed (Inhoff & Starr 2004; White 2008). Whereas word form and lexical meaning are processed simultaneously with RSVP and auditory presentation, the parafoveal preview in natural reading allows for a temporal decoupling such that word forms are processed before meaning. This is one reason for the faster information uptake in reading. The present dissertation is the first to systematically investigate the influence of the parafoveal preview in sentence processing. Participants read sentences such as in (1)-(3), in which two adjectives were either antonyms (1), semantically related non-antonyms (2), or semantically unrelated non-antonyms (3). ERPs were computed for the last fixation before the target word (the sentence-final word in 1-3), which was assumed to capture parafoveal processing, and for the first fixation on the target, that should reflect foveal processing. The results were compared to two experiments using identical stimuli with auditory and RSVP presentation, and the parafoveal preview clearly led to different ERP results. While the RSVP and auditory presentations replicated the finding of a P300 to the second antonym in (1) (Kutas & Iragui 1998; Roehm et al. 2007), there was no P300 in response to antonyms at any fixation position in natural reading. However, the dissociation of parafoveal and foveal processing in reading also made it possible to disentangle different processes underlying the N400. There was a reduced parafoveal N400 for (1,2) compared with (3), which could be attributed to the preactivation of the word forms of the expected antonyms and of semantically related non-antonyms. In foveal vision, all non-antonyms (2,3) showed an enhanced N400 compared with (1) because they were unexpected and implausible in the sentence context. This dissociation between the preactivation of a word-form and the contextual fit of a word’s meaning is impossible with the other two modes of presentation, because orthographic and semantic information become available almost at the same time and are thus processed simultaneously. Furthermore, the parafoveal N400 effect was not accompanied by changes in the duration of the corresponding fixation, whereas the foveal N400 was. Similarly, with the concurrent measurement of ERPs and eye movements, the temporal paradox described above remained, as effects in the eye movement record preceded the neuronal ERP effects. Further support for these central findings came from two additional experiments that investigated different stimuli with concurrent ERP-eye tracking measures. Altogether, the experiments revealed that the previous findings on the language-related N400 can be replicated with natural reading, but they can also be differentiated qualitatively by virtue of the characteristics of natural reading. Although the behavioral and neuronal effects mirrored one another, not every neuronal effect necessarily translates into a behavioral output. Finally, even concurrent ERP-eye tracking measures cannot resolve the temporal paradox

Precursors and downstream consequences of prediction in language comprehension

During language comprehension, the brain rapidly integrates incoming linguistic stimuli to not only incrementally build a contextual representation, but also predict upcoming information. This predictive mechanism leads to behavioral facilitation of processing of expected words, as well as a reduction in amplitude of the N400, a neural response reflecting access of semantic memory. However, little research has identified a behavioral or neurophysiological cost of errors in prediction. Additionally, only recent work has begun to investigate neural activity related to prediction prior to encountering a predicted stimulus. Most work has focused on what happens immediately after a predicted or unpredicted stimulus is encountered. Here, I explore new avenues of research by examining downstream consequences of prediction during language comprehension on future recognition memory. Additionally, I test whether these consequences occur following any violation of predictions, or whether the semantic fit of the violation to the established context plays a role. Finally, I adapt a classic paradigm, word stem completion, to investigate electrophysiological activity following a cue that is modulated by how predictive the outcome is. With this work, I not only have discovered costs of failed and successful predictions and identified neural signals potentially related to generation of predictions, but also have researched prediction in novel ways that can continue to expand and further elucidate how this mechanism affects cognition and changes across populations

Predicting form and meaning:Evidence from brain potentials

We used ERPs to investigate the pre-activation of form and meaning in language comprehension. Participants read high-cloze sentence contexts (e.g., “The student is going to the library to borrow a…”), followed by a word that was predictable (book), form-related (hook) or semantically related (page) to the predictable word, or unrelated (sofa). At a 500 ms SOA (Experiment 1), semantically related words, but not form-related words, elicited a reduced N400 compared to unrelated words. At a 700 ms SOA (Experiment 2), semantically related words and form-related words elicited reduced N400 effects, but the effect for form-related words occurred in very high-cloze sentences only. At both SOAs, form-related words elicited an enhanced, post-N400 posterior positivity (Late Positive Component effect). The N400 effects suggest that readers can pre-activate meaning and form information for highly predictable words, but form pre-activation is more limited than meaning pre-activation. The post-N400 LPC effect suggests that participants detected the form similarity between expected and encountered input. Pre-activation of word forms crucially depends upon the time that readers have to make predictions, in line with production-based accounts of linguistic prediction

Verbing and nouning in French : toward an ecologically valid approach to sentence processing

La présente thèse utilise la technique des potentiels évoqués afin d’étudier les méchanismes neurocognitifs qui sous-tendent la compréhension de la phrase. Plus particulièrement, cette recherche vise à clarifier l’interaction entre les processus syntaxiques et sémantiques chez les locuteurs natifs et les apprenants d’une deuxième langue (L2). Le modèle “syntaxe en premier” (Friederici, 2002, 2011) prédit que les catégories syntaxiques sont analysées de façon précoce: ce stade est reflété par la composante ELAN (Early anterior negativity, Négativité antérieure gauche), qui est induite par les erreurs de catégorie syntaxique. De plus, ces erreurs semblent empêcher l’apparition de la composante N400 qui reflète les processus lexico-sémantiques. Ce phénomène est défini comme le bloquage sémantique (Friederici et al., 1999). Cependant, la plupart des études qui observent la ELAN utilisent des protocoles expérimentaux problématiques dans lesquels les différences entre les contextes qui précèdent la cible pourraient être à l’origine de résultats fallacieux expliquant à la fois l’apparente “ELAN” et l’absence de N400 (Steinhauer & Drury, 2012). La première étude rééevalue l’approche de la “syntaxe en premier” en adoptant un paradigme expériemental novateur en français qui introduit des erreurs de catégorie syntaxique et les anomalies de sémantique lexicale. Ce dessin expérimental équilibré contrôle à la fois le mot-cible (nom vs. verbe) et le contexte qui le précède. Les résultats récoltés auprès de locuteurs natifs du français québécois ont révélé un complexe N400-P600 en réponse à toutes les anomalies, en contradiction avec les prédictions du modèle de Friederici. Les effets additifs des manipulations syntaxique et sémantique sur la N400 suggèrent la détection d’une incohérence entre la racine du mot qui avait été prédite et la cible, d’une part, et l’activation lexico-sémantique, d’autre part. Les réponses individuelles se sont pas caractérisées par une dominance vers la N400 ou la P600: au contraire, une onde biphasique est présente chez la majorité des participants. Cette activation peut donc être considérée comme un index fiable des mécanismes qui sous-tendent le traitement des structures syntagmatiques. La deuxième étude se concentre sur les même processus chez les apprenants tardifs du français L2. L’hypothèse de la convergence (Green, 2003 ; Steinhauer, 2014) prédit que les apprenants d’une L2, s’ils atteignent un niveau avancé, mettent en place des processus de traitement en ligne similaires aux locuteurs natifs. Cependant, il est difficile de considérer en même temps un grand nombre de facteurs qui se rapportent à leurs compétences linguistiques, à l’exposition à la L2 et à l’âge d’acquisition. Cette étude continue d’explorer les différences inter-individuelles en modélisant les données de potentiels-évoqués avec les Forêts aléatoires, qui ont révélé que le pourcentage d’explosition au français ansi que le niveau de langue sont les prédicteurs les plus fiables pour expliquer les réponses électrophysiologiques des participants. Plus ceux-ci sont élevés, plus l’amplitude des composantes N400 et P600 augmente, ce qui confirme en partie les prédictions faites par l’hypothèse de la convergence. En conclusion, le modèle de la “syntaxe en premier” n’est pas viable et doit être remplacé. Nous suggérons un nouveau paradigme basé sur une approche prédictive, où les informations sémantiques et syntaxiques sont activées en parallèle dans un premier temps, puis intégrées via un recrutement de mécanismes contrôlés. Ces derniers sont modérés par les capacités inter-individuelles reflétées par l’exposition et la performance.The present thesis uses event-related potentials (ERPs) to investigate neurocognitve mechanisms underlying sentence comprehension. In particular, these two experiments seek to clarify the interplay between syntactic and semantic processes in native speakers and second language learners. Friederici’s (2002, 2011) “syntax-first” model predicts that syntactic categories are analyzed at the earliest stages of speech perception reflected by the ELAN (Early left anterior negativity), reported for syntactic category violations. Further, syntactic category violations seem to prevent the appearance of N400s (linked to lexical-semantic processing), a phenomenon known as “semantic blocking” (Friederici et al., 1999). However, a review article by Steinhauer and Drury (2012) argued that most ELAN studies used flawed designs, where pre-target context differences may have caused ELAN-like artifacts as well as the absence of N400s. The first study reevaluates syntax-first approaches to sentence processing by implementing a novel paradigm in French that included correct sentences, pure syntactic category violations, lexical-semantic anomalies, and combined anomalies. This balanced design systematically controlled for target word (noun vs. verb) and the context immediately preceding it. Group results from native speakers of Quebec French revealed an N400-P600 complex in response to all anomalous conditions, providing strong evidence against the syntax-first and semantic blocking hypotheses. Additive effects of syntactic category and lexical-semantic anomalies on the N400 may reflect a mismatch detection between a predicted word-stem and the actual target, in parallel with lexical-semantic retrieval. An interactive rather than additive effect on the P600 reveals that the same neurocognitive resources are recruited for syntactic and semantic integration. Analyses of individual data showed that participants did not rely on one single cognitive mechanism reflected by either the N400 or the P600 effect but on both, suggesting that the biphasic N400-P600 ERP wave can indeed be considered to be an index of phrase-structure violation processing in most individuals. The second study investigates the underlying mechanisms of phrase-structure building in late second language learners of French. The convergence hypothesis (Green, 2003; Steinhauer, 2014) predicts that second language learners can achieve native-like online- processing with sufficient proficiency. However, considering together different factors that relate to proficiency, exposure, and age of acquisition has proven challenging. This study further explores individual data modeling using a Random Forests approach. It revealed that daily usage and proficiency are the most reliable predictors in explaining the ERP responses, with N400 and P600 effects getting larger as these variables increased, partly confirming and extending the convergence hypothesis. This thesis demonstrates that the “syntax-first” model is not viable and should be replaced. A new account is suggested, based on predictive approaches, where semantic and syntactic information are first used in parallel to facilitate retrieval, and then controlled mechanisms are recruited to analyze sentences at the interface of syntax and semantics. Those mechanisms are mediated by inter-individual abilities reflected by language exposure and performance

Cognitive and neural mechanisms underlying word retrieval and integration during sentence comprehension

O processamento das palavras é facilitado quando estas surgem numa frase com um contexto que lhes dá suporte. Porém, a natureza dos processos subjacentes a este efeito é ainda pouco clara. Numa abordagem multimodal, os estudos empíricos desta dissertação investigaram como é que os processos de acesso e integração de palavras são implementados e interagem durante a compreensão de frases, atendendo ao contexto frásico e à expectativa das palavras. Nos estudos comportamental e eletrofisiológico, observou-se uma facilitação na leitura de todas as palavras esperadas, i.e., não apenas as mais esperadas, mas também as segundas palavras mais esperadas. Os potenciais evocados mostraram que o efeito de facilitação é inicialmente restrito às palavras mais esperadas e manifesta-se mais tardiamente para as segundas palavras mais esperadas. Adicionalmente, estes efeitos não foram modelados pelas características da tarefa nem pelas capacidades dos leitores, tratando-se de um efeito robusto e consistente. O estudo de fMRI revelou padrões de ativação distintos em duas sub-regiões do córtex frontal inferior esquerdo. Na sub-região anterior, a ativação foi maior durante o processamento de frases completadas com palavras inesperadas comparativamente a palavras esperadas, apenas quando o contexto frásico era muito restritivo e ainda durante o processamento de frases com palavras esperadas em que o contexto frásico era pouco restritivo relativamente a contextos muito restritivos, sugerindo que esta região está associada ao acesso semântico de palavras. A sub-região posterior, além de apresentar resultados semelhantes aos da sub-região anterior, revelou maior ativação para frases completadas com palavras inesperadas em contextos muito restritivos comparativamente a contextos pouco restritivos. Logo, esta região parece ser recrutada em frases que promovem a ativação de várias palavras e, consequentemente, requerem um esforço adicional para selecionar e integrar a palavra alvo. Estes resultados enfatizam o papel dos mecanismos preditivos no acesso às palavras durante a compreensão de frases. Estes mecanismos ocorrem de forma serial e gradativa, são desencadeados automaticamente e são consistentes, independentemente das capacidades dos leitores. Esta evidência contribui para o desenvolvimento dos modelos de compreensão da linguagem.The processing of words is facilitated when they appear in a sentence with a supportive context. Yet, the nature of the processes underlying that effect is still unclear. In a multimodal approach, the empirical studies of this dissertation investigate how word retrieval and integration are implemented and interact during sentence comprehension, according to the sentence context and word’s expectation. In the behavioural and electrophysiological studies, the facilitation effect during sentence reading was found for all expectable candidates, i.e., not only for the most expected words, but also for the second-best words. The event-related potentials revealed that the facilitation effect is initially restricted to the most expected word, with the second-best word demonstrating a delayed effect. Additionally, the facilitation effect was not affected by task demands or readers’ abilities, showing that the effect is robust and consistent. The fMRI study showed distinct patterns of activation in subregions of the LIFG. In the anterior subregion, the expectancy effect was restricted to high constraint sentences and the expected words triggered more activation in low than in high constraint sentences, suggesting that this region is involved in lexical-semantic retrieval. The posterior subregion, besides showing similar results as the ones found in the anterior subregion, also revealed an enhanced activation for unexpected words in high constraint sentences compared with low constraint sentences. Thus, the posterior LIFG was recruited by sentences that endorsed the retrieval of multiple words and, consequently, require greater effort to select and integrate the candidate word. The findings highlight the impact of the predictive mechanisms on word retrieval during sentence comprehension. These mechanisms seem to occur in a serial and graded way, are triggered automatically and are consistent despite readers capacities. This evidence contributes to the further development of language comprehension models

Predicting while comprehending language:A theory and review

Researchers agree that comprehenders regularly predict upcoming language, but they do not always agree on what prediction is (and how to differentiate it from integration) or what constitutes evidence for it. After defining prediction, we show that it occurs at all linguistic levels from semantics to form, and then propose a theory of which mechanisms comprehenders use to predict. We argue that they most effectively predict using their production system (i.e., prediction-by-production): They covertly imitate the linguistic form of the speaker’s utterance and construct a representation of the underlying communicative intention. Comprehenders can then run this intention through their own production system to prepare the predicted utterance. But doing so takes time and resources, and comprehenders vary in the extent of preparation, with many groups of comprehenders (non-native speakers, illiterates, children, and older adults) using it less than typical native young adults. We thus argue that prediction-by-production is an optional mechanism, which is augmented by mechanisms based on association. Support for our proposal comes from many areas of research (electrophysiological, eye-tracking, and behavioral studies of reading, spoken language processing in the context of visual environments, speech processing, and dialogue)

Prediction during native and non-native language comprehension: the role of mediating factors

Psycholinguistic evidence suggests that people predict upcoming words during language comprehension. While many studies have addressed what information people predict, less is known about the role of factors that potentially mediate predictive processing. This thesis examines predictions of semantic information and word form information. It investigates whether predictive processing is mediated by availability of cognitive resources and time to generate predictions, and compares predictive processing in native (L1) speakers and non-native (L2) speakers. This thesis presents two major lines of work. Two eye-tracking studies investigate prediction of semantic and word form information using a visual world paradigm. In further two ERP studies, we address the interplay of semantic and word form information in a paradigm which combines both possibilities. Experiments 1 and 2 were an eye-tracking study conducted on L1 and L2 speakers of English. The study has demonstrated that L1 and L2 speakers predict semantic information, but their predictive eye movements are delayed when they are under a cognitive load. The effects of cognitive load on predictive eye movements suggest a role of cognitive resources in language prediction in both L1 and L2 speakers. Experiments 3 and 4 were another eye-tracking study conducted on L1 and L2 speakers. The study has shown that L1 speakers predict word form information, but L2 speakers do not. Experiments 5 and 6 were an ERP study, which investigated the interplay of prediction of semantic and word form information in L1 English speakers. Consistent with the two sets of eye-tracking experiments, L1 speakers predicted both semantic and word form information, but word form was only predicted when sentences were presented at a slower rate, while semantic information was predicted at standard and slow presentation rates. Experiments 7 and 8 used the same method as Experiments 5 and 6, conducted on L2 English speakers. L2 speakers comprehended sentences incrementally, but there was no clear evidence that they predicted semantic information or word form information. Experiments 5 – 8 suggest that prediction of word form information is mediated both by nativeness of the target language and by reading rates. To conclude, both L1 and L2 speakers make predictions, but prediction of semantic information occurs only when there are enough cognitive resources available. Prediction of word form can occur in L1 speakers, but it occurs only when there is enough time available. There is no evidence that L2 speakers predict word form, suggesting a role of nativeness of the target language. The findings are consistent with the production-based prediction model of language prediction, in that prediction of word form is less likely to occur compared to prediction of semantic information. Furthermore, the findings are also consistent with the claim that not everyone makes predictions, and predictions do not always occur. The thesis concludes that prediction is additional processing for the comprehension system, and is not always implicated in the comprehension system