Waves and Words: Oscillatory activity and language processing

Successful language comprehension depends not only on the involvement of different domain-specific linguistic processes, but also on their respective time-courses. Both aspects of the comprehension process can be examined by means of event-related brain potentials (ERPs), which not only provide a direct reflection of human brain activity within the millisecond range, but also allow for a qualitative dissociation between different language-related processing domains. However, recent ERP findings indicate that the desired one-to-one mapping between ERP components and linguistic processes cannot be upheld, thus leading to an interpretative uncertainty. This thesis presents a fundamentally new analysis technique for language-based ERP components, which aims to address the ambiguity associated with traditional language-related ERP effects. It is argued that this new method, which supplements ERP measures with corresponding frequency-based analyses, not only allows for a differentiation of ERP components on the basis of activity in distinct frequency bands and underlying dynamic behaviour (in terms of power changes and/or phase locking), but also provides further insights into the functional organisation of the language comprehension system and its inherent complexity. On the basis of 5 EEG experiments, I show (1) that it is possible to dissociate two superficially indistinguishable language-related ERP components on the basis of their respective underlying frequency characteristics (Experiment 1), thereby resolving the vagueness of interpretation inherent to the ERP components themselves; (2) that the processing nature of the ‘classical’ semantic N400 effect can be unambiguously specified in terms of its underlying frequency characteristics, i.e. in terms of (evoked and whole) power and phase-locking differences in specific frequency bands, thereby allowing for a first interpretative categorisation of the N400 effect with respect to its underlying neuronal processing dynamics; and (3) that frequency-based analyses may be employed to distinguish the semantic N400 effect from N400-like effects that appear in contexts which cannot readily be characterised as semantic-interpretative processes. Experiments 2 – 5 investigated the processing of antonym relations under different task conditions. Whereas in Experiment 2, the processing of antonym pairs (black – white) was compared to that of related (black – yellow) and non-related (black – nice) word pairs in a sentence context, Experiments 3 to 5 presented isolated word pairs. The frequency-based analysis showed that the observed N400 effects were not uniform in nature, but rather resulted from the superposition of functionally different frequency components. Task-relevant targets elicited a specific frequency modulation, which showed up as a P300-like positivity in terms of ERP measures. In addition, lexical-semantic processing elicited a pronounced increase in a different frequency range that was independent of the experimental context. For antonyms (Experiments 2 and 3), the task-related positive component appeared almost simultaneously to the N400 deflection for non-related words, thereby giving rise to a substantial N400 effect. In contrast, for pseudowords (Experiment 5), this positivity appeared in temporal succession to the N400. In sum, in the present results provide converging evidence that N400 effects should not be regarded as functionally uniform. Depending on the respective task and stimulus manipulations, the N400 effect appears as a result of the superposition of functionally different activities, which can be clearly distinguished in terms of their underlying frequency characteristics. In this way, the proposed frequency-based methods directly bear upon the interpretation of language-related ERP effects and thus have straightforward consequences for psycholinguistic theory. In view of the phenomenon that language-related processes have, in a number of cases, been directly attributed to the lexical-semantic processing domain on account of the observation of an N400, these results not only call for a reinterpretation of previous findings but also for a reinterpretation of their theoretical consequences

EEG analysis based on dynamic visual stimuli: best practices in analysis of sign language data

This paper reviews best practices for experimental design and analysis for sign language research using neurophysiological methods, such as electroencephalography (EEG) and other methods with high temporal resolution, as well as identifies methodological challenges in neurophysiological research on natural sign language processing. In particular, we outline the considerations for generating linguistically and physically well-controlled stimuli accounting for 1) the layering of manual and non-manual information at different timescales, 2) possible unknown linguistic and non-linguistic visual cues that can affect processing, 3) variability across linguistic stimuli, and 4) predictive processing. Two specific concerns with regard to the analysis and interpretation of observed event related potential (ERP) effects for dynamic stimuli are discussed in detail. First, we discuss the “trigger/effect assignment problem”, which describes the difficulty of determining the time point for calculating ERPs. This issue is related to the problem of determining the onset of a critical sign (i.e., stimulus onset time), and the lack of clarity as to how the border between lexical (sign) and transitional movement (motion trajectory between individual signs) should be defined. Second, we discuss possible differences in the dynamics within signing that might influence ERP patterns and should be controlled for when creating natural sign language material for ERP studies. In addition, we outline alternative approaches to EEG data analyses for natural signing stimuli, such as the timestamping of continuous EEG with trigger markers for each potentially relevant cue in dynamic stimuli. Throughout the discussion, we present empirical evidence for the need to account for dynamic, multi-channel, and multi-timescale visual signal that characterizes sign languages in order to ensure the ecological validity of neurophysiological research in sign languages

Visual boundaries in sign motion: processing with and without lip-reading cues

Sign languages demonstrate a higher degree of iconicity than spoken languages. Studies on a number of unrelated sign languages show that the event structure of verb signs is reflected in the phonological form of the signs (Wilbur (2008), Malaia & Wilbur (2012), Krebs et al. (2021)). Previous research showed that hearing non-signers (with no prior exposure to sign language) can use the iconicity inherent in the visual dynamics of a verb sign to correctly identify its event structure (telic vs. atelic). In two EEG experiments, hearing non-signers were presented with telic and atelic verb signs unfamiliar to them, which they had to classify in a two-choice lexical decision task in their native language. The first experiment assessed the timeline of neural processing mechanisms in non-signers processing telic/atelic signs without access to lip-reading cues in their native language, to understand the pathways for incorporation of physical perceptual motion features into linguistic processing. The second experiment further probed the impact of visual information provided by lip-reading (speech decoding based on visual information from the face of the speaker, most importantly, the lips) on the processing of telic/atelic signs in non-signers

Waves and Words: Oscillatory activity and language processing

Successful language comprehension depends not only on the involvement of different domain-specific linguistic processes, but also on their respective time-courses. Both aspects of the comprehension process can be examined by means of event-related brain potentials (ERPs), which not only provide a direct reflection of human brain activity within the millisecond range, but also allow for a qualitative dissociation between different language-related processing domains. However, recent ERP findings indicate that the desired one-to-one mapping between ERP components and linguistic processes cannot be upheld, thus leading to an interpretative uncertainty. This thesis presents a fundamentally new analysis technique for language-based ERP components, which aims to address the ambiguity associated with traditional language-related ERP effects. It is argued that this new method, which supplements ERP measures with corresponding frequency-based analyses, not only allows for a differentiation of ERP components on the basis of activity in distinct frequency bands and underlying dynamic behaviour (in terms of power changes and/or phase locking), but also provides further insights into the functional organisation of the language comprehension system and its inherent complexity. On the basis of 5 EEG experiments, I show (1) that it is possible to dissociate two superficially indistinguishable language-related ERP components on the basis of their respective underlying frequency characteristics (Experiment 1), thereby resolving the vagueness of interpretation inherent to the ERP components themselves; (2) that the processing nature of the ‘classical’ semantic N400 effect can be unambiguously specified in terms of its underlying frequency characteristics, i.e. in terms of (evoked and whole) power and phase-locking differences in specific frequency bands, thereby allowing for a first interpretative categorisation of the N400 effect with respect to its underlying neuronal processing dynamics; and (3) that frequency-based analyses may be employed to distinguish the semantic N400 effect from N400-like effects that appear in contexts which cannot readily be characterised as semantic-interpretative processes. Experiments 2 – 5 investigated the processing of antonym relations under different task conditions. Whereas in Experiment 2, the processing of antonym pairs (black – white) was compared to that of related (black – yellow) and non-related (black – nice) word pairs in a sentence context, Experiments 3 to 5 presented isolated word pairs. The frequency-based analysis showed that the observed N400 effects were not uniform in nature, but rather resulted from the superposition of functionally different frequency components. Task-relevant targets elicited a specific frequency modulation, which showed up as a P300-like positivity in terms of ERP measures. In addition, lexical-semantic processing elicited a pronounced increase in a different frequency range that was independent of the experimental context. For antonyms (Experiments 2 and 3), the task-related positive component appeared almost simultaneously to the N400 deflection for non-related words, thereby giving rise to a substantial N400 effect. In contrast, for pseudowords (Experiment 5), this positivity appeared in temporal succession to the N400. In sum, in the present results provide converging evidence that N400 effects should not be regarded as functionally uniform. Depending on the respective task and stimulus manipulations, the N400 effect appears as a result of the superposition of functionally different activities, which can be clearly distinguished in terms of their underlying frequency characteristics. In this way, the proposed frequency-based methods directly bear upon the interpretation of language-related ERP effects and thus have straightforward consequences for psycholinguistic theory. In view of the phenomenon that language-related processes have, in a number of cases, been directly attributed to the lexical-semantic processing domain on account of the observation of an N400, these results not only call for a reinterpretation of previous findings but also for a reinterpretation of their theoretical consequences

Psycho-/Neurolinguistik: Neuronale Korrelate der Verarbeitung grammatischer Variation

Interindividuelle Unterschiede bei der Verarbeitung sprachlicher Strukturen haben bei experimentellen Untersuchungen zur Sprachverarbeitung mittels neurobasierter Verfahren lange Zeit keine oder bestenfalls eine untergeordnete Rolle gespielt. Während individuelle Verarbeitungsstrategien in Abhängigkeit von experimentellen Faktoren (z.B. Aufgabenstellung) relativ gut belegt sind (z.B. probandenspezifisches strategisches Verhalten bei der Verarbeitung von semantischen Relationen; Roehm et al. 2007), wurde der Einfluss von Variation in der Grammatik des Standarddeutschen in Korrelation zu Hirnprozessen bisher kaum berücksichtigt. In diesem Beitrag werde ich auf der Basis dreier EEG-Experimente aus unterschiedlichen Bereichen (Synästhesie, semantische Relationen, Auxiliarselektion bei intransitiven Verben) Beispiele für Verarbeitungskorrelate interindividueller Variation vorstellen und diskutieren

Neural mechanisms of Event Visibility in sign languages

Event structure in sign languages is reflected in the manual dynamics of verb production. As signed event structure is visible (iconic), non-signers are able to recognize it, despite having no sign lexicon. In this EEG study, hearing non-signers were presented with telic and atelic verb signs, followed by a lexical classification task in their native language. Behavioral data confirmed that non-signers classified both telic and atelic signs with above-chance accuracy. ERP waveforms indicated that non-signers identified the perceptual differences in motion features when viewing telic/atelic signs, and used different processing mechanisms when integrating the perceptual information with linguistic concepts in their native language. Non-signers appeared to segment visual sign language input into discrete events, as they attempted to map the observed visual forms to concepts, and label them linguistically. This mechanism suggests a potential evolutionary pathway for co-optation of perceptual features into the linguistic structure of sign languages