Electroencephalogram-Based Single-Trial Detection of Language Expectation Violations in Listening to Speech

We propose an approach for the detection of language expectation violations that occur in communication. We examined semantic and syntactic violations from electroencephalogram (EEG) when participants listened to spoken sentences. Previous studies have shown that such event-related potential (ERP) components as N400 and the late positivity (P600) are evoked in the auditory where semantic and syntactic anomalies occur. We used this knowledge to detect language expectation violation from single-trial EEGs by machine learning techniques. We recorded the brain activity of 18 participants while they listened to sentences that contained semantic and syntactic anomalies and identified the significant main effects of these anomalies in the ERP components. We also found that a multilayer perceptron achieved 59.5% (semantic) and 57.7% (syntactic) accuracies

The reliability of the N400 in single subjects: Implications for patients with disorders of consciousness

Functional neuroimaging assessments of residual cognitive capacities, including those that support language, can improve diagnostic and prognostic accuracy in patients with disorders of consciousness. Due to the portability and relative inexpensiveness of electroencephalography, the N400 event-related potential component has been proposed as a clinically valid means to identify preserved linguistic function in non-communicative patients. Across three experiments, we show that changes in both stimuli and task demands significantly influence the probability of detecting statistically significant N400 effects — that is, the difference in N400 amplitudes caused by the experimental manipulation. In terms of task demands, passively heard linguistic stimuli were significantly less likely to elicit N400 effects than task-relevant stimuli. Due to the inability of the majority of patients with disorders of consciousness to follow task commands, the insensitivity of passive listening would impede the identification of residual language abilities even when such abilities exist. In terms of stimuli, passively heard normatively associated word pairs produced the highest detection rate of N400 effects (50% of the participants), compared with semantically-similar word pairs (0%) and high-cloze sentences (17%). This result is consistent with a prediction error account of N400 magnitude, with highly predictable targets leading to smaller N400 waves, and therefore larger N400 effects. Overall, our data indicate that non-repeating normatively associated word pairs provide the highest probability of detecting single-subject N400s during passive listening, and may thereby provide a clinically viable means of assessing residual linguistic function. We also show that more liberal analyses may further increase the detection-rate, but at the potential cost of increased false alarms

Unconstrained multivariate EEG decoding can help detect lexical-semantic processing in individual children

Funder: Macquarie UniversityAbstract: In conditions such as minimally-verbal autism, standard assessments of language comprehension are often unreliable. Given the known heterogeneity within the autistic population, it is crucial to design tests of semantic comprehension that are sensitive in individuals. Recent efforts to develop neural signals of language comprehension have focused on the N400, a robust marker of lexical-semantic violation at the group level. However, homogeneity of response in individual neurotypical children has not been established. Here, we presented 20 neurotypical children with congruent and incongruent visual animations and spoken sentences while measuring their neural response using electroencephalography (EEG). Despite robust group-level responses, we found high inter-individual variability in response to lexico-semantic anomalies. To overcome this, we analysed our data using temporally and spatially unconstrained multivariate pattern analyses (MVPA), supplemented by descriptive analyses to examine the timecourse, topography, and strength of the effect. Our results show that neurotypical children exhibit heterogenous responses to lexical-semantic violation, implying that any application to heterogenous disorders such as autism spectrum disorder will require individual-subject analyses that are robust to variation in topology and timecourse of neural responses

Unconstrained multivariate EEG decoding can help detect lexical-semantic processing in individual children

Funder: Macquarie UniversityAbstract: In conditions such as minimally-verbal autism, standard assessments of language comprehension are often unreliable. Given the known heterogeneity within the autistic population, it is crucial to design tests of semantic comprehension that are sensitive in individuals. Recent efforts to develop neural signals of language comprehension have focused on the N400, a robust marker of lexical-semantic violation at the group level. However, homogeneity of response in individual neurotypical children has not been established. Here, we presented 20 neurotypical children with congruent and incongruent visual animations and spoken sentences while measuring their neural response using electroencephalography (EEG). Despite robust group-level responses, we found high inter-individual variability in response to lexico-semantic anomalies. To overcome this, we analysed our data using temporally and spatially unconstrained multivariate pattern analyses (MVPA), supplemented by descriptive analyses to examine the timecourse, topography, and strength of the effect. Our results show that neurotypical children exhibit heterogenous responses to lexical-semantic violation, implying that any application to heterogenous disorders such as autism spectrum disorder will require individual-subject analyses that are robust to variation in topology and timecourse of neural responses

Semantic radical consistency and character transparency effects in Chinese: an ERP study

BACKGROUND: This event-related potential (ERP) study aims to investigate the representation and temporal dynamics of Chinese orthography-to-semantics mappings by simultaneously manipulating character transparency and semantic radical consistency. Character components, referred to as radicals, make up the building blocks used dur...postprin

The neurocognitive processing of plausibility and real-world knowledge:A cross-linguistic investigation

Our knowledge about concepts and meanings is at the very heart of human cognition. In everyday life, we have to interact with our environment in a variety of different ways. Our actions are guided by what we know and believe about the world and this knowledge derives primarily from previous sensory and perceptual experiences. The fact that we are capable of engaging with our environment in an appropriate and efficient way means that we have learnt (how) to make sense of the events and entities we are faced with in day-to-day life. We are thus able to recognise and name both physical objects and abstract concepts, to categorise and associate them based on their specific properties, to interpret other people’s intentions, and to judge cause and effect of their actions as well as our own. Moreover, the ability to represent this wealth of knowledge about the real world in the conceptualised and symbolic form of language is believed to be exclusive to humans. Our language capacity allows us to communicate with others about past and future events or to describe fictitious scenarios by combining previously acquired concepts in a novel way without the need for external stimulation. Thus language forms a primary means of interacting with those around us by allowing us to express our own thoughts and comprehend those of others. As long as language processing proceeds in an undisturbed manner, we are largely unaware of the underlying mechanisms that support the seemingly effortless interpretation of linguistic input. The importance of these processes for successful communication, however, becomes all the more apparent when language processing is disrupted, for example, by brain lesions that render semantic analysis difficult or impossible. Scientific research that aims to uncover and define cognitive or neural mechanisms underlying semantic processing is inevitably faced with the complexity and wealth of semantic relationships that need to be taken into account. In absence of noninvasive neurocognitive methods and insights gleaned from modern neurobiology, early research had a limited impact on our understanding of how semantic processing is implemented in the human brain. Traditional neurological models of language have been based primarily on lesion-deficit data, and thus supported the view that certain areas of the brain were exclusively dedicated to the processing of language-specific functions (Geschwind, 1970; Lichtheim, 1885; Wernicke, 1874). Furthermore, classical theories of sensory processing viewed the brain as a purely stimulus-driven system that retrieves and combines individual low-level aspects or features in an automated, passive and context-independent manner (Biederman, 1987; Burton & Sinclair, 1996; Hubel & Wiesel, 1965; Massaro, 1998). After a recent paradigm shift in the cognitive neurosciences, current theories of sensory processing are now based on the concept of the brain as a highly active, adaptive and dynamic device. In this sense, language comprehension, like many other higher-cognitive functions, is shaped by a flexible interaction of a number of different processes and information sources that include so-called bottom-up signals, i.e., the actual sensory input and processes related to their forward propagation, and top-down processes that generate predictions and expectations based on prior experience and perceived probabilities. Therefore, accounts that view semantic processing as a dynamic and active construction of meaning that is highly sensitive to contextual influences seem most probable from a neurobiological perspective. Results from electrophysiological and neuroimaging research on semantic analysis in sentence and discourse context have provided evidence for top-down influences from the very beginning. In addition, recent ERP results have suggested that the interaction between topdown and bottom-up information is more flexible and dynamic than previously assumed. Yet, the importance of predictions and expectations has long been neglected in models of semantic processing and language comprehension in general. Neuroimaging data have provided us with a long list of brain regions that have been implicated in different aspects of semantic analysis. We are only beginning to understand the role(s) that these regions play and how they interact to support the flexible and efficient construction of meaning. The aim of the present thesis is to gain a more comprehensive view on the computational mechanisms underlying language processing by investigating how bottom- up and top-down information and processes interactively contribute to the semantic analysis in sentences and discourse. To this end, we conducted a total of five studies that used either event-related potentials or functional neuroimaging to shed light on this matter from different perspectives. The thesis is divided into two main parts: Part I (chapters 1-5) provides an overview on previous results from electrophysiology and neuroimaging on semantic processing as well as a description and discussion of the studies conducted in the present thesis. Part II (chapters 6-9) consists of three research articles that describe and discuss the results of five experimental studies. In Part I, Chapter 2 gives a brief introduction to the event-related potential and functional neuroimaging techniques and reviews the most relevant results and theories that have emerged from studies on sentence and discourse processing. Chapter 3 highlights the research questions targeted in each of the experimental studies and describes and discusses the most relevant findings against the background established by Chapter 2. Chapters 4 and 5 conclude Part I by placing the presented results in a broader context and by briefly outlining future directions. Part II begins with a survey of the three studies reported in the subsequent chapters. Chapter 7 highlights the results of the first study, a German ERP experiment that investigated the impact of capitalisation, i.e., a purely form-based and contextually independent bottom-up manipulation, on the processing of semantic anomalies in single sentences. Chapter 8 comprises three ERP experiments that used both easy and hard to detect semantic anomalies in German and English to corroborate the assumption that the weighting of top-down and bottom-up information cues might be determined in a language-specific way. Chapter 9, the final chapter of the thesis, describes and discusses the results of the third study, in which the impact of embedding context on the required depth of semantic processing was examined using functional neuroimaging

Electrophysiological Correlates of Natural Language Processing in Children and Adults

To understand the causes of differences in language ability we must measure the specific and separable processes that contribute to natural language comprehension. Specifically, we need measures of the three language subsystems – semantics, syntax, and phonology – as they are used during the comprehension of real speech. Event-Related Potentials (ERPs) are a promising approach to reaching this level of specificity. Previous research has identified distinct ERP effects for each of the subsystems – the N400 to semantic anomalies, the Anterior Negativity and P600 to syntactic anomalies, and the Phonological Mapping Negativity to unexpected speech sounds. However, these studies typically use stimuli and tasks that encourage processing that differs from real-world language comprehension. Further, previous ERP studies indexing language processing in young children not only use unfamiliar tasks, but also typically exclude data from the large proportion of children. We need to measure language-related ERPs in a context as close as possible to real-world processing, and in a manner that includes data from representative rather than highly-selected samples of children. The experiments described in this dissertation achieve that goal. Adults and five-year-old children listened to a child-directed story while answering comprehension questions. Infrequent violations were included to independently probe the three language subsystems. In children and adults, the canonical N400 response was evident in response to semantic violations. Morphosyntactic violations elicited a long-duration Anterior Negativity without a later P600. Phonological violations on suffixes elicited a Phonological Mapping Negativity in adults. This is the first report of this phonological effect outside of highly-predictable lexical contexts. Popular normed behavioral assessments were also administered to the children who participated in this study. Results from these assessments confirmed that performance on tasks claiming to measure categorically different abilities are correlated with one another, and that language measures correlate with so-called nonverbal measures. ERPs indexing different language subsystem did not correlate with each other or with measures of nonverbal cognitive ability. Using multiple ERP measures during natural language comprehension, we are able to isolate specific aspects of language processing, increasing the possibility of making meaningful connections between biology, experience, and resulting language ability

Expectancy effects in the EEG during joint and spontaneous word-by-word sentence production in German

Our aim in the present study is to measure neural correlates during spontaneous interactive sentence production. We present a novel approach using the word-by-word technique from improvisational theatre, in which two speakers jointly produce one sentence. This paradigm allows the assessment of behavioural aspects, such as turn-times, and electrophysiological responses, such as event-related-potentials (ERPs). Twenty-five participants constructed a cued but spontaneous four-word German sentence together with a confederate, taking turns for each word of the sentence. In 30% of the trials, the confederate uttered an unexpected gender-marked article. To complete the sentence in a meaningful way, the participant had to detect the violation and retrieve and utter a new fitting response. We found significant increases in response times after unexpected words and – despite allowing unscripted language production and naturally varying speech material – successfully detected significant N400 and P600 ERP effects for the unexpected word. The N400 EEG activity further significantly predicted the response time of the subsequent turn. Our results show that combining behavioural and neuroscientific measures of verbal interactions while retaining sufficient experimental control is possible, and that this combination provides promising insights into the mechanisms of spontaneous spoken dialogue