Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading

How is syntactic analysis implemented by the human brain during language comprehension? The current study combined methods from computational linguistics, eyetracking, and fMRI to address this question. Subjects read passages of text presented as paragraphs while their eye movements were recorded in an MRI scanner. We parsed the text using a probabilistic context-free grammar to isolate syntactic difficulty. Syntactic difficulty was quantified as syntactic surprisal, which is related to the expectedness of a given word's syntactic category given its preceding context. We compared words with high and low syntactic surprisal values that were equated for length, frequency, and lexical surprisal, and used fixation-related (FIRE) fMRI to measure neural activity associated with syntactic surprisal for each fixated word. We observed greater neural activity for high than low syntactic surprisal in two predicted cortical regions previously identified with syntax: left inferior frontal gyrus (IFG) and less robustly, left anterior superior temporal lobe (ATL). These results support the hypothesis that left IFG and ATL play a central role in syntactic analysis during language comprehension. More generally, the results suggest a broader cortical network associated with syntactic prediction that includes increased activity in bilateral IFG and insula, as well as fusiform and right lingual gyri

Tapahtumasegmentaation aivovasteet hippokampuksessa ja aivokuorella äänitarinan kuuntelun aikana

Tapahtumasegmentaatio jäsentää sekä arkista kokemustamme että muistiamme. Parhaillaan meneillään olevan tapahtuman hahmotus ja prosessointi tapahtuu todennäköisesti aivokuorella, mutta ilman toimivaa hippokampusta tilanteesta ei voi syntyä pysyvää muistoa. On olennainen kysymys, missä kohtaa ja miten hippokampus osallistuu tapahtumien prosessointiin ja mieleen painamiseen. Aiemmin on magneettikuvaustutkimuksin osoitettu, että hippokampus reagoi tapahtumien välisiin rajoihin aktivaatiopiikein. On ehdotettu, että ne ilmentäisivät aistimodaliteetista riippumattoman tason prosessia, jossa hippokampus kokoaa yhteen ja vahvistaa koetun tilanteen kokonaisrepresentaation, jotta se voidaan painaa muistiin. Aiemmat tutkimukset on kuitenkin toteutettu yksinomaan audiovisuaalisilla ärsykkeillä, ja koska hippokampuksen tiedetään osallistuvan myös visuaaliseen prosessointiin, ei ole täysin selvää, etteivätkö havaitut aktivaatiot voisi selittyä alemman, aistitietoa käsittelevän tason prosesseilla. Tämän kysymyksen ratkaisemiseksi tässä tutkimuksessa selvitettiin reagoiko hippokampus tapahtumarajoihin puhtaasti auditiivisessa ärsykkeessä. Ärsykkeenä oli 71-minuuttinen tarinallinen äänikirja, jonka osallistujat kuuntelivat passiivisesti fMRI-rekisteröinnin aikana, ja jonka tapahtumarajat määriteltiin kokeellisesti erillisen koehenkilöryhmän avulla. Aivokuvausaineisto analysoitiin aivoalueittain sekä hippokampuksesta että eksploratiivisesti myös kaikilta aivokuoren alueilta. Hippokampuksen havaittiin reagoivan tapahtumarajoihin aktivaatiopiikein. Aivokuorella voimakkaasti reagoivia alueita olivat mm. posteriorinen mediaalinen aivokuori, ventromediaalinen prefrontaalialue, parahippokampaalinen poimu sekä etummainen pihtipoimu. Monien näistä alueista uskotaan osallistuvan meneillään olevan tapahtuman mallintamiseen ja hahmottamiseen, ja osa mahdollisesti osallistuu huomion siirtämiseen sisäisen ja ulkoisen välillä. Etummaisen pihtipoimun tiedetään osallistuvan odotusten ja havaintojen välisten konfliktien monitorointiin, mikä saattaisi tukea teoriaa, jonka mukaan segmentaatio olisi riippuvaista havaituista ennustevirheistä. Tätä ei kuitenkaan tämän tutkimuksen perusteella voida varmasti päätellä, vaan asiaa tulisi tutkia tarkemmin. Tämän tutkimuksen tulokset tukevat näkemystä, jonka mukaan hippokampuksen lisääntynyt toiminta tapahtumarajoilla liittyy korkean tason abstraktiin segmentaatioon ja mahdollisesti episodisen muiston luomiseen. Tämä prosessi mahdollisesti tapahtuu yhteistyössä aivokuoren aktiivisten alueiden kanssa, mutta kausaaliset suhteet ja informaation kulku näiden alueiden välillä on selvitettävä myöhemmissä tutkimuksissa.Event segmentation structures our experience as well as our memories. The representation of the currently ongoing event is likely dependent on a network of cortical areas, but the ability to retain a memory of the event requires an intact hippocampus. It is thus a relevant question how and when this hippocampal episodic encoding happens. It has previously been shown that the hippocampus is sensitive to event boundaries and responds to them with transient fMRI activation peaks. It has been proposed that these hippocampal end-of-event activations represent a high-level, modality-independent process of sharpening or “printing out” of the memory trace of the situation. However, the studies reporting hippocampal peaks have been conducted on audio-visual stimuli, so it is unclear whether these results generalise to narratives without a visual component, as the hippocampus is known to support visual processing as well as episodic encoding. In this study I aim to answer this question by analysing fMRI data from participants experiencing a purely auditory narrative. The stimulus was a 71-minute-long audio book, and it was segmented behaviourally by a separate group of participants with a naïve intuitive segmentation paradigm. The data was analysed with a region of interest (ROI) analysis in the hippocampus, as well as in an exploratory manner on all areas from a cortical atlas. The hippocampus was found to respond significantly to event boundaries in the story. Strong responses were also found in areas of the posterior medial cortex (PMC), as well as in ventromedial prefrontal cortex (vmPFC), parahippocampal gyrus, anterior cingulate (ACC) and the insula. Many of these are known to be involved in representing the event model, and some with switching between internal and external processing modes. ACC in particular is known to be involved in conflict monitoring – this might link with the proposal that segmentation in general is driven by prediction error and would merit further study. I conclude that the hippocampus does detect and respond to event boundaries in a naturalistic auditory narrative, which is in line with the “print out” hypothesis and implies that these activations are related to domain-general episodic encoding. The increased hippocampal processing is likely to happen in collaboration with cortical areas involved in signalling change and representing the working event model. However, the causal connections between these areas during the boundary-related processing cascade needs to be elaborated in future studies

Buchstaben und Wörter im Kontext

Acknowledgments VI Zusammenfassung VIII Summary XIV General Introduction 1 Extending the theoretical base: Interactive Activation Models 2 The original interactive activation model and the identification of letters 3 The Multiple Read-Out Model and the recognition of words 5 Dual-route models and phonological representations in visual word recognition and reading aloud 11 How to model (semantic) associations during word recognition? 17 Overview of the present studies and their methods 22 Study 1: Sub-lexical frequency measures provided by corpus analyses 22 Study 2: Word frequency, lexicality and optical imaging 24 Study 3: Modeling electrophysiological responses to conflicting lexical representations 29 Study 4: Affective connotation of lexical representations, ERPs, and pupillometry 31 Study 5: Modeling associations between lexical representations and Receiver Operation Characteristics 34 Study 1: Sub-lexical frequency measures for orthographic and phonological units in German 41 Introduction 42 Grain sizes, domains, databases, and measures 46 Grain sizes: Syllable, dual unit, or single unit 47 Processing domains: Orthography or phonology 49 Databases: Lemma or word form 50 Measures: Type or token 51 Method 53 Results 55 Syllable frequencies of the lemma database 56 Syllable frequencies of the word form database 56 Dual unit frequencies of the lemma database 57 Dual unit frequencies of the word form database 58 Single unit frequencies of the lemma database 58 Single unit frequencies of the word form database 59 Discussion 60 Study 2: Differential activation of frontal and parietal regions during visual word recognition: An optical topography study 65 Introduction 67 Methods 71 Participants 71 Materials 71 Experimental procedure 72 Data acquisition 73 Data analysis 73 Results 76 Behavioral results 76 fNIRS (optical topography) 78 Discussion 82 The lexicality effect 83 The word frequency effect 85 Optical topography as a tool to investigate word recognition 86 Appendix 90 Study 3: Conflict monitoring engages the mediofrontal cortex during nonword processing 91 Introduction 92 Methods 95 Participants 95 Materials 95 Procedure 95 Data acquisition 96 Data analysis 97 Results 98 Behavioral 98 ERPs 98 sLORETA 98 Discussion 101 Conclusion 103 Study 4: Affective processing within 1/10th of a second: High-Arousal is necessary for early facilitative processing of negative but not positive words 105 Introduction 107 Method 112 Participants 112 Materials 112 Procedure 113 Data acquisition 114 Data analysis 114 Results 117 Behavioral 117 ERPs 117 sLORETA 118 Discussion 122 Study 5: Remembering words in context as predicted by an Associative Read-Out Model 127 Introduction 129 Does associative spreading activate ‘false memories’? 130 Can each item’s signal be detected in an explicit memory task? 134 Simulation methods: The AROM and its predictions 137 Experimental methods: Testing the AROM’s predictions 140 Participants 140 Corpus 140 Stimuli 140 Procedure 143 Experimental and modeling results 144 Discussion 148 Conclusions 155 General Discussion: Summary and outlook 159 Sub-lexical frequencies 160 Sub-lexical frequency measures constrained the interpretation of effects! 160 Can the matching of global features be replaced by specific ones? 161 Word frequency and optical imaging 163 Optical imaging revealed greater “neural activations” to low frequency words! 163 What is “neural activation” in the IFG? 164 Lexical conflicts 167 Lexical conflicts predicted behavioral data and ACC activation! 167 Does associative-semantic competition predict IFG activation? 170 Affective word features 171 Affective lexical features elicited behavioral and ERP, but no pupil dilation effects! 171 Can semantic cohesiveness account for affective effects? 173 Associative-semantic representations 176 Modeling associations between the word stimuli of an experiment predicted false and veridical memories! 176 Going beyond measurement models of familiarity and recollection? 179 The rebirth of a mental lexicon: How to answer the challenge of fixing the structure of time? 182 Does the mind construct semantic taxonomies from associations? 186 Conclusions 189 References 191 Erklärung I Curriculum Vitae III Wissenschaftlicher Werdegang III Lehre III Vorträge III Poster V Publizierte Konferenzbeiträge und Buchkapitel VI Gutachtertätigkeiten VI Zeitschriftenartikel VIIThis dissertation investigated visual word recognition based on the theoretical framework of interactive activation models (IAMs, McClelland & Rumelhart, 1981). Study 1 provided sub-lexical frequency measures for German, which were used as control variables in the Studies 2, 4, and 5. Study 2 was the first of three studies using the lexical decision task. The optical imaging results showed that words elicit greater neural responses than nonwords in the left inferior parietal gyrus, which suggests a role of this region during the integration of orthographic, phonological and semantic representations. Greater activations for word stimuli in the superior frontal gyrus can be interpreted in terms of decision-related processes during visual word recognition. Moreover, rare words elicited greater neural activation than common words in the left inferior frontal gyrus. This word frequency effect suggests a role of this region during the selection of a semantic representation from many co-activated semantic candidates. Study 3 used an IAM to calculate the conflicts between orthographic representations, and set this model-generated measure of lexical competition into a direct relation with an event-related potentials (ERP) negativity between 400 and 600 ms post- stimulus. The electric sources of the ERP-conflict effects were attributed to the anterior cingulate cortex. The model accounted for a significant portion of item-level variance in reaction times, error rates and mean amplitudes. Study 4 showed that positive and high-arousal negative words elicit response facilitation and an early ERP effect between 80 and 120 ms post-stimulus, when compared to neutral words. The ERP-effect in high-arousal negative words was source-localized in the left fusiform and middle temporal gyri. The latter finding may be explained by the larger amount of associative relations of affective words. Study 5 captures associative relations between words for IAMs. Two words were defined as 'associated', when they co-occur significantly often together in the sentences of a large corpus. This corresponds to Hebbian learning: Items being repeatedly presented together are likely to be associated. The results of a recognition memory task showed that learned and non-learned words elicit greater 'yes' response rates when they provide a larger amount of associated items in the stimulus set. The co-occurrence statistics were further used to implement associations between words in a contextual representation layer. This IAM-model predicted which word is recognized with which probability on an item-level. Because many of the most strongly co-activated words revealed a semantic relation to the presented word (e.g., synonymy), the resulting 'Associative Read-Out Model' is the first IAM with a fully implemented semantic representation layer.Diese Dissertation untersucht die visuelle Worterkennung auf der theoretischen Grundlage des 'Interactive Activation Models' (IAM, McClelland und Rumelhart, 1981). Studie 1 stellt sub-lexikalische Häufigkeitsmaße für das Deutsche zur Verfügung: Orthographische und phonologische Silbenfrequenzen, Bigramm- Biphonemfrequenzen, sowie Buchstaben- und Phonemfrequenzen. Solche Maße dienen in den Studien 2, 4, und 5 als Kontroll-Variablen. Studie 2 ist die erste von drei Studien, welche die visuelle Worterkennung mit der lexikalischen Entscheidungsaufgabe untersucht. Die optischen Bildgebungsbefunde zeigen, dass Wörter höhere neuronale Aktivierungen im linken inferioren Parietallappen auslösen, was auf die Rolle dieser Region bei der Integration orthographischer, phonologischer und semantischer Repräsentationen hinweist. Die höheren Aktivierungen für Wörter im superioren Frontallappen weisen auf die entscheidungsrelevanten Prozesse der Worterkennung hin. Seltene Wörter lösten höhere Aktivierungen im linken inferioren Frontallappen im Vergleich zu häufigen Wörtern aus, was die Beteiligung dieser Region bei der Auswahl einer semantischen Repräsentation aus verschiedenen konfligierenden Kandidaten nahelegt. Studie 3 setzt simulierte Konflikte zwischen orthographischen Repräsentationen bei der Verarbeitung von Nichtwörtern in eine direkte Beziehung zu einer Negativierung des ereigniskorrelierten Potentials (EKP) zwischen 400 und 600 ms. Die Quellen dieser Aktivierungen wurden im anterioren cingulären Cortex verortet. Das Modell klärte signifikante Varianzanteile für Reaktionszeiten, Fehlerraten und EKP-Negativierungen auf. Studie 4 zeigt, dass positive und hocherregend negative Wörter Antworterleichterungen im Vergleich zu neutralen Wörtern und eine frühen EKP-Negativierung zwischen 80 und 120 ms nach Reizdarbietung auslösen. Der EKP-Effekt hocherregend negativer Wörter konnte im linken fusiformen und im mittleren temporalen Gyrus verortet werden, was dafür spricht, dass affektiv konnotierte Wörter mehr assoziativ verknüpfte Wörter aufweisen. Studie 5 erschließt assoziative Verknüpfungen zwischen Wörtern für IAMs. Zwei Wörter wurden als 'assoziiert' definiert, wenn sie in den Sätzen eines großen Satzkorpus signifikant häufig gemeinsam auftreten. Dies entspricht der Hebb'schen Lernregel: Wörter, die häufig gemeinsam auftreten, sind wahrscheinlich assoziiert. Die Ergebnisse einer Wiedererkennens-Gedächtnisaufgabe zeigen, dass gelernte und nicht-gelernte Wörter mehr 'Ja-'Antworten auslösen, wenn sie eine größere Anzahl assoziierte Wörter im Reizmaterial aufweisen. Die ko-okkurrenzstatistischen Maße wurden benutzt, um eine kontextuelle Modell-Repräsentationsebene mit assoziativen Verbindungsgewichten auszustatten. Das Modell sagt auf dem Item-level voraus, welches Wort mit welcher Wahrscheinlichkeit wiedererkannt wird. Da viele der am stärksten assoziierten Wörter zum präsentierten Wort eine semantische Verknüpfung aufweisen (z.B. Synonymie), ist das so gewonnene 'Associative Read-Out Model', das erste IAM mit einer semantischen Repräsentationsebene

An electrophysiological analysis of the time course of phonological and orthographic encoding in written word production

Recent evidence suggests that individuals generate written words based on both spelling and sound. The present study used event-related potentials (ERPs) to examine the relative time course of orthographic and phonological activation. We adopted Chinese as a target language in which spelling and sound are largely dissociated. Native speakers of Chinese Mandarin were presented with coloured pictures and wrote down colour and picture names as adjective-noun phrases. Colour and picture names were either phonologically related, orthographically related, or unrelated. EEG revealed phonological effects in the 200-500 ms time window, starting at 206 ms after picture onset, and orthographic effects in the 300-400 ms time window, starting at 298 ms. The results of our study suggest that activation of phonological codes takes place approximately 100 ms earlier than access to orthographic codes, which provides evidence for phonological encoding as early sources of constraint in written word production

Invariants for neural automata

Computational modeling of neurodynamical systems often deploys neural networks and symbolic dynamics. One particular way for combining these approaches within a framework called vector symbolic architectures leads to neural automata. Specifically, neural automata result from the assignment of symbols and symbol strings to numbers, known as Gödel encoding. Under this assignment, symbolic computation becomes represented by trajectories of state vectors in a real phase space, that allows for statistical correlation analyses with real-world measurements and experimental data. However, these assignments are usually completely arbitrary. Hence, it makes sense to address the problem which aspects of the dynamics observed under a Gödel representation is intrinsic to the dynamics and which are not. In this study, we develop a formally rigorous mathematical framework for the investigation of symmetries and invariants of neural automata under different encodings. As a central concept we define patterns of equality for such systems. We consider different macroscopic observables, such as the mean activation level of the neural network, and ask for their invariance properties. Our main result shows that only step functions that are defined over those patterns of equality are invariant under symbolic recodings, while the mean activation, e.g., is not. Our work could be of substantial importance for related regression studies of real-world measurements with neurosymbolic processors for avoiding confounding results that are dependant on a particular encoding and not intrinsic to the dynamics.RTI2018-093860-BC21 funded by (AEI/FEDER, UE) and acronym MathNEURO PID2020-117281GB-I00 PID2019-107444GA-I00 European Regional Development Fund (ERDF) Basque Government, grant IT1483-2

Experience-dependent plasticity in the auditory domain: effects of expertise and training on functional brain organization

The present dissertation aims at systematically investigating manifestations of experience-dependent plasticity in the auditory domain, resulting from intensive musical training, utilizing analytical tools from network neuroscience. The dissertation is based on data acquired in the course of a longitudinal study investigating structural and functional changes in the auditory domain due to music training. A group of aspiring professional musicians, attending preparatory courses for entrance exams at universities of arts, and a group of amateur musicians, actively practicing in their everyday life, completed up to 5 behavioral and neuroimaging assessments in the course of one year. The dissertation consists of three studies addressing cross-sectional and longitudinal aspects of functional plastic differences and changes, respectively, ranging from a specific auditory process over unconstrained music listening to longitudinal changes in functional organization

Uncovering the myth of learning to read Chinese characters: phonetic, semantic, and orthographic strategies used by Chinese as foreign language learners

Oral Session - 6A: Lexical modeling: no. 6A.3Chinese is considered to be one of the most challenging orthographies to be learned by non-native speakers, in particular, the character. Chinese character is the basic reading unit that converges sound, form and meaning. The predominant type of Chinese character is semantic-phonetic compound that is composed of phonetic and semantic radicals, giving the clues of the sound and meaning, respectively. Over the last two decades, psycholinguistic research has made significant progress in specifying the roles of phonetic and semantic radicals in character processing among native Chinese speakers …postprin

(Dis)connections between specific language impairment and dyslexia in Chinese

Poster Session: no. 26P.40Specific language impairment (SLI) and dyslexia describe language-learning impairments that occur in the absence of a sensory, cognitive, or psychosocial impairment. SLI is primarily defined by an impairment in oral language, and dyslexia by a deficit in the reading of written words. SLI and dyslexia co-occur in school-age children learning English, with rates ranging from 17% to 75%. For children learning Chinese, SLI and dyslexia also co-occur. Wong et al. (2010) first reported on the presence of dyslexia in a clinical sample of 6- to 11-year-old school-age children with SLI. The study compared the reading-related cognitive skills of children with SLI and dyslexia (SLI-D) with 2 groups of children …postprin

Speech-brain synchronization: a possible cause for developmental dyslexia

152 p.Dyslexia is a neurological learning disability characterized by the difficulty in an individual¿s ability to read despite adequate intelligence and normal opportunities. The majority of dyslexic readers present phonological difficulties. The phonological difficulty most often associated with dyslexia is a deficit in phonological awareness, that is, the ability to hear and manipulate the sound structure of language. Some appealing theories of dyslexia attribute a causal role to auditory atypical oscillatory neural activity, suggesting it generates some of the phonological problems in dyslexia. These theories propose that auditory cortical oscillations of dyslexic individuals entrain less accurately to the spectral properties of auditory stimuli at distinct frequency bands (delta, theta and gamma) that are important for speech processing. Nevertheless, there are diverging hypotheses concerning the specific bands that would be disrupted in dyslexia, and which are the consequences of such difficulties on speech processing. The goal of the present PhD thesis was to portray the neural oscillatory basis underlying phonological difficulties in developmental dyslexia. We evaluated whether phonological deficits in developmental dyslexia are associated with impaired auditory entrainment to a specific frequency band. In that aim, we measured auditory neural synchronization to linguistic and non-linguistic auditory signals at different frequencies corresponding to key phonological units of speech (prosodic, syllabic and phonemic information). We found that dyslexic readers presented atypical neural entrainment to delta, theta and gamma frequency bands. Importantly, we showed that atypical entrainment to theta and gamma modulations in dyslexia could compromise perceptual computations during speech processing, while reduced delta entrainment in dyslexia could affect perceptual and attentional operations during speech processing. In addition, we characterized the links between the anatomy of the auditory cortex and its oscillatory responses, taking into account previous studies which have observed structural alterations in dyslexia. We observed that the cortical pruning in auditory regions was linked to a stronger sensitivity to gamma oscillation in skilled readers, but to stronger theta band sensitivity in dyslexic readers. Thus, we concluded that the left auditory regions might be specialized for processing phonological information at different time scales (phoneme vs. syllable) in skilled and dyslexic readers. Lastly, by assessing both children and adults on similar tasks, we provided the first evaluation of developmental modulations of typical and atypical auditory sampling (and their structural underpinnings). We found that atypical neural entrainment to delta, theta and gamma are present in dyslexia throughout the lifespan and is not modulated by reading experience