The Neural Mechanisms Supporting Structure and Inter-Brain Connectivity In Natural Conversation

Conversation is the height of human communication and social interaction, yet little is known about the neural mechanisms supporting it. To date, there have been no ecologically valid neuroimaging studies of conversation, and for good reason. Until recently, imaging techniques were hindered by artifact related to speech production. Now that we can circumvent this problem, I attempt to uncover the neural correlates of multiple aspects of conversation, including coordinating speaker change, the effect of conversation type (e.g. cooperative or argumentative) on inter-brain coupling, and the relationship between this coupling and social coherence. Pairs of individuals underwent simultaneous fMRI brain scans while they engaged in a series of unscripted conversations, for a total of 40 pairs (80 individuals). The first two studies in this dissertation lay a foundation by outlining brain regions supporting comprehension and production in both narrative and conversation - two aspects of discourse level communication. The subsequent studies focus on two unique features of conversation: alternating turns-at-talk and establishing inter-brain coherence through speech. The results show that at the moment of speaker change, both people are engaging attentional and mentalizing systems - which likely support orienting toward implicit cues signaling speaker change as well as anticipating the other person's intention to either begin or end his turn. Four networks were identified that are significantly predicted by a novel measure of social coherence; they include the posterior parietal cortex, medial prefrontal cortex, and right angular gyrus. Taken together, the findings reveal that natural conversation relies on multiple cognitive networks besides language to coordinate or enhance social interaction. &#8195

A Model of the Network Architecture of the Brain that Supports Natural Language Processing

For centuries, neuroscience has proposed models of the neurobiology of language processing that are static and localised to few temporal and inferior frontal regions. Although existing models have offered some insight into the processes underlying lower-level language features, they have largely overlooked how language operates in the real world. Here, we aimed at investigating the network organisation of the brain and how it supports language processing in a naturalistic setting. We hypothesised that the brain is organised in a multiple core-periphery and dynamic modular architecture, with canonical language regions forming high-connectivity hubs. Moreover, we predicted that language processing would be distributed to much of the rest of the brain, allowing it to perform more complex tasks and to share information with other cognitive domains. To test these hypotheses, we collected the Naturalistic Neuroimaging Database of people watching full length movies during functional magnetic resonance imaging. We computed network algorithms to capture the voxel-wise architecture of the brain in individual participants and inspected variations in activity distribution over different stimuli and over more complex language features. Our results confirmed the hypothesis that the brain is organised in a flexible multiple core-periphery architecture with large dynamic communities. Here, language processing was distributed to much of the rest of the brain, together forming multiple communities. Canonical language regions constituted hubs, explaining why they consistently appear in various other neurobiology of language models. Moreover, language processing was supported by other regions such as visual cortex and episodic memory regions, when processing more complex context-specific language features. Overall, our flexible and distributed model of language comprehension and the brain points to additional brain regions and pathways that could be exploited for novel and more individualised therapies for patients suffering from speech impairments

Neural correlates of creative thinking : conceptual expansion processing

In contemporary scientific discourse, creative thinking is seen as arising from a medley of normative cognitive processes that are not exclusive to highly creative people. Following this modern continuum-based view, one functional magnetic resonance imaging (fMRI) study and one event-related potential (ERP) study, implementing a novel task design, were carried out to investigate the neural correlates of conceptual expansion, acritical facet of creative thinking. Conceptual expansion is one of the core processes in the invention of creative ideas and describes the ability to widen the boundaries of existing semantic concepts beyond conventional limits. The new task design used a modified version of the alternate uses task and required subjects to rate a given object and a described use for this object into one of three possible categories: High-unusual and low-appropriate (nonsense), low-unusual and high-appropriate (common), and high-unusual and high-appropriate (creative). Brain activation during trials rated as high-unusual and high-appropriate (creative) reflected conceptual expansion processing and were contrasted with trials reflecting pure novelty processing (nonsense object-use combinations) or appropriateness processing (creative and common uses). As hypothesized, the fMRI results showed conceptual expansion related activation in a semantic and conceptual integration network comprising the frontopolar cortex, anterior inferior frontal gyrus, anterior cingulate cortex and the temporal poles. The ERP study investigated whether conceptual expansion related activation could be observed in the N400-time window. Results showed a post-N400 effect, differentiating conceptual expansion specific processing from the processing of mere novelty or appropriateness, implicating semantic integration brain mechanisms. Both studies together led to the uncovering of relevant brain networks and cerebral time flow of conceptual expansion processing. These studies illustrate a new and viable approach by which to investigate predefined facets of creative thinking using neuroscientific methods, which allowed for more consistent and specific results to surface compared to prior creativity research

Perceptual coupling and decoupling of the default mode network during mind-wandering and reading

While reading, our mind can wander to unrelated autobiographical information, creating a perceptually decoupled state detrimental to narrative comprehension. To understand how this mind-wandering state emerges, we asked whether retrieving autobiographical content necessitates functional disengagement from visual input. In Experiment 1, brain activity was recorded using functional magnetic resonance imaging (fMRI) in an experimental situation mimicking naturally occurring mind-wandering, allowing us to precisely delineate neural regions involved in memory and reading. Individuals read expository texts and ignored personally relevant autobiographical memories, as well as the opposite situation. Medial regions of the default mode network (DMN) were recruited during memory retrieval. In contrast, left temporal and lateral prefrontal regions of the DMN, as well as ventral visual cortex, were recruited when reading for comprehension. Experiment two used functional connectivity both at rest and during tasks to establish that (i) DMN regions linked to memory are more functionally decoupled from regions of ventral visual cortex than regions in the same network engaged when reading; and (ii) individuals with more self-generated mental contents and poorer comprehension, while reading in the lab, showed more decoupling between visually connected DMN sites important for reading and primary visual cortex. A similar pattern of connectivity was found in Experiment 1, with greater coupling between this DMN site and visual cortex when participants reported greater focus on reading in the face of conflict from autobiographical memory cues; moreover, the retrieval of personally rele- vant memories increased the decoupling of these sites. These converging data suggest we lose track of the narrative when our minds wander because generating autobiographical mental content relies on cortical regions within the DMN which are functionally decoupled from ventral visual regions engaged during reading

The neural processing of sentences in bilinguals and monolinguals: an fMRI study of portugues-english bilinguals and portuguese monolinguals reading comprehension effects on brain activation

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão, Programa de Pós-Graduação em Inglês: Estudos Linguísticos e Literários, Florianópolis, 2016.Abstract : Although considerable research has been devoted to language representation and processing in monolinguals, rather less attention has been paid to language representation and processing in bilinguals and even less, to the comparison in brain activation between bilinguals and monolinguals reading in their L1. Since the majority of studies on bilingualism investigate processes at the word level, the present dissertation, a cross-sectional, quantitative and exploratory study in nature, aimed at investigating monolingual and bilingual brains and their neuroanatomical response to the processing of written sentences. More specifically, it sought to explore (1) whether and to what extent Portuguese and English are represented and processed in the same areas of the brain in late bilinguals; (2) whether Portuguese is represented and processed in same brain areas in bilinguals and monolinguals; (3) whether the semantic neural representation of sentences in one language can be identified based on the brain activation for the same sentences in another language; (4) whether individual differences, namely proficiency in the second language and working memory capacity, modulate activation in bilinguals, whether working memory capacity modulate activation in monolinguals; and (5), whether word length and lexical frequency have an effect on brain activation. Twelve Brazilian Portuguese-English late bilinguals and ten Brazilian Portuguese monolinguals participated in the study. Data collection took place at Carnegie Mellon University during a PhD internship. The stimuli consisted of 60 sentences in English and their translation-equivalent sentences in Portuguese (e.g., The diplomat negotiated at the embassy/O diplomata negociou na embaixada). Bilingual participants read the sentences while functional images were acquired on two separate days while monolinguals only read the Portuguese sentences in a single session. Data were analyzed statistically and revealed, in general terms, that language representation and processing engages a complex network of brain areas in monolinguals and bilinguals. For processing the L2, bilinguals recruit a more widely distributed set of areas bilaterally than for processing the L1 (more left-lateralized). For processing the L1, in comparison with monolinguals, bilinguals recruited additional bilateral areas for dealing with the phonological and semantic aspects of the L1. In spite of the small differences in processing the languages, the commonalities in concept representations across languages were sufficient to allow decoding of sentences using multi-voxel pattern analysis and machine learning techniques. The model generated reasonable accurate predictions of the neural representation of words in the context of sentences based on simple addition of words, semantic features and semantic cuboids derived from an independent study. Variables as proficiency in the L2, working memory capacity, word length and lexical frequency modulated brain activation. In a nutshell, findings add support to the literature about bilingual and monolingual language comprehension and contribute to the area by suggesting that there are common neural areas involved in the representation of different languages and cultures.Embora um considerável número de pesquisas tem se dedicado à representação e ao processamento da linguagem em monolíngues, menos atenção tem sido dada à representação e ao processamento da linguagem em bilíngues e ainda menos, à comparação da ativação cerebral de bilíngues e monolíngues ao ler sua L1. Já que a maioria dos estudos sobre bilinguismo investigam processos no nível da palavra, a presente tese, um estudo de natureza transversal, quantitativa e exploratória, objetivou investigar cérebros de monolíngues e bilíngues e sua resposta neuroanatômica ao processamento de frases escritas. Mais especificamente, procurou explorar (1) se as duas línguas, português e inglês, são representadas e processadas nas mesmas áreas do cérebro de bilíngues tardios e em que medida; (2) se a L1 (português) é representada e processada nas mesmas áreas cerebrais de bilíngues e monolíngues; (3) se a representação neural semântica de frases em uma língua pode ser identificada baseada na ativação cerebral das mesmas frases na outra língua; (4) se diferenças individuais como a proficiência na segunda língua e a capacidade de memória de trabalho modulam a ativação em bilíngues e se a capacidade de memória de trabalho modula a ativação em monolíngues; e (5) se a extensão das palavras e a frequência lexical têm efeito na ativação cerebral. Doze bilíngues tardios do par linguístico português brasileiro-inglês e 10 monolíngues do português brasileiro participaram do estudo. Os dados foram coletados na Carnegie Mellon University durante o doutorado sanduíche. Os estímulos consistiram de 60 frases em inglês e frases equivalentes em português (ex.: The diplomat negotiated at the embassy/O diplomata negociou na embaixada). Os participantes bilíngues leram as frases enquanto imagens funcionais do cérebro eram adquiridas em dois dias distintos enquanto os participantes monolíngues apenas leram as frases em português numa única sessão. Os dados foram analisados estatisticamente e revelam, em termos gerais, que a representação e o processamento da linguagem engaja uma rede complexa de áreas cerebrais em monolíngues e bilíngues. Para processar a L2, os bilíngues recrutam um conjunto mais amplamente distribuído de áreas bilaterais que para processar a L1 (mais lateralizada à esquerda). Para processar a L1, em comparação com os monolíngues, os bilíngues recrutaram áreas adicionais bilateralmente para lidar com os aspectos fonológicos e semânticos da L1. Apesar das pequenas diferenças no processamento das línguas, as semelhanças na representação dos conceitos entre as línguas foram suficientes para permitir a decodificação de frases usando técnicas de aprendizagem de máquina e de análise de padrão multi-voxel. O modelo gerou predições razoavelmente precisas da representação neural de palavras no contexto de frases baseado na adição simples de palavras, características semânticas e cuboides semânticos derivados de um estudo independente. Variáveis como a proficiência na L2, a capacidade de memória de trabalho, a extensão das palavras, e a frequência lexical influenciaram a ativação cerebral. Em suma, os achados corroboram a literatura sobre compreensão de linguagem em monolíngues e bilíngues e contribuem com a área ao sugerir que há áreas neurais comuns envolvidas na representação de diferentes línguas e culturas

An Electroencephalogram Investigation of Two Modes of Reasoning

The use of electroencephalography (EEG) to exam the electrical brain activity associated with reasoning provides an opportunity to quantify the functional and temporal aspects of this uniquely human capability, and at the same time expand our knowledge about what a given event-related potential (ERP) might measure. The question of what form of mental representation and transformational processes underlie human reasoning has been a central theme in cognitive psychology since its inception (Chomsky, 1957; McCarthy, 1955; Miller, 1956; Newell, Shaw, Simon, 1958). Two prominent, but competing views remain at the forefront of the discussion, one positing that human inference making is principally syntactic (Braine & O'Brien, 1998; Fodor, 1975; Pylyshyn, 1984; Rips, 1994), and the other that it is, fundamentally, semantic in nature (Gentner & Stevens, 1983; Johnson-Laird, 1983). The purpose of the proposed study is to investigate the neurophysiology of mental model (MM) and mental rule (MR) reasoning using high-density electroencephalography (EEG), with the goal of providing a characterization of the time course and a general estimate of the spatial dimensions of the brain activations correlated with these specific instances of two classic views of reasoning. The research is motivated by two questions: 1) Will violations of expectancy established by the devised MM and MR reasoning strategies evoke the N400 and P600 ERPs, respectively, and 2) Will topographical scalp distributions associated with each reasoning strategy suggest distinct psychological representations and processes? A finding of a N400 response in the MM condition suggests that reasoning about the relations between entities in the type of problems presented engages a network of cortical areas previously shown to be involved in processing violations of semantic expectancies in studies of language comprehension. By comparison, incongruent events in the MR condition are expected to evoke a bilateral anterior P600, a component previously associated with recognizing and restructuring syntactic anomalies or incongruities in sentence comprehension. If the hypothesized results are obtained they would provide potentially insightful information about the chronometry of mental processes associated with the different representations and inference making mechanisms postulated to support each mode of reasoning, and as well, broaden our understanding of the neural functionality associated with the N400 and P600 ERP

Principled Explanations in Comparative Biomusicology – Toward a Comparative Cognitive Biology of the Human Capacities for Music and Language

The current thesis tackles the question “Why is music the way it is?” within a comparative biomusicology framework by focusing on musical syntax and its relation to syntax in language. Comparative biomusicology integrates different comparative approaches, biological frameworks as well as levels of analysis in cognitive science, and puts forward principled explanations, regarding cognitive systems as different instances of the same principles, as its central research strategy. The main goal is to provide a preliminary answer to this question in form of hypotheses about neurocognitive mechanisms, i.e., cognitive and neural processes, underlying a core function of syntactic computation in language and music, i.e., mapping hierarchical structure and temporal sequence. In particular, the relationship between language and music is discussed on the basis of a top-down approach taking syntax as combinatorial principles and a bottom-up approach taking neural structures and operations as implementational principles. On the basis of the top-down approach, the thesis identifies computational problems of musical syntax, cognitive processes and neural correlates of music syntactic processing, and the relationship to language syntax and syntactic processing. The neural correlates of music syntactic processing are investigated by ALE meta-analyses. The bottom-up approach then studies the relationship between language and music on the basis of neural processes implemented in the cortico-basal ganglia-thalamocortical circuits. The main result of the current thesis suggests that the relationship between language and music syntactic processing can be explained in terms of the same neurocognitive mechanisms with different expressions on the motor-to-cognitive gradient. The current thesis, especially its bottom-up approach, opens up a possible way going toward comparative cognitive biology, i.e., a comparative approach to cognitive systems with a greater emphasis on the biology

Les corrélats comportementaux et neurofonctionnels de la thérapie par analyse des composantes phonologiques

L'aphasie est un trouble acquis du langage qui cause des difficultés de communication pouvant affecter la compréhension et l'expression. Lorsque l'aphasie survient par suite d’un accident vasculaire cérébral, des difficultés importantes de communication sont initialement constatées, puis une amélioration graduelle dans les semaines et les mois qui suivent. Toutefois, pour certaines personnes, l’aphasie peut rester sévère un an, l’aphasie est alors considérée comme chronique. L'une des manifestations les plus courantes de l'aphasie est l'anomie - un manque de mots. Un nombre croissant d'études se penchent sur l'impact de la thérapie sur l'aphasie chronique et ont montré une amélioration du langage après plusieurs années. Des thérapies sont développées et étudiées en anglais, mais peu d’écrits scientifiques décrivent les effets des thérapies spécifiques et formalisées en français. Le but du présent travail doctoral est d'examiner les effets comportementaux et d’explorer les substrats neurologies du protocole Analyse des Composantes Phonologiques (ACP) ainsi que l’influence des facteurs individuels sur les gains chez dix-huit personnes vivant avec une aphasie chronique. Ainsi, la thèse comporte trois articles qui abordent, chacun, des facettes différentes des effets de l’ACP. Dans le premier article, les effets comportementaux de l’ACP sont explorés ainsi que le rôle des facteurs individuels sur la réponse à l’intervention. Le deuxième considère l’effet de l’ACP sur la connectivité fonctionnelle du cerveau, tout en considérant l’effet des facteurs individuels. Enfin, le troisième article s’intéresse à la réponse par suite de l’intervention ACP en considérant les facteurs individuels du bilinguisme et des caractéristiques neuroanatomiques comme la localisation des lésions et l’épaisseur corticale. L'ensemble de résultats démontre une amélioration significative de la dénomination des noms parmi les dix-huit participant·e·s inclus·e·s. Par ailleurs, cette amélioration a été mesurée sur des mots non-traités, ce qui reflète un effet de généralisation. Par surcroit, cette amélioration se maintient à trois mois et six mois après la fin de l’intervention. L’âge, la sévérité de l’anomie et la présence d’une apraxie de la parole sévère influencent les gains mesurés. Les analyses neurofonctionnelles de connectivité fonctionnelle au repos auprès de dix participant·e·s montrent une meilleure intégration des réseaux visuels et langagiers, associés lors d’une tâche de dénomination des noms. Les changements de connectivité observés dépendent de la sévérité de l'aphasie. Plus spécifiquement, l’intégration interhémisphérique et au sein de l’hémisphère droit est plus importante chez les personnes ayant une aphasie plus légère. Finalement, être bilingue influence aussi les gains observés par suite du protocole Analyse des Composantes Phonologiques. Les personnes bilingues s’améliorent davantage, cet avantage est associé une épaisseur corticale supérieure dans l’hémisphère droit comparativement aux personnes monolingues. Ces données préliminaires suggèrent que les changements comportementaux observés résultent de la combinaison des composantes phonologiques et orthographiques de l'intervention, et que l’intervention requiert et entraîne les fonctions exécutives. Les données cueillies dans le cadre de cette thèse sont prometteuses en ce qui concerne les effets bénéfiques de l’Analyse des Composantes Phonologiques en français. Plus précisément, les gains mesurés auprès d’un échantillon présentant des symptômes variés, la généralisation et le maintien supportent l’utilisation de ce protocole d’intervention auprès de personnes francophones. Il est à noter que la taille du groupe étudié est modeste, mais les méthodologies complémentaires et la convergence des résultats avec les données existantes dans les écrits scientifiques et à travers les méthodes d'étude de cette thèse (niveaux comportemental et neurofonctionnel) justifient des études subséquentes pour mieux comprendre les mécanismes d'action et l’influence des facteurs individuels sur les gains.Aphasia is an acquired language disorder that causes communication difficulties that can affect understanding and expression. When aphasia occurs following a stroke, significant communication difficulties are initially noted, followed by gradual improvement over the following weeks and months. However, for some people, the aphasia may remain severe after a year even when recovery has been seen, in which case the aphasia is considered chronic. One of the most common symptoms in aphasia is anomia - a difficulty finding and saying words. A growing number of studies are looking at the impact of speech and language therapy on chronic aphasia and have shown improvement in language after several years. Therapies are developed and studied in English, but little scientific literature describes the effects of specific, formalized therapies in French. The purpose of the present doctoral work is to examine the behavioural effects and to explore the neurological substrates of the Phonological Component Analysis (PCA) protocol as well as the influence of individual factors on gains in 18 adults living with chronic aphasia. Thus, the dissertation consists of three articles, each of which addresses different facets of the effects of PCA. In the first article, the behavioural effects of PCA are explored as well as the role of individual factors on the response to the intervention. The second considers the effect of PCA on resting-state functional brain connectivity, while considering the effect of individual factors. Finally, the third paper looks at the response to the PCA intervention by considering the individual factors of bilingualism and neuroanatomical characteristics associated such as cortical thickness, and also lesion location. The overall results show a significant improvement in naming among the eighteen participants included in the thesis. Moreover, this improvement was measured on untreated words, which reflects a generalization effect and was maintained at three and six months after the end of the intervention. Age, severity of impairment and the presence of severe apraxia of speech influenced the measured gains. Neurofunctional analyses of resting-state functional connectivity in ten participants showed improved integration of the visual and language networks, associated during a naming task. The connectivity changes observed depend on the severity of the aphasia. More specifically, interhemispheric and right hemisphere integration is greater in individuals with milder aphasia. Finally, being bilingual also influences the gains observed following the Phonological Component Analysis protocol. Bilinguals improved more, and this advantage was associated with greater cortical thickness in frontal regions of the right hemisphere compared to monolinguals. These preliminary data suggest that the observed behavioural changes result from the combination of the phonological and orthographic components of the intervention, and that the intervention requires and trains executive functions. The data collected in this thesis show promise for the beneficial effects of Phonological Component Analysis in French. Specifically, the gains, generalization, and maintenance measured in a sample of adults living with aphasia displaying varied symptoms support the use of this intervention protocol with French speakers. It should be noted that the size of the study group is modest, but the complementary methodologies and the convergence of the results with existing data in the scientific literature and through the study methods of this thesis (behavioural and neurofunctional levels) justify subsequent studies to better understand the mechanisms of action and the influence of individual factors on gains

One Model to Rule them all: Multitask and Multilingual Modelling for Lexical Analysis

When learning a new skill, you take advantage of your preexisting skills and knowledge. For instance, if you are a skilled violinist, you will likely have an easier time learning to play cello. Similarly, when learning a new language you take advantage of the languages you already speak. For instance, if your native language is Norwegian and you decide to learn Dutch, the lexical overlap between these two languages will likely benefit your rate of language acquisition. This thesis deals with the intersection of learning multiple tasks and learning multiple languages in the context of Natural Language Processing (NLP), which can be defined as the study of computational processing of human language. Although these two types of learning may seem different on the surface, we will see that they share many similarities. The traditional approach in NLP is to consider a single task for a single language at a time. However, recent advances allow for broadening this approach, by considering data for multiple tasks and languages simultaneously. This is an important approach to explore further as the key to improving the reliability of NLP, especially for low-resource languages, is to take advantage of all relevant data whenever possible. In doing so, the hope is that in the long term, low-resource languages can benefit from the advances made in NLP which are currently to a large extent reserved for high-resource languages. This, in turn, may then have positive consequences for, e.g., language preservation, as speakers of minority languages will have a lower degree of pressure to using high-resource languages. In the short term, answering the specific research questions posed should be of use to NLP researchers working towards the same goal.Comment: PhD thesis, University of Groninge