Gestures Enhance Foreign Language Learning

Language and gesture are highly interdependent systems that reciprocally influence each other. For example, performing a gesture when learning a word or a phrase enhances its retrieval compared to pure verbal learning. Although the enhancing effects of co-speech gestures on memory are known to be robust, the underlying neural mechanisms are still unclear. Here, we summarize the results of behavioral and neuroscientific studies. They indicate that the neural representation of words consists of complex multimodal networks connecting perception and motor acts that occur during learning. In this context, gestures can reinforce the sensorimotor representation of a word or a phrase, making it resistant to decay. Also, gestures can favor embodiment of abstract words by creating it from scratch. Thus, we propose the use of gesture as a facilitating educational tool that integrates body and mind

Understanding language and attention: brain-based model and neurophysiological experiments

This work concerns the investigation of the neuronal mechanisms at the basis of language acquisition and processing, and the complex interactions of language and attention processes in the human brain. In particular, this research was motivated by two sets of existing neurophysiological data which cannot be reconciled on the basis of current psycholinguistic accounts: on the one hand, the N400, a robust index of lexico-semantic processing which emerges at around 400ms after stimulus onset in attention demanding tasks and is larger for senseless materials (meaningless pseudowords) than for matched meaningful stimuli (words); on the other, the more recent results on the Mismatch Negativity (MMN, latency 100-250ms), an early automatic brain response elicited under distraction which is larger to words than to pseudowords. We asked what the mechanisms underlying these differential neurophysiological responses may be, and whether attention and language processes could interact so as to produce the observed brain responses, having opposite magnitude and different latencies. We also asked questions about the functional nature and anatomical characteristics of the cortical representation of linguistic elements. These questions were addressed by combining neurocomputational techniques and neuroimaging (magneto-encephalography, MEG) experimental methods. Firstly, a neurobiologically realistic neural-network model composed of neuron-like elements (graded response units) was implemented, which closely replicates the neuroanatomical and connectivity features of the main areas of the left perisylvian cortex involved in spoken language processing (i.e., the areas controlling speech output – left inferior-prefrontal cortex, including Broca’s area – and the main sensory input – auditory – areas, located in the left superior-temporal lobe, including Wernicke’s area). Secondly, the model was used to simulate early word acquisition processes by means of a Hebbian correlation learning rule (which reflects known synaptic plasticity mechanisms of the neocortex). The network was “taught” to associate pairs of auditory and articulatory activation patterns, simulating activity due to perception and production of the same speech sound: as a result, neuronal word representations distributed over the different cortical areas of the model emerged. Thirdly, the network was stimulated, in its “auditory cortex”, with either one of the words it had learned, or new, unfamiliar pseudoword patterns, while the availability of attentional resources was modulated by changing the level of non-specific, global cortical inhibition. In this way, the model was able to replicate both the MMN and N400 brain responses by means of a single set of neuroscientifically grounded principles, providing the first mechanistic account, at the cortical-circuit level, for these data. Finally, in order to verify the neurophysiological validity of the model, its crucial predictions were tested in a novel MEG experiment investigating how attention processes modulate event-related brain responses to speech stimuli. Neurophysiological responses to the same words and pseudowords were recorded while the same subjects were asked to attend to the spoken input or ignore it. The experimental results confirmed the model’s predictions; in particular, profound variability of magnetic brain responses to pseudowords but relative stability of activation to words as a function of attention emerged. While the results of the simulations demonstrated that distributed cortical representations for words can spontaneously emerge in the cortex as a result of neuroanatomical structure and synaptic plasticity, the experimental results confirm the validity of the model and provide evidence in support of the existence of such memory circuits in the brain. This work is a first step towards a mechanistic account of cognition in which the basic atoms of cognitive processing (e.g., words, objects, faces) are represented in the brain as discrete and distributed action-perception networks that behave as closed, independent systems

Modelling concrete and abstract concepts using brain-constrained deep neural networks

A neurobiologically constrained deep neural network mimicking cortical areas relevant for sensorimotor, linguistic and conceptual processing was used to investigate the putative biological mechanisms underlying conceptual category formation and semantic feature extraction. Networks were trained to learn neural patterns representing specific objects and actions relevant to semantically ‘ground’ concrete and abstract concepts. Grounding sets consisted of three grounding patterns with neurons representing specific perceptual or action-related features; neurons were either unique to one pattern or shared between patterns of the same set. Concrete categories were modelled as pattern triplets overlapping in their ‘shared neurons’, thus implementing semantic feature sharing of all instances of a category. In contrast, abstract concepts had partially shared feature neurons common to only pairs of category instances, thus, exhibiting family resemblance, but lacking full feature overlap. Stimulation with concrete and abstract conceptual patterns and biologically realistic unsupervised learning caused formation of strongly connected cell assemblies (CAs) specific to individual grounding patterns, whose neurons were spread out across all areas of the deep network. After learning, the shared neurons of the instances of concrete concepts were more prominent in central areas when compared with peripheral sensorimotor ones, whereas for abstract concepts the converse pattern of results was observed, with central areas exhibiting relatively fewer neurons shared between pairs of category members. We interpret these results in light of the current knowledge about the relative difficulty children show when learning abstract words. Implications for future neurocomputational modelling experiments as well as neurobiological theories of semantic representation are discussed

Gestures Enhance Foreign Language Learning

Language and gesture are highly interdependent systems that reciprocally influence each other. For example, performing a gesture when learning a word or a phrase enhances its retrieval compared to pure verbal learning. Although the enhancing effects of co-speech gestures on memory are known to be robust, the underlying neural mechanisms are still unclear. Here, we summarize the results of behavioral and neuroscientific studies. They indicate that the neural representation of words consists of complex multimodal networks connecting perception and motor acts that occur during learning. In this context, gestures can reinforce the sensorimotor representation of a word or a phrase, making it resistant to decay. Also, gestures can favor embodiment of abstract words by creating it from scratch. Thus, we propose the use of gesture as a facilitating educational tool that integrates body and mind

Neurobiological mechanisms for language, symbols and concepts: Clues from brain-constrained deep neural networks

Neural networks are successfully used to imitate and model cognitive processes. However, to provide clues about the neurobiological mechanisms enabling human cognition, these models need to mimic the structure and function of real brains. Brain-constrained networks differ from classic neural networks by implementing brain similarities at different scales, ranging from the micro- and mesoscopic levels of neuronal function, local neuronal links and circuit interaction to large-scale anatomical structure and between-area connectivity. This review shows how brain-constrained neural networks can be applied to study in silico the formation of mechanisms for symbol and concept processing and to work towards neurobiological explanations of specifically human cognitive abilities. These include verbal working memory and learning of large vocabularies of symbols, semantic binding carried by specific areas of cortex, attention focusing and modulation driven by symbol type, and the acquisition of concrete and abstract concepts partly influenced by symbols. Neuronal assembly activity in the networks is analyzed to deliver putative mechanistic correlates of higher cognitive processes and to develop candidate explanations founded in established neurobiological principles

Prädiktoren erfolgreichen Sprachlernens im Alter

Sprachliche Fertigkeiten sind der Schlüssel zu akademischem und beruflichem Erfolg. Bislang ist weitestgehend unbekannt, welche kognitiven Leistungen das Sprachlernen beeinflussen. Ziel der Studie war es, Prädiktoren für den erfolgreichen Spracherwerb gesunder Menschen im Alter (65-80 Jahren) zu identifizieren. Die Kenntnis über Zusammenhänge von Kognition und Spracherwerb sind eine unerlässliche Voraussetzung für die zielgerichtete Therapiezuweisung von Schlaganfallpatienten mit Aphasie. Die Teilnehmer lernten mittels assoziativer Lernprinzipien ein Miniaturlexikon. Eine neuropsychologische Testbatterie erfasste den kognitiven Status aller Probanden. Beste Prädiktoren für ein erfolgreiches intensives Sprachtraining waren allgemeine sprachliche, visuell-räumliche und gute frontal exekutive Leistungen, gute assoziative Lernleistungen sowie eine gute Merkfähigkeit für Geschichten. Das Fehlen von Hinweisen auf eine dementielle Entwicklung wirkte sich ebenfalls positiv aus

Dinamica di Latching in una Memoria Associativa di Potts

Scopo di questo lavoro di tesi e' costruire un modello di rete neurale che mimi il comportamento della corteccia cerebrale in due particolari processi cognitivi: il processo di riconoscimento, ovvero come le informazioni memorizzate vengano recuperate a seguito della presentazione di uno stimolo esterno, e il processo di latching, ovvero lo spontaneo passaggio da una memoria ad un'altra ad essa correlata. La modellizazione di questi processi si basa sulla descrizione della corteccia in termini di rete neurale e sfrutta il formalismo del modello di Ising. In questi termini la rete diventa una memoria associativa e un 'concetto' coincide con una particolare configurazione della rete (pattern). Punto di partenza e' il modello classico di Hopfield, modificato di volta in volta secondo due criteri guida fondamentali: da un lato le caratteristiche strutturali biologiche del sistema considerato, la corteccia, dall'altro l'esigenza di descrivere non solo il recupero di un concetto all'interno della rete, ma anche il fenomeno di passaggio spontaneo da un concetto all'altro. Rispettivamente, i vincoli legati alla struttura biologica portano all'utilizzo del modello di Potts, mentre il fenomeno di passaggio, o latching, comporta l'introduzione di patterns correlati e di un complesso meccanismo di adattamento della rete. Sono stati prodotti dunque quattro modelli, evoluti uno dall'altro. Per ognuno dei essi e' stata sviluppata una simulazione numerica e una derivazione teorica delle equazioni della dinamica. Il modello finale consiste in una rete di Potts in grado di recuperare un pattern tra i memorizzati e da questo passare spontaneamente ad un altro ad esso correlato. Il motivo che ci ha spinto allo studio di tale processo e' legato alla possibilita' che questo rappresenti l'unita' di base nel fenomeno che noi percepiamo come susseguirsi di pensieri

A Neurobiologically Constrained Model

Understanding the meaning of words and its relationship with the outside world involves higher cognitive processes unique of the human brain. Despite many decades of research on the neural substrates of semantic processing, a consensus about the functions and components of the semantic system has not been reached among cognitive neuroscientists. This issue is mainly influenced by two sets of neurocognitive empirical findings that have shown (i) the existence of several regions acting as ’semantic hubs’, where the meaning of all types of words is processed and (ii) the presence of other cortical regions specialised for the processing of specific semantic word categories, such as animals, tools, or actions. Further evidence on semantic meaning processing comes from neuroimaging and transcranial magnetic stimulation studies in visually deprived population that acquires semantic knowledge through non-sensory modalities. These studies have documented massive neural changes in the visual system that is in turn recruited for linguistic and semantic processing. On this basis, this dissertation investigates the neurobiological mechanism that enables humans to acquire, store and processes linguistics meaning by means of a neurobiologically constrained neural network and offers an answer to the following hotly debated questions: Why both semantic hubs and modality-specific regions are involved in semantic meaning processing in the brain? Which biological principles are critical for the emergence of semantics at the microstructural neural level and how is the semantic system implemented under deprived conditions, in particular in congenitally blind people? First, a neural network model closely replicating the anatomical and physiological features of the human cortex was designed. At the micro level, the network was composed of 15,000 artificial neurons; at the large-scale level, there were 12 areas representing the frontal, temporal, and occipital lobes relevant for linguistic and semantic processing. The connectivity structure linking the different cortical areas was purely based on neuroanatomical evidence. Two models were used, each simulating the same set of cortical regions but at different level of details: one adopted a simple connectivity structure with a mean-field approach (i.e. graded-response neurons), and the other used a fully connected model with adaptation-based spiking cells. Second, the networks were used to simulate the process of learning semantic relationships between word-forms, specific object perceptions, and motor movements of the own body in deprived and undeprived visual condition. As a result of Hebbian correlated learning, distributed word-related cell assembly circuits spontaneously emerged across the different cortical semantic areas exhibiting different topographical distribution. Third, the network was reactivated with the learned auditory patterns (simulating word recognition processes) to investigate the temporal dynamics of cortical semantic activation and compare them with real brain responses. In summary, the findings of the present work demonstrate that meaningful linguistic units are represented in the brain in the form of cell assemblies that are distributed over both semantic hubs and category-specific regions spontaneously emerged through the mutual interaction of a single set of biological mechanisms acting within specific neuroanatomical structures. These biological principles acting together also offer an explanation of the mechanisms underlying massive neural changes in the visual cortex for language processing caused by blindness. The present work is a first step in better understanding the building blocks of language and semantic processing in sighted and blind populations by translating biological principles that govern human cognition into precise mathematical neural networks of the human brain.Um die Bedeutung von Wörtern und ihre Beziehung zur Außenwelt zu verstehen, müssen die kognitiven Prozesse betrachtet werden, die einzigartig für das menschliche Gehirn sind. Trotz jahrzehntelanger Forschungen an den neuronalen Substraten der semantischen Verarbeitung im menschlichen Gehirn wurde bisher kein Konsens über die Funktionen und Komponenten des semantischen Systems in den kognitiven Neurowissenschaftlern erreicht. Dieses Problem gründet darin, dass neurokognitive empirische Studien zumeist zu zwei Endergebnissen kamen: (i) der Existenz von mehrere Regionen, die als ‘semantische Hubs’ fungieren, in denen die Bedeutung aller Wortarten verarbeitet wird, und (ii) dem Vorhandensein weiterer kortikaler Regionen, die auf die Verarbeitung spezifischer semantischer Kategorien wie Tiere, Werkzeuge oder Aktionswörtern spezialisiert sind. Ein weiterer Beweis für die Verarbeitung semantischer Bedeutungen lässt sich aus Bildgebungsstudien und Studien mit transkranialer Magnetstimulation an visuell benachteiligten Probanden entnehmen, die die linguistische Bedeutung nicht durch sensorische Modalitäten erwerben. Diese Studien konnten massive neuronale Veränderungen im visuellen System dokumentieren, die wiederum für die sprachliche und semantische Verarbeitung verwendet werden. Die vorliegende Dissertation untersucht mittels eines biologischen neuronalen Netzwerkes jene kognitiven Prozesse, die es Menschen ermöglichen, linguistische Bedeutungen in der täglichen Kommunikation zu erfassen, zu speichern und zu verarbeiten. Sie schlägt Antworten auf die folgenden neurowissenschaftlich heiß diskutierten Fragen vor: Warum sind sowohl semantische Hubs als auch modalitätsspezifische Regionen relevant für die sprachliche und semantische Informationsverarbeitung im Gehirn? Welche biologischen Prinzipien sind von entscheidender Bedeutung für die Entstehung von Semantik auf mikrostruktureller neuronaler Ebene? Und Wie ist das semantische System unter benachteiligten Bedingungen repräsentiert? Zunächst wurde ein neuronales Netzwerkmodell implementiert, das die anatomischen und physiologischen Merkmale des menschlichen Kortex präzise widerspiegelt. Auf der Mikroebene besteht das Netzwerkmodel aus 15.000 künstlichen Neuronen, auf der Großebene aus 12 Arealen der Frontal-, Temporal- und Okzipitallappen, die für die sprachliche und semantische Verarbeitung relevant sind. Die Verbindungsstruktur zwischen den verschiedenen kortikalen Arealen wurde rein auf Grundlage von neuroanatomischen Befunden implementiert. Zwei Modelle wurden verwendet, die jeweils die gleichen kortikalen Regionen simulierten, allerdings in verschiedenen Varianten: Das erste Modell ging von einer einfachen Konnektivitätsstruktur mit einem Mean-field Ansatz (graded-response neurons) aus, während das zweite einen vollständig verbundenen Aufbau mit adaptionsbasierten Spiking-Zellen (Aktionspotential) verwendete. Anschließend dienten die neuronalen Netzwerke dazu, den Lernprozess der semantischen Verlinkung zwischen Wortformen, bestimmten Objektwahrnehmungen und motorischen Bewegungen des eigenen Körpers zu simulieren, sowohl in gesundem als auch in benachteiligtem Sehzustand. Als Ergebnis des Hebbschen Korrelationslernens traten spontan verteilte Neuronenverbindungen (cell assemblies) in den verschiedenen kortikalen semantischen Bereichen auf, die unterschiedliche topografische Verteilungen zeigten. Zuletzt wurde das Netzwerkmodell mit den erlernten auditorischen Mustern reaktiviert (Worterkennungsprozesse), um die zeitliche Dynamik kortikaler semantischer Aktivierung zu untersuchen und sie mit realen Gehirnantworten zu vergleichen. Die vorliegende Arbeit kam zu folgenden Ergebnissen: Die neuronale Repräsentation linguistischer Bedeutung wird im Gehirn in Form von cell assemblies dargestellt, welche über semantische Hubs und modalitätsspezifische Regionen verteilt sind. Diese entstehen spontan durch die Interaktion einer Reihe von biologischen Mechanismen, die innerhalb spezifischer neuroanatomischer Strukturen wirken. Das Zusammenwirken dieser biologischen Prinzipien bietet zusätzlich eine Erklärung für jene Faktoren, die für die massiven neuronalen Veränderungen in der sprachlichen und semantischen Netzwerke bei Blindheit verantwortlich sind. Die in dieser Dissertation dokumentierten Studien sind ein erster Schritt in Richtung eines besseren Verständnisses der sprachlichen und semantischen Informationsverarbeitung bei sehenden und blinden Menschen, basierend auf einer Übersetzung der biologischen Prinzipien der menschlichen Kognition in präzise mathematische neuronale Netzwerke des menschlichen Gehirns

A neuronal model of the language cortex

We modelled language-learning processes in a brain-inspired model of the language cortex. The network consisted of neuron-like elements (graded-response units) and mimicked the neuroanatomical areas in the perisylvian language cortex and the intrinsic and mutual connections within and between them. Speaking words creates correlated activity in motor and auditory cortical systems. This correlated activity might play an important role in setting up word representations at the neuronal level [D.B. Fry, The development of the phonological system in the normal and deaf child, in: F. Smith, F.A. Miller, (Eds.), The genesis of language MIT Press, Cambridge, MA, 1966, pp. 187–206; F. Pulvermüller, Words in the Brain’s Language, Behav. Brain Sci. 22 (1999) 253–279]. We simulated this language-learning process and used the network to simulate neurophysiological brain responses to words and meaningless ‘‘pseudowords’’ as they have been documented using EEG and MEG