Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

The role of feedback and instruction on the cross-situational learning of vocabulary and morphosyntax:Mixed effects models reveal local and global effects on acquisition

First language acquisition is implicit, in that explicit information about the language structure to be learned is not provided to children. Instead, they must acquire both vocabulary and grammar incrementally, by generalizing across multiple situations that eventually enable links between words in utterances and referents in the environment to be established. However, this raises a problem of how vocabulary can be acquired without first knowing the role of the word within the syntax of a sentence. It also raises practical issues about the extent to which different instructional conditions – about grammar in advance of learning or feedback about correct decisions during learning – might influence second language acquisition of implicitly experienced information about the language. In an artificial language learning study, we studied participants learning language from inductive exposure, but under different instructional conditions. Language learners were exposed to complex utterances and complex scenes and had to determine the meaning and the grammar of the language from these co-occurrences with environmental scenes. We found that learning was boosted by explicit feedback, but not by explicit instruction about the grammar of the language, compared to an implicit learning condition. However, the effect of feedback was not general across all aspects of the language. Feedback improved vocabulary, but did not affect syntax learning. We further investigated the local, contextual effects on learning, and found that previous knowledge of vocabulary within an utterance improved learning but that this was driven only by certain grammatical categories in the language. The results have implications for theories of second language learning informed by our understanding of first language acquisition as well as practical implications for learning instruction and optimal, contingent adjustment of learners’ environment during their learning

Acquiring and processing verb argument structure : distributional learning in a miniature language

Adult knowledge of a language involves correctly balancing lexically-based and more language-general patterns. For example, verb argument structures may sometimes readily generalize to new verbs, yet with particular verbs may resist generalization. From the perspective of acquisition, this creates significant learnability problems, with some researchers claiming a crucial role for verb semantics in the determination of when generalization may and may not occur. Similarly, there has been debate regarding how verb-specific and more generalized constraints interact in sentence processing and on the role of semantics in this process. The current work explores these issues using artificial language learning. In three experiments using languages without semantic cues to verb distribution, we demonstrate that learners can acquire both verb-specific and verb-general patterns, based on distributional information in the linguistic input regarding each of the verbs as well as across the language as a whole. As with natural languages, these factors are shown to affect production, judgments and real-time processing. We demonstrate that learners apply a rational procedure in determining their usage of these different input statistics and conclude by suggesting that a Bayesian perspective on statistical learning may be an appropriate framework for capturing our findings

The interactive nature of second-language word learning in non-instructed environments

Gaining the command of a second language is a difficult task for an adult. Understanding and learning novel words is challenging, particularly in non-instructed situations: Words are often parts of complex linguistic contexts and potential referents are embedded in rich visual scenes. To overcome this challenge learners can potentially exploit the richness of their multi-modal environment through a range of different word-learning mechanisms and based on automatic sentence-processing mechanisms.Despite numerous investigations of word learning by researchers from a range of disciplines, few have examined the interplay of different learning and processing mechanisms. Such an approach, however, potentially both oversimplifies and overcomplicates the scenario. The main goal of this thesis is to study word learning in adults in a more situated and interactive manner, considering different mechanisms, processes, and information sources in parallel. This enterprise is driven by the motivation to contribute to a more complete theory of second-language word learning, to bridge research traditions, and to draw implications for the development of practical learning applications. In particular, we examined the interaction of the two important and visually situated word-learning mechanisms, cross-situational word learning (CSWL, Yu & Smith, 2007) and sentence-level constraint learning (SLCL). CSWL is a bottom-up, associative manner of word learning: people make connections between visual objects and spoken words by tracking their co-occurrence frequencies. SLCL, on the contrary, is a top-down strategy, which is based on making inferences about likely word meanings given a linguistic context (and word knowledge). SLCL in this thesis refers to inferring the meanings of object nouns (e.g., the corn) based on restrictive verbs (e.g., eat), a visual context, and people\u27s world knowledge. Our studies exploit a novel experimental paradigm which integrates teaching German adults a semi-natural miniature language in a step-wise procedure. Participants were familiarized with a set of verbs (e.g., bermamema "to eat\u27;) before they were exposed to noun-learning trials. These trials consisted of pairs of visual scenes and auditory transitive sentences, in which novel nouns were embedded (e.g. Si laki bermamema si sonis.}, "The man will eat the corn\u27;). Finally, participants performed a forced-choice vocabulary test (with confidence ratings). Eye-movements were recorded during learning and testing. In Experiment 1, we evaluated the use of CSWL and SLCL in this naturalized situation. We found that participants applied both mechanisms in a complementary manner to learn the vocabulary. In Experiment 2, we introduced a second word order (OVS), which is characterized by a verb which follows rather than precedes the syntactic object (that denotes a visual object). Results are in accordance with the hypothesis that verb-based prediction of referents has a positive influence on noun learning. In Experiment 3, we re-addressed the question whether SLCL boosts noun learning and examined the interaction of CSWL and SLCL by manipulating the degree of referential uncertainty. Results provide evidence for the hypotheses that, firstly, SLCL boosts noun learning and secondly, SLCL and CSWL interact in that they jointly contribute to the identification of noun meanings. Experiment 4 was conducted in order to investigate the interaction of CSWL and SLCL when both mechanisms are in conflict. Learning rates clearly reveal that CSWL and SLCL were similarly influential with regard to learners\u27 decisions in the vocabulary test. The aim of Experiment 5 was to examine the nature of both mechanisms by studying the interaction of CSWL and SLCL when both are independently applicable. Results clearly provide evidence for the hypothesis that SLCL completely blocks learner\u27s sensitivity to smaller difference in co-occurrence frequencies, which characterizes pure CSWL learning. This pattern confirms the hypothesis that while CSWL is a parallel and probabilistic way of learning, SLCL is more deterministic. Results from a vocabulary test one day after learning reveal that learning rates were still clearly above chance.Taken together, our experimental data clearly shows that CSWL and SLCL are powerful mechanisms for word learning in adults in non-instructed environments, which may lead into long-lasting retention. Importantly, these mechanisms interact in multiple ways due to differences in their nature: They can be used in a complementary way, they influence word learning about equally strongly when they are in conflict, and SLCL blocks CSWL when both mechanisms are independently applicable. We conclude that adult word learners employ as many resources in parallel as necessary but ignore the less direct and helpful cue when information is redundant. However, when the relevance of different cues is unclear, they consider all of them.Eine zweite Sprache zu erlernen, ist eine schwierige Aufgabe für Erwachsene. Das Verstehen und Lernen unbekannter Wörter ist mühsam, insbesondere in nicht-instruierten Situationen. Um dies zu überwinden, haben Lerner jedoch die Möglichkeit, die Reichhaltigkeit ihrer multi-modalen Umgebung zu nutzen und Wortlern-Mechanismen anzuwenden. Automatische Sprach-Verarbeitungs-Mechanismen unterstützen deren zügige Integration. Obwohl viele Wissenschaftler aus diversen Bereichen Studien zum Wortlernen durchgeführt haben, gibt es nur sehr wenige Untersuchungen zum Zusammenspiel verschiedener Wortlern- und Verarbeitungs-Mechanismen. Das Hauptziel dieser Arbeit ist es, Wortlernen bei Erwachsenen in einer stärker integrativen und interaktiven Weise zu erforschen. Hierbei sollen verschiedene Informationsquellen sowie Wortlern-Mechanismen und -Prozesse gleichzeitig berücksichtigt werden. Die Motivation für diese Auseinandersetzung ist es zu einer vollständigeren Theorie des Wortlernens in einer Zweitsprache beizutragen, Forschungstraditionen zu verbinden und Rückschlüsse für die Entwicklung praktischer Lern-Anwendungen zu ziehen. Die Interaktion zweier bedeutender und visuell integrierter Wortlern-Mechanismen wurde untersucht cross-situational word learning (CSWL, Yu and Smith, 2007) und sentence-level constraint learning (SLCL). CSWL funktioniert bottom-up: Lerner ziehen Verbindungen zwischen visuellen Objekten und gesprochenen Wörtern, indem sie die Häufigkeiten ihres gemeinsamen Auftretens (Mit-Auftretens) verfolgen. SLCL vollzieht sich im Gegensatz dazu top-down, denn es folgt dem Grundsatz des Inferierens basierend auf dem linguistischen Kontext. SLCL in dieser Arbeit bezieht sich auf das Inferieren von Objekt-Nomina-Bedeutungen (z.B. der Maiskolben) auf Grund restriktiver Verben (z.B. essen), einem visuellen Kontext, und dem Weltwissen des Lerners. Der Kern unserer Studien ist ein neuartiges experimentelles Paradigma, in dessen Rahmen erwachsenen Deutschen Muttersprachlern in einer stufenweisen Prozedur eine semi-natürliche Mini-Sprache gelehrt wird. Partizipanten wurden zunächst mit einer Reihe Verben vertraut gemacht (z.B. bermamema "essen"), bevor ihnen Materialien zum Nomina-Lernen dargeboten wurden. Die experimentellen Items bestanden aus visuellen Szenen, die mit gesprochenen transitiven Sätzen gepaart waren (z.B. Si laki bermamema si sonis. "Der Mann isst den Maiskolben"). Am Ende des Experiments wurde ein selektiver Vokabeltest (mit Konfidenz-Selbst-Wertung) durchgeführt. Die Augenbewegungen wurden aufgezeichnet. In Experiment 1 wurde der Gebrauch von CSWL und SLCL in diesem Paradigma evaluiert. Die Teilnehmer wendeten beide Mechanismen komplementär an. In Experiment 2 wurde eine zweite Wortreihenfolge eingeführt (OVS). Die gewonnen Resultate stimmen mit der Hypothese überein, dass die Verb-basierte Vorhersage von Referenten einen positiven Einfluss auf das Wortlernen. In Experiment 3 gingen wir der Frage nach, ob SLCL Nomina-Lernen verstärkt und wendeten uns der Interaktion von CSWL und SLCL zu. Die Ergebnisse liefern Evidenz dafür, dass SLCL das Erlernen von Nomina verstärkt und, dass CSWL und SLCL gemeinsam zu der Identifizierung von Nomina-Bedeutungen verhelfen. Experiment 4 wurde durchgeführt um die Interaktion von CSWL und SLCL zu erforschen, wenn beide Mechanismen konfligieren. CSWL und SLCL hatten einen gleich starken Einfluss. Das Ziel von Experiment 5 war es, der Beschaffenheit von CSWL und SLCL auf den Grund zu gehen, indem sie als voneinander unabhängig erforscht wurden. Die Ergebnisse belegen, dass SLCL die Sensibilität für kleinere Unterschiede in der Mit-Auftretens-Wahrscheninlichkeit blockiert, durch welche reines CSWL-Lernen gekennzeichnet ist. Dieses Muster bestätigt die Hypothesen, dass CSWL parallel und probabilistisch verläuft, während sich SLCL eher deterministisch vollzieht. Die Resultate von einem Vokabeltest einen Tag nach der Lernprozedur zeigen, dass die Lernraten noch immer gut waren. Zusammengenommen belegen unsere experimentellen Daten eindeutig, dass CSWL und SLCL wirkungsstarke Mechanismen für das Wortlernen bei sind, welche wahrscheinlich langfristige Lernerfolge nach sich ziehen. Auf Grund ihrer unterschiedlichen Beschaffenheit interagieren diese Mechanismen in vielerlei Hinsichten: Sie können komplementär angewendet werden, ihr Einfluss auf das Erlernen eines Wortes ist ungefähr gleich groß, wenn sie konfligieren und SLCL blockiert CSWL in Fällen, wenn beide Mechanismen unabhängig voneinander angewendet werden können. Wir ziehen die Schlussfolgerung, dass erwachsene Wortlerner einer effizienten Strategie folgen: Sie machen von so vielen Ressourcen wie notwendig parallel Gebrauch, ignorieren aber weniger direkte und hilfreiche Hinweise, wenn redundante Informationen zur Verfügung stehen. Ist die Relevanz verschiedener Hinweise jedoch unklar, so berücksichtigen sie jede dieser Informationsquellen

Visual attention-capture cue in depicted scenes fails to modulate online sentence processing

Everyday communication is enriched by the visual environment that listeners concomitantly link to the linguistic input. If and when visual cues are integrated into the mental meaning representation of the communicative setting, is still unclear. In our earlier findings, the integration of linguistic cues (i.e., topic-hood of a discourse referent) reduced discourse updating costs of the mental representation as indicated by reduced sentence-initial processing costs of the non-canonical word order in German. In the present study we tried to replicate our earlier findings by replacing the linguistic cue by a visual attention-capture cue presented below the threshold of perception in order to direct participant’s attention to a depicted referent. While this type of cue has previously been shown to modulate word order preferences in sentence production, we found no effects on sentence comprehension. We discuss possible theory-based reasons for the null effect of the implicit visual cue as well as methodological caveats and issues that should be considered in future research on multimodal meaning integration

Prediction error boosts retention of novel words in adults but not in children

How do we update our linguistic knowledge? In seven experiments, we asked whether error-driven learning can explain under what circumstances adults and children are more likely to store and retain a new word meaning. Participants were exposed to novel object labels in the context of more or less constraining sentences or visual contexts. Both two-to-four-year-olds (Mage = 38 months) and adults were strongly affected by expectations based on sentence constraint when choosing the referent of a new label. In addition, adults formed stronger memory traces for novel words that violated a stronger prior expectation. However, preschoolers' memory was unaffected by the strength of their prior expectations. We conclude that the encoding of new word-object associations in memory is affected by prediction error in adults, but not in preschoolers

The long road of statistical learning research: past, present and future

Published 21 November 2016 http://rstb.royalsocietypublishing.org/content/372/1711/20160047http://rstb.royalsocietypublishing.org/content/372/1711/20160047This paper was supported by the Israel Science Foundation (grant no. 217/14 awarded to R.F.), by the National Institute of Child Health and Human Development (RO1 HD 067364 awarded to Ken Pugh and R.F., PO1-HD 01994 awarded to Haskins Laboratories) and by the European Research Council (project ERC-ADG- 692502 awarded to R.F.)

Brain Signatures of Embodied Semantics and Language: A Consensus Paper

According to embodied theories (including embodied, embedded, extended, enacted, situated, and grounded approaches to cognition), language representation is intrinsically linked to our interactions with the world around us, which is reflected in specific brain signatures during language processing and learning. Moving on from the original rivalry of embodied vs. amodal theories, this consensus paper addresses a series of carefully selected questions that aim at determining when and how rather than whether motor and perceptual processes are involved in language processes. We cover a wide range of research areas, from the neurophysiological signatures of embodied semantics, e.g., event-related potentials and fields as well as neural oscillations, to semantic processing and semantic priming effects on concrete and abstract words, to first and second language learning and, finally, the use of virtual reality for examining embodied semantics. Our common aim is to better understand the role of motor and perceptual processes in language representation as indexed by language comprehension and learning. We come to the consensus that, based on seminal research conducted in the field, future directions now call for enhancing the external validity of findings by acknowledging the multimodality, multidimensionality, flexibility and idiosyncrasy of embodied and situated language and semantic processes

Cross-situational learning from ambiguous egocentric input is a continuous process: Evidence using the human simulation paradigm

Recent laboratory experiments have shown that both infant and adult learners can acquire word-referent mappings using cross-situational statistics. The vast majority of the work on this topic has used unfamiliar objects presented on neutral backgrounds as the visual contexts for word learning. However, these laboratory contexts are much different than the real-world contexts in which learning occurs. Thus, the feasibility of generalizing cross-situational learning beyond the laboratory is in question. Adapting the Human Simulation Paradigm, we conducted a series of experiments examining cross-situational learning from children's egocentric videos captured during naturalistic play. Focusing on individually ambiguous naming moments that naturally occur during toy play, we asked how statistical learning unfolds in real time through accumulating cross-situational statistics in naturalistic contexts. We found that even when learning situations were individually ambiguous, learners' performance gradually improved over time. This improvement was driven in part by learners' use of partial knowledge acquired from previous learning situations, even when they had not yet discovered correct word-object mappings. These results suggest that word learning is a continuous process by means of real-time information integration

'Being the Teacher': identity and classroom conversation

Recent debate on the standard classroom Initiation–Response–Follow-up pattern has focused particular attention on the final move and the contribution it can make to productive interaction in teacher-fronted situations. This paper suggests that current research in this area has tended to exaggerate the pedagogic impact of changes based on specifiable discourse moves, proposing instead an approach to analysis which takes account of the dynamic nature of identity construction and its relationship to the development of ongoing talk. It challenges the view that the concept of classroom conversation is inherently contradictory and, drawing on the work of Zimmerman (1998) related to the broader field of Membership Categorization Analysis, demonstrates how shifts in the orientation to different aspects of identity produce distinctively different interactional patterns in teacher-fronted talk. Using Zimmerman's distinction between discourse, situated and transportable identities in talk, extracts from classroom exchanges from different educational contexts are analysed as the basis for claiming that conversation involving teacher and students in the classroom is indeed possible. The paper concludes with a discussion of the pedagogical implications of this