Self-Organizing Grammar Induction Using a Neural Network Model

This paper presents a self-organizing, real-time, hierarchical neural network model of sequential processing, and shows how it can be used to induce recognition codes corresponding to word categories and elementary grammatical structures. The model, first introduced in Mannes (1992), learns to recognize, store, and recall sequences of unitized patterns in a stable manner, either using short-term memory alone, or using long-term memory weights. Memory capacity is only limited by the number of nodes provided. Sequences are mapped to unitized patterns, making the model suitable for hierarchical operation. By using multiple modules arranged in a hierarchy and a simple mapping between output of lower levels and the input of higher levels, the induction of codes representing word category and simple phrase structures is an emergent property of the model. Simulation results are reported to illustrate this behavior.National Science Foundation (IRI-9024877

Computational and Robotic Models of Early Language Development: A Review

We review computational and robotics models of early language learning and development. We first explain why and how these models are used to understand better how children learn language. We argue that they provide concrete theories of language learning as a complex dynamic system, complementing traditional methods in psychology and linguistics. We review different modeling formalisms, grounded in techniques from machine learning and artificial intelligence such as Bayesian and neural network approaches. We then discuss their role in understanding several key mechanisms of language development: cross-situational statistical learning, embodiment, situated social interaction, intrinsically motivated learning, and cultural evolution. We conclude by discussing future challenges for research, including modeling of large-scale empirical data about language acquisition in real-world environments. Keywords: Early language learning, Computational and robotic models, machine learning, development, embodiment, social interaction, intrinsic motivation, self-organization, dynamical systems, complexity.Comment: to appear in International Handbook on Language Development, ed. J. Horst and J. von Koss Torkildsen, Routledg

Word Learning in 6-16 Month Old Infants

Understanding words requires infants to not only isolate words from the speech around them and delineate concepts from their world experience, but also to establish which words signify which concepts, in all and only the right set of circumstances. Previous research places the onset of this ability around infants\u27 first birthdays, at which point they have begun to solidify their native language phonology, and have learned a good deal about categories, objects, and people. In this dissertation, I present research that alters this accepted timeline. In Study 1, I find that by 6 months of age, infants demonstrate understanding of around a dozen words for foods and body parts. Around 13-14 months of age, performance increases significantly. In Study 2, I find that for a set of early non-nouns, e.g. `uh-oh\u27 and `eat\u27, infants do not show understanding until 10 months, but again show a big comprehension boost around 13-14 months. I discuss possible reasons for the onset of noun-comprehension at 6 months, the relative delay in non-noun comprehension, and the performance boost for both word-types around 13-14 months. In Study 3, I replicate and extend Study 1\u27s findings, showing that around 6 months infants also understand food and body-part words when these words are spoken by a new person, but conversely, by 12 months, show poor word comprehension if a single vowel in the word is changed, even when the speaker is highly familiar. Taken together, these results suggest that word learning begins before infants have fully solidified their native language phonology, that certain generalizations about words are available to infants at the outset of word comprehension, and that infants are able to learn words for complex object and event categories before their first birthday. Implications for language acquisition and cognitive development more broadly are discussed

Segmentation and UR Acquisition with UR Constraints

This paper presents a model that treats segmentation and underlying representation acquisition as parallel, interacting processes. A probability distribution over mappings from underlying to surface forms is defined us- ing a Maximum Entropy grammar which weights a set of underlying representation constraints (URCs) (Apoussidou, 2007; Pater et al., 2012). URCs are induced from observed surface strings and used to generate candidates. Structural ambiguity arising from the com- parison of segmented outputs to unsegmented surface strings is handled with Expectation Maximization (Dempster et al., 1977; Jarosz, 2013). The model successfully learns a simple voicing assimilation rule and segmentation via correspondences between surface phones and input meanings. The trained grammar is also able to segment novel forms affixed with familiar morphemes

Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media

Recognizing named entities in a document is a key task in many NLP applications. Although current state-of-the-art approaches to this task reach a high performance on clean text (e.g. newswire genres), those algorithms dramatically degrade when they are moved to noisy environments such as social media domains. We present two systems that address the challenges of processing social media data using character-level phonetics and phonology, word embeddings, and Part-of-Speech tags as features. The first model is a multitask end-to-end Bidirectional Long Short-Term Memory (BLSTM)-Conditional Random Field (CRF) network whose output layer contains two CRF classifiers. The second model uses a multitask BLSTM network as feature extractor that transfers the learning to a CRF classifier for the final prediction. Our systems outperform the current F1 scores of the state of the art on the Workshop on Noisy User-generated Text 2017 dataset by 2.45% and 3.69%, establishing a more suitable approach for social media environments.Comment: NAACL 201

Early word learning through communicative inference

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Brain and Cognitive Sciences, 2010.Cataloged from PDF version of thesis.Includes bibliographical references (p. 109-122).How do children learn their first words? Do they do it by gradually accumulating information about the co-occurrence of words and their referents over time, or are words learned via quick social inferences linking what speakers are looking at, pointing to, and talking about? Both of these conceptions of early word learning are supported by empirical data. This thesis presents a computational and theoretical framework for unifying these two different ideas by suggesting that early word learning can best be described as a process of joint inferences about speakers' referential intentions and the meanings of words. Chapter 1 describes previous empirical and computational research on "statistical learning"--the ability of learners to use distributional patterns in their language input to learn about the elements and structure of language-and argues that capturing this abifity requires models of learning that describe inferences over structured representations, not just simple statistics. Chapter 2 argues that social signals of speakers' intentions, even eye-gaze and pointing, are at best noisy markers of reference and that in order to take advantage of these signals fully, learners must integrate information across time. Chapter 3 describes the kinds of inferences that learners can make by assuming that speakers are informative with respect to their intended meaning, introducing and testing a formalization of how Grice's pragmatic maxims can be used for word learning. Chapter 4 presents a model of cross-situational intentional word learning that both learns words and infers speakers' referential intentions from labeled corpus data.by Michael C. Frank.Ph.D