Learning language through pictures

We propose Imaginet, a model of learning visually grounded representations of language from coupled textual and visual input. The model consists of two Gated Recurrent Unit networks with shared word embeddings, and uses a multi-task objective by receiving a textual description of a scene and trying to concurrently predict its visual representation and the next word in the sentence. Mimicking an important aspect of human language learning, it acquires meaning representations for individual words from descriptions of visual scenes. Moreover, it learns to effectively use sequential structure in semantic interpretation of multi-word phrases.Comment: To appear at ACL 201

Computational and Robotic Models of Early Language Development: A Review

We review computational and robotics models of early language learning and development. We first explain why and how these models are used to understand better how children learn language. We argue that they provide concrete theories of language learning as a complex dynamic system, complementing traditional methods in psychology and linguistics. We review different modeling formalisms, grounded in techniques from machine learning and artificial intelligence such as Bayesian and neural network approaches. We then discuss their role in understanding several key mechanisms of language development: cross-situational statistical learning, embodiment, situated social interaction, intrinsically motivated learning, and cultural evolution. We conclude by discussing future challenges for research, including modeling of large-scale empirical data about language acquisition in real-world environments. Keywords: Early language learning, Computational and robotic models, machine learning, development, embodiment, social interaction, intrinsic motivation, self-organization, dynamical systems, complexity.Comment: to appear in International Handbook on Language Development, ed. J. Horst and J. von Koss Torkildsen, Routledg

Developmental Stages of Perception and Language Acquisition in a Perceptually Grounded Robot

The objective of this research is to develop a system for language learning based on a minimum of pre-wired language-specific functionality, that is compatible with observations of perceptual and language capabilities in the human developmental trajectory. In the proposed system, meaning (in terms of descriptions of events and spatial relations) is extracted from video images based on detection of position, motion, physical contact and their parameters. Mapping of sentence form to meaning is performed by learning grammatical constructions that are retrieved from a construction inventory based on the constellation of closed class items uniquely identifying the target sentence structure. The resulting system displays robust acquisition behavior that reproduces certain observations from developmental studies, with very modest “innate” language specificity

Imaging, Keyboarding, and Posting Identities: Young People and New Media Technologies

Part of the Volume on Youth, Identity, and Digital Media Clicking, posting, and text messaging their way through a shifting digital landscape, young people are bending and blending genres, incorporating old ideas, activities, and images into new bricolages, changing the face, if not the substance, of social interaction and altering how they see themselves and each other. From data collected in Britain, Canada, and South Africa, we have selected cases that involve a range of technologies and contexts, from adult-mediated activities in schools and community centers to spontaneous media production done in private at home. Whether it be postings on websites, improvisations in video production, or the incorporation of objects in a multi-media presentation, these cases illustrate that, like digital cultural production, identity processes are multifaceted and in flux, constructed and deconstructed through a process of bricolage that we label as "identities-in-action." Analysis of the cases reveals certain shared features of digital production that contribute to identities-in-action: the "constructedness" of production, the collective and social aspects of individual productions, the neglected but crucial element of embodiment, the reflexivity and negotiation involved in producing and consuming one's own images, the creativity in media convergence, and the value of constructivist models of learning