Towards a complete multiple-mechanism account of predictive language processing [Commentary on Pickering & Garrod]

Although we agree with Pickering & Garrod (P&G) that prediction-by-simulation and prediction-by-association are important mechanisms of anticipatory language processing, this commentary suggests that they: (1) overlook other potential mechanisms that might underlie prediction in language processing, (2) overestimate the importance of prediction-by-association in early childhood, and (3) underestimate the complexity and significance of several factors that might mediate prediction during language processing

An integrated theory of language production and comprehension

Currently, production and comprehension are regarded as quite distinct in accounts of language processing. In rejecting this dichotomy, we instead assert that producing and understanding are interwoven, and that this interweaving is what enables people to predict themselves and each other. We start by noting that production and comprehension are forms of action and action perception. We then consider the evidence for interweaving in action, action perception, and joint action, and explain such evidence in terms of prediction. Specifically, we assume that actors construct forward models of their actions before they execute those actions, and that perceivers of others' actions covertly imitate those actions, then construct forward models of those actions. We use these accounts of action, action perception, and joint action to develop accounts of production, comprehension, and interactive language. Importantly, they incorporate well-defined levels of linguistic representation (such as semantics, syntax, and phonology). We show (a) how speakers and comprehenders use covert imitation and forward modeling to make predictions at these levels of representation, (b) how they interweave production and comprehension processes, and (c) how they use these predictions to monitor the upcoming utterances. We show how these accounts explain a range of behavioral and neuroscientific data on language processing and discuss some of the implications of our proposal

TOWARDS THE GROUNDING OF ABSTRACT CATEGORIES IN COGNITIVE ROBOTS

The grounding of language in humanoid robots is a fundamental problem, especially in social scenarios which involve the interaction of robots with human beings. Indeed, natural language represents the most natural interface for humans to interact and exchange information about concrete entities like KNIFE, HAMMER and abstract concepts such as MAKE, USE. This research domain is very important not only for the advances that it can produce in the design of human-robot communication systems, but also for the implication that it can have on cognitive science. Abstract words are used in daily conversations among people to describe events and situations that occur in the environment. Many scholars have suggested that the distinction between concrete and abstract words is a continuum according to which all entities can be varied in their level of abstractness. The work presented herein aimed to ground abstract concepts, similarly to concrete ones, in perception and action systems. This permitted to investigate how different behavioural and cognitive capabilities can be integrated in a humanoid robot in order to bootstrap the development of higher-order skills such as the acquisition of abstract words. To this end, three neuro-robotics models were implemented. The first neuro-robotics experiment consisted in training a humanoid robot to perform a set of motor primitives (e.g. PUSH, PULL, etc.) that hierarchically combined led to the acquisition of higher-order words (e.g. ACCEPT, REJECT). The implementation of this model, based on a feed-forward artificial neural networks, permitted the assessment of the training methodology adopted for the grounding of language in humanoid robots. In the second experiment, the architecture used for carrying out the first study was reimplemented employing recurrent artificial neural networks that enabled the temporal specification of the action primitives to be executed by the robot. This permitted to increase the combinations of actions that can be taught to the robot for the generation of more complex movements. For the third experiment, a model based on recurrent neural networks that integrated multi-modal inputs (i.e. language, vision and proprioception) was implemented for the grounding of abstract action words (e.g. USE, MAKE). Abstract representations of actions ("one-hot" encoding) used in the other two experiments, were replaced with the joints values recorded from the iCub robot sensors. Experimental results showed that motor primitives have different activation patterns according to the action's sequence in which they are embedded. Furthermore, the performed simulations suggested that the acquisition of concepts related to abstract action words requires the reactivation of similar internal representations activated during the acquisition of the basic concepts, directly grounded in perceptual and sensorimotor knowledge, contained in the hierarchical structure of the words used to ground the abstract action words.This study was financed by the EU project RobotDoC (235065) from the Seventh Framework Programme (FP7), Marie Curie Actions Initial Training Network

A multimodal corpus for the evaluation of computational models for (grounded) language acquisition

Gaspers J, Panzner M, Lemme A, Cimiano P, Rohlfing K, Wrede S. A multimodal corpus for the evaluation of computational models for (grounded) language acquisition. In: EACL Workshop on Cognitive Aspects of Computational Language Learning. 2014

Processing Narratives by Means of Action Languages

In this work we design a narrative understanding system Text2ALM that can be used in Question Answering domains. System Text2ALM utilizes an action language ℒℳ to perform inferences on complex interactions of events described in narratives. The methodology that Text2ALM follows in its implementation was originally outlined by Yuliya Lierler, Daniela Inclezan, and Michael Gelfond in 2017 via a manual process, and this work serves as a proof of concept in a large-scale environment. Our system automates the conversion of a narrative to an ℒℳ model containing facts about the narrative. We make use of the VerbNet lexicon that we annotated with interpretable semantics in ℒℳ. Text2ALM also utilizes Text2DRS system developed by Gang Ling at UNO in 2018. These resources are used to produce an ℒℳ program with a system description containing information on the narrative’s entities, events, and their relations, as well as a history of the narrative’s events. The ℒℳ logic is used in tandem with a basic commonsense library of ℒℳ modules to generate a formal structure capturing the narrative’s properties. The CALM system designed by researchers at Texas Tech in 2018 and is used by Text2ALM to process the ℒℳ program. The effectiveness of this approach is measured by the system’s ability to correctly answer questions from the QA bAbI tasks published by Facebook Research in 2015. The Text2ALM system matched or exceeded the performance of state-of-the-art machine learning methods in six of the seven tested tasks