Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

We propose a computational model of situated language comprehension based on the Indexical Hypothesis that generates meaning representations by translating amodal linguistic symbols to modal representations of beliefs, knowledge, and experience external to the linguistic system. This Indexical Model incorporates multiple information sources, including perceptions, domain knowledge, and short-term and long-term experiences during comprehension. We show that exploiting diverse information sources can alleviate ambiguities that arise from contextual use of underspecific referring expressions and unexpressed argument alternations of verbs. The model is being used to support linguistic interactions in Rosie, an agent implemented in Soar that learns from instruction.Comment: Advances in Cognitive Systems 3 (2014

Hungarian Gyerekestül versus Gyerekkel (‘with [the] kid’)

The paper analyzes the various uses of the Hungarian -stUl (‘together with’, ‘along with’) sociative (associative) suffix (later in the paper referred to simply as “sociative”), as in the example gyerekestül. As opposed to its comitative-instrumental suffix -vAl (‘with’), the - stUl suffix cannot express instrumentality. The paper aims to demonstrate the difference in use between the comitative-instrumental -vAl and the -stUl suffix in contemporary Hungarian, and to illuminate the historical emergence of the suffix as well as its grammatical status. It is argued on the basis of Antal (1960) and Kiefer (2003) that -stUl cannot be analyzed as an inflectional case suffix (such as the -vAl suffix, or -ed, -ing, or the plural in English), but should rather be categorized as a derivational suffix (such as English dis-, re-, in-, -ance, - able, -ish, -like, etc.). The paper also tries to shed light on the hypothetical cognitive psychological distinction between the comitative and the sociative. It is suggested that the sociative is based on the amalgam image schema which is derived from the LINK schema of the comitative. The ironical reading of the sociative is an implicature in the sense of Grice (1989) and Sperber and Wilson (1987). Psycholinguistic experimentation is proposed to follow up on the mental representation of the sociative

To dash or to dawdle: verb-associated speed of motion influences eye movements during spoken sentence comprehension

In describing motion events verbs of manner provide information about the speed of agents or objects in those events. We used eye tracking to investigate how inferences about this verb-associated speed of motion would influence the time course of attention to a visual scene that matched an event described in language. Eye movements were recorded as participants heard spoken sentences with verbs that implied a fast (“dash”) or slow (“dawdle”) movement of an agent towards a goal. These sentences were heard whilst participants concurrently looked at scenes depicting the agent and a path which led to the goal object. Our results indicate a mapping of events onto the visual scene consistent with participants mentally simulating the movement of the agent along the path towards the goal: when the verb implies a slow manner of motion, participants look more often and longer along the path to the goal; when the verb implies a fast manner of motion, participants tend to look earlier at the goal and less on the path. These results reveal that event comprehension in the presence of a visual world involves establishing and dynamically updating the locations of entities in response to linguistic descriptions of events

An embodied approach to language comprehension in probable Alzheimer’s Disease: could perceptuo-motor processing be a key to better understanding?

One of the central tenets of the embodied theory of language comprehension is that the process of understanding prompts the same perceptuo-motor activity involved in actual perception and action. This activity is a component of comprehension that is not memory–dependent and is hypothesized to be intact in Alzheimer’s Disease (AD). Each article in this thesis is aimed at answering the question whether individuals with probable AD, healthy older adults and younger adults show differences in their performance on tests where perceptual and motoric priming take place during language comprehension. The second question each article asks is whether language comprehension in AD can be facilitated by the specific use of this perceptual and motoric priming. Article I examines whether the way individuals with pAD represent verbs spatially matches the way healthy older and younger adults do, and how stable these representations are. It also explores in what way spatial representations may relate to verb comprehension, more specifically, whether representations matching the norms translate into a better quality of verb comprehension. Article II tests the interaction between the verbs’ spatial representations taking place during comprehension and perceptual cues - compatible and incompatible to the representations - in order to investigate whether individuals with pAD show differences in susceptibility to perceptual cues, compared to healthy older and younger participants. The second aim of this article is to explore in what way performance on a word-picture verification task can be affected, with reference to the fact that in previous studies on young participants, both priming and interference have resulted from the interaction of linguistic and perceptual processing. Article III explores the Action Compatibility Effect (ACE) (Glenberg & Kaschak, 2002) with the aim of finding out whether the ACE exists for volunteers with pAD and whether it can facilitate language comprehension. The order of presentation of language and movement is manipulated to establish whether there is a reciprocal relationship between them. This information could be crucial in view of possible applications to individuals with pAD. These articles test, for the first time, the effects of the manipulation of the perceptuo-motor component during language comprehension in individuals with pAD; they are intended as a methodological exploration contributing to a better understanding of the potential of embodiment principles to support language comprehension changes associated with pAD. Embodiment effects need to be studied further with a view to putting them to use in either clinical or real-life applications

Three symbol ungrounding problems: Abstract concepts and the future of embodied cognition

A great deal of research has focused on the question of whether or not concepts are embodied as a rule. Supporters of embodiment have pointed to studies that implicate affective and sensorimotor systems in cognitive tasks, while critics of embodiment have offered nonembodied explanations of these results and pointed to studies that implicate amodal systems. Abstract concepts have tended to be viewed as an important test case in this polemical debate. This essay argues that we need to move beyond a pretheoretical notion of abstraction. Against the background of current research and theory, abstract concepts do not pose a single, unified problem for embodied cognition but, instead, three distinct problems: the problem of generalization, the problem of flexibility, and the problem of disembodiment. Identifying these problems provides a conceptual framework for critically evaluating, and perhaps improving upon, recent theoretical proposals

On staying grounded and avoiding Quixotic dead ends

The 15 articles in this special issue on The Representation of Concepts illustrate the rich variety of theoretical positions and supporting research that characterize the area. Although much agreement exists among contributors, much disagreement exists as well, especially about the roles of grounding and abstraction in conceptual processing. I first review theoretical approaches raised in these articles that I believe are Quixotic dead ends, namely, approaches that are principled and inspired but likely to fail. In the process, I review various theories of amodal symbols, their distortions of grounded theories, and fallacies in the evidence used to support them. Incorporating further contributions across articles, I then sketch a theoretical approach that I believe is likely to be successful, which includes grounding, abstraction, flexibility, explaining classic conceptual phenomena, and making contact with real-world situations. This account further proposes that (1) a key element of grounding is neural reuse, (2) abstraction takes the forms of multimodal compression, distilled abstraction, and distributed linguistic representation (but not amodal symbols), and (3) flexible context-dependent representations are a hallmark of conceptual processing

A neural blackboard architecture of sentence structure

We present a neural architecture for sentence representation. Sentences are represented in terms of word representations as constituents. A word representation consists of a neural assembly distributed over the brain. Sentence representation does not result from associations between neural word assemblies. Instead, word assemblies are embedded in a neural architecture, in which the structural (thematic) relations between words can be represented. Arbitrary thematic relations between arguments and verbs can be represented. Arguments can consist of nouns and phrases, as in sentences with relative clauses. A number of sentences can be stored simultaneously in this architecture. We simulate how probe questions about thematic relations can be answered. We discuss how differences in sentence complexity, such as the difference between subject-extracted versus object-extracted relative clauses and the difference between right-branching versus center-embedded structures, can be related to the underlying neural dynamics of the model. Finally, we illustrate how memory capacity for sentence representation can be related to the nature of reverberating neural activity, which is used to store information temporarily in this architecture

Simulating Speed in Language: Contributions from vision, audition and action

Embodied theories propose that understanding meaning in language requires the mental simulation of entities being referred to. These mental simulations would make use of the same modality-specific systems involved in perceiving and acting upon such entities in the world, grounding language in the real world. However, embodied theories are currently underspecified in terms of how much information from an event is contained in mental simulations, and what features of experience are included. The thesis addresses comprehension of language that describes speed of events. Investigating speed allows embodied theories to be extended to a more complex feature of events. Further, speed is a fine-grained feature and thus testing an embodied theory of speed will reveal whether or not mental simulations include the fine details of real-world experience. Within the thesis four main methods of investigation were used, assessing simulation of speed with different types of speed language under different conditions: behavioural testing combining speed in language with speed in perception and action, eye-tracking investigating whether eye-movements to a visual scene are affected by speed in sentences, a psychophysics paradigm assessing whether speed in language affects visual perception processes, and finally, as a crucial test of embodiment, whether or not Parkinson’s patients, who have difficulty moving speedily, also have problems with comprehension of speed language. The main findings of the thesis are that: (1) speed, a fine-grained and abstract dimension, is simulated during comprehension, (2) simulations are dynamic and context-dependent, and (3) simulations of speed are specific to biological motion and can encode specific effectors used in an action. These results help to specify current embodied theories in terms of what the nature of simulations are and what factors they are sensitive to, in addition to broadly providing support for the sharing of cognitive/neural processes between language, action and perception

Head up, foot down : object words orient attention to the objects' typical location

Many objects typically occur in particular locations, and object words encode these spatial associations. We tested whether such object words (e.g., head, foot) orient attention toward the location where the denoted object typically occurs (i.e., up, down). Because object words elicit perceptual simulations of the denoted objects (i.e., the representations acquired during actual perception are reactivated), we predicted that an object word would interfere with identification of an unrelated visual target subsequently presented in the object's typical location. Consistent with this prediction, three experiments demonstrated that words denoting objects that typically occur high in the visual field hindered identification of targets appearing at the top of the display, whereas words denoting low objects hindered target identification at the bottom of the display. Thus, object words oriented attention to and activated perceptual simulations in the objects' typical locations. These results shed new light on how language affects perception