Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

We propose a computational model of situated language comprehension based on the Indexical Hypothesis that generates meaning representations by translating amodal linguistic symbols to modal representations of beliefs, knowledge, and experience external to the linguistic system. This Indexical Model incorporates multiple information sources, including perceptions, domain knowledge, and short-term and long-term experiences during comprehension. We show that exploiting diverse information sources can alleviate ambiguities that arise from contextual use of underspecific referring expressions and unexpressed argument alternations of verbs. The model is being used to support linguistic interactions in Rosie, an agent implemented in Soar that learns from instruction.Comment: Advances in Cognitive Systems 3 (2014

From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction.

Intelligent collaborative agents are becoming common in the human society. From virtual assistants such as Siri and Google Now to assistive robots, they contribute to human activities in a variety of ways. As they become more pervasive, the challenge of customizing them to a variety of environments and tasks becomes critical. It is infeasible for engineers to program them for each individual use. Our research aims at building interactive robots and agents that adapt to new environments autonomously by interacting with human users using natural modalities. This dissertation studies the problem of learning novel tasks from human-agent dialog. We propose a novel approach for interactive task learning, situated interactive instruction (SII), and investigate approaches to three computational challenges that arise in designing SII agents: situated comprehension, mixed-initiative interaction, and interactive task learning. We propose a novel mixed-modality grounded representation for task verbs which encompasses their lexical, semantic, and task-oriented aspects. This representation is useful in situated comprehension and can be learned through human-agent interactions. We introduce the Indexical Model of comprehension that can exploit extra-linguistic contexts for resolving semantic ambiguities in situated comprehension of task commands. The Indexical model is integrated with a mixed-initiative interaction model that facilitates a flexible task-oriented human-agent dialog. This dialog serves as the basis of interactive task learning. We propose an interactive variation of explanation-based learning that can acquire the proposed representation. We demonstrate that our learning paradigm is efficient, can transfer knowledge between structurally similar tasks, integrates agent-driven exploration with instructional learning, and can acquire several tasks. The methods proposed in this thesis are integrated in Rosie - a generally instructable agent developed in the Soar cognitive architecture and embodied on a table-top robot.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/111573/1/shiwali_1.pd

Mind and Matter

Do brains create material reality in thinking processes or is it the other way around, with things shaping the mind? Where is the location of meaning-making? How do neural networks become established by means of multimodal pattern replications, and how are they involved in conceptualization? How are resonance textures within cellular entities extended in the body and the mind by means of mirroring processes? In which ways do they correlate to consciousness and self-consciousness? Is it possible to explain out-of-awareness unconscious processes? What holds together the relationship between experiential reality, bodily processes like memory, reason, or imagination, and sign-systems and simulation structures like metaphor and metonymy visible in human language? This volume attempts to answer some of these questions

Keeping Track of Time:The Role of Spatial and Embodied Cognition in the Comprehension of Nonlinear Storyworlds

ABSTRACT: What allows an audience to make sense of stories with complex nonlinear time structures that are radically different from everyday experience? To address this question, we distinguish between two types of narrative nonlinearity: nonlinear storytelling (a non-chronological presentation of events in the narration) and nonlinear storyworlds (nonlinearity as a feature of the narrated world, for instance by way of time-travel or temporal loops). With most scholarly attention focusing on the former, here we focus on the latter, as the question of what allows audiences to make sense of strange and impossible storyworld temporalities has remained somewhat overlooked. Drawing on the available research on text comprehension, we first discuss how both strategies of nonlinearity affect narrative comprehension differently. We then ask what cognitive abilities allow spectators to engage with nonlinear storyworlds. Drawing on insights from conceptual metaphor theory and mental timeline theory, we propose that the comprehension of nonlinear storyworlds is facilitated by the cognitive ability to mentally represent time in terms of space. By metaphorically blending spatial and embodied concepts into narrative timelines, strategies of spatial mental representation allow spectators to conceive and comprehend various forms of phenomenologically non-experienceable time structures—a hypothesis we seek to demonstrate through several cases of nonlinear storyworlds from contemporary complex cinema

Using Gestures and Body Movements for Thinking and Learning

Gestures have been found to be helpful to people in many cognitive and daily activities, such as speaking, counting, learning, and problem solving. However, different gestures benefit people to different degrees, and people use gestures in different ways to assist thinking and learning. From an embodied cognition perspective, gesture is seen as a simulated action. Therefore, to further understand the mechanisms of gesture’s effects on thinking will directly help us harness embodied cognition theories to guide teaching and learning. In the literature, it is widely known that gesture not only reflects thinking, but also actively promotes thinking and learning. However, the mechanisms that account for gesture’s effects on cognition remained obscure to us. To better understand how different types of gestures benefit thinking and learning, Study 1 was conducted with 31 participants to investigate how teaching big (n=15) and small gestures (n=16) as a problem solving strategy influenced the actual gesture use and performance. The results suggested that the small gesture might possibly be a more effective gesture, because people who were taught small and used small gestures had the highest accuracy percentage on the primary task. However, using the small gesture did not significantly lower cognitive load compared to using the big gesture. Based on these findings, Study 2 was conducted with 100 adults to further investigate how teaching different types of gestures influenced learners’ gesture use, performance, learning, and cognitive load. In this study, the participants were randomly assigned to three groups, where they were taught to solve a molecular structure problem using small (n=25), big (n=50), or no gestures (n=25). Then they were left in a quiet room to solve 15 molecule questions independently. Their answers and time spent on each question were recorded. A dual-task paradigm was used as an objective measure of cognitive load, and a NASA Questionnaire was used as a subjective measure of cognitive load. At the end, participants were asked to answer some transfer questions. Throughout the study, all participants’ gestures and body movements were recorded by two cameras. The findings from the two studies suggested that teaching different types of gestures had some influence on people’s gesture use, performance, learning, and cognitive load. Specifically, small gestures taught as a problem-solving strategy were adopted more easily and more effectively used than big gestures and body movements. Questions that were answered through small gestures seemed to have a slightly higher accuracy percentage, but were not necessarily related to lowered cognitive load. The study also found that when people were taught gesture as a problem solving strategy and then asked to use it, they took some time at the very beginning to try and practice, and then gradually transitioned to using no gestures. In both studies, their thinking time, gesture time, gesturing density decreased gradually, without sacrificing accuracy. These findings contributed to both embodied cognition theories and gesture literature, and also shed light on instructional design in an educational setting

Shifting the Perspectival Landscape: Methods for Encoding, Identifying, and Selecting Perspectives

This dissertation explores the semantics and pragmatics of perspectival expressions. Perspective, or point-of-view, encompasses an individual’s thoughts, perceptions, and location. Many expressions in natural language have components of their meanings that shift depending on whose perspective they are evaluated against. In this dissertation, I explore two sets of questions relating to perspective sensitivity. The first set of questions relate to how perspective is encoded in the semantics of perspectival expressions. The second set of questions relate to how conversation participants treat perspectival expressions: the speaker’s selection of a perspective and the listener’s identification of the speaker’s perspective. In Part I, I explore the landscape of perspectival expressions by exploring different semantic mechanisms for encoding the perspective holder. In Chapter 2, I introduce key properties of perspectival expressions through a discussion of one canonical perspectival expression: the motion verb come. In Chapter 3, I discuss the various ways of encoding the perspective holder in the semantics of perspectival expressions. I contrast the predictions of these approaches and lay out a set of diagnostics to guide the analysis of perspectival expressions. I present two case studies using this set of diagnostics. In Chapter 3, I probe the semantics of the well-studied perspectival expression come in American English, and argue in favor of a perspective-anaphoric analysis. In Chapter 4, I focus on an expression that has not previously been recognized as perspectival, the temporal adverbial tomorrow. Through a series of experimental studies, I make the case that tomorrow is perspective-sensitive for some American English speakers, and narrow the hypothesis space for a perspectival account of tomorrow. I sketch a perspective-anaphoric semantics for tomorrow, while leaving open the possibility of a logophoric analysis. I conclude Part I with a discussion of how perspectival expressions fit into the broader landscape of context sensitivity. In Part II, I turn to a fresh set of questions about perspective: how do conversation participants select and identify perspectives? In Chapter 6, I discuss previous models of perspective production and comprehension, and factors that affect these processes, such as a bias towards the perspective of the speaker. I argue that although the selection and identification of perspective holders may be guided by simple heuristics some of the time, certain cases require a more involved reasoning system. In Chapters 7 and 8, I develop models of perspectival reasoning in comprehension and production rooted in a leading framework for pragmatic reasoning: the Rational Speech Acts framework. In Chapter 7, I propose and implement a computational model of perspective identification. I posit that listeners reason jointly about the speaker’s intended message and their adopted perspective using a mental model of the speaker’s production process. I present two comprehension studies that support a key assumption of the proposed Perspectival Rational Speech Acts model: that listeners reason simultaneously over multiple perspectives to better understand the speaker’s intended meaning. In Chapter 8, I propose a model of perspective selection that mirrors the Perspectival Rational Speech Acts comprehension model. I posit that speakers reason about the listener’s comprehension process in order to pick a perspective and an utterance that will maximize their chance of being understood. However, the results of the production study do not match the model’s predictions. I conclude with a discussion of the challenges that the attested asymmetry between speaker and listeners poses for the Rational Speech Acts framework. The main contributions of this dissertation are as follows: (1) a comparison of four approaches to encoding the semantics of perspective, leading to a diagnostic toolkit for perspectival expressions; (2) an experimental case study that employs the diagnostics to identify a novel perspectival expression; (3) an implemented computational model of perspective identification, supported by experimental evidence; and (4) an implemented computational model of perspective selection, which reveals further challenges in perspective production

Relativized propositions (draft 1)

Can we solve the problem of the essential indexical, and account for de se belief, by appealing to 'relativized propositions' (functions from rich indices to truth-values)? According to John Perry, we cannot. This paper offers a detailed examination and a critique of Perry's argument

Transmedial Narration

This open access book is a methodical treatise on narration in different types of media. A theoretical rather than a historical study, Transmedial Narration is relevant for an understanding of narration in all times, including our own. By reconstructing the theoretical framework of transmedial narration, this book enables the inclusion of all kinds of communicative media forms on their own terms. The treatise is divided into three parts. Part I presents established and newly developed concepts that are vital for formulating a nuanced theoretical model of transmedial narration. Part II investigates the specific transmedial media characteristics that are most central for realizing narratives in a plenitude of different media types. Finally, Part III contains brief studies in which the narrative potentials of painting, instrumental music, mathematical equations, and guided tours are illuminated with the aid of the theoretical framework developed throughout the book. Suitable for advanced students and scholars, this book provides tools to disentangle the narrative potential of any form of communication

Situated Sentence Processing: The Coordinated Interplay Account and a Neurobehavioral Model

Crocker MW, Knoeferle P, Mayberry M. Situated Sentence Processing: The Coordinated Interplay Account and a Neurobehavioral Model. Brain and Language. 2010;112(3):189-201