Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

We propose a computational model of situated language comprehension based on the Indexical Hypothesis that generates meaning representations by translating amodal linguistic symbols to modal representations of beliefs, knowledge, and experience external to the linguistic system. This Indexical Model incorporates multiple information sources, including perceptions, domain knowledge, and short-term and long-term experiences during comprehension. We show that exploiting diverse information sources can alleviate ambiguities that arise from contextual use of underspecific referring expressions and unexpressed argument alternations of verbs. The model is being used to support linguistic interactions in Rosie, an agent implemented in Soar that learns from instruction.Comment: Advances in Cognitive Systems 3 (2014

Embodiment and embodied design

Picture this. A preverbal infant straddles the center of a seesaw. She gently tilts her weight back and forth from one side to the other, sensing as each side tips downward and then back up again. This child cannot articulate her observations in simple words, let alone in scientific jargon. Can she learn anything from this experience? If so, what is she learning, and what role might such learning play in her future interactions in the world? Of course, this is a nonverbal bodily experience, and any learning that occurs must be bodily, physical learning. But does this nonverbal bodily experience have anything to do with the sort of learning that takes place in schools - learning verbal and abstract concepts? In this chapter, we argue that the body has everything to do with learning, even learning of abstract concepts. Take mathematics, for example. Mathematical practice is thought to be about producing and manipulating arbitrary symbolic inscriptions that bear abstract, universal truisms untainted by human corporeality. Mathematics is thought to epitomize our species’ collective historical achievement of transcending and, perhaps, escaping the mundane, material condition of having a body governed by haphazard terrestrial circumstance. Surely mathematics is disembodied

Pedagogies of Design and Multiliterate Learner Identities

In an era of multiliteracies, teaching and learning have become knowledge performances at multiple levels. Instead of a singular, linear focus upon print technologies, the techno-oriented philosophy of teaching aims at providing a rhizomatic network of texts where there is a close link between, and often an overlap of, different designs—linguistic, visual, spatial, and gestural—to construct the multiliterate learner. In this paper, I discuss the role of multimodal literacies in a primary classroom, affirming the role of multiliteracies and decentring the pre-dominance of linguistic at the cost of other designs. While the print media are acknowledged as significant to literacy, the multimodality of print is enhanced through visual and spatial design (Kenner, 2004). Through graphic examples of ICT applications of designs in a primary classroom, I demonstrate that students are operating through multitextual and digitextual (Everett, 2003) practices. What follows is the complex positioning and re-situating of teacher and learner identities engaged in learning through the knowledge processes of experiencing, identifying, applying and critiquing concepts (Kalantzis & Cope, 2004). In particular, I argue that within the diversity of present day classrooms, the digital oriented, multiliterate learner is implicated in constant identity construction by drawing upon macro and micro social practices. I conclude by reiterating the significance of new technologies and new literacy practices as essential to the construction of new learner identities

Socialising Epistemic Cognition

We draw on recent accounts of social epistemology to present a novel account of epistemic cognition that is ‘socialised’. In developing this account we foreground the: normative and pragmatic nature of knowledge claims; functional role that ‘to know’ plays when agents say they ‘know x’; the social context in which such claims occur at a macro level, including disciplinary and cultural context; and the communicative context in which such claims occur, the ways in which individuals and small groups express and construct (or co-construct) their knowledge claims. We frame prior research in terms of this new approach to provide an exemplification of its application. Practical implications for research and learning contexts are highlighted, suggesting a re-focussing of analysis on the collective level, and the ways knowledge-standards emerge from group-activity, as a communicative property of that activity

From Verbs to Tasks: An Integrated Account of Learning Tasks from Situated Interactive Instruction.

Intelligent collaborative agents are becoming common in the human society. From virtual assistants such as Siri and Google Now to assistive robots, they contribute to human activities in a variety of ways. As they become more pervasive, the challenge of customizing them to a variety of environments and tasks becomes critical. It is infeasible for engineers to program them for each individual use. Our research aims at building interactive robots and agents that adapt to new environments autonomously by interacting with human users using natural modalities. This dissertation studies the problem of learning novel tasks from human-agent dialog. We propose a novel approach for interactive task learning, situated interactive instruction (SII), and investigate approaches to three computational challenges that arise in designing SII agents: situated comprehension, mixed-initiative interaction, and interactive task learning. We propose a novel mixed-modality grounded representation for task verbs which encompasses their lexical, semantic, and task-oriented aspects. This representation is useful in situated comprehension and can be learned through human-agent interactions. We introduce the Indexical Model of comprehension that can exploit extra-linguistic contexts for resolving semantic ambiguities in situated comprehension of task commands. The Indexical model is integrated with a mixed-initiative interaction model that facilitates a flexible task-oriented human-agent dialog. This dialog serves as the basis of interactive task learning. We propose an interactive variation of explanation-based learning that can acquire the proposed representation. We demonstrate that our learning paradigm is efficient, can transfer knowledge between structurally similar tasks, integrates agent-driven exploration with instructional learning, and can acquire several tasks. The methods proposed in this thesis are integrated in Rosie - a generally instructable agent developed in the Soar cognitive architecture and embodied on a table-top robot.PhDComputer Science and EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/111573/1/shiwali_1.pd

Designing software to maximize learning1

This paper starts from the assumption that any evaluation of educational software should focus on whether or not, and the extent to which, it maximizes learning. It is particularly concerned with the impact of software on the quality of learning. The paper reviews key texts in the literature on learning, including some which relate directly to software development, and suggests ways in which a range of learning theories can inform the process of software design. The paper sets out to make a contribution to both the design and the evaluation of educational software

From conditioning to learning communities: Implications of fifty years of research in e‐learning interaction design

This paper will consider e‐learning in terms of the underlying learning processes and interactions that are stimulated, supported or favoured by new media and the contexts or communities in which it is used. We will review and critique a selection of research and development from the past fifty years that has linked pedagogical and learning theory to the design of innovative e‐learning systems and activities, and discuss their implications. It will include approaches that are, essentially, behaviourist (Skinner and Gagné), cognitivist (Pask, Piaget and Papert), situated (Lave, Wenger and Seely‐Brown), socio‐constructivist (Vygotsky), socio‐cultural (Nardi and Engestrom) and community‐based (Wenger and Preece). Emerging from this review is the argument that effective e‐learning usually requires, or involves, high‐quality educational discourse, that leads to, at the least, improved knowledge, and at the best, conceptual development and improved understanding. To achieve this I argue that we need to adopt a more holistic approach to design that synthesizes features of the included approaches, leading to a framework that emphasizes the relationships between cognitive changes, dialogue processes and the communities, or contexts for e‐learning