Action Composition for the Animation of Natural Language Instructions

This research project investigates the relationship between computer animation and language; specifically, developing utilities to generate animation from natural language instructions. Methods for specifying simulations at a task-level rather than at the level of individual motions are discussed. We envision a system which would allow engineers or technical staff who currently write instruction manuals to instead generate animations which illustrate the task. However it is unlikely that these engineers would have sufficient knowledge of animation techniques. For this reason, such a system must provide high-level tools to permit the engineer to animate a task without becoming entangled in low-level animation issues

CLiFF Notes: Research in the Language, Information and Computation Laboratory of the University of Pennsylvania

One concern of the Computer Graphics Research Lab is in simulating human task behavior and understanding why the visualization of the appearance, capabilities and performance of humans is so challenging. Our research has produced a system, called Jack, for the definition, manipulation, animation and human factors analysis of simulated human figures. Jack permits the envisionment of human motion by interactive specification and simultaneous execution of multiple constraints, and is sensitive to such issues as body shape and size, linkage, and plausible motions. Enhanced control is provided by natural behaviors such as looking, reaching, balancing, lifting, stepping, walking, grasping, and so on. Although intended for highly interactive applications, Jack is a foundation for other research. The very ubiquitousness of other people in our lives poses a tantalizing challenge to the computational modeler: people are at once the most common object around us, and yet the most structurally complex. Their everyday movements are amazingly fluid, yet demanding to reproduce, with actions driven not just mechanically by muscles and bones but also cognitively by beliefs and intentions. Our motor systems manage to learn how to make us move without leaving us the burden or pleasure of knowing how we did it. Likewise we learn how to describe the actions and behaviors of others without consciously struggling with the processes of perception, recognition, and language. Present technology lets us approach human appearance and motion through computer graphics modeling and three dimensional animation, but there is considerable distance to go before purely synthesized figures trick our senses. We seek to build computational models of human like figures which manifest animacy and convincing behavior. Towards this end, we: Create an interactive computer graphics human model; Endow it with reasonable biomechanical properties; Provide it with human like behaviors; Use this simulated figure as an agent to effect changes in its world; Describe and guide its tasks through natural language instructions. There are presently no perfect solutions to any of these problems; ultimately, however, we should be able to give our surrogate human directions that, in conjunction with suitable symbolic reasoning processes, make it appear to behave in a natural, appropriate, and intelligent fashion. Compromises will be essential, due to limits in computation, throughput of display hardware, and demands of real-time interaction, but our algorithms aim to balance the physical device constraints with carefully crafted models, general solutions, and thoughtful organization. The Jack software is built on Silicon Graphics Iris 4D workstations because those systems have 3-D graphics features that greatly aid the process of interacting with highly articulated figures such as the human body. Of course, graphics capabilities themselves do not make a usable system. Our research has therefore focused on software to make the manipulation of a simulated human figure easy for a rather specific user population: human factors design engineers or ergonomics analysts involved in visualizing and assessing human motor performance, fit, reach, view, and other physical tasks in a workplace environment. The software also happens to be quite usable by others, including graduate students and animators. The point, however, is that program design has tried to take into account a wide variety of physical problem oriented tasks, rather than just offer a computer graphics and animation tool for the already computer sophisticated or skilled animator. As an alternative to interactive specification, a simulation system allows a convenient temporal and spatial parallel programming language for behaviors. The Graphics Lab is working with the Natural Language Group to explore the possibility of using natural language instructions, such as those found in assembly or maintenance manuals, to drive the behavior of our animated human agents. (See the CLiFF note entry for the AnimNL group for details.) Even though Jack is under continual development, it has nonetheless already proved to be a substantial computational tool in analyzing human abilities in physical workplaces. It is being applied to actual problems involving space vehicle inhabitants, helicopter pilots, maintenance technicians, foot soldiers, and tractor drivers. This broad range of applications is precisely the target we intended to reach. The general capabilities embedded in Jack attempt to mirror certain aspects of human performance, rather than the specific requirements of the corresponding workplace. We view the Jack system as the basis of a virtual animated agent that can carry out tasks and instructions in a simulated 3D environment. While we have not yet fooled anyone into believing that the Jack figure is real , its behaviors are becoming more reasonable and its repertoire of actions more extensive. When interactive control becomes more labor intensive than natural language instructional control, we will have reached a significant milestone toward an intelligent agent

Research in the Language, Information and Computation Laboratory of the University of Pennsylvania

This report takes its name from the Computational Linguistics Feedback Forum (CLiFF), an informal discussion group for students and faculty. However the scope of the research covered in this report is broader than the title might suggest; this is the yearly report of the LINC Lab, the Language, Information and Computation Laboratory of the University of Pennsylvania. It may at first be hard to see the threads that bind together the work presented here, work by faculty, graduate students and postdocs in the Computer Science and Linguistics Departments, and the Institute for Research in Cognitive Science. It includes prototypical Natural Language fields such as: Combinatorial Categorial Grammars, Tree Adjoining Grammars, syntactic parsing and the syntax-semantics interface; but it extends to statistical methods, plan inference, instruction understanding, intonation, causal reasoning, free word order languages, geometric reasoning, medical informatics, connectionism, and language acquisition. Naturally, this introduction cannot spell out all the connections between these abstracts; we invite you to explore them on your own. In fact, with this issue it’s easier than ever to do so: this document is accessible on the “information superhighway”. Just call up http://www.cis.upenn.edu/~cliff-group/94/cliffnotes.html In addition, you can find many of the papers referenced in the CLiFF Notes on the net. Most can be obtained by following links from the authors’ abstracts in the web version of this report. The abstracts describe the researchers’ many areas of investigation, explain their shared concerns, and present some interesting work in Cognitive Science. We hope its new online format makes the CLiFF Notes a more useful and interesting guide to Computational Linguistics activity at Penn

The Topic is \u3cem\u3eOpen\u3c/em\u3e

We can open a door, open an umbrella, or open a jar; we open the safe (open the door of the safe), open the cookies (open the box and plastic pouch) and open a valve (turn the knob and open the pipe). I take open to mean CAUSE X TO BE OPEN and to describe the process of opening. given the diversity that occurs, is it possible to characterize the objects of a verb like open? In this paper I argue that there are similarities among the possible physical objects of the verb open, based on the objects; underlying geometric structure. This regularity, derived from an analysis of tokens extracted from the Brown corpus, is used in the remainder of the paper to analyze similarities amongst abstract use of open: (open a meeting, open a gulf). The proposed analysis also explain limits in usage: #open the chair; #open the question

Doing What You\u27re Told: Following Task Instructions in Changing, but Hospitable Environments

The AnimNL project (Anim ation from N atural L anguage) has as its goal the automatic creation of animated task simulations from natural-language instructions. The question addressed in this paper is how agents can perform tasks in environments about which they have only partial relevant knowledge. The solution we describe involves enabling such agents to * develop expectations through instruction understanding and plan inference, and use those expectations in deciding how to act; * exploit generalized abilities in order to deal with novel geometric situations. The AnimNL project builds on an animation system, Jack™, that has been developed at the Computer Graphics Research Lab at the University of Pennsylvania, and draws upon a range of recent work in Natural Language semantics, planning and plan inference, philosophical studies of intention, reasoning about knowledge and action, and subsumption architectures for autonomous agents

Connecting Planning And Acting Via Object-Specific Reasoning

Instructions from a high-level planner are in general too abstract for a behavioral simulator to execute. In this dissertation I describe an intermediate reasoning system -- the Object-Specific Reasoner -- which bridges the gap between high-level task-actions and action directives of a behavioral system. It decomposes task-actions and derives parameter values for each action directive, thus enabling existing high-level planners to instruct synthetic agents with the same task-action commands that they currently produce. The Object-Specific Reasoner's architecture follows directly from the hypothesis that action representations are underspecified descriptions, and that objects in the same functional category are manipulated in similar ways. The action representation and the object representation are combined to complete the action interpretation, thereby grounding plans in action. The Object-Specific Reasoner provides evidence that a small number of object functional categories, organize..

Connecting Planning and Acting: Towards an Architecture for Object-Specific Reasoning

Controlling synthetic agents requires specifying details of the desired actions which are rarely specified in high-level commands. And while simulation or robotic systems provide a motion language for controlling and coordinating an agent's movements, each command must be completely specified in order to be executed. Although a human agent can interpret a command such as (pickup fred glass), current synthetic agents cannot. Asking the instructor to learn to give fully specified commands to synthetic agents is impractical for a number of reasons, including unavailability of objectand situation-specific information (which is readily apparent in the case of a remote agent). Instead, I will suggest the need for an intermediate reasoning system which can determine the missing command parameters. This system, the Object Specific Reasoner (OSR), tailors high-level plans to the specifics of the agent and objects. As plans are elaborated, the OSR generates a sequence of action directives which ..

CLiFF Notes - Research in the Language, Information and Computation Laboratory of the . . .

s 1 Breck Baldwin 2 Tilman Becker 4 Betty J. Birner 6 Sandra Carberry 8 Christine Doran 10 Dania Egedi 13 Jason Eisner 15 Christopher Geib 17 Abigail S. Gertner 19 James Henderson 21 Beth Ann Hockey 23 Beryl Hoffman 26 Paul S. Jacobs 28 Aravind K. Joshi 29 Jonathan M. Kaye 32 Albert Kim 34 Nobo Komagata 36 Seth Kulick 37 Sadao Kurohashi 38 Libby Levison 39 D. R. Mani 41 Mitch Marcus 43 I. Dan Melamed 46 ii Michael B. Moore 47 Charles L. Ortiz, Jr. 48 Martha Palmer 49 Hyun S Park 52 Jong Cheol Park 54 Scott Prevost 55 Ellen F. Prince 58 Lance A. Ramshaw 60 Lisa F. Rau 62 Jeffrey C. Reynar 64 Francesc Ribas i Framis 66 James Rogers 68 Bernhard Rohrbacher 70 Joseph Rosenzweig 72 Deborah Rossen-Knill 73 Anoop Sarkar 76 B. Srinivas 77 Mark Steedman 80 Matthew Stone 82 Anne Vainikka 84 Bonnie Lynn Webber 86 Michael White 89 David Yarowsky 91 III Projects and Working Groups 93 The AnimNL Project 94 The Gesture Jack project 96 Information Theory Reading Group 99 iii The STAG Machine Tran..