From Visual Input to Verbal Output in the Visual Translator

The project VITRA (VIsual TRAnslator) deals with the relationship between natural language and vision. Experimental studies are being carried out in the way of designing an interface between image understanding and natural language systems, with the aim of developing systems for the natural language description of image sequences. During the last ten years several domains of application have been investigated and an approach for integrating language and vision has evolved, which allows incremental evaluation and simultaneous natural language description of real world image sequences. This contribution provides a compact description of the VI- TRA conception and presents some of the results obtained so far. Current limitations of the approach and future research tasks will be discussed as well. In: Proc. of the AAAI Fall Symposium on Computational Models for Integrating Language and Vision, Cambridge, MA, 1995. 1 Introduction Despite the intrinsic difficulty of both natural langua..

Resource-Adaptive Action Planning in a Dialogue System for Repair Support

. For reasonable application in critical domains dialogue systems not only have to pay attention to user knowledge and dialogue goals but they also have to adapt their dialogue behaviour to current limitations of the user's cognitive processing capabilities. In case of a help system the point of concern is on the one hand the aspect as to what kind of help is appropriate for the user P (What to do?). This action planning has to take into account P's resource limitations like time pressure and working memory load. On the other hand the system's utterances should also be adapted to P's language processing capabilities (What to say -- and how?). This work presents primarily approaches to adaptive action planning. Its interaction with dialogue planning is outlined. 1 Introduction 1.1 Issues In the field of human-computer-communication considerable progress has been made since the first approaches to user modelling in adapting system behaviour to individual user knowledge and current dial..

Utilizing Interval-Based Event Representations for Incremental High-Level Scene Analysis

Within the project Vitra (VIsual TRAnslator) we are concerned with the design and construction of integrated knowledge-based systems capable of translating visual information into natural language descriptions. In this contribution 1 the focus will be on high-level scene analysis, i.e., the step from a geometrical representation, as it might be provided by a vision component, into conceptual descriptions of object motions. In a combined vision and natural language system aiming at simultaneous natural language descriptions of dynamic imagery the recognition of such motion events has to be done incrementally in order to be able to talk about events even while they are progressing. We present course diagrams as a means for such an incremental motion analysis and show how they can be constructed automatically from interval-based event representations using temporal constraint propagation techniques. To appear in: Proc. of the 4th International Workshop on Semantics of Time, Space, and..

Numerical Uncertainty Management in User and Student Modeling: An Overview of Systems and Issues

. A rapidly growing number of user and student modeling systems have employed numerical techniques for uncertainty management. The three major paradigms are those of Bayesian networks, the Dempster-Shafer theory of evidence, and fuzzy logic. In this overview, each of the first three main sections focuses on one of these paradigms. It first introduces the basic concepts by showing how they can be applied to a relatively simple user modeling problem. It then surveys systems that have applied techniques from the paradigm to user or student modeling, characterizing each system within a common framework. The final main section discusses several aspects of the usability of these techniques for user and student modeling, such as their knowledge engineering requirements, their need for computational resources, and the communicability of their results. Key words: numerical uncertainty management, Bayesian networks, Dempster-Shafer theory, fuzzy logic, user modeling, student modeling 1. Introdu..

Exploiting Models of Musical Structure for Automatic Intention-Based Composition of Background Music

Music is known to be able to support certain aims of presentations. In addition, there are a number of approaches to automatic music composition. Thus, it is a natural consequence, to establish automatic, intention-based music composition as an additional tool for multimedia presentation systems. However, many related applications show unsatisfactory results. An explanation for this phenomenon is that the consideration of musical structure at different abstraction levels is often neglected although many relevant theories do exist and have been refined to a high level. The aim of the project described in this paper is to build a system for the automatic composition of music that supports the intentions of a given presentation. The idea underlying the system design is that the composition process works on different abstraction levels, so that each subset of desired intentions can be realized at the appropriate level. The resulting coordination problems make great demands on the level of ..

Spatial Information in Instructions and Questions to an Autonomous System

The task of supplying user-friendly access to an autonomous system using natural language is an interesting field of research which can be partitioned into access to autonomous robots and simulated agents. The perceptual information of such a system is to be related to language expressions in order to be able to talk about spatial configurations. The multiplicity of possible utterances is one of the main difficulties with natural language in general. In this paper, we concentrate on spatial expressions in different kinds of instructions and questions using anytime algorithms in order to consider resource limitations. 1 Introduction Autonomous systems are characterized by their sometimes unpredictable behavior. E.g., their error recovery feature can cause understanding problems for the user: he possibly does not know why the system acts in a certain way. Thus, intelligent systems should also have intelligent man-machine interfaces in order to improve cooperativeness [14]. Talking about..

Human-Machine Interaction for Intelligent Robots Using Natural Language

In this paper, a new natural language interface is presented that can be applied to make the use of intelligent robots more flexible. This interface was developed for the autonomous mobile two-arm robot KAMRO, which uses several camera systems to generate an environment model and to perform assembly tasks. A fundamental requirement in human-machine interaction for intelligent robots is the ability to refer to objects in the robot’s environment. Hence, the interface and the intelligent system need similar environment models and it is necessary to provide current sensor information. Additional flexibility can be achieved by integrating the man-machine interface into the control architecture of the robot and to give it access to all internal information and to the models that the robot uses for an autonomous behaviour. In order to fully exploit the capabilities of a natural language access, we favour a dialogue-based approach, i.e., for the interface, KANTRA, presented here, the human-machine interaction is not restricted to unidirectional communication.

KANTRA: Human-Machine Interaction for Intelligent Robots Using Natural Language

In this paper, a new natural language interface is presented that can be applied to make the use of intelligent robots more flexible. This interface was developed for the autonomous mobile two-arm robot KAMRO, which uses several camera systems to generate an environment model and to perform assembly tasks. A fundamental requirement in human-machine interaction for intelligent robots is the ability to refer to objects in the robot's environment. Hence, the interface and the intelligent system need similar environment models and it is necessary to provide current sensor information. Additional flexibility can be achieved by integrating the man-machine interface into the control architecture of the robot and to give it access to all internal information and to the models that the robot uses for an autonomous behaviour. In order to fully exploit the capabilities of a natural language access, we favour a dialogue-based approach, i.e., for the interface, KANTRA, presented here, the human-mac..