Generation of multi-modal dialogue for a net environment

In this paper an architecture and special purpose markup language for simulated affective face-to-face communication is presented. In systems based on this architecture, users will be able to watch embodied conversational agents interact with each other in virtual locations on the internet. The markup language, or Rich Representation Language (RRL), has been designed to provide an integrated representation of speech, gesture, posture and facial animation

Automatic design of multimodal presentations

We describe our attempt to integrate multiple AI components such as planning, knowledge representation, natural language generation, and graphics generation into a functioning prototype called WIP that plans and coordinates multimodal presentations in which all material is generated by the system. WIP allows the generation of alternate presentations of the same content taking into account various contextual factors such as the user\u27s degree of expertise and preferences for a particular output medium or mode. The current prototype of WIP generates multimodal explanations and instructions for assembling, using, maintaining or repairing physical devices. This paper introduces the task, the functionality and the architecture of the WIP system. We show that in WIP the design of a multimodal document is viewed as a non-monotonic process that includes various revisions of preliminary results, massive replanning and plan repairs, and many negotiations between design and realization components in order to achieve an optimal division of work between text and graphics. We describe how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. Finally, we discuss the generation of cross-modal expressions that establish referential relationships between text and graphics elements

RRL: A Rich Representation Language for the Description of Agent Behaviour in NECA

In this paper, we describe the Rich Representation Language (RRL) which is used in the NECA system. The NECA system generates interactions between two or more animated characters. The RRL is a formal framework for representing the information that is exchanged at the interfaces between the various NECA system modules

Representing Conversations for Scalable Overhearing

Open distributed multi-agent systems are gaining interest in the academic community and in industry. In such open settings, agents are often coordinated using standardized agent conversation protocols. The representation of such protocols (for analysis, validation, monitoring, etc) is an important aspect of multi-agent applications. Recently, Petri nets have been shown to be an interesting approach to such representation, and radically different approaches using Petri nets have been proposed. However, their relative strengths and weaknesses have not been examined. Moreover, their scalability and suitability for different tasks have not been addressed. This paper addresses both these challenges. First, we analyze existing Petri net representations in terms of their scalability and appropriateness for overhearing, an important task in monitoring open multi-agent systems. Then, building on the insights gained, we introduce a novel representation using Colored Petri nets that explicitly represent legal joint conversation states and messages. This representation approach offers significant improvements in scalability and is particularly suitable for overhearing. Furthermore, we show that this new representation offers a comprehensive coverage of all conversation features of FIPA conversation standards. We also present a procedure for transforming AUML conversation protocol diagrams (a standard human-readable representation), to our Colored Petri net representation

Human Machine Interaction

In this book, the reader will find a set of papers divided into two sections. The first section presents different proposals focused on the human-machine interaction development process. The second section is devoted to different aspects of interaction, with a special emphasis on the physical interaction

Automated Visual Discourse Synthesis: Coherence, Versatility, and Interactivity

In this proposal, we present comprehensive and systematic approaches of building systems that can automatically generate coherent visual discourse for interactive environments. A visual discourse refers to a series of connected visual displays. A coherent visual discourse requires smooth transitions between displays, consistent designs within and among displays, and effective integration of various components. While our main research goal is to develop approaches to automatically create coherent, versatile, and interactive visual discourse, we also emphasize integrating these approaches into a general framework to provide a reference model in which a specific system is considered an instantiation of the framework. In other words, the framework should consist of a knowledge base, an inference engine, a visual realizer, and an interaction handler. As a consequence, not only can a general framework serve as a template from which a specific generation system can be instantiated,but the framework also can be used as a base for comparing or evaluating different systems. We concentrate on the basic issues involved in developing comprehensive and systematic approaches to ensure a visual discourse's coherence, versatility, and interactivity. In particular, To ensure coherence, we have established a set of comprehensive design criteria to measure both expressiveness and effectiveness of a visual discourse. To provide versatility, we address the design of visual discourse for heterogeneous information. Within such discussions, heterogeneous information refers to both quantitative and qualitative, or static and dynamic information. In addition, we are also concerned with characterizing and employing different visual media and a wide variety of visual techniques. To support interaction, we integrate many conventional user interface metaphors and styles into visual discourse design and explore reactive planning approaches to provide proper response to user interactions. To establish the framework, we identify various knowledge sources and determine effective knowledge representation paradigms in constructing the knowledge base. We emphasize the efficiency, usability, and flexibility issues in modeling the inference engine. We are concerned with portability and parallelization issues in building the visual realizer, and we also take into account interaction capabilities for interactive environments. We describe a system called IMPROVISE (Illustrative Metaphor Production in Reactive Object-oriented VISual Environments) that serves as a proof-of-concept prototype. IMPROVISE is built based on our framework, aiming to automatically generate coherent visual discourse for various application domains in interactive environments. IMPROVISE has been used in two testbed application domains to demonstrate its generality and flexibility. Examples from both domains will be given to illustrate IMPROVISE's generation process and to identify the future research areas