Constraint-based graphical layout of multimodal presentations

When developing advanced multimodal interfaces, combining the characteristics of different modalities such as natural language, graphics, animation, virtual realities, etc., the question of automatically designing the graphical layout of such presentations in an appropriate format becomes increasingly important. So, to communicate information to the user in an expressive and effective way, a knowledge-based layout component has to be integrated into the architecture of an intelligent presentation system. In order to achieve a coherent output, it must be able to reflect certain semantic and pragmatic relations specified by a presentation planner to arrange the visual appearance of a mixture of textual and graphic fragments delivered by mode-specific generators. In this paper we will illustrate by the example of LayLab, the layout manager of the multimodal presentation system WIP, how the complex positioning problem for multimodal information can be treated as a constraint satisfaction problem. The design of an aesthetically pleasing layout is characterized as a combination of a general search problem in a finite discrete search space and an optimization problem. Therefore, we have integrated two dedicated constraint solvers, an incremental hierarchy solver and a finite domain solver, in a layered constraint solver model CLAY, which is triggered from a common metalevel by rules and defaults. The underlying constraint language is able to encode graphical design knowledge expressed by semantic/pragmatic, geometrical/topological, and temporal relations. Furthermore, this mechanism allows one to prioritize the constraints as well as to handle constraint solving over finite domains. As graphical constraints frequently have only local effects, they are incrementally generated by the system on the fly. Ultimately, we will illustrate the functionality of LayLab by some snapshots of an example run

Generating multimedia presentations: from plain text to screenplay

In many Natural Language Generation (NLG) applications, the output is limited to plain text – i.e., a string of words with punctuation and paragraph breaks, but no indications for layout, or pictures, or dialogue. In several projects, we have begun to explore NLG applications in which these extra media are brought into play. This paper gives an informal account of what we have learned. For coherence, we focus on the domain of patient information leaflets, and follow an example in which the same content is expressed first in plain text, then in formatted text, then in text with pictures, and finally in a dialogue script that can be performed by two animated agents. We show how the same meaning can be mapped to realisation patterns in different media, and how the expanded options for expressing meaning are related to the perceived style and tone of the presentation. Throughout, we stress that the extra media are not simple added to plain text, but integrated with it: thus the use of formatting, or pictures, or dialogue, may require radical rewording of the text itself

LAYLAB : a constraint-based layout manager for multimedia presentations

When developing advanced intelligent user interfaces composing text, graphics, animation, hypermedia etc., the question of automatically designing the graphical layout of such multimedia presentations in an appropriate format plays a crucial role. This paper introduces the task, the functionality and the architecture of the constraint-based multimedia layout manager LayLab

Automatic design of multimodal presentations

We describe our attempt to integrate multiple AI components such as planning, knowledge representation, natural language generation, and graphics generation into a functioning prototype called WIP that plans and coordinates multimodal presentations in which all material is generated by the system. WIP allows the generation of alternate presentations of the same content taking into account various contextual factors such as the user\u27s degree of expertise and preferences for a particular output medium or mode. The current prototype of WIP generates multimodal explanations and instructions for assembling, using, maintaining or repairing physical devices. This paper introduces the task, the functionality and the architecture of the WIP system. We show that in WIP the design of a multimodal document is viewed as a non-monotonic process that includes various revisions of preliminary results, massive replanning and plan repairs, and many negotiations between design and realization components in order to achieve an optimal division of work between text and graphics. We describe how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. Finally, we discuss the generation of cross-modal expressions that establish referential relationships between text and graphics elements

Using graphical style and visibility constraints for a meaningful layout in visual programming interfaces

In the expanding field of visual applications, layout design and graphical editing tasks are crucial points. In this paper, we address the incorporation of AI aspects in the visual software design process and the automatic layout and beautification of informational graphics such as visual programs and chart diagrams. Since layout in dynamic settings frequently requires a direct manipulation responsiveness, an incremental redesign of the generated visual material is necessary. Following our previous work on constraint-based multimodal design, we show how powerful constraint processing techniques, such as constraint hierarchies and dynamic constraint satisfaction, can be applied to visual programming environments in order to maintain graphical style and consistency for a meaningful layout. We describe InLay, a system for constraint-based presenting and editing visual programs. Finally, we will have a short look at some extensions with regard to advanced interaction and visualization techniques

Research in constraint-based layout, visualization, CAD, and related topics : a bibliographical survey

The present work compiles numerous papers in the area of computer-aided design, graphics, layout configuration, and user interfaces in general. There is nearly no conference on graphics, multimedia, and user interfaces that does not include a section on constraint-based graphics; on the other hand most conferences on constraint processing favour applications in graphics. This work of bibliographical pointers may serve as a basis for a detailed and comprehensive survey of this important and challenging field in the intersection of constraint processing and graphics. In order to reach this ambitious aim, and also to keep this study up-to-date, the authors appreciate any comment and update information

Perspektiven zur Kombination von automatischem Animationsdesign und planbasierter Hilfe

Aktuelle Themen auf dem Gebiet der intelligenten Benutzerschnittstellen behandeln derzeit die automatische Planung multimodaler Präsentationen. Hierbei stand bisher im wesentlichen die koordinierte Generierung von Text und Graphik im Vordergrund. In Zukunft wird hier aufgrund der Komplexität der zu präsentierenden Information zunehmend auch die Einbeziehung realistischer animierter 3D-Graphiken gefordert sein. Einen anderen wichtigen Forschungsschwerpunkt bildet der Einsatz graphischer Ausgabekomponenten für planbasierte Hilfesysteme. Die vorliegende Arbeit hat zum Ziel zunächst einen Überblick über den derzeitigen Stand der Forschung in diesen beiden Bereichen zu geben, als auch neue Anforderungen an die automatische Animationsgenerierung und an Systeme zur planbasierten graphischen Hilfe zu formulieren. Anschließend wollen wir, basierend auf Ergebnissen und Erfahrungen aus WIP und PLUS, Perspektiven für eine mögliche Weiterentwicklung und Integration von Techniken der Animationsplanung und graphischen Hilfe präsentieren

Constraint-basierte Verarbeitung graphischen Wissens

Bei der Entwicklung neuerer intelligenter Benutzerschnittstellen, die wie im Beispiel des multimodalen Präsentationssystems WIP natürliche Sprache und Graphik kombinieren, spielt insbesondere die wissensbasierte Gestaltung des Layouts multimodaler Dokumente eine wichtige Rolle. Am Beispiel des Layout-Managers in WIP soll gezeigt werden, wie aufgrund der von einem Präsentationsplaner spezifizierten semantischen und pragmatischen Relationen, die von den media-spezifischen Generatoren erzeugten Graphik- und Textfragmente in einem Dokument automatische arrangiert werden können. Dabei wird das Layoutproblem als Constraint-Satisfaction-Problem behandelt. Es wird hier gezeigt, wie der Constraint-Ansatz sowohl zur Repräsentation von graphischem Wissen, als auch zur Berechnung der Platzierung der Layoutobjekte auf einem Design-Grid verwendet werden kann. So werden semantische Kohärenzrelationen wie etwa "sequence" oder "contrast" durch entsprechende Design-Constraints reflektiert, die perzeptuelle Kriterien (Alignierung, Gruppierung, Symmetrie, etc.) spezifizieren. Zur Realisierung wird in WIP ein mehrschichtiger inkrementeller Constraint-Solver mit lokaler Propagierung verwendet, der es erlaubt, Constraints dynamisch zu generieren