LAYLAB : a constraint-based layout manager for multimedia presentations

When developing advanced intelligent user interfaces composing text, graphics, animation, hypermedia etc., the question of automatically designing the graphical layout of such multimedia presentations in an appropriate format plays a crucial role. This paper introduces the task, the functionality and the architecture of the constraint-based multimedia layout manager LayLab

Automatic design of multimodal presentations

We describe our attempt to integrate multiple AI components such as planning, knowledge representation, natural language generation, and graphics generation into a functioning prototype called WIP that plans and coordinates multimodal presentations in which all material is generated by the system. WIP allows the generation of alternate presentations of the same content taking into account various contextual factors such as the user\u27s degree of expertise and preferences for a particular output medium or mode. The current prototype of WIP generates multimodal explanations and instructions for assembling, using, maintaining or repairing physical devices. This paper introduces the task, the functionality and the architecture of the WIP system. We show that in WIP the design of a multimodal document is viewed as a non-monotonic process that includes various revisions of preliminary results, massive replanning and plan repairs, and many negotiations between design and realization components in order to achieve an optimal division of work between text and graphics. We describe how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. Finally, we discuss the generation of cross-modal expressions that establish referential relationships between text and graphics elements

Constraint-based graphical layout of multimodal presentations

When developing advanced multimodal interfaces, combining the characteristics of different modalities such as natural language, graphics, animation, virtual realities, etc., the question of automatically designing the graphical layout of such presentations in an appropriate format becomes increasingly important. So, to communicate information to the user in an expressive and effective way, a knowledge-based layout component has to be integrated into the architecture of an intelligent presentation system. In order to achieve a coherent output, it must be able to reflect certain semantic and pragmatic relations specified by a presentation planner to arrange the visual appearance of a mixture of textual and graphic fragments delivered by mode-specific generators. In this paper we will illustrate by the example of LayLab, the layout manager of the multimodal presentation system WIP, how the complex positioning problem for multimodal information can be treated as a constraint satisfaction problem. The design of an aesthetically pleasing layout is characterized as a combination of a general search problem in a finite discrete search space and an optimization problem. Therefore, we have integrated two dedicated constraint solvers, an incremental hierarchy solver and a finite domain solver, in a layered constraint solver model CLAY, which is triggered from a common metalevel by rules and defaults. The underlying constraint language is able to encode graphical design knowledge expressed by semantic/pragmatic, geometrical/topological, and temporal relations. Furthermore, this mechanism allows one to prioritize the constraints as well as to handle constraint solving over finite domains. As graphical constraints frequently have only local effects, they are incrementally generated by the system on the fly. Ultimately, we will illustrate the functionality of LayLab by some snapshots of an example run

Von Textgeneratoren zu Intellimedia-Präsentationssystemen

Während Textgeneratoren ausschließlich auf das Medium Sprache zurückgreifen, nutzen Intellimedia-Präsentationssysteme die individuellen Stärken unterschiedlicher Medien, wie Text, Graphik, Gestik und Animation, für die Informationsdarbietung. Die aus kommunikationstheoretischer Sicht allgemeinere Aufgabe wirft einerseits neue interessante Probleme auf, etwa die Selektion und die Koordination von Medien, führt aber andererseits zu einer umfassenderen Behandlung von Fragestellungen, die bereits von der Textgenerierung her bekannt sind. Dieses Papier stellt die erste Generation von sprachverarbeitenden Intellimedia Präsentationssystemen vor, skizziert die neuen Problemstellungen und beschäftigt sich insbesondere mit der Frage, inwiefern sich Methoden zur Textgenerierung verallgemeinern lassen, damit sie für die Informationspräsentation mit mehreren Medien anwendbar sind.While text generators exclusively rely on a single medium, intellimedia presentation systems take advantage of the individual strength of several media, such as text, graphics, gestures and animations, to present information. On the one hand, new interesting problems arise in the broader context of multimedia communication, in particular the selection and the coordination of media. On the other hand, this research leads to a more general treatment of problems already known from text generation. The paper discusses the first generation of NL processing intellimedia presentation systems and sketches the new problems. Particular emphasis is given to the question of how to generalize methods for text generation in such a way that they become useful for the production of multimedia presentations, too

Plan-based integration of natural language and graphics generation

Multimodal interfaces combining natural language and graphics take advantage of both the individual strength of each communication mode and the fact that several modes can be employed in parallel. The central claim of this paper is that the generation of a multimodal presentation system WIP which allows the generation of alternate presentations of the same content taking into account various contextual factors. We discuss how the plan-based approach to presentation design can be exploited so that graphics generation influences the production of text and vice versa. We show that well-known concepts from the area of natural language processing like speech acts, anaphora, and rhetorical relations take on an extended meaning in the context of multimodal communication. Finally, we discuss two detailed examples illustrating and reinforcing our theoretical claims

MOG 2007:Workshop on Multimodal Output Generation: CTIT Proceedings

This volume brings together presents a wide variety of work offering different perspectives on multimodal generation. Two different strands of work can be distinguished: half of the gathered papers present current work on embodied conversational agents (ECA’s), while the other half presents current work on multimedia applications. Two general research questions are shared by all: what output modalities are most suitable in which situation, and how should different output modalities be combined

An information-based approach to punctuation

Ankara : Department of Computer Engineering and Information Science and the Institute of Engineering and Science of Bilkent University, 1998.Thesis (Ph. D.) -- Bilkent University, 1998.Includes bibliographical references leaves 83-93.Say, BilgePh.D

Punctuations in Maintaining The Coherence of Discourse: The Case of J.K. Rowling’s Harry Potter Fictions

ABSTRACT Communication consists of two forms: verbal and written. The term discourse encompasses both forms. It is easier for the hearer of verbal communication to convey the message performed by the speaker because those supporting clues are eligible for instance gesture and intonation. Such a situation cannot be fulfilled by the written discourse. At least, there are two kinds of relations in discourse that can help the readers fulfill this need called cohesion and coherence. While cohesion is the unity of text related to syntax and grammar, coherence lies in the people. This abstract description of coherence leads the writer to do an extra work to give the clues for the readers. One of the clues is the use of punctuation. It is a challenge to investigate the functions of punctuation in the direct speech of Harry Potter fictions. How do the presence of punctuations support the coherence of discourse as written by J.K.Rowling in Harry Potter fictions?What punctuations are used by J. K. Rowling in the direct speech of Harry Potter fictions?What are the functions of punctuations in the direct speech as written by J.K. Rowling in Harry Potter fictions? Theoretically, this study uses the theory of punctuation and discourse markers to answer these problems. The theory of punctuation is used to recognize the functions of punctuations in the direct speech of Harry Potter fictions while the theory of discourse markers leads to the type of functions of punctuations in signaling the kind of discourse relation it act upon. Methodologically, this study practiced qualitative study as the research design; the data are collected by skimming and scanning, and analyzed by applying the ideal type analysis trough the reading process of the data. The findings shows that J.K.Rowling uses period (.), comma (,), colon (:), semicolon (;), question mark (?), exclamation mark (!), quotation marks (‘ ’ and “ ”), ellipses (…), and dash (–). The function of period is to end a word, phrase, and sentence in an utterance therefore it belongs to signaling alteration relation. Comma is used to separate elements or unit of talks and it is included in signaling continuation. Colon is used to introduce a list, to signal an explanation, to signal an example and to notify an emerging idea.The function of colon to introduce a list and to signal an example is classified as signaling instances relation. Whereas the function of colon to signal explanation belongs to signaling explanations, the function of colon to notify an emerging idea fits in the function of signaling elaboration relation. The functions of semicolon are to end a direct question, to show uncertainty and to indicate another unit of utterance appeared. Thus, the first two function of question mark is categorized as signaling alteration, while the last function is determine as signaling continuation. The functions of question marks are as follows: to end a direct question, to show uncertainty and to indicate another unit of utterance appeared. Thus, the first two function of question mark is categorized as signaling alteration, while the last function is determine as signaling continuation. Exclamation mark functions only in term of it mark any word, phrase, sentence that have a strong emotion. Therefore, similar to question mark, this function is also put as signaling alteration. The function of quotation marks is to set off word or phrase, thus this belong to signaling additional relation. Ellipses functions to indicate an interruption and to indicate an incomplete statement. All those functions can be classified as signaling continuation. Dash performs four functions. Those are to indicate a sudden interruption, to notify the reader of a surprising element, to notify the reader of an important idea and to emphasize a definition. The first function is categorized as signaling continuation, while the rest three functions belong to the category of signaling elaboration. In summary, this study concludes that the functions of punctuations in the direct speech of Harry Potter fictions contributes to the coherence of discourse by signaling the relation between the discourse unit it act upon