Welcome to Aberdeen at the Symposium on Multimodal Output Generation (MOG 2008)! In this volume the papers presented at the MOG 2008 international symposium are collected

André, E.

Bachvarova, Y.S.

Sluis, I. van der

Theune, M.

English

Theune, Mariët

van der Sluis, Ielka

Bachvarova, Yulia

André, Elisabeth

University of Twente Research Information

The AISB’08 Symposium on Multimodal Output Generation (MOG 2008)Welcome to Aberdeen at the Symposium on Multimodal Output Generation (MOG 2008)! This year MOG is held as apart of the AISB Convention of the Society for the Study of Artificial Intelligence and the Simulation of Behaviour that isorganised around the theme Communication, Interaction and Social Intelligence. Similar to MOG 2007, this year’s MOGaims to present the state of the art and identify future research needs in multimodal output generation. MOG 2008 provesagain to be successful in bringing together work from different disciplines which is usually scattered across various events.Besides contributions from research fields such as multimodal language generation and embodied conversational agents(ECA’s), MOG 2008 has an additional angle by investigating how research on multimodal output generation can benefitfrom a non-engineering perspective on multimodality. For example, how can insights from psychology and cognitive sci-ences, related to understanding how humans perceive and process multimodal information, be properly formalized for thepurposes of intelligent multimodal output generation? And to what extent is it possible to formalize existing theories abouthow meaning is made in multimodal communication?This year, we are pleased to welcome two invited speakers: Dr. Michelle Zhou (IBM T. J. Watson Research Center)and Professor Justine Cassell (Northwestern University).In this volume the papers presented at the MOG 2008 international symposium are collected. Five papers introducedifferent approaches to two of the most fundamental topics for multimodal output generation-output planning and modalitychoice. Gerrit Kahl et al. propose three multimodal output generation strategies: user-defined (output modalities explicitlyselected by the user), symmetric multimodal (using the same modalities for output as the ones used for input) and context-based (generating an optimal multimodal output based on different factors in the user’s current situational context). YujiaCao proposes a modality planning framework which can optimally allocate one or more modalities to a chunk of informa-tion and also calculate an optimal combination of modalities. Central to the framework is a computational optimizationmodel which takes as input the information to be conveyed, modality availability, and the user profile, translates those intoconstraints, and outputs a modality plan. In his contribution Flavio Soares Correa da Silva proposes a knowledge-basedlayer which selects the output modalities that present medical information on mobile devices of a telehealth system. Theproposed layer does modality selection based on device capabilities, type of interaction, type of information and featuresdescribing modalities. The enlisted knowledge sources are described by respective ontologies or reference models. VerenaRieser and Oliver Lemon adopt a machine learning approach to modality selection. They develop and evaluate adaptivemultimodal dialogue strategies using simulation-based reinforcement learning showing that the latter approach, which pro-vides additional information about the user in the reward function, overperforms supervised learning which only mimicsthe data. Bosma et al. present a method for automatic text illustration, based on the similarity between the text to be illus-trated and picture-related text. A user study conducted to evaluate their method showed that when compared to manuallyselected pictures, automatically selected pictures were rated similarly to decorative pictures, but worse than informativepictures.Four papers are dedicated to the topic of ECAs’ speech and gesture. Kirsten Bergmann and Stefan Kopp present acomputational perspective on the joint production process for speech and gesture. Based on empirical evidence indicatinga mutual influence of speech and gesture in utterance production, they propose an interface between imagistic and propo-sitional knowledge at the level of content representation. Beatriz López et al. explore the possibilities of using ECAsto improve the robustness and the interactive fluency in spoken dialogue systems. Results of a comparison between twointerfaces, one with an ECA and one with a voice-only output, show that user frustration is lower and interaction flowsmore smoothly when an ECA is present in the interface. In their paper, Rieks op den Akker and Mariët Theune discuss themain literature on addressing and present both qualitative and quantitative analyses of addressing in multi-party, face-to-face conversations. Based on these findings they sketch a model for the generation of multimodal addressing behaviour.Finally, Werner Breitfuss et al. introduce a system that automatically adds gaze and gesture to a given dialogue script be-tween two virtual embodied agents in the roles of speaker and listener. The quality of the gaze generation was empiricallytested showing that the naturalness of the agent’s behaviour was not increased when compared to randomly selected gazebehaviour, but the quality of the communication between the two agents was perceived as significantly enhanced.Topics related to corpora studies are investigated in two papers. In their contribution Michael Barclay and AntonyGalton describe initial steps in the design of a scene corpus for training and testing spatial communication systems. Such ascene corpus needs to allow a full range of spatial relations to be expressed over a range of scale spaces, the scenes shouldbe sufficiently complex to allow the construction of sequential spatial descriptions and the integration of listener modelsand reference frame variations should be possible. Ielka van der Sluis et al. describe the experiment setup with which atransparent multimodal dialogue corpus was collected. The corpus is currently being transcribed and will be used to testhypotheses about human production as well as hypotheses about human perception of referring expressions that includepointing gestures.And finally two papers extend the interdisciplinary approach to multimodal output generation into semiotics and prag-matics by discussing the incorporation of principles informed by these two disciplines. In his paper Frédéric Landraginiiiproposes a set of principles for the design of multimedia presentation systems that are based on pragmatics and humanfactors such as the specificities of human perception, attention, memory, conceptualization and language. Arianna Maio-rani applies the multimodal discourse analysis method to the study of Internet as a multimodal semiotic system. The papertests the validity of the functional framework used in linguistics to study online multimodal communication.Thanks are due to the programme committee members: Adrian Bangerter, Ellen Gurman Bard, John Bateman, HarryBunt, Stephan Kopp, Emiel Krahmer, Theo van Leeuwen, Anton Nijholt, Jon Oberlander, Niels Ole Bernsen, Paul Piwek,Ehud Reiter, Jan Peter de Ruiter, Jacques Terken, Eija Ventola, Ipke Wachsmuth, and Marilyn Walker. We would alsolike to thank the additional reviewers: Christian Becker-Asano, Nikolaus Bee, Kirsten Bergmann, Trung Bui, Yujia Cao,Nadine Pfeiffer-Lessmann and Mannes Poel.MOG 2007 is endorsed by SIGGEN (ACL Special Interest Group on Generation) and SIGMedia (ACL Special Inter-est Group on Multimedia Language Processing). The symposium is also sponsored by NWO via IMOGEN (InteractiveMultimodal Output Generation), a research project within the NWO-IMIX research programme. We are grateful to allthese supporting organizations. At the University of Aberdeen a smooth AISB organisation team made it possible to or-ganise this years MOG symposium. Very special thanks are due to Frank Guerin and Wamberto Vasconcelos.Mariët TheuneIelka van der SluisYulia BachvarovaElisabeth AndréSymposium Organisers:Mariët Theune, University of Twente, The NetherlandsYulia Bachvarova, University of Twente, The NetherlandsElisabeth André, University of Augsburg, GermanyIelka van der Sluis, University of Aberdeen, UKProgramme Committee:Adrian Bangerter, University of Neuchâtel, SwitzerlandEllen Gurman Bard, University of Edinburgh, UKJohn Bateman, University of Bremen, GermanyHarry Bunt, Tilburg University, The NetherlandsStephan Kopp, University of Bielefeld, GermanyEmiel Krahmer, Tilburg University, The NetherlandsTheo van Leeuwen, University of Technology Sydney, AustraliaAnton Nijholt, University of Twente, The NetherlandsJon Oberlander, University of Edinburgh, UKNiels Ole Bernsen, University of Southern DenmarkPaul Piwek, Open University, Milton KeynesEhud Reiter, University of Aberdeen, UKJan Peter de Ruiter, MPI, The NetherlandsJacques Terken, Eindhoven University, The NetherlandsIpke Wachsmuth, University of Bielefeld, GermanyMarilyn Walker, University of Sheffield, UKiv

The AISB’08 Symposium on Multimodal Output Generation (MOG 2008)

http://purl.utwente.nl/publications/64707

The AISB’08 Symposium on Multimodal Output Generation (MOG 2008)

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information