Heylen, Dirk K.J.

Nijholt, Antinus

van Dijk, Elisabeth M.A.G.

English

Within the AMI (Augmented Multi-party Interaction) project technologies will be developed that can facilitate human interaction in the context of instrumented meeting rooms, which includes remote participant support and the possibility to browse through past meetings. The project collects data on people engaged in meetings in order to describe, analyze and theorize about meeting behavior and collaborative work. This symposium addresses research methods and techniques used in the AMI project. It deals with the collection of a richly annotated corpus, the design of annotation tools that are focused on specific annotation tasks, methods for user requirements elicitation and browser evaluation, and the observation and interpretation of behavior to obtain recognized behavioral aspects that are visualized and evaluated in a virtual meeting room

University of Twente Research Information

Annotating and Measuring Meeting Behavior  Betsy van Dijk, Anton Nijholt and Dirk Heylen  Human Media Interaction Group, Department of Computer Science, University of Twente, Enschede, The Netherlands   Abstract Within the AMI (Augmented Multi-party Interaction) project technologies will be developed that can facilitate human interaction in the context of instrumented meeting rooms, which includes remote participant support and the possibility to browse through past meetings.  The project collects data on people engaged in meetings in order to describe, analyze and theorize about meeting behavior and collaborative work. This symposium addresses research methods and techniques used in the AMI project. It deals with the collection of a richly annotated corpus, the design of annotation tools that are focused on specific annotation tasks, methods for user requirements elicitation and browser evaluation, and the observation and interpretation of behavior to obtain recognized behavioral aspects that are visualized and evaluated in a virtual meeting room.   Keywords Meetings, human interaction, annotation, multimodal, user requirements  1 Introduction The focus of the Augmented Multi-party Interaction (AMI) project is on meeting research and on assisting participants of meetings with technology. When people meet they interact and hence the project needs to understand the details of human communicative behavior in the realistic and comparatively well-structured setting of a meeting. The overall objective of AMI is to develop technologies that are able to understand and support multimodal, multiparty, human communication in meetings. The primary focus is on the development of meeting browsers that enable users to navigate and search through past meetings and on remote meeting assistants that enable remote participants to have a richer interaction with the meeting or to monitor the meeting while they are engaged in other activities in parallel.   AMI is a multi-disciplinary research project which includes modeling of human-human interaction and group dynamics, multimodal (speech and vision) processing and recognition, content abstraction and human-computer interaction.  2 Analysis of human interactions Automatic analysis of human interactions is an emerging domain in meeting research [2]. Building upon the findings of social psychologists (amongst others [4]) computer understanding of human interactions is used to accomplish more natural human-machine interaction and computer enhanced human-human communication [3]. For the automatic analysis of natural interactive communicative behavior in meetings, between humans as well as between humans and machines, the availability of large annotated databases of multimodal meeting recordings is essential. The AMI project has a number of meeting rooms equipped with multimodal sensors and computers where meetings are recorded. The collection of recorded meetings is transcribed and annotated on a wide range of properties, ranging from higher level features (e.g. dialogue acts, gestures, emotions, focus of attention) to lower level features (e.g. words, hand and arm movements, facial display elements). Many coding schemes for annotating various aspects of behavior already exist (and have been tested). In the AMI project existing coding schemes are used, improved and extended and new coding schemes for phenomena which have not been investigated before (e.g. emotion) are developed.  AMI builds on previous European projects in the field of multimodal meeting modeling and annotation. Almost the same partners have collaborated in the M4 (Multimodal Meeting Manager) project [6]. Part of the AMI partners participated in the NITE (Natural Interactivity Tools Engineering) project [7] and ICSI (the ICSI Meeting Project [1]) participates in AMI as well. Other projects that work on the computational modeling of meetings and the development of tools for meeting support are for example the Meeting Room project at Carnegie Mellon University [5] and the NIST Meeting Room Project [8].   3 Access to meeting data The annotated meetings can be used in the meeting browser and remote meeting assistant to enable efficient access to the full multimodal content. Navigation may for instance use identification of meeting participants, identification of focus of attention, or degree of involvement of participants at a particular time.  If the assistance can be offered real-time during a meeting, a remote participant that simultaneously is engaged in other activities might be warned when a topic of interest is discussed or the chairman can get support in summarizing the decisions and action points.  Though in theory the possibilities of the use of technologies to facilitate meeting browsing and access seem promising, the actual success depends on the (potential) users. Therefore user requirements elicitation and evaluation of meeting browser concepts and demonstrators of (remote) meeting assistants are increasingly emphasized in the project.  One of the main problems in deriving requirements for new technology is that potential users are often unaware of the possibilities. Consequently, when asked directly what they need, users will hardly ever express the need for new technology. In AMI the requirements gathering methods will try to deal with this well-known observation.   4 Outline of the symposium  This symposium presents a selection of methods and techniques used in the AMI project. Though far from complete this selection gives an impression of the wide range of subjects, disciplines and methods AMI deals with.       McCowan et al. give an overview of AMI research themes and discuss corpus design, data collection, annotation and distribution of the AMI meeting corpus.   The creation of large annotated multimodal corpora of meetings is very time consuming. Reidsma et al. stress the need for annotation tools that reduce the amount of work. They discuss the requirements for such tools and propose to design tools that are focused on specific tasks. An overview of properties of annotation problems is presented.  The methodology used to gather user requirements is discussed by Tucker et al. They propose a hybrid approach, combining the examination of current practices with user evaluation of new technology.    Finally, Rienks et al. present the measurement of the behavior of meeting participants. They describe the process from automatic observation of behavioral aspects through interpretations resulting in recognized behavior. They also discuss the use of a virtual meeting room as a research tool. A virtual meeting room can be used for visualization and evaluation of annotations and their interpretations, and for simulation to test models of behavior.       References:  1. Janin, A.; Ang, J.; Bhagat, S.; Dhillon, R.; Edwards, J.; Macias-Guarasa, J.; Morgan, N.; Peskin, B.; Shriberg, E.; Stolcke, A.; Wooters, C.; Wrede, B. (2004). The ICSI Meeting Project: Resources and Research. NIST ICASSP Meeting Recognition Workshop, Montreal, Canada, May 2004 2. McCowan, I.; Gatica-Perez, D.; Bengio, S.; Lathoud, G.; Barnard, M.; Zhang, D. (2005). Automatic analysis of multimodal group actions in meetings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 305-317. 3. McCowan, I., Gatica-Perez, D., Bengio, S., Moore, D. and Bourlard, H. Towards Computer Understanding of Human Interactions. In: Ambient Intelligence, E. Aarts, R. Collier, E. van Loenen & B. de Ruyter (eds.), Lecture Notes in Computer Science, 2875, Springer-Verlag Heidelberg, 235-251. 4. McGrath, J.E. (1984). Groups: Interaction and Performance, Prentice Hall, Englewood Cliffs, NJ.  5. Meeting room project Carnegy Mellon University, http://www.is.cs.cmu.edu/meeting room/ 6. Multimodal Meeting Manager project, 2002-2005, www.m4project.org  7. Natural Interactivity Tools Engineering project, 2001-2003, http://nite.nis.sdu.dk 8. NIST meeting room project http://www.nist.gov/speech/test beds/mr proj/     

Annotating and measuring meeting behavior

https://ris.utwente.nl/ws/files/5398117/DijkNijholtHeylenDef.pdf

Annotating and measuring meeting behavior

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information