Development of the User-State Conventions for the Multimodal Corpus in SmartKom

Abstract

This contribution deals with the problems and solutions of finding procedures for the labeling of a multimodal data corpus that is created within the SmartKom project. The goal of the SmartKom project is the development of an intelligent computer-user interface that should allow almost natural communication with an adaptive and selfexplanatory machine. The system does not only accept input in form of natural speech but also in form of gestures. Additionally the facial expression and prosody of speech is analyzed. To train recognizers and to explore how users interact with the system, data is collected in so-called Wizard-of-Oz experiments. Speech is transliterated and gestures as well as user-states 2 are labeled. At the start of the project only the speech transliteration conventions existed. Therefore conventions and procedures to label the video data had to be newly created. In this contribution we will describe the development process of the User-State Labeling Conventions 3 as an example for our strategy of "functional labeling". The development and structure of the gesture labeling is described in detail in Steininger et al. [1]. The transliteration conventions can be found in Beringer et al. [2]. The special problem of combining the information of the different labeling steps and the transliteration is discussed in Schiel et al. at this workshop