Location of Repository

Construction of bilingual multimodal corpora of referring expressions in collaborative problem solving

By Takenobu Tokunaga, Ryu Iida, Masaaki Yasuhara, Asuka Terai, David Morris and Anja Belz


This paper presents on-going work on constructing bilingual multimodal corpora of referring expressions in collaborative problem solving for English and Japanese. The corpora were collected from dialogues in which two participants collaboratively solved Tangram puzzles with a puzzle simulator. Extra-linguistic information such as operations on puzzle pieces, mouse cursor position and piece positions were recorded in synchronisation with utterances. The speech data was transcribed and time-aligned with the extra-linguistic information. Referring expressions in utterances that refer to puzzle pieces were annotated in terms of their spans, their referents and their other attributes. The Japanese corpus has already been completed, but the English counter-part is still undergoing annotation. We have conducted a preliminary comparative analysis of both corpora, mainly with respect to task completion time, task success rates and attributes of referring expressions. These corpora showed significant differences in task completion time and success rate

Topics: G400 Computing, Q100 Linguistics
Publisher: University of Michigan
Year: 2010
OAI identifier: oai:eprints.brighton.ac.uk:7706

Suggested articles



  1. (2009). A Japanese corpus of referring expressions used in a situated collaboration task. doi
  2. (1993). A speech-first model for repair identification and correction. doi
  3. (2009). Accessibility and attention in situated dialogue: Roles and regulations. doi
  4. (2002). Anaphora Resolution. doi
  5. (2006). Building a semantically transparent corpus for the generation of referring expressions. doi
  6. (1995). Collaborating on referring expressions. doi
  7. (2007). Corpusbased generation of head and eyebrow motion for an embodied conversational agent. Language Resources and Evaluation, doi
  8. (2001). Cue-readiness in insight problem-solving.
  9. (2006). Effects of reflective verbalization on insight problem solving.
  10. (2010). Eric Kow, Jette Viethen, and Albert Gatt. doi
  11. (2007). How do adults solve digital tangram problems? Analyzing cognitive strategies through eye tracking approach. doi
  12. (2010). Incorporating extra-linguistic information into reference resolution in collaborative task dialogue.
  13. (2007). Modality choise for generation of referring acts. doi
  14. (2008). Multiple purpose annotation using SLAT – Segment and link-based annotation tool.
  15. (2009). Producing referring expressions in dialogue: Insights from a translation exercise.
  16. (1986). Referring as a collaborative process. doi
  17. (1978). Resolving pronoun references. doi
  18. (2008). SCARE: A situated corpus with annotated referring expressions. doi
  19. (2010). Situated dialogue processing for human-robot interaction. In Cognitive Systems: Final report of the CoSy project, doi
  20. (2007). Special issue on Mulitmodal corpora for modeling human multimodal behaviour. Language Resources and Evaluation, doi
  21. (2000). The agreement process: An empirical investigation of human-human computer-mediated collaborative dialogues. doi
  22. (2010). The give2 corpus of giving instructions in virtual environments.
  23. (1996). The Nature of Insight. doi
  24. (2005). The OSU Quake
  25. (2008). The roles of haptic-ostensive referring expressions in cooperative, task-based human-robot dialogue. doi
  26. (2010). Towards an extrinsic evaluation of referring expressions in situated dialogs. doi
  27. (2009). Using extra linguistic information for generating demonstrative pronouns in a situated collaboration task. doi

To submit an update or takedown request for this paper, please submit an Update/Correction/Removal Request.