Hough J, de Ruiter L, Betz S, Schlangen D. Disfluency and Laughter Annotation in a Light-weight Dialogue Mark-up Protocol. Presented at the The 7th Workshop on Disfluency in Spontaneous Speech (DiSS), Edinburgh, UK

Betz, Simon 

de Ruiter, Laura

Hough, Julian

Schlangen, David 

Publications at Bielefeld University

DISFLUENCY AND LAUGHTER ANNOTATION IN A LIGHT-WEIGHTDIALOGUE MARK-UP PROTOCOLJulian Hough*, Laura de Ruiter**, Simon Betz** and David Schlangen*Bielefeld University, *Dialogue Systems Group, **Workgroup Phonetics and Phonologyjulian.hough@uni-bielefeld.deABSTRACTDespite a great deal of research effort, disfluencyand laughter annotation is still an unsolved prob-lem, both in terms of consensus for a general appli-cable system, and in terms of annotation agreementmetrics. In this paper we present a new annotationscheme within a light-weight mark-up for sponta-neous speech. We show, despite the low overheadrequired for understanding the annotation protocol,it allows for good inter-annotator agreement and canbe used to map onto existing disfluency categoriza-tion, with no loss of information.Keywords: Disfluency annotation, laughter, Ger-man corpora, inter-annotator agreement, sponta-neous speech1. INTRODUCTIONAnnotating spontaneous speech material is alwaysconstrained by a trade-off between time and efforton the one hand and coverage on the other. Here wedevelop a system for annotating spontaneous speechthat reduces effort while increasing coverage of dis-fluency and laughter phenomena.The low effort required is due to (1) a minimalis-tic and comprehensive vocabulary to be learned bythe annotator and (2) the application directly on thetranscription text at transcription time. This makesthe scheme easy to learn and handle for non-expertswho can facilitate the markup of spontaneous speechphenomena on the fly while transcribing. Furtheranalysis and correction is left to interested expertswho can focus on their own research questions moredirectly than in existing annotation systems wheremore expert work for identification and labeling ofthese phenomena might be necessary from the out-set. Analyses of agreement confirm that non-expertsindeed have little trouble in comprehension and ap-plication of this system.Despite the existence of numerous annotation sys-tems, some of which even focus on spontaneousspeech phenomena (see §2 for an overview), thesesystems never cover all the phenomena potentiallyof interest to disfluency and dialogue researchers.Our system encompasses a light-weight way of cov-ering all disfluency and laughter phenomena andthe potential to be mapped onto existing annotationschemes.2. EXISTING DISFLUENCY ANDLAUGHTER ANNOTATION SCHEMESThere is a plethora of existing speech annotationschemes, some of which have been developed espe-cially to capture disfluency phenomena. They varyin terms of practical implementation, the use of cate-gory labels and whether or not they mark non-verbalor paralinguistic events.[15]’s thesis and the ensuing Switchboard disflu-ency annotation manual [9] are perhaps the mostwell known for disfluency mark-up. Other moregeneral schemes, such as that used in the GermanVerbmobil corpus for task oriented dialogue anno-tation, have various conventions, covering a widerange of spontaneous speech phenomena, including,but not explicitly focusing on, disfluencies [2].Technically, some schemes annotate disfluencieson a separate tier, for instance [4], [2] or [10]), whileothers use an in-text annotation on the transcriptiontier, for instance [9], [12], [13], with more recentschemes often doing this by means of an XML-styleannotation [12], [1]. Several schemes require anno-tators to assign disfluency category labels like “re-pair", “insertion" or “stutter" from a pre-specifiedset (vocabulary) as they go through the data ([12],[2], [1], [4]), so training costs can be substantial.3. A LIGHT-WEIGHT DISFLUENCY ANDLAUGHTER ANNOTATION TECHNIQUEIn an effort to both improve the ease of annotationfor non disfluency experts, and allow subsumptionof existing categorical approaches, here we proposea more light-weight approach than those mentionedabove which does not reduce the mark-up informa-tion. It combines transcription and annotation in onepass, so that turn segmentation and word transcrip-tion can be aided by directly observing their inter-Table 1: Exemplary DF annotations and their equivalent labels in [1] and [8].Annotation [1] [8](Ich + ich) will <repetition> Covert repair(I + I) want <rm> Ich </rm><rs> ich </rs></repetition>(nicht verwinkelt so dass +) und breit <restart><rm> Fresh start(not contorted so that+) and wide nicht verwinkelt so dass</rm><rs/></restart> und breitdie (<p="Küche">Krü-</p> + Küche)the (kri- + kitchen)die <sot> Krü- </sot> Küche Phonetic repairaction with disfluencies. Transcription and annota-tion can be conducted based on perception withoutlengthy training: transcribers do not have to learncategorical labels for the phenomena because suchanalyses can be carried out afterwards with the useof automatic search, dependent on the given interestof the researcher.The advantage of avoiding category labels in theannotation process is that the annotated data is moreversatile and more ‘theory-neutral’, allowing re-searchers to map it onto different existing classifi-cation schemes should they so wish, but not con-straining them to do so. Labels like “repetition" or“replacement" (used for instance by [1]) can easilybe derived from our bracketed annotation, which issimilar to [15]. Other categories like those suggestedby [8] can also be derived from our mark-up.Table 1 provides some examples of how disfluen-cies annotated according to our scheme can be bemapped onto other classification schemes. Whilethe bracketing of our mid-utterance repair disfluen-cies is based on [15], with reparandum, interregnumand repair phases available, with a strictly right-branching mark-up of chaining disfluencies such asin (4), we emphasize in our annotation manual thedistinction between abandonment of turns (wherebyannotators will simply transcribe an utterance final ‘-’) as in (1) and (2) below, and turn-initial fresh startsmarked with no repair phase such as in (3) belowand in Table 1. Note that while not a focus of this pa-per, we account for phonetic variants of lexical itemswith <v="..">..</v> tags.(1) Oder das <v="ist">is’</v> im -Or that is in the -(2) Ja eigentlich <v="wäre es"> wär’s </v> cool in derKüche <v="einen">’n</v> kleinen Tisch zu habenwo man -Yes actually it would be cool to have a small tablein the kitchen where you -We include filled pauses, marked simply bya {F } bracketing and other fillers simply use{ }. We also include laughed speech withsimple XML-style tags spanning the affectedspeech <laughter>...</laughter> and a<laughterOffset/> tag for the often audibledeep inhalation of breath after laughed speech or alaughter bout marked <laughter/> (see (3)).(3) (Und mit einem +) mit vielleicht Sachen die nicht<laughter> auseinander brechen </laughter><laughterOffset/> -(And with a +) with perhaps things that don’t fallapart -For partial words, we encourage transcribers toguess the complete form of the word where possi-ble, again using a simple tag <p="..">..-</p>, asbelow:(4) ( <p="Wohnzimmer">Wohn-</p> + . {ja also} (die + ( die + das ) ) {F äh} ... Wohnzimmer )<p="living room">liv-</p> yes well the the the uhliving room -4. INTER-ANNOTATOR AGREEMENTCategory Agreement κ f reereparandum 0.9477 0.8954repair 0.9677 0.9353filled pause 0.9968 0.9937laughed speech 0.9558 0.9117Table 2: Inter-annotator agreement scores for dis-fluent word types using κ f reeWe test our annotation scheme on a corpus ofdyadic interactions between German speakers, theDream Apartment (DAP) corpus [7], which in con-trast to existing corpora used to studies disfluenciesClass SWBD % of words DAP % of wordsReparandum words 5.16 5.51Partial words* 0.75 1.10Filled pauses* 1.12 1.81Laughed words* 0.45 6.06Table 3: The frequency of disfluent and laughed words in Switchboard (SWBD) and the Dream Apartment (DAP)corpora. Starred categories indicate a significantly different frequency between the two corpora.like [6],[14] or [13], is a relatively domain-generalcorpus of face-to-face interactions. The DAP con-sists of 9 dialogues of 15 minutes in length. Inthe task, participant pairs were instructed to discusstheir ideal apartment they could jointly design suchthat they could describe it to an architect. They aregiven a substantial budget of 500,000 Euros. Thefamiliarity of the subjects varied with 2 of the pairsbeing strangers and the others varying in familiarity.All participants were students.To test agreement we use one transcript and 3annotators: one was the second author, while theother two were non experts. We compared the inter-annotator agreement of words being part of differentdisfluency and laughed speech elements using themarginal-free multi-rater metric κ f ree [11]– we usethis metric as other multi-rater agreement measureslike Fleiss’ κ suffer from an assumption annotatorsknow a priori how many cases they should assign toeach category, which is not the case here.The results shown in Table 2 are both interest-ing and encouraging. Filled pauses and repair phasewords have very good agreement, while the lowerreparandum word agreement shows a deviation inthe way annotators perceive the extent of repairs,and consequently their discourse effects– see [5] fora similar finding. The lower agreement for laughedspeech segmentation is not detrimental, as it is stillgood enough to provide search terms for subsequentstand-off annotation.5. USE CASE 1: DISFLUENCY ANDLAUGHED SPEECH RATES IN PHONE ANDFACE-TO-FACE CONVERSATIONSOne of the benefits of our scheme is that it is directlycompatible with established schemes, including theSwitchboard disfluency annotation mark up [9]. Wecan therefore directly compare the rates of the dis-fluency and laughter phenomena in the DAP withSwitchboard.In Switchboard we use the held-out data for dis-fluency detection (all files named sw4[5-9]* in thePenn TB III release: 52 transcripts, 6.5K utterances,49K words) marked up according to the scheme in[9]. The DAP is smaller with fewer, but longer, dia-logues (9 transcripts, 4.1K utterances, 20k words).The proportion of reparandum words ineach corpus was not significantly different(χ2(1)=3.568,p=0.06) however the proportion offilled pauses, laughed words and partial words ofthe total word tokens was significantly lower in theSwitchboard corpus.The most striking difference is in the proportionof laughed words. We hypothesize this may havebeen due to the difference in topics between the DAPand Switchboard, and also due the familiarity of theparticipants. Upon inspection there were many op-portunities for laughables based primarily on the in-congruity of the situation of being students with avast amounts of spending money.6. USE CASE 2: DISFLUENCY ANDLAUGHTER INTERACTIONA second use case we investigate for our annotationprotocol is a qualitative analysis of the interactionof laughed speech, laughter and disfluency in theDAP corpus. In Figure 1 three extracts from the cor-pus with our mark-up scheme are shown with theirEnglish translations. In Example 1 we see someevidence to explain the high frequency of laughedspeech described above being due to the topic ofconversation. Here the subject of intimacy and pri-vacy with one’s partner in the dream apartment isnot being fully addressed but is jointly laughed at.A filled pause is employed after the joint laughter.In Example 2, disfluency and laughed speech in-teract again on the same topic as in Example 1, witha chaining replacement repair directly following thelaughter. In Example 3, a laughable is taken upby both participants in response to a self-answeredquestion by A who mocks her own lack of intelligi-bility in her explanation to B. Following this the turnis immediately held by A by use of a filled pause.While these few examples do not provide a thor-ough analysis of the interactions, we hope they illus-trate that Conversation Analysts and dialogue theo-rists may also benefit from our simple mark-up.Example 1: Laughter and laughed speech (joint):A okay also ( wenn wir + wenn wir ) <v="eine">’ne</v> Küche haben ein {F ähm } Wohnzimmer zusammen haben und{F äh } ein [ Badezimmer ] das ist ja so wie in <v="einer">’ner</v> WG [ ] <breathing/> brauchen wir auf jeden Fall<breathing/> jeder so grosse Zimmer dass jeweils unsere <laughter> Partner die dann ja auch noch zu [ Besuchkommen ] </laughter> <breathing/> {F ähm } auch noch Platz findenA-en okay so ( when we + when we ) have a kitchen, a {F uh } livingroom and {F uh } a [ bathroom ] together which is theway it is in a shared flat [ ], we must definitely have big rooms so when our <laughter> partners come [to visit]</laughter> <breathing/>{F um} they have a place to go.B [mhm] / [ja] / [<laughter>sehr richtig</laughter>]B-en [mhm] /[yes] / [<laughter>very true</laughter>]Example 2: Chaining substitution repair after laughed speech:A dann hat jeder genug Privatsphäre .. mit seinem <laughter> Partner </laughter> / ( und die Küche + ( und die + {F ähm} ( und die + ... und das Wohnzimmer ) ) ) ist quasi so ... mittigA-en then everyone has some privacy ... with their <laughter> partner </laughter> / ( and the kitchen + ( and the + {F um} ( and the + ... and the living room ) ) ) is kind of ... centralExample 3: Laughter at embarrassment of disagreement followed by turn hold filled pause:A und vom Wohnzimmer kannst du halt in die Küche gehen / <v="verstehst du">verstehste</v> das ? / [ okay ] nee / [gut ] / {F ä:hm }A-en and from the livingroom can you sort of go to the kitchen / do you understand ? / [ okay ] no / [ good ] / {F u:m }B [ nee ] / [ <laughter/> ] <laughterOffset/>B-en [ no ] / [ <laughter/> ] <laughterOffset/>Figure 1: Examples of the interaction between disfluency and laughter in the dream apartment corpus7. CONCLUSIONWe present a light-weight and reliable protocol fordisfluency and laughter annotation which is cur-rently being used in the DUEL (DisflUencies, Ex-clamations and Laughter in dialogue) project [3]. Itis both fast and easy to use for non-experts, and sub-sumes existing schemes. Stand-off timing informa-tion for detailed investigation into phenomena of in-terest is derivable from this automatically using theMINT tools [7] software, among others. We haveshown two use cases of the scheme, one which al-lows direct comparability to other schemes, and onewhich allows fast mark-up of dialogues at transcrip-tion time for quantitative and qualitative analysis.8. REFERENCES[1] Besser, J., Alexandersson, J. 2007. A comprehen-sive disfluency model for multi-party interaction.SigDial 2007 volume 8 182–189.[2] Burger, S., Weilhammer, K., Schiel, F., Tillmann,H. G. 2000. Verbmobil data collection and anno-tation. In: Verbmobil: Foundations of Speech-to-Speech Translation. Springer 537–549.[3] Ginzburg, J., Tian, Y., Amsili, P., Beyssade, C.,Hemforth, B., Mathieu, Y., Saillard, C., Hough,J., Kousidis, S., Schlangen, D. 2014. The Dis-fluency, Exclamation and Laughter in Dialogue(DUEL) Project. Proceedings of the 18th SemDialWorkshop (DialWatt) Herriot Watt University, Ed-inburgh. 176–178.[4] Hedeland, H., Schmidt, T. 2012. Technologicaland methodological challenges in creating, anno-tating and sharing a learner corpus of spoken ger-man. Multilingual Corpora and Multilingual Cor-pus Analysis 14, 25.[5] Hough, J., Purver, M. Dec. 2013. Modelling expec-tation in the self-repair processing of annotat-, um,listeners. Proceedings of the 17th SemDial Work-shop (DialDam) Amsterdam. 92–101.[6] Kohler, K. J. 1996. Labelled data bank ofspoken standard german: the kiel corpus ofread/spontaneous speech. ICSLP 96 volume 3.IEEE 1938–1941.[7] Kousidis, S., Pfeiffer, T., Schlangen, D. 2013. Mint.tools: Tools and adaptors supporting acquisition,annotation and analysis of multimodal corpora. In-terspeech 2013.[8] Levelt, W. J. 1983. Monitoring and self-repair inspeech. Cognition 14(1), 41–104.[9] Meteer, M. W., Taylor, A. A., MacIntyre, R., Iyer,R. 1995. Dysfluency annotation stylebook for theswitchboard corpus. University of Pennsylvania.[10] Moniz, H., Batista, F., Mata, A. I., Trancoso, I.2014. Speaking style effects in the production ofdisfluencies. Speech Communication 65, 20–35.[11] Randolph, J. J. 2005. Free-marginal multiraterkappa (multirater k [free]): An alternative to fleiss’fixed-marginal multirater kappa. Online Submis-sion.[12] Rodríguez, L. J., Torres, I., Varona, A. 2001. Anno-tation and analysis of disfluencies in a spontaneousspeech corpus in spanish. ISCA 2001.[13] Schiel, F., Heinrich, C., Barfüsser, S. 2012. Alco-hol language corpus: the first public corpus of al-coholized german speech. Language resources andevaluation 46(3), 503–521.[14] Schmidt, T., Hedeland, H., Lehmberg, T., Wörner,K. 2010. Hamatac–the hamburg maptask corpus.[15] Shriberg, E. E. 1994. Preliminaries to a theory ofspeech disfluencies. PhD thesis.

Disfluency and Laughter Annotation in a Light-weight Dialogue Mark-up Protocol

https://pub.uni-bielefeld.de/download/2758948/2764630

Disfluency and Laughter Annotation in a Light-weight Dialogue Mark-up Protocol

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University