Wagner P, Bryhadyr N, Schröer M, Ludusan B. Does information-structural acoustic prosody change under different visibility conditions? In:  Proceedings of ICPhS. 2019.It is well-known that the effort invested in prosodic expression can be adjusted to the information structure in a message, but also to the characteristics of the transmission channel. To investigate wether visibly accessible cues to information structure or facial prosodic expression have a differentiated impact on acoustic prosody, we modified the visibility conditions in a spontaneous dyadic interaction task, i.e. a verbalized version of TicTacToe. The main hypothesis was that visibly accessible cues should lead to a decrease in prosodic effort. While we found that - as expected - information structure is expressed throughout a number of acoustic-prosodic cues, visible accessibility to context information makes accents shorter, while accessability to an interlocutor's facial expression slightly increases the mean F0 of an accent

Bryhadyr, Nataliya

Ludusan, Bogdan

Schröer, Marin

Wagner, Petra 

English

Ludusan, Bogdan 

Publications at Bielefeld University

DOES INFORMATION-STRUCTURAL ACOUSTIC PROSODY CHANGEUNDER DIFFERENT VISIBILITY CONDITIONS?Petra Wagner, Nataliya Bryhadyr, Marin Schröer, Bogdan LudusanPhonetics and Phonology Workgroup, Faculty of Linguistics and Literary StudiesBielefeld Universitypetra.wagner@uni-bielefeld.deABSTRACTIt is well-known that the effort invested in prosodicexpression can be adjusted to the information struc-ture in a message, but also to the characteristics ofthe transmission channel. To investigate whethervisibly accessible cues to information structure orfacial prosodic expression have a differentiated im-pact on acoustic prosody, we modified the visibilityconditions in a spontaneous dyadic interaction task,i.e. a verbalized version of TicTacToe. The main hy-pothesis was that visibly accessible cues should leadto a decrease in prosodic effort. While we found that- as expected - information structure is expressedthroughout a number of acoustic-prosodic cues, vis-ible accessibility to context information makes ac-cents shorter, while accessibility to an interlocutor’sfacial expression slightly increases the mean f0 of anaccent.Keywords: prosody, dialogue, information struc-ture, visibility, speech economy.1. INTRODUCTIONIt is well described that contextually novel, surpris-ing, important, contrastive or somehow discourse-relevant words are made prosodically prominentacross a number of typologically diverse languages[10, 15, 2, 12].What is less well understood is whether these var-ious prominence-cueing sources should be modeledas a single one-dimensional concept of “informa-tion structure”, or whether the different functions ofprominence are phonologically differentiated [11].Watson et al. [14] investigated whether differenttypes of information-structure trigger different typesof prosodic prominence. They operationalized thedifference between relevance accents (roughly cor-responding to “focus” in information theory) andunpredictability accents (roughly corresponding to“novelty” in information theory)) by measuring ver-balized game moves in games of TicTacToe. In earlystages of the game, the moves are relatively unpre-dictable, hence tend to be accented, but also less rel-evant, as they are not decisive for the outcome ofthe game. Later on, the game moves are highly pre-dictable, but relevant, as they typically prevent theinterlocutor from winning, or may constitute win-ning moves (cf. Figure 1). For American English,[14] found a difference in the prosodic realizationsof these two types of prominence: accents express-ing unpredictability are longer in duration and areproduced with a higher f0 excursion, while accentsrelated to relevance are louder. A replication of thisstudy for another Germanic language, namely Ger-man, constitutes the first goal of this paper. We ex-pect a similar accentual differentiation.Figure 1: An unpredictable move on field “5”(left), followed by a relevant move on field “7”(right) in TicTacToe.Most studies (including [14]) investigate the in-fluence of information structure in ways neutraliz-ing the influence of contextual factors that may af-fect information transfer, e.g. information that is ac-cessible via a visual channel. However, in authenticcommunicative settings, interlocutors often can seetheir interlocutor’s facial expression and have vis-ible access to the information that is verbally ex-pressed (e.g. in a TV weather report, speech is ac-companied by a visual illustration). The interplayof visually and verbally accessible information inprosodic expression constitutes the second goal ofour paper. Optimization models of speech commu-nication would predict a decrease in overall prosodiceffort if interlocutors have visible access to informa-tion that is verbally expressed [5, 7], but there areseveral candidates that may trigger such an effect.When interlocutors see each other in the TicTacToesetting, they can see both the facial expression andthe manually played game moves. Access to themanual game moves makes the verbal message fullyredundant and might therefore lead to its pronun-ciation with reduced effort. However, visibility offacial movements have independently shown to en-hance a message’s intelligibility [9, 8, 4] and to con-vey prosodic prominence [4]. Therefore, a reductionin prosodic expression under mutual visibility couldalso result from exploiting a cumulative effect of vi-sually and verbally conveyed prosodic prominence.We therefore hypothesize that (1) if the visibly ac-cessible information makes the verbal message re-dundant (which is the case if the manual moves toa game target field are visible), acoustic prosodiceffort should be reduced. We further hypothesizethat (2) if facial movements are visible (but not man-ual ones), these may provide cues to prosodic struc-ture and thus cue a lack of vocal effort likewise.However, given the strong findings with respect tocongruency across the visual and verbal modality[6, 3, 13], and given the fact that facially conveyedprosody does not render the verbal message redun-dant, we predict that the effect of facial visibility isless strong than the effect of visibility to relevantcontext information. Our subsequent study sets outto investigate our hypotheses.2. METHODS2.1. Recording SetupThe recordings were carried out at the faculty’srecording studio using Sennheiser neckband micro-phones. We recorded 40 participants (native speak-ers of the Northern German Standard Variety) form-ing 20 dyadic pairs. Interlocutors within dyadswere of equal social status, typically friends, and ofsame or mixed gender. Recordings of one speakerwere excluded from further analyses due to techni-cal problems resulting in a poor recording quality.Each player received a set of cut outs in the formof blue or red felt squares to mark their moves on ashared vertical grid (cf. Figure 2). The game boardlooked like a normal TicTacToe grid, however withevery cell being numbered. This was introduced inorder to enable the interlocutors to unambiguouslyrefer to the different cells on the game board us-ing the digits 1− 9. That way, a typical verbalizedmove is produced by placing a sentence or nuclearaccent on the target of the move, which correspondsto one of the numbers available on the game boardand is realized sentence-finally in the vast majorityof cases, e.g.(1) Mein nächster Zug geht auf FÜNF.(Engl.: My next move goes on FIVE.)The verbalizations of game board targets (“1-9”)were later analyzed with respect to their prosodicrealization. However, speakers were not instructedto use a particular sentence structure or use specificwords to refer to their targets.2.2. Visibility ConditionsIn each dyad, participants performed 4 games ofTicTacToe in each of 4 different recording setups.The game board was placed horizontally betweenthe speakers . In each game, the initial move waspreset (randomly) by the experimenter. The order ofgame initializations rotated and the order of record-ing setups was shifted for each dyad. The 4 differentrecording conditions are specified in the following:1. Manual and facial visibility: transparent gameboard, full view of interlocutor’s head and fa-cial expression.2. Manual visibility, no facial visibility: trans-parent game board, but obstructed view of in-terlocutor’s head and facial expression (with alight curtain)3. Facial visibility, no manual visibility: non-transparent game board, but full view of inter-locutor’s head and facial expression4. Neither manual nor facial visibility: non-transparent game board, obstructed view ofpartner’s head and facial expression (with alight curtain)Figure 2: The recording setup under the full visi-bility condition.2.3. Information Structure ConditionsLargely following [14], we used the TicTacToe Set-ting to disentangle two aspects of information struc-ture, namely predictability and relevance. As theinitial moves were predefined by the experimenter,openly told to both participants and then simply re-peated by the participant, they were defined as fullypredictable (‘8’). The second move was annotated asleast predictable (‘1’), as the participants still havemany options to choose from on the game board.The following moves were annotated with increas-ing predictability (‘2—8’) in course of the game.For statistical analysis, predictability was recodedin a binary fashion, with predictability > 4 = ‘pre-dictable’, the remaining moves as ‘unpredictable’.Relevance was operationalized in a binary fashion,with moves that prevent or constitute a winningmove being annotated as ‘relevant’, others as ‘irrel-evant’ (for the outcome of the game). As game deci-sive, relevant moves tend to come later in the game,and typically are predictable, the presence of eitherfeature predicts the absence of the other, allowingfor a contrastive analysis of both. For an illustrationof predictability and relevance, cf. Figure 1.2.4. Annotations and AnalysesIn each dyad, the verbalizations of the game movetargets (numbers 1-9) as well as the correspondingmove’s relevance and predictability were annotatedmanually using Praat [1]. In the vast majority ofcases, these targets corresponded to the final word ofan utterance, coinciding with a (nuclear) accent. Werestricted our analyses to the phrase final, accentedverbalizations of target moves. Using a Praat script,we then carried out a number of acoustic analysesof these target move verbalizations (duration (ms),mean f0 (st rel 1 Hz), f0 range (st), RMS inten-sity). These acoustic features served as dependentvariables in the subsequent analyses. In a first step,it was then determined which participants were in-fluenced by different visibility conditions by calcu-lating a set of Linear Mixed Effect Models on alldependent variables, using the interaction of partici-pant and both visibility conditions (‘manual visibil-ity’, ‘facial visibility’) as fixed factors, and ‘item’and ‘dyad’ as random factors. For the subset ofparticipants who showed an interaction with visi-bility on any dependent variable, we then calcu-lated a series of Linear Mixed Effects Models, us-ing the above mentioned acoustic features as depen-dent variables, ‘relevance’, ‘predictability’, ‘manualvisibility’ and ‘facial visibility’ as fixed factors, and‘item’, ‘participant’ and ‘dyad’ as random factors.For a direct comparison of relevance and unpre-dictability accents, we coded a factor ‘accent con-trast’, containing the labels ‘important’ and ‘unpre-dictable’, but excluding the cases where accents areneither ‘unpredictable’ nor ‘relevant’ (mostly initialmoves), or both.3. RESULTS3.1. Influence of Visibility on Individual ParticipantsOut of the 39 remaining participant recordings, 37showed an interaction effect between facial visibilityor manual visibility on at least one of the examinedacoustic parameters. Data from these 37 participantswere entered into the analysis of potential effects ofvisibility and the prosodic expression of informationstructure.3.2. Information Structure EffectsBoth predictability and relevance have a signifi-cant influence on the duration of accented words,with relevant accents being significantly longer (β =15,SE = 5.36, t = 2.9) and predictable accents be-ing significantly shorter (β = −32,SE = 5.37, t =−6.0). When contrasting accents of unpredictabil-ity and accents of relevance, unpredictable accentsare significantly longer than relevant ones (β =13.8,SE = 6.32, t = 2.18, cf. Figure 3). These re-sults are in line with previous research on AmericanEnglish [14]. Also, f0 range is influenced both bypredictability and relevance, with relevant accentsbeing produced with a higher f0 excursion (β =0.69, SE=0.31, t=2.2) and predictable accents witha more compressed one (β = −0.9,SE = 0.31, t =−2.84). However, a direct comparison of unpre-dictable and relevant accents fails to show a signifi-cant distinction: Both unpredictability and relevanceincrease f0 range in a similar fashion. This differsfrom results for American English, where unpre-dictable accents showed a larger f0 excursion. Glob-ally, neither relevance nor predictability had an im-pact on RMS or Mean f0. Again, this finding differsfrom American English, where intensity was usedto mark relevance accents rather than unpredictableaccents.3.3. Visibility Condition EffectsVisibility conditions had a significant influenceon two acoustic parameters: If participants couldsee the interlocutor’s hands, they produced theirverbalized target moves significantly faster (β =−10,SE = 5.14, t = −2.0, cf. Figure 3). If partic-Figure 3: Accent duration distribution dependingon accent function (relevance vs. unpredictabilityaccent) and absence (left) or presence (right) ofmanual game moves.invisible moves visible movesrelevant unpredictable relevant unpredictable300600900Accent FunctionDuration (ms)Figure 4: Distribution of mean f0 in accents bothwith and without access to facial visibility.8090100110invisible face visible faceFacial VisibilityMean Pitch (st rel 1 Hz)ipants could see each other’s faces, they producedtheir target moves with a slightly higher average f0(β = 0.52,SE = 0.26, t = 2.0, cf. Figure 4). RMSintensity was not affected by visibility, and neitherwas f0 range.4. DISCUSSIONAccent functions are acoustically differentiated inGerman, but different from American English. Forthe latter, it was found that unpredictability accentsare longer in duration and produced with a higherf0 range, but that relevance accents are louder. Asin American English, German speakers produce un-predictable accents comparatively longer than rele-vant ones, but do not use f0 excursion to differentiatethem further. Also different from American English,German speakers do not use intensity cues to signalaccent function. As our analysis only looks at shal-low acoustic features and does not take into accountthe fine detail of f0 movements or accent types, werefrain from concluding that predictability and rel-evance are not differentiated using f0 cues in Ger-man. A more fine-grained analysis of f0 movementsis needed in the future.With respect to the interplay of different visibilityconditions and accent realization, we had hypothe-sized a comparatively stronger effect of manual visi-bility on the prosodic expression of accents. Indeed,this was supported by a decreased duration of ac-cented words given access to manual visibility. Ifinterlocutors have visible access to the informationthat is verbally transmitted, they invest less (dura-tional) effort into their accent realization. Interest-ingly, no other prosodic features of prominence ex-pression were affected, and f0 range stayed similar.Our findings thus provide some, but not full, sup-port for speech optimization theories. Our secondhypothesis was that visible access to the interlocu-tor’s facial expression would also reduce prosodiceffort, but less so. Here, we only found a verysmall positive impact of facial visibility on Mean f0,which could be interpreted as an increase in over-all prosodic “engagement”, possibly modulated bya strengthened interlocutor’s co-presence. Our hy-pothesized reduced prosodic effort under facial visi-bility conditions was not corroborated by the results.Maybe the most surprising result was the lack ofinfluence of different visibility conditions on RMSintensity. We interpret this in such a way that speak-ers react with intensity modulations mostly to noisein the transmission channel, in a classic Lombard-like fashion. Lacking visibility of an interlocutordoes not automatically create Lombard speech.5. REFERENCES[1] Boersma, P., Weenink, D. 2019. Praat: doing pho-netics by computer [computer program]. version6.0.49.[2] Féry, C., Kügler, F. 2008. Pitch accent scalingon given, new and focused constituents in German.Journal of Phonetics 36, 680 – 703.[3] Jannedy, S., Mendoza-Denton, N. 2005. Structur-ing information through gesture and intonation. In-terdisciplinary Studies on Information Structure 3,199–244.[4] Krahmer, E., Swerts, M. 2007. The effects of visualbeats on prosodic prominence: Acoustic analyses,auditory perception and visual production. Journalof Memory and Language 57, 396–414.[5] Lindblom, B. 1990. Explaining Phonetic Variation:A Sketch of the H&H Theory 403–439. KluwerAcademic Publishers.[6] Loehr, D. 2012. Temporal, structural, and prag-matic synchrony between intonation and gesture.Laboratory Phonology. Journal of the Associationfor Laboratory Phonology 3, 71–89.[7] Lombard, É. 1911. Le signe de l’élévation de lavoix. Annales des Maladies de l’Oreille et du Lar-ynx XXXVII(2), 101–109.[8] McGurk, H., McDonald, J. 1976. Hearing lips andseeing voices. Nature 264(746–748).[9] Munhall, K., Jones, J. A., Callan, D. E., Kuratate,T., Vatikiotis-Bateson, E. 2004. Visual prosody andspeech intelligibility – head movement improvesauditory speech perception. Psychological Science15(2).[10] Pierrehumbert, J., Hirschberg, J. 1990. The mean-ing of intonational contours in the interpretationof discourse. In: Cohen, P., Morgan, J., Pol-lack, M., (eds), Intentions in Communication. Cam-bridge MA: MIT Press 271–311.[11] Riester, A., Baumann, S. 2013. Focus triggers andfocus types from a corpus perspective. Dialogueand Discourse 4(2), 215–248.[12] Skopeteas, S., Féry, C. 2010. Effect of narrow focuson tonal realization in Georgian. Proceedings ofSpeech Prosody 2010 Chicago, Illinois.[13] Wagner, P., Malisz, Z., Kopp, S. 2014. Gesture andSpeech in Interaction: An Overview. Speech Com-munication 57(Special Iss.), 209–232.[14] Watson, D., Arnold, J., Tanenhaus, M. K. 2008.Tic tac TOE: Effects of predictability and impor-tance on acoustic prominence in language produc-tion. Cognition 106(3), 1548–1557.[15] Xu, Y. 1999. Effects of tone and focus on the for-mation and alignment of f0 contours. Journal ofPhonetics (27), 55– 105.

Does information-structural acoustic prosody change under different visibility conditions?

https://pub.uni-bielefeld.de/download/2933719/2936367

Does information-structural acoustic prosody change under different visibility conditions?

Abstract

Similar works

Full text

Available Versions

Publications at Bielefeld University