It was previously observed [1] that the order of presentation of paired stimuli influenced the number of different responses in same-different tasks in speech synthesis evaluation. This paper investigates this phenomenon within the context of cognitive psychology and demonstrates that, as the cognitive psychology literature suggests, there is an effect relating to the prototypicality of the stimulus. Index Terms: speech synthesis, evaluation, perception, Blizzard Challeng

Clark, Robert A. J.

Jacobsen, Thomas

Janska, Anna C

Schröger, Erich

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Asymmetries in the perception of synthesized speechCitation for published version:Janska, AC, Schröger, E, Jacobsen, T & Clark, RAJ 2012, Asymmetries in the perception of synthesizedspeech. in Proceedings Interspeech 2012, Portland. INTERSPEECH 2012 - 13th Annual Conference of theInternational Speech Communication Association, Portland, Oregon, United States, 9/09/12.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:Proceedings Interspeech 2012, PortlandGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019Asymmetries in the perception of synthesized speechAnna C. Janska1, Erich Schro¨ger2, Thomas Jacobsen3, Robert A. J. Clark41IMPRS NeuroCom, University of Leipzig, Germany2 University of Leipzig, Germany3 Helmut Schmidt University, Hamburg, Germany4CSTR, The University of Edinburgh, U.K.janska@rz.uni-leipzig.de, robert@cstr.ed.ac.ukAbstractIt was previously observed [1] that the order of presen-tation of paired stimuli influenced the number of differ-ent responses in same-different tasks in speech synthe-sis evaluation. This paper investigates this phenomenonwithin the context of cognitive psychology and demon-strates that, as the cognitive psychology literature sug-gests, there is an effect relating to the prototypicality ofthe stimulus.Index Terms: speech synthesis, evaluation, perception,Blizzard Challenge1. IntroductionWhen listeners are asked to decide whether two pairedstimuli sound equally natural, they are basically askedto position the stimuli in their mental space in relationto their abstract representation of naturalness [2]. Thenthese two distances are gauged and compared, to decidewhether two stimuli are equally good samples of thecategory naturalness. This task is not trivial and subjectto perceptual biases: Rosch [3] introduced the idea thatcategories are graded, i.e. that some category membersare more representative of a category than others, andthat discriminative ability is influenced by a stimulus’distance to the prototype or reference level1. Further-more, she found that stimuli close to the reference pointare assimiliated to the reference stimulus to a higherdegree than stimuli further away from it [4].Such warping of the perceptual space has been ob-served in a systematic manner in different modalities aswell as stimulus types: using linguistic hedges, Rosch[4] found that given a sentence such as “A is al-most a ”, participants would insert the less prototypi-cal item in the first, and the prototypical item in the sec-ond slot. This effect was found reliably across 6 stimu-lus sets. From this order effect Rosch deduced that therewas an asymmetrical relation between reference stimuliand somewhat deviant stimuli. These order effects were1In the following article, the two terms will be used interchangeablyabsent in neutral stimuli. In harmonies, Schellenbergfound that shifts from in-tune sequences to out-of-tunesequences were more noticable than vice versa [5]. Intone durations, Hellstro¨m found that repeated presen-tation of a reference tone could establish that tone as areference level; deviations from the reference tone lengthwere detected more easily if they followed the referencetone, rather than preceded it. Hellstro¨m argues that ordereffects are more than a bias, and not simply additive [6].If this order effect also influences the evaluationof synthesized speech, this has some strong implica-tions when employing pairwise comparisons in evalua-tion studies, such as multidimensional scaling: It is vitalthat in evaluations, the order of comparisons is carefullybalanced, as neglect of this can create a strong bias in thedata. This re-emphasizes the need for concise and ex-haustive testing and justifies the additional cost of timeand money this requires. Furthermore should findings ofsuch an asymmetry rekindle discussions about whetherand if so how aggregation across listeners as well asstimuli can be done to generate replicable and valid re-sults. It also raises the question: is any order effect smallenough that averaging across both orders is a sufficientway to deal with any effect? And whether anything canbe learned from the order effect itself?2. Modelling the order effect in theperception of paired stimuliHellstro¨m [7] accounts for the perceptual asymmetry inAX discrimination (i.e. same-different judgment) tasksby weighting the perception of the individual stimuli to bediscriminated, as they are positioned in mental space inreference to a prototypical representation. The magnitudeof the perceptual weights depends on a stimulus’ positioninside the stimulus pair :dab =(pXi=1[s1( ai   ri)  s2( bi   ri)]m)1/ms1 < s2(1)dba =(pXi=1[s1( bi   ri)  s2( ai   ri)]m)1/ms1 < s2(2)where dab is the perceived distance between the twoitems of the stimulus pair a-b, and dba is the perceiveddistance between the same two items, presented in re-verse order.  are stimulus representations: represen-tations with subscripts a and b refer to the memory traceof stimuli a and b.  r is the reference level, i.e. the ref-erence point in the category against which other stimuliare compared. i to p indicates the number of dimensions.s1 and s2 are sensation weights, for which the first is as-sumed to be smaller than the second, thus attributing asmaller weight to the distance between the first stimulusand the prototype than to the distance between the secondone and the prototype. m is the Minkowski constant [7].Thus, when the first stimulus is closer to the refer-ence level, the distance between the two stimuli appearssmaller than when the second stimulus is closer to the ref-erence level, albeit the acoustic difference remained con-stant. Furthermore should the asymmetries be larger forpairs in which both stimuli are fairly distant to the refer-ence level, than for pairs in which one stimulus more orless coincides with the reference level; as the distance ofboth stimuli to the reference level is larger, the rescalingdone by the sensation weights becomes more noticable.The current study applies this reasoning to a re-analysis of a subset of the Blizzard 2008 data [8]. Meanopinion scores (MOS) are used to guide the directionof hypotheses for which order effects are expected insame-different judgments after classifying stimuli intotwo groups: high and low scorers. If we consider pairsof stimuli where one stimuli is picked from each group,we can make predictions:The perceived distance between paired stimuli willbe larger if the higher scoring stimulus is presented firstcompared to when it is presented second; within the highscoring stimuli group, which consists of natural speechexamples and very good synthesized speech examples,the natural examples are still superior, and thus we canexpect the order effect to be larger for comparisons in-volving the good synthetic examples than those of naturalexamples.Figure 1 shows the expected relationships schemati-cally: As the distance between natural stimuli and lowranking synthesized speech is larger than the distance be-tween high and low ranked synthesized speech, compar-isons of the former kind will receive more different judg-ments than comparisons of the latter kind. This should begenerally true, regardless of the direction of comparison(see arrow lengths in Fig. 1).More different judgments should also be found if theless natural stimulus is presented in second, rather thanin first position, regardless of which kind of more naturalstimulus is presented first (cf. lengths of paired arrows inopposing directions). However, this asymmetry should bemore pronounced if both stimuli are synthesized than ifone stimulus is natural (cf. relative differences in lengthsof paired arrows).highlownaturalFigure 1: Hypotheses for the order effect in the percep-tion of synthesized speech; the base of the arrow is thefirst stimulus in a pair, the tip is the second3. MethodsWe are re-analysing a dataset that was collected as partof a previous study [9] to address asymmetries betweenorders of presentation with a guided hypothesis. Thedataset comprises of the evaluations from 30 native En-glish speakers who produced judgements on 10 stimuli,2 of which were natural speech (stimuli T1, B1) and 8 ofwhich where synthetic (T2–T5, B2–B5) covering a widerange of naturalness.For the set of stimuli, subjects provided both same-different judgements for each pair of stimuli in each pre-sentation orders, and Mean Opinion Scores (MOS) foreach of the stimuli. See [9] for the full experimental pro-cedure.4. Initial Re-analysisOrdinal MDS plots (using the Identity Euclidian distancemetric) were generated for the same-different responses,one for the lower, and one for the upper matrix triangle.The thus generated coordinates were overlayed in figure 2with the corresponding T1 points aligned as anchors toget a first impression of whether order of presentationplayed any role in the data.4.1. ResultsInitial inspection of the MDS graphs, shows that there aresome differences between the plots for each order.When ranking the stimuli according to their distanceto the natural stimulus T1, we already see that betweenthe two configurations, the ranks of T3 and B5 are in-terchanged. This demonstrates that testing stimulus pairsin only one order of presentation can have strong effectsFigure 2: The two MDS configurations, once on lower,once on upper triangle of the data matrix, and their dis-tance to T1Figure 3: Boxplots of MOS scores collected for the 10stimuli; grey boxes were not used in the asymmetry ana-lysis- even for such a low number of stimuli and despite thefact that we have a full matrix for every participant, andthe same stimulus set has been tested on all listeners.5. Further AnalysisMOSs are often recorded as conditional similarity data,which means that values cannot be compared directly be-tween subjects [10]. In order to allow comparisons be-tween listeners, MOS were transformed into z-scores forevery listener and then averaged across participants. Thusrelations between stimuli were maintained, even thoughthe ranges of the rating scales participants used, differed.In order to make clear predictions, the data were par-titioned into two groups (see Figure 3):• a high group: whose MOS were better than aver-age. As a minimum, a z-score of 0.25 was selected.• a low group: whoseMOSwere worse than average.As a maximum, a z-score of -0.25 was selected.This constraint excludes two stimuli, T2 and T4, astheir medians lie close to z = 0 . This constraint alsoensures that we have two distinct groups with no overlap.The high ranking stimuli T1, B1, B4, T5 were paired inturn with each of the low ranking stimuli B2, B3, B5,and T3 to test the hypothesis that pairs consisting of amember of the high ranking group presented first, and amember of the low ranking group second, will receivemore different judgments than the same stimuli presentedin the reverse order.Additionally, since B4 and T5, the synthetic stimuliin the high ranking group, were rated as less natural thanthe natural example stimuli in this group, we hypothesisethat the asymmetries found in pairs involving B4 and T5should be larger than those comparisons which involvestimuli B1 and T1.5.1. ResultsTo get a clearer picture of the effect order of presenta-tion has, we test our hypothesis using a McNemar test(which tests the difference between paired judgments) onthe individual same-different response data. This showsthat the presentation order of the more natural-soundingstimulus inside a stimulus pair has a significant effect onthe perceived similarity of a pair ( 2(1)=8.348, p<.01,the odds-ratio is 11.4). This indicates that the asymme-tries we have found between the two MDS configurationsabove are systematic, rather than noise, determined by theposition of the more typical items in stimulus pairs.To investigate how the selected stimulus pairs con-tribute to this order effect, the trials were pooled acrosslisteners for every contrast. To ensure that the asymmetryeffect is constituted by several stimulus pairings, ratherthan one which is very pronounced, a Wilcoxon signedrank test is run2. It takes the proportions of different judg-ments in stimulus pairs as input and conducts pairwisecomparisons between the two orders of presentation.It revealed that there are significantly more differentjudgments in pairs in which the more natural soundingstimulus came in first (Mdn=27), as opposed to in sec-ond position (Mdn=26.5), T=24M p<.05, r=-0.516. Thisshows that the effect is not constituted by a single stimu-lus pair, but overall supports the asymmetry hypothesis.Looking at the proportions of different judgments (seeFigure 4), the highest bars are the ones in B1 and T1 com-parisons, since the differences between these two naturalstimuli and the synthesized sentences were largest. Bluebars show contrasts, in which the more natural stimu-lus was presented first, while the reverse is true for redbars. So according to the order effect we expect thatblue bars are higher than red bars. Generalizing, this istrue. However, while this effect is very pronounced forcomparisons between synthesized sentences, this is not2As the Shapiro-Wilk test for the data was significant for the variableof directional change towards the prototype, a non-parametric test hadto be chosenB1−B2 B1−B5 B4−B2 B4−B5 T1−B2 T1−B5 T5−B2 T5−B500.20.40.60.81  low−highhigh−lowB1−B3 B1−T3 B4−B3 B4−T3 T1−B3 T1−T3 T5−B3 T5−T3Figure 4: Number of different judgments for stimuluspairs across 30 listenersindiscriminately true for natural stimuli: T1-B3 showsequal perception of naturalness, and in most B1 contraststhe effect is reversed! B1 comparisons clearly contradictthe hypothesis. While we assumed that the order effectwould be smaller for comparisons involving natural stim-uli than only synthesized ones, we still expected an effectin the same direction. As for the order effects in morenatural-sounding synthesized pairs B4 and T5, the ordereffect is present, and more pronounced than in the naturalstimuli, which supports the model of the order effect.6. DiscussionIt is an interesting finding to see that the presentationorder effect can be consistently found across listeners,rather than only in a repeated measurements in a withinsubjects design. This supports the assumption that na-tive speakers of English have a fairly similar idea of whatnatural sounds like. This is the precondition for testingthe naturalness of synthesized speech. The finding alsostresses the need to gather discrimination data in both di-rections when conducting MDS analysis.It is particularly interesting that while the order ef-fect was demonstrated robustly for pairs including thehigh ranking synthesized examples, it was not found forpairs involving natural recordings, especially in the caseof B1. Why our predictions do not hold true entirely forthe recordings of a natural voice, we can only hypothesizeat this point. It could be that we have insufficient data tosee this effect clearly, or it could be that the natural speechhas a different status to that of synthesized speech, whichboth results in the higher number of different responsesand the loss of the order effect. It is also worth consider-ing that these data were not originally collected with thiskind of investigation in mind, and thus several parameterswere left uncontrolled for: there was no control over theamount of time that elapsed between playing the first andthe second stimulus of a pair, for which an order effectmay be susceptible to [12]; neither were repeated playsof a stimulus recorded.A question raised for future work is as to whether theorder effect itself can be used to pin-point the location ordirection of prototypical, natural speech in an MDS de-rived space where there were no natural stimuli includedin the experiment.7. AcknowledgementsThe authors would like to thank The Blizzard Challengeorganisers and participants for providing the data usedfor this work. The research leading to these results hasreceived funding from the International Max Planck Re-search School Neuroscience of Communication and theEuropean Community’s Seventh Framework Programme(FP7/2007-2013) under grant agreement no. 287678.8. References[1] Janska, A.C., “Further Investigation of MDS as a Tool forEvaluation of Speech Quality of Synthesized Speech”, MScDissertation, The University of Edinburgh, 2009. Online:www.era.lib.ed.ac.uk/handle/1842/3624[2] Minda, J.P., Smith, J.D., “Prototype models of categorization: Ba-sic formulation, predictions, and limitations“, in Pothos, E.M.,and Willis, A.J. [Eds], Formal Approaches in Categorization, 40–64, Cambridge University Press, 2011.[3] Rosch, E., Mervis, C.B., “Family resemblances: Studies in the in-ternal structure of categories”, Cognitive Psychology, 7(4): 573–605, 1975.[4] Rosch, E., “Cognitive Reference Points”, Cognitive Psychology,7(4): 532–47, 1975.[5] Schellenberg, E. G., “Asymmetries in the discrimination of mu-sical intervals: Going out-of-tune is more noticeable than goingin-tune”, Music Perception, 19(2): 223–248, 2001.[6] Hellstro¨m, A., “Anatomy of stimulus comparison“, in J.S. Mon-ahan, S.M. Sheffert & J.T. Townsend [Eds], Fechner Day 2005,113–118. Proceedings of the 21st Annual Meeting of the Interna-tional Society for Psychophysics, Traverse City, Michigan, USA,October 19-22, 2005.[7] Hellstro¨m, A, ”Temporal asymmetry and ”magnet effect” in simi-larity and discrimination of prototypical and nonprototypical stim-uli: Consequences of differential sensation weighting”, in S.Mori, T. Miyaoka, & W. Wong [Eds], Fechner Day 2007, 283–288. Proceedings of the 23rd Annual Meeting of the InternationalSociety for Psychophysics. Tokyo: International Society for Psy-chophysics, 2007.[8] Karaiskos, V., King, S., Clark, R., Mayo, C., “The Blizzard Chal-lenge 2008”. URL: http://festvox.org/blizzard/bc2008/summaryBlizzard2008.pdf, 2008[9] Janska, A.C., Clark, R.A.J., “Further exploration of the possibili-ties and pitfalls of multidimensional scaling as a tool for the evalu-ation of the quality of synthesized speech”. The 7th ISCA Tutorialand Research Workshop on Speech Synthesis: 142–147, 2010[10] Coxon, A., Jackson, J.E., Davies, P.M., Smith H.V., Sachs, L.,Schmee, J., “User’s guide to multidimensional scaling”, Heine-man Education books, 1982.[11] Janska, A.C, Clark, R.A.J., “Native and non-native speaker judge-ments on the quality of synthesized speech”. Proc. Interspeech2010: 1121–1124, 2010.[12] Hellstro¨m, A., Rammsayer, T.H., “Effects of time-order, inter-stimulus interval, and feedback in duration discrimination of noisebursts in the 50- and 1000-ms ranges”, Acta Psychologica, 116:1–20, 2004.

     Edinburgh Research Explorer                                      Asymmetries in the perception of synthesized speechCitation for published version:Janska, AC, Schröger, E, Jacobsen, T & Clark, RAJ 2012, Asymmetries in the perception of synthesizedspeech. in Proceedings Interspeech 2012, Portland. INTERSPEECH 2012 - 13th Annual Conference of theInternational Speech Communication Association, Portland, Oregon, United States, 9-13 September.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:Proceedings Interspeech 2012, PortlandGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 28. Apr. 2017Asymmetries in the perception of synthesized speechAnna C. Janska1, Erich Schro¨ger2, Thomas Jacobsen3, Robert A. J. Clark41IMPRS NeuroCom, University of Leipzig, Germany2 University of Leipzig, Germany3 Helmut Schmidt University, Hamburg, Germany4CSTR, The University of Edinburgh, U.K.janska@rz.uni-leipzig.de, robert@cstr.ed.ac.ukAbstractIt was previously observed [1] that the order of presen-tation of paired stimuli influenced the number of differ-ent responses in same-different tasks in speech synthe-sis evaluation. This paper investigates this phenomenonwithin the context of cognitive psychology and demon-strates that, as the cognitive psychology literature sug-gests, there is an effect relating to the prototypicality ofthe stimulus.Index Terms: speech synthesis, evaluation, perception,Blizzard Challenge1. IntroductionWhen listeners are asked to decide whether two pairedstimuli sound equally natural, they are basically askedto position the stimuli in their mental space in relationto their abstract representation of naturalness [2]. Thenthese two distances are gauged and compared, to decidewhether two stimuli are equally good samples of thecategory naturalness. This task is not trivial and subjectto perceptual biases: Rosch [3] introduced the idea thatcategories are graded, i.e. that some category membersare more representative of a category than others, andthat discriminative ability is influenced by a stimulus’distance to the prototype or reference level1. Further-more, she found that stimuli close to the reference pointare assimiliated to the reference stimulus to a higherdegree than stimuli further away from it [4].Such warping of the perceptual space has been ob-served in a systematic manner in different modalities aswell as stimulus types: using linguistic hedges, Rosch[4] found that given a sentence such as “A is al-most a ”, participants would insert the less prototypi-cal item in the first, and the prototypical item in the sec-ond slot. This effect was found reliably across 6 stimu-lus sets. From this order effect Rosch deduced that therewas an asymmetrical relation between reference stimuliand somewhat deviant stimuli. These order effects were1In the following article, the two terms will be used interchangeablyabsent in neutral stimuli. In harmonies, Schellenbergfound that shifts from in-tune sequences to out-of-tunesequences were more noticable than vice versa [5]. Intone durations, Hellstro¨m found that repeated presen-tation of a reference tone could establish that tone as areference level; deviations from the reference tone lengthwere detected more easily if they followed the referencetone, rather than preceded it. Hellstro¨m argues that ordereffects are more than a bias, and not simply additive [6].If this order effect also influences the evaluationof synthesized speech, this has some strong implica-tions when employing pairwise comparisons in evalua-tion studies, such as multidimensional scaling: It is vitalthat in evaluations, the order of comparisons is carefullybalanced, as neglect of this can create a strong bias in thedata. This re-emphasizes the need for concise and ex-haustive testing and justifies the additional cost of timeand money this requires. Furthermore should findings ofsuch an asymmetry rekindle discussions about whetherand if so how aggregation across listeners as well asstimuli can be done to generate replicable and valid re-sults. It also raises the question: is any order effect smallenough that averaging across both orders is a sufficientway to deal with any effect? And whether anything canbe learned from the order effect itself?2. Modelling the order effect in theperception of paired stimuliHellstro¨m [7] accounts for the perceptual asymmetry inAX discrimination (i.e. same-different judgment) tasksby weighting the perception of the individual stimuli to bediscriminated, as they are positioned in mental space inreference to a prototypical representation. The magnitudeof the perceptual weights depends on a stimulus’ positioninside the stimulus pair :dab =(pXi=1[s1( ai   ri)  s2( bi   ri)]m)1/ms1 < s2(1)dba =(pXi=1[s1( bi   ri)  s2( ai   ri)]m)1/ms1 < s2(2)where dab is the perceived distance between the twoitems of the stimulus pair a-b, and dba is the perceiveddistance between the same two items, presented in re-verse order.  are stimulus representations: represen-tations with subscripts a and b refer to the memory traceof stimuli a and b.  r is the reference level, i.e. the ref-erence point in the category against which other stimuliare compared. i to p indicates the number of dimensions.s1 and s2 are sensation weights, for which the first is as-sumed to be smaller than the second, thus attributing asmaller weight to the distance between the first stimulusand the prototype than to the distance between the secondone and the prototype. m is the Minkowski constant [7].Thus, when the first stimulus is closer to the refer-ence level, the distance between the two stimuli appearssmaller than when the second stimulus is closer to the ref-erence level, albeit the acoustic difference remained con-stant. Furthermore should the asymmetries be larger forpairs in which both stimuli are fairly distant to the refer-ence level, than for pairs in which one stimulus more orless coincides with the reference level; as the distance ofboth stimuli to the reference level is larger, the rescalingdone by the sensation weights becomes more noticable.The current study applies this reasoning to a re-analysis of a subset of the Blizzard 2008 data [8]. Meanopinion scores (MOS) are used to guide the directionof hypotheses for which order effects are expected insame-different judgments after classifying stimuli intotwo groups: high and low scorers. If we consider pairsof stimuli where one stimuli is picked from each group,we can make predictions:The perceived distance between paired stimuli willbe larger if the higher scoring stimulus is presented firstcompared to when it is presented second; within the highscoring stimuli group, which consists of natural speechexamples and very good synthesized speech examples,the natural examples are still superior, and thus we canexpect the order effect to be larger for comparisons in-volving the good synthetic examples than those of naturalexamples.Figure 1 shows the expected relationships schemati-cally: As the distance between natural stimuli and lowranking synthesized speech is larger than the distance be-tween high and low ranked synthesized speech, compar-isons of the former kind will receive more different judg-ments than comparisons of the latter kind. This should begenerally true, regardless of the direction of comparison(see arrow lengths in Fig. 1).More different judgments should also be found if theless natural stimulus is presented in second, rather thanin first position, regardless of which kind of more naturalstimulus is presented first (cf. lengths of paired arrows inopposing directions). However, this asymmetry should bemore pronounced if both stimuli are synthesized than ifone stimulus is natural (cf. relative differences in lengthsof paired arrows).highlownaturalFigure 1: Hypotheses for the order effect in the percep-tion of synthesized speech; the base of the arrow is thefirst stimulus in a pair, the tip is the second3. MethodsWe are re-analysing a dataset that was collected as partof a previous study [9] to address asymmetries betweenorders of presentation with a guided hypothesis. Thedataset comprises of the evaluations from 30 native En-glish speakers who produced judgements on 10 stimuli,2 of which were natural speech (stimuli T1, B1) and 8 ofwhich where synthetic (T2–T5, B2–B5) covering a widerange of naturalness.For the set of stimuli, subjects provided both same-different judgements for each pair of stimuli in each pre-sentation orders, and Mean Opinion Scores (MOS) foreach of the stimuli. See [9] for the full experimental pro-cedure.4. Initial Re-analysisOrdinal MDS plots (using the Identity Euclidian distancemetric) were generated for the same-different responses,one for the lower, and one for the upper matrix triangle.The thus generated coordinates were overlayed in figure 2with the corresponding T1 points aligned as anchors toget a first impression of whether order of presentationplayed any role in the data.4.1. ResultsInitial inspection of the MDS graphs, shows that there aresome differences between the plots for each order.When ranking the stimuli according to their distanceto the natural stimulus T1, we already see that betweenthe two configurations, the ranks of T3 and B5 are in-terchanged. This demonstrates that testing stimulus pairsin only one order of presentation can have strong effectsFigure 2: The two MDS configurations, once on lower,once on upper triangle of the data matrix, and their dis-tance to T1Figure 3: Boxplots of MOS scores collected for the 10stimuli; grey boxes were not used in the asymmetry ana-lysis- even for such a low number of stimuli and despite thefact that we have a full matrix for every participant, andthe same stimulus set has been tested on all listeners.5. Further AnalysisMOSs are often recorded as conditional similarity data,which means that values cannot be compared directly be-tween subjects [10]. In order to allow comparisons be-tween listeners, MOS were transformed into z-scores forevery listener and then averaged across participants. Thusrelations between stimuli were maintained, even thoughthe ranges of the rating scales participants used, differed.In order to make clear predictions, the data were par-titioned into two groups (see Figure 3):• a high group: whose MOS were better than aver-age. As a minimum, a z-score of 0.25 was selected.• a low group: whoseMOSwere worse than average.As a maximum, a z-score of -0.25 was selected.This constraint excludes two stimuli, T2 and T4, astheir medians lie close to z = 0 . This constraint alsoensures that we have two distinct groups with no overlap.The high ranking stimuli T1, B1, B4, T5 were paired inturn with each of the low ranking stimuli B2, B3, B5,and T3 to test the hypothesis that pairs consisting of amember of the high ranking group presented first, and amember of the low ranking group second, will receivemore different judgments than the same stimuli presentedin the reverse order.Additionally, since B4 and T5, the synthetic stimuliin the high ranking group, were rated as less natural thanthe natural example stimuli in this group, we hypothesisethat the asymmetries found in pairs involving B4 and T5should be larger than those comparisons which involvestimuli B1 and T1.5.1. ResultsTo get a clearer picture of the effect order of presenta-tion has, we test our hypothesis using a McNemar test(which tests the difference between paired judgments) onthe individual same-different response data. This showsthat the presentation order of the more natural-soundingstimulus inside a stimulus pair has a significant effect onthe perceived similarity of a pair ( 2(1)=8.348, p<.01,the odds-ratio is 11.4). This indicates that the asymme-tries we have found between the two MDS configurationsabove are systematic, rather than noise, determined by theposition of the more typical items in stimulus pairs.To investigate how the selected stimulus pairs con-tribute to this order effect, the trials were pooled acrosslisteners for every contrast. To ensure that the asymmetryeffect is constituted by several stimulus pairings, ratherthan one which is very pronounced, a Wilcoxon signedrank test is run2. It takes the proportions of different judg-ments in stimulus pairs as input and conducts pairwisecomparisons between the two orders of presentation.It revealed that there are significantly more differentjudgments in pairs in which the more natural soundingstimulus came in first (Mdn=27), as opposed to in sec-ond position (Mdn=26.5), T=24M p<.05, r=-0.516. Thisshows that the effect is not constituted by a single stimu-lus pair, but overall supports the asymmetry hypothesis.Looking at the proportions of different judgments (seeFigure 4), the highest bars are the ones in B1 and T1 com-parisons, since the differences between these two naturalstimuli and the synthesized sentences were largest. Bluebars show contrasts, in which the more natural stimu-lus was presented first, while the reverse is true for redbars. So according to the order effect we expect thatblue bars are higher than red bars. Generalizing, this istrue. However, while this effect is very pronounced forcomparisons between synthesized sentences, this is not2As the Shapiro-Wilk test for the data was significant for the variableof directional change towards the prototype, a non-parametric test hadto be chosenB1−B2 B1−B5 B4−B2 B4−B5 T1−B2 T1−B5 T5−B2 T5−B500.20.40.60.81  low−highhigh−lowB1−B3 B1−T3 B4−B3 B4−T3 T1−B3 T1−T3 T5−B3 T5−T3Figure 4: Number of different judgments for stimuluspairs across 30 listenersindiscriminately true for natural stimuli: T1-B3 showsequal perception of naturalness, and in most B1 contraststhe effect is reversed! B1 comparisons clearly contradictthe hypothesis. While we assumed that the order effectwould be smaller for comparisons involving natural stim-uli than only synthesized ones, we still expected an effectin the same direction. As for the order effects in morenatural-sounding synthesized pairs B4 and T5, the ordereffect is present, and more pronounced than in the naturalstimuli, which supports the model of the order effect.6. DiscussionIt is an interesting finding to see that the presentationorder effect can be consistently found across listeners,rather than only in a repeated measurements in a withinsubjects design. This supports the assumption that na-tive speakers of English have a fairly similar idea of whatnatural sounds like. This is the precondition for testingthe naturalness of synthesized speech. The finding alsostresses the need to gather discrimination data in both di-rections when conducting MDS analysis.It is particularly interesting that while the order ef-fect was demonstrated robustly for pairs including thehigh ranking synthesized examples, it was not found forpairs involving natural recordings, especially in the caseof B1. Why our predictions do not hold true entirely forthe recordings of a natural voice, we can only hypothesizeat this point. It could be that we have insufficient data tosee this effect clearly, or it could be that the natural speechhas a different status to that of synthesized speech, whichboth results in the higher number of different responsesand the loss of the order effect. It is also worth consider-ing that these data were not originally collected with thiskind of investigation in mind, and thus several parameterswere left uncontrolled for: there was no control over theamount of time that elapsed between playing the first andthe second stimulus of a pair, for which an order effectmay be susceptible to [12]; neither were repeated playsof a stimulus recorded.A question raised for future work is as to whether theorder effect itself can be used to pin-point the location ordirection of prototypical, natural speech in an MDS de-rived space where there were no natural stimuli includedin the experiment.7. AcknowledgementsThe authors would like to thank The Blizzard Challengeorganisers and participants for providing the data usedfor this work. The research leading to these results hasreceived funding from the International Max Planck Re-search School Neuroscience of Communication and theEuropean Community’s Seventh Framework Programme(FP7/2007-2013) under grant agreement no. 287678.8. References[1] Janska, A.C., “Further Investigation of MDS as a Tool forEvaluation of Speech Quality of Synthesized Speech”, MScDissertation, The University of Edinburgh, 2009. Online:www.era.lib.ed.ac.uk/handle/1842/3624[2] Minda, J.P., Smith, J.D., “Prototype models of categorization: Ba-sic formulation, predictions, and limitations“, in Pothos, E.M.,and Willis, A.J. [Eds], Formal Approaches in Categorization, 40–64, Cambridge University Press, 2011.[3] Rosch, E., Mervis, C.B., “Family resemblances: Studies in the in-ternal structure of categories”, Cognitive Psychology, 7(4): 573–605, 1975.[4] Rosch, E., “Cognitive Reference Points”, Cognitive Psychology,7(4): 532–47, 1975.[5] Schellenberg, E. G., “Asymmetries in the discrimination of mu-sical intervals: Going out-of-tune is more noticeable than goingin-tune”, Music Perception, 19(2): 223–248, 2001.[6] Hellstro¨m, A., “Anatomy of stimulus comparison“, in J.S. Mon-ahan, S.M. Sheffert & J.T. Townsend [Eds], Fechner Day 2005,113–118. Proceedings of the 21st Annual Meeting of the Interna-tional Society for Psychophysics, Traverse City, Michigan, USA,October 19-22, 2005.[7] Hellstro¨m, A, ”Temporal asymmetry and ”magnet effect” in simi-larity and discrimination of prototypical and nonprototypical stim-uli: Consequences of differential sensation weighting”, in S.Mori, T. Miyaoka, & W. Wong [Eds], Fechner Day 2007, 283–288. Proceedings of the 23rd Annual Meeting of the InternationalSociety for Psychophysics. Tokyo: International Society for Psy-chophysics, 2007.[8] Karaiskos, V., King, S., Clark, R., Mayo, C., “The Blizzard Chal-lenge 2008”. URL: http://festvox.org/blizzard/bc2008/summaryBlizzard2008.pdf, 2008[9] Janska, A.C., Clark, R.A.J., “Further exploration of the possibili-ties and pitfalls of multidimensional scaling as a tool for the evalu-ation of the quality of synthesized speech”. The 7th ISCA Tutorialand Research Workshop on Speech Synthesis: 142–147, 2010[10] Coxon, A., Jackson, J.E., Davies, P.M., Smith H.V., Sachs, L.,Schmee, J., “User’s guide to multidimensional scaling”, Heine-man Education books, 1982.[11] Janska, A.C, Clark, R.A.J., “Native and non-native speaker judge-ments on the quality of synthesized speech”. Proc. Interspeech2010: 1121–1124, 2010.[12] Hellstro¨m, A., Rammsayer, T.H., “Effects of time-order, inter-stimulus interval, and feedback in duration discrimination of noisebursts in the 50- and 1000-ms ranges”, Acta Psychologica, 116:1–20, 2004.

https://www.pure.ed.ac.uk/ws/files/4888301/janskaetal_IS_2012.pdf

Asymmetries in the perception of synthesized speech

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Explorer