This paper highlights on a method that provides a new
prosodic feature called ‘F0 reliability field’ based on a reliability
function of the fundamental frequency (F0). The
proposed method does not employ any correction process
for F0 estimation errors that occur during automatic F0
extraction. By applying this feature as a score function
for prosodic analyses like prosodic structure estimation
or superpositional modeling of prosodic commands, these
prosodic information could be acquired with higher accuracy.
The feature has been applied to ‘F0 template matching
method’, which detects accent phrase boundaries in
Japanese continuous speech. The experimental results
show that compared to the conventional F0 contour, the
proposed feature overcomes the harmful influence caused
by F0 errors

Nakai, Mitsuru

Shimodaira, Hiroshi

Edinburgh Research Archive

ON REPRESENTATION OF FUNDAMENTAL FREQUENCY OF SPEECH FOR PROSODYANALYSIS USING RELIABILITY FUNCTIONMitsuru NAKAI and Hiroshi SHIMODAIRAJapan Advanced Institute of Science and Technology, Hokuriku1-1 Asahidai, Tatsunokuchi, Nomi, Ishikawa, 923-12 JapanABSTRACTThis paper highlights on a method that provides a newprosodic feature called ‘F0 reliability field’ based on a re-liability function of the fundamental frequency (F0). Theproposed method does not employ any correction processfor F0 estimation errors that occur during automaticF0extraction. By applying this feature as a score functionfor prosodic analyses like prosodic structure estimationor superpositional modeling of prosodic commands, theseprosodic information could be acquired with higher accu-racy. The feature has been applied to ‘F0 template match-ing method’, which detects accent phrase boundaries inJapanese continuous speech. The experimental resultsshow that compared to the conventionalF0 contour, theproposed feature overcomes the harmful influence causedby F0 errors.1. INTRODUCTIONProsody is a very important information for speech un-derstanding. Several researches have been performed forobtaining prosodic information and they are reported inboth of speech synthesis and speech recognition field. Wehave also proposed an automatic scheme called ‘F0 tem-plate matching method[1]’ for estimating prosodic phraseboundaries of Japanese continuous speech. It is largelyhoped that these kind of prosodic information will be use-ful for constructing a high-performanced speech recogni-tion system with low-costed CPU power.Among several prosodic features,F0 contour (pitch pat-tern) has been widely used for prosodic analysis and manyPitch Determination Algorithms (PDA) have been pro-posed. However, PDAs tend to yield some gross errors,such as harmonic errors of double pitch or half pitch, anderror correction is one of the laborious postprocessing taskin any automatic PDA.In the present approach, our segmentation system isbased on a pattern matching technique that employs somedistance measure between reference templates of typicalaccentF0 pattern and an observedF0 contour for whichthe prosodic boundaries are unknown. Therefore, the pre-viously mentioned errors can largely effect the boundarysegmentation results as the number of boundary insertionincrease. However, this problem is inevitable as long as0.5 0.51 0.52 0.53 0.54 (sec)0 6000 (Hz)0 0.01 0.02 (sec)0 0.01 0.02 (sec)0 6000 (Hz)0 6000 (Hz)0 6000 (Hz)0 0.01 0.02 (sec)X(t): speech wave?FFTIN (λ): power spectrum?IFFTv(τ): correlation function? lag window:wt(τ)¾ṽ(τ)?FFTĨN (λ): smoothedpower spectrum?IN (λ)/ĨN (λ)?¾ spectrum bandfilter: wf (λ)?IFFTY (τ)Fig. 1: A process ofF0 reliability analysis based on lag-window method ofF0 determination.the F0 value is fixed to some unique value per temporalanalyzing frame. In the present approach, this problemhas been thoroughly investigated and an‘F0 reliabilityfield’ has been designed in which there is no process ofdecidingF0 value distinctly.2. ANALYSIS OF F0 AND ∆F0RELIABILITY FIELDEuclidean distance is one of the most widely used measurefor pattern matching technique. The proposed prosodicsegmentation system is based on Dynamic Programming(DP) method that employs Euclidean distance. The dis-tance between twoF0 values at timet is defined byd(pt, p̄t) = (pt − p̄t)2, (1)wherep̄t is a referenceln F0 value andpt is an observedln F0 value. This distortion function has a continuous dis-tribution on the frequency axis. However, as the detectedF0 valuept sometimes has an error of half pitch or dou-ble pitch, the distortion does not vary monotonously onthe temporal sequence. In the present approach, theF0reliability field does not have any process forF0 determi-nation and therefore, it is not necessary to consider suchproblems.Fig.1 shows a process ofF0 reliability analysis basedon lag-window method[2] which is one of PDAs. The de-sirable smoothed functionY (τ) can be obtained by in-corporating a narrow spectrum band filter[3]. We haveapplied rectangle band filter as a window function in pre-vious works, but here we use the Hanning windowwf (λ)on the frequency domain.By using a temporal sequence of the functionY (τ), theF0 reliability field can be represented as shown in Fig.2.Here, the horizontal axis represents the logarithmic fre-quencyln(1/τ) and it can be seen that the contours ofthe harmonic peak lie in a fixed interval. This field is de-fined as a distribution of the formS(t, p), wheret andp stand for time and the logarithmic value ofF0 respec-tively. Here, similar to the characteristic of lag-windowmethod, the maximum value of the fieldS(t, p) is nor-malized to 1.0. If we determine the frequency value bythe following equation,ft = arg maxpS(t, p), (2)thenft would be the exactF0 value at timet and the al-gorithm would become equivalent to PDA.We have also defined∆F0 as a regression coefficientof ln F0. Similarly, ∆F0 vector can be represented as adirection along the ridge of anF0 reliability field and it isshown in Fig.3(a). If we assume that the vector(vt, vp)lies in the direction of the maximum slope on the pointlnF0 with valuep0 at timet0, then the component of thevector along the temporal direction can be represented bythe following equation,vt =M∑i=−Mi 6=0N∑j=−Nwt(ti, pj)(S(ti, pj)− S(t0, pj)ti − t0),(3)and the component of the vector along the frequency di-rection can be represented by the following equation.vp =M∑i=−MN∑j=−Nj 6=0wp(ti, pj)(S(ti, pj)− S(ti, p0)pj − p0).(4)Here, M and N are the window width,wt and wp areweighting function for each direction. In Fig.3(b) the∆F0 vector is represented by a vector that is orthogonaloF0 (log scale)Time (sec.)50 100 200 300 4000.00.10.20.30.40.5Fig. 2: An example ofF0 reliability filed.0.0 0.1 0.2 0.3 0.4 0.550100200300Time (sec)Frequency (Hz)(a) Observation ofF0 reliability pattern. The radius ofthe circle represents the reliability measure.0.0 0.1 0.2 0.3 0.4 0.550100200300Time (sec)Frequency (Hz)(b) A vector field ofF0 reliability pattern.Fig. 3: F0 reliability pattern and its∆F0 vector field.odelta F0Time (sec.)-0.05 0.0 0.050.00.10.20.30.40.5Fig. 4: An example of∆F0 reliability filed. The maxi-mum value is normalized to 1.0.to (vt, vp) and it has an angle ofa(t,p) = −vt/vp with thetemporal axis. Fig.4 shows an example of∆F0 reliabilityfield which is calculated by the summation of Gaussiandistribution per each temporal frame:∑p√v2t + v2p N(a(t,p), σ2). (5)3. PROSODIC SEGMENTATION USINGRELIABILITY FIELDThe proposed prosodic segmentation system[1] has twophases; training phase and segmentation phase, and the re-liability field is only used during the segmentation phase.During training, a set ofK templates{R0, · · · , RK−1}are created from a large number of accentF0 patterns us-ing k-means clustering algorithm, and they are approxi-mated semi-automatically by the commands of the super-positional model[4]. On the other hand, the segmenta-tion phase is performed automatically by One-Stage DPmatching between the referenceF0 templates and the tar-get F0 reliability field. The location of acquired refer-ence template connection boundary is considered to be thephrase boundary. The N-best sequences ofF0 templatescan also be searched by using the criterion of theN mostscores. The matching score between the reference tem-plate Ri = (p̄1, · · · , p̄n) and an input speech started attime ts is defined asDi =n∑t=1S(ts + t, p̄t), (6)whereS(t, p) is a reliability field ofF0 (or ∆F0). If thematching template forms the sequenceRc1 ⊕Rc2 ⊕ · · · ⊕Rci ⊕ · · · ⊕ RcM (0 ≤ ci ≤ K − 1), then the score issummed up toD =M∑i=1Dci + γM−1∑i=1ln P (ci+1|ci) (7)whereP (ci+1|ci) is the transition probability from thetemplateRci to the templateRci+1 andγ is the strengthfactor of their bigram constraints. Furthermore, ifS(t, p)is replaced to−d(pt, p̄t) in Eq.(1), then the algorithmbecomes equivalent to the conventional segmentationmethod which is based on Euclidean measure.4. EVALUATION OF PROSODIC FEATURESFor the evaluation of prosodic features, prosodic seg-mentation accuracy has been used under the environmentof 1-best search forF0 template sequence. The speechdatabase used in the evaluation test is the ATR’s contin-uous speech database of phoneme balanced 503 Japanesesentences uttered by each of 3 male speakers and 1 femalespeaker. Out of them, a total of 575 utterances from 3 malespeakers (MHT, MSH, MTK) is used for constructing 8F0 templates and the bigram probabilities between thetemplates were estimated. Automatic prosodic segmenta-tion was performed for each 50 sentences from the speakerMHT (male) and the speaker FKN (female), which are dif-ferent in contents from the training sentences.The result is shown in Fig.5 and Fig.6. They-axis is thecorrectly detected rate which is defined bycorrect rate=# correctly detected boundaries# accent boundariesand thex-axis is the boundary insertion error rate per ac-cent phrase:insertion rate=# incorrectly detected boundaries# accent phrases.Here, we treat the detected boundaries located within 100ms from the hand labeled boundaries as the correct one.Varying the bigram strengthγ, we can reduce many inser-tion errors while the correct detected rate decrease a little.Apart from the proposed reliability field (‘©’ in figure),the following two features have been used as the targetpattern for DP matching. One is an automatically detectedF0 contour using PDA and the postprocessing ofF0 errorcorrection is not applied (‘+’ in figure). The other is anidealF0 contour produced byF0 generating function[4],and it is regarded as the best case with noF0 errors (‘4’in figure).In reality, it is not possible to detect all of the prosodicboundaries, because the perception of the same depend onthe listener and the hand labeled boundaries used duringthe experiments are not always correct. Therefore, 70%correct rate with less than 25 % insertion rate is consid-ered to be sufficiently efficient for prosodic segmentationaccuracy of Japanese speech. It is also reported in theexperimental results of Takahashi and Matsunaga[5], thatthese numerical values are similar to the human hearingability for distinguishing accent segments.It can be seen that the segmentation accuracy using thereliability field (4) is better than the conventionalF0 con-tour (+) and the result does not depend onF0 or ∆F0, andmale or female speakers. In the experiments of speakerMHT, the segmentation accuracy came up to 70 % of cor-rect rate with low insertion error rate, and it is close to theaccuracy of idealF0 contour which has noF0 estimationerror. On the other hand, due to severalF0 extraction er-rors, the segmentation accuracy of speaker FKN has beenalways lower than the other speakers. However, in case ofher utterance, we can see the improvement of segmenta-tion accuracy in Fig.6. It is concluded that the reliabilityfield is enable to overcome the harmful influences causedby F0 errors.5. CONCLUSIONA new prosodic feature‘F0 reliability field’ has been pro-posed. The analysis of this feature is very simple and it isbasically the same as the conventionalF0 determinationInsertion rate (%)Correct rate (%)0 20 40 60020406080(a) Segmentation accuracy withF0 reliability fieldInsertion rate (%)Correct rate (%)0 20 40 60020406080(b) Segmentation accuracy with∆F0 reliability fieldFig. 5: A relationship of correctly detected rate and boundary insertion rate for speaker MHT. ‘+’ is using an automaticallydetectedF0 contour, ‘4’ is using an idealF0 contour withoutF0 errors, and ‘©’ is using a reliability field.Insertion rate (%)Correct rate (%)0 20 40 60020406080(a) Segmentation accuracy withF0 reliability fieldInsertion rate (%)Correct rate (%)0 20 40 60020406080(b) Segmentation accuracy with∆F0 reliability fieldFig. 6: A relationship of correctly detected rate and boundary insertion rate for speaker FKN. ‘+’ is using an automaticallydetectedF0 contour, and ‘©’ is using a reliability field.algorithm, like lag-window method. In case of the pro-posed reliability field feature, it is not necessary to cor-rect theF0 extraction errors due to the cause of focusingon a specificF0 decision value. It has been also shownthrough experiments using prosodic segmentation, that theproposed feature is effective for prosodic analysis. Thenext step would be to apply this feature for other prosodicinformation analysis, like prosodic structure estimation orsuperpositional modeling of prosodic commands.REFERENCES[1] M. Nakai and H. Singer and Y. Sagisaka andH. Shimodaira: “Automatic Prosodic Segmentationby F0 Clustering Using Superpositional Modeling”,ICASSP-95, pp.624–627, (1995).[2] S. Sagayama and S. Furui: “A technique for pitch ex-traction by the lag-window method”, Proc. Conf. IE-ICE, 1235 (1978) (in Japanese).[3] H. Shimodaira and M. Nakai: “Robust Pitch Detec-tion by Narrow Band Spectrum Analysis”,ICSLP-92,pp.1597–1600, (1992).[4] H. Fujisaki and H. Kawai: “Realization of Lin-guistic Information in the Voice Fundamental Fre-quency Contour of the Spoken Japanese”,ICASSP-88, pp.663–666, (1988).[5] S. Takahashi and S. Matsunaga: “Stochastic ProsodyModeling for Accent Phrase Boundary Detection inContinuous Speech”, In Papers of Technical Group ofSpeech, IEICE, SP90-71 (1990) (in Japanese).

On Representation of Fundamental Frequency of Speech for Prosody Analysis Using Reliability Function.

A technique for pitch extraction by the lag-window method”,

Automatic Prosodic Segmentation by F0 Clustering Using Superpositional Modeling”, ICASSP-95,

Realization of Linguistic Information

Robust Pitch Detection by Narrow Band Spectrum Analysis”, ICSLP-92,

Stochastic Prosody Modeling for Accent Phrase Boundary Detection in Continuous Speech”,

https://era.ed.ac.uk/bitstream/1842/1226/1/Nakai1997Eurospeech.pdf

On Representation of Fundamental Frequency of Speech for Prosody Analysis Using Reliability Function.

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Archive