Geutner, Petra

Scheytt, Peter

Waibel, Alex

English

KITopen

SERBO-CROATIAN LVCSR ON THE DICTATION ANDBROADCAST NEWS DOMAINPeter Scheytt1,2, Petra Geutner2, Alex Waibel1,2Interactive Systems Laboratories1Carnegie Mellon University, Pittsburgh PA, USA2University of Karlsruhe, Karlsruhe, Germanyscheytt@cs.cmu.edu, pgeutner@ira.uka.de, waibel@cs.cmu.eduABSTRACTThis paper describes the development of a Serbo-Croatiandictation and broadcast news speech recognizer. The intention isto generate an automatic text transcription of a news show,which will be submitted to a multilingual Informedia database.We outline the complete system development process using theJanusRTk, beginning with data collection, design and training ofparameters, tuning and evaluation. We report on generalrecognition techniques like segmentation, adaptation andlanguage model interpolation, as well as language specificproblems, e.g. high OOV rate due to inflected word forms. Weshow that even a low amount of acoustic training data,combined with Web based interpolated language models, issufficient to build up a fairly reliable automatic newstranscription system, which yields a performance of 36.0% WE.1. INTRODUCTIONThe goal of our work was to develop a speech recognizer forSerbo-Croatian broadcast news. The automatic transcriptiongenerated by our system will be submitted to a multilingualInformedia database. Thus adding not only diversity ofinformation, but also the possibility of trans-linguistic queriesand multiple language document retrieval. First we show thedevelopment of a Serbo-Croatian dictation system for read newstext. We outline the data collection, show how we determinedthe speech recognizer's base parameters for this new language:Phones and phone classes, pronunciation dictionary and Webbased language models. Even with a rather low amount ofacoustic data and in short time, we were able to train a quitestable system, which yielded a performance of 28.8% WE. Thedictation recognizer served as the baseline for our broadcastnews system, which we developed on the combined trainingdata of both tasks. We report the effects of Vocal Tract LengthNormalization (VTLN) and MLLR Adaptation, which were notas favorable as observed by other groups for English broadcastnews. We present the gains obtained by language modelinterpolation and describe the language specific problems, e.g.high OOV rate due to rapid vocabulary growth and inflectedword forms. Again, the few training utterances were no obstacleto build up a system, which surprised us positively with arecognition performance of 36.0% WE.2. DATA COLLECTION2.1 Speech Recording and TranscriptionThe audio data for the dictation system was collected in Croatiaand Bosnia-Hercegovina. Native speakers were asked to read 20minutes of news texts, extracted from the HRT (Croatian Radioand Television) web site and Obzor Nacional, a Croatiannewspaper. The speech was digitally recorded using a portableDAT-recorder at a sampling rate of 48 kHz in stereo quality andfurther sampled down to 16 kHz with 16 bit resolution in monoquality. The read utterances were checked against the originaltext to eliminate major errors and mark spontaneous effects.This data was originally collected for the GlobalPhone project atKarlsruhe University.Spkrs. Articles Rec. Length Words Vocab.85 131 18 h 89 K 17 KTable 1: Dictation System DatabaseThe broadcast news data was collected at Karlsruhe Universityin Germany. A satellite dish and a dedicated PC, equipped withan MPEG encoder board, were installed to record the HRTevening news show, which is transmitted from Croatia via theEutelsat satellite. The television signal was digitally recorded inMPEG format (target bit rate: 1.008 Mbit/s, audio bit rate:0.192 Mbit/s, sampling rate: 44.1 kHz). For speech recognitionthe audio signal was uncompressed and sampled down to 16kHz with 16 bit resolution. Three native speakers transcribedthe news broadcasts, using a software tool developedspecifically for this task. Similar to the HUB4 corpus forEnglish broadcast news data, the Serbo-Croatian recordingswere divided into segments, in which the acoustic conditionsremained constant, and tagged according to the speaker, thechannel quality and the background noises. The different tags inthese three categories are shown in Table 2, where “Non-Serbo-Croatian” identifies a person speaking in another language thanSerbo-Croatian, most often English. In addition to these acoustictags, only the most frequent and clearly audible spontaneouseffects were transcribed: Hesitation, breathing, some otherhuman and non-human noises. The diacritical letters in Serbo-Croatian were transcribed applying the rules in Table 3.Speaker Channel NoiseMale Clean MusicFemale Telephone Second SpeakerNon-Serbo-Croatian Distorted ConferenceUnknown Unknown StreetNone Static NoiseOtherNoneTable 2: Acoustic TagsDiacritic û ü ý þ  ÿ Š   š ä åASCII C1 c1 C5 c5 D1 d1 S5 s5 Z5 z5Table 3: Serbo-Croatian DiacriticsTranscription time varied between 13 and 18 hours per newsbroadcast (approx. 40 min.). This is fairly long and there areseveral reasons for that: No close caption or teletext was available Speakers speak very fast Many noisy segments, which take longer to transcribe Acoustic labeling of the segments consumes a lot of timeIn addition to the television broadcasts, we downloaded someradio news from the Radio Free Europe/Radio Liberty web sitein Realaudio format and converted them to 16 kHz, 16 bit Waveformat for speech processing.Source Broadcasts Rec. Length Words Vocab.HRT (MPEG) 23 15 h 118K 24 KRFE/RL  (RA) 7 0.5 h 7 K 2.5 KTotal 30 15.5 h 125 K 25 KTable 4: Broadcast News System Database2.2 Text Acquisition and PreparationFor language modeling we searched the Internet for news textsin Serbo-Croatian. There were few sites at the beginning of theproject, but the amount of available data later increased almostdaily. We retrieved text data from 20 different sources(television and radio stations, newspapers and news agencies).During text preparation, we encountered one major problem:Many sites simply map diacritics onto their corresponding non-GLDFULWLFDO OHWWHU LH ü DQG þ EHFRPH F ÿ EHFRPHV G ã LVUHSODFHG ZLWK V DQG å ZLWK ] ZKLFK LV QR SUREOHP IRU QDWLYHspeakers. For language modeling, however, we have to insertthe diacritics at the right position. To accomplish this wecollected as many Serbo-Croatian texts with diacritics as wereavailable, not necessarily only news texts. From these texts wegenerated a list, LC, of correct words, which served as referenceto convert the second list, LF, of both correct and false wordforms, which were extracted from the texts without specialcharacters. First, all words in LF that did not contain the lettersc, d, s and z were marked as correct. Then all words, whichoccurred in LF and LC, were labeled correct. For some words thismight be wrong in certain situations; depending on the context,‘grada’ converts into ‘grada’ (grad, gen. sg WRZQ RU µJUDÿD¶JUDÿD QRP VJ PDWHULDO $ WULJUDP PRGHO VKRXOG KHOS WRimprove the converting accuracy in such cases. In the next step,all remaining words in LF were assigned to the their nearestneighbor in LC. When the Levenstein editing distance exceededa certain threshold, this word pair was thought to be valid andthe necessary conversions were performed (Table 5).No Diacritics Diacritics Conversionafirmisanog afirmisanoj NOafricki DIULþNL  c Æ þTable 5: Conversion PairsWhen applying this operation to a separate test text, 2% of thewords among the qualifying pairs were not converted correctly.In the last step of the text conversion algorithm, we generated aletter trigram model. This model was used to score thelikelihood of the different possible character sequences(switching the potential diacritic candidates c, d, s and z) for theremaining words in LF. The sequence with the highest score waschosen. In the test text, 25% of the words were convertedincorrectly using this mechanism, which is better than justleaving the words as they are, which produces an error rate of70%. Thus the combined conversion error rate of the wholealgorithm on the test text was 5%.Character Set Web Sites Words VocabDiacritics 7 5 M 236 KNo Diacritics 13 6 M 216 KTotal 20 11 M 353 KTable 6: Internet Text Database3. DICTATION SYSTEM3.1 System OverviewWe used the JanusRTk speech recognizer to build up the Serbo-Croatian dictation system. We determined a phone set andgenerated a pronunciation dictionary almost automatically. Webootstrapped a first context independent system with an existingGerman recognizer. Our first language model was based on thefew training utterances. Even with the low amount of availabledata given, we were able to develop a chain of improvingcontext dependent systems, applying techniques like vocal tractlength normalization (VTLN), adaptation (MLLR) and multiplecorpora language model interpolation.3.2 Bootstrapping and Initial SystemsOne principle of Serbo-Croatian is “Write as you speak, andspeak as you write”. Thus the phone set corresponds almostexactly to the alphabet. The set of phones used in the JanusRTkspeech recognizer is based upon 30 language phones, 4 noisephones and the silence phone (Table 7). Each phone wasmodeled by a left-to-right HMM with 16 diagonal Gaussians. Alinear discriminant analysis (LDA) was used to reduce the 43feature dimensions to a 32 dimensional input vector. Thepronunciation dictionary was created by an automatic grapheme-to-phoneme tool, whose mapping rules were also derived fromTable 7. Some manual adjustments were necessary for numbers,abbreviations, foreign words and names.SCR A B C C1 C5 D D1 DZ5Letter A B C û ý D  'äGER A B TS TSCH TSCH D TSCH TSCHSCR E F G H I J K L LJ M NLetter E F G H I J K L LJ M NGER E F G X I J K L L M NSCR NJ O P R S S5 T U V Z Z5Letter NJ O P R S Š T U V Z äGER N O P R S SCH T U V Z SCHPhone +QK +hGH +hBR +nGN SILDescr. Garb. Hmn. Noise Breath. Oth.Noise SilenceTable 7: Serbo-Croatian Phone SetThe first context independent Serbo-Croatian dictation systemwas trained upon the labels generated by JanusRTk for aGerman scheduling task using “label boosting”. The Serbo-Croatian phones were initialized by their closest Germanequivalent (Table 7).System Data Labels Vocab. OOV LM WED-CI-0 5.5 h GSST 10 K 13.6% D-LM-0 52.7%D-CI-1 5.5 h D-CI-0 10 K 13.6% D-LM-0 50.4%Table 8: Results for Context Independent SystemsThe labels written by this first recognizer turned out to be moreaccurate and were used to train the next context independentsystem. Both recognizers used a single trigram language modelbased on the training transcriptions (Table 8).3.3 Advanced SystemsWe introduced 54 acoustic and articulatory phone classes, whichguided the training of phonetically tied Gaussian mixtures onthe base of pentaphones.System Data Labels Vocab OOV LM WED-CD-0 5.5 h DCI-1 10 K 13.6% D-LM-0 39.4%D-CD-1 12.5 h DCD-0 18 K 8.5% D-LM-1 36.6%Table 9: Results for Context Dependent SystemsWhile the first polyphone system was still based on a lownumber of training utterances, more audio data was available forthe second context dependent system. Together with anincreased vocabulary and speaker dependent VTLN duringtraining, the WE rate went down significantly (Table 9).Some more tests with the same vocabulary were performed onthe D-CD-1 system. They showed the positive effect of VTLNand MLLR adaptation during testing, especially when theamount of training data is limited. Interpolated language modelsfrom different corpora also helped improve the systemperformance. The system particularly benefits from languagemodel interpolation when the acoustic performance is weaker(Table 10). See Table 11 for details on language modelinterpolation.System VTLN/MLLR LM WED-CD-1 NO D-LM-0 36.6%D-CD-1 NO D-LM-1 34.2%D-CD-1 SPK. DEP. D-LM-0 28.4%D-CD-1 SPK. DEP. D-LM-1 28.2%Table 10: Results for System D-CD-1 with VTLN,Adaptation and Interpolated LMLM Crpra. Words Single PP PPD-LM-0 1 42 K 480 N/AD-LM-1 1 98 K 313 N/AD-LM-2 2 98 K/1.6 M 313/322 260B-LM-0 1 2.5 M 426 N/AB-LM-1 3 2.5 M/80 K/8 M 426/1598/998 281B-LM-2 2 2.5M/220 K Classes 426/344 268Table 11: LM Interpolation4. Broadcast News System4.1 System OverviewThe Serbo-Croatian broadcast news recognizer was based on theabove described dictation system. We started with a contextdependent system that was trained on the broadcast news dataalone; later we also included the dictation data. We studied theeffects of segment clustering for MLLR adaptation duringtesting. Several word and class based language modelinterpolations were performed. We observed a very strongvocabulary growth (Figure 1) and a high OOV rate.Figure 1: Vocabulary GrowthThe reported results correspond with the PE (partitionedevaluation) test in the last HUB4 evaluation (December 1996),in which the segments and their constant acoustic propertieswere given for training and testing.4.2 Bootstrapping and Initial SystemIn a baseline experiment we evaluated the performance of thedictation system D-CD-1 on the broadcast news test set. Due tothe noisy conditions, even in the clean segments, the recognitionperformance dropped off. We used this system, however, tolabel our broadcast news data and trained a first recognizer B-CD-0 solely on 10 hours of transcribed recordings. This context0200004000060000800001000001200001 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21Broadcastsdependent system was set up with 2 K codebook vectors over 24input features. The vocabulary size was 29 K, the OOV rate14.0% (Table 12).System Data WE D-LM-1WED-LM-2WEB-LM-0WEB-LM-1D-CD-1 12.5 h (D) 75.9% 73.6% N/A N/AB-CD-0 10.0 h (B) N/A N/A 45.2% 43.6%Table 12: Baseline Tests on the Broadcast News DataWe also observed the positive effect of language modelinterpolation. Its influence however reduced with the systembecoming more stable and seemed to remain at an improvementlevel of about 1.5%, compared to a single language model. Weapplied three criteria do divide our text data into differentcorpora: Geographical origin (Serbia and Croatia), contentsource (television and radio stations on one side, newspapersand news agencies on the other) and language model perplexity.An interpolation of three different corpora yielded the bestrecognition results. We also noticed that the increased numberof words we used to create the language models did notnecessarily boost the recognition improvement.4.3 Advanced SystemThe next system, B-CD-1, was developed on the combineddictation (12 h) and broadcast news data (13 h), with the latteroccurring twice in the training set. We augmented the testvocabulary (31 K), which slightly reduced the OOV rate(13.5%), examined the effect of interpolated language modelsand also applied different adaptation techniques on the test data.Classes Portion of Datamale_clean_quiet 28.0%male_clean_noisy 20.0%male_dirty_quiet 1.5%male_dirty_noisy 7.5%female_clean_quiet 20.0%female_clean_noisy 12.5%female_dirty_quiet 7.0%female_dirty_noisy 0.5%nospeaker 3.5%Table 13: Segment ClustersFor that reason the segments that shared the same acousticconditions were assigned to one of the clusters in Table 13. Thewarping factor for VTLN and the adaptation were calculated onthe hypothesis generated by the recognizer. The recognition gainby simply adapting over single segments was very low.Although observed for English broadcast news systems,adaptation on clustered segments, before adapting the singlesegments, did not improve recognition results.System MLLR LM WEB-CD-1 SNGL B-LM-1 36.0%B-CD-1 CLST+SNGL B-LM-1 36.2%B-CD-1 SNGL B-LM-0 37.6%B-CD-1 SNGL B-LM-2 38.7%Table 14: Results for System B-CD-1 with DifferentTest ParametersSerbo-Croatian is a strongly inflected language (Table 15), sothe recognizer might be able to benefit from a class basedlanguage model, in which a class represents all inflections ofone stem form. Although this approach reduced the languagemodel perplexity, the overall system did not improve.hotelhotelhotele, hotela, hotelu, hotelom, hotelu, hoteli, hotelimagovoritito speakgovorim, govoriš, govori, govorimo, govorite, govoreTable 15: Samples of InflectionsThe number of inflections is one reason for the high OOV rate,the dominating factor in the WE rate. If we had a 64 Kfrequency based vocabulary, the Serbo-Croatian OOV would be9.0%, for English broadcast an OOV rate of 0.6% was reported.This seems to be the most challenging problem to be solved inour future work.5. SUMMARYWe described the development of a Serbo-Croatian dictation andbroadcast news speech recognizer. We showed that the acousticmaterial of both domains can be combined to train a fairlyreliable automatic transcription system, which generates outputfor a multilingual Informedia database. We showed the wholesystem development process, starting with data collection anddesign of the speech recognizer's base parameters for acompletely new language. We examined the effects ofsegmentation, adaptation and language interpolation. Futurework will concentrate on improvement of the systemperformance, by further attacking language specific problems,e.g. inflections and OOV rate.AcknowledgementsThis work was partly supported by the Advanced ResearchProjects Agency under contract No. N66001-97-D-8502,Delivery Order 0001. The views and findings contained in thismaterial are those of the authors and do not necessarily reflectthe position or policy of the Government and no officialendorsement should be inferred.Thanks to Radio B92, Belgrade, Serbia, HRT, Zagreb, Croatia,Radio Free Europe/Radio Liberty, South Slavic Service and allother organizations that provided data for this project.7KDQNV WR $OHNVDQGUD 6ODYNRYLü 6DQGUD <RRQ /MXERPLU&YHWNRYLü %RULV 7RPD] DQG 0DQIUHG :HEHUThanks to Michael Finke for his time, ideas and motivation.Without his work this paper would not have been possible.6. REFERENCES[1] Broadcast News Transcription Using HTK. P.C. Woodland,M.J.F. Gales, D. Pye & S.J. Young. Proc. of ARPA SRWorkshop, Feb. 1997.[2] The JanusRTk Switchboard/Callhome 1997 EvaluationSystem. Michael Finke, Jürgen Fritsch, Petra Geutner,Klaus Ries and Torsten Zeppenfeld. Proceedings of LVCSRHub5-e Workshop, May 13-15, Baltimore, Maryland.

Serbo-Croatian LVCSR on the dictation and broadcast news domain

https://publikationen.bibliothek.kit.edu/13598/1889

Serbo-Croatian LVCSR on the dictation and broadcast news domain

Abstract

Similar works

Full text

Available Versions

KITopen