We present a conversational telephone speech data set designed to support research on novel acoustic models. Small vocabulary tasks from 10 words up to 500 words are defined using subsets of the Switchboard-1 corpus; each task has a completely closed vocabulary (an OOV rate of 0%). We justify the need for these tasks, describe the algorithm for selecting them from a large corpus, give a statistical analysis of the data and present baseline whole-word hidden Markov model recognition results. The goal of the paper is to define a common data set and to encourage other researchers to use it

Bartels, Chris

Bilmes, Jeff

King, Simon

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      SVitchboard 1: Small Vocabulary Tasks from Switchboard 1Citation for published version:King, S, Bartels, C & Bilmes, J 2005, SVitchboard 1: Small Vocabulary Tasks from Switchboard 1. inInterspeech 2005 - Eurospeech: 9th European Conference on Speech Communication and Technology .International Speech Communication Association, pp. 3385-3388.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:Interspeech 2005 - EurospeechGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019SVitchboard 1: Small Vocabulary Tasks from Switchboard 1Simon King∗Centre for Speech Technology ResearchUniversity of Edinburgh, UKwww.cstr.ed.ac.ukChris Bartels, Jeff Bilmes†Department of Electrical EngineeringUniversity of Washington, USAssli.ee.washington.eduAbstractWe present a conversational telephone speech data setdesigned to support research on novel acoustic models.Small vocabulary tasks from 10 words up to 500 wordsare defined using subsets of the Switchboard-1 corpus;each task has a completely closed vocabulary (an OOVrate of 0%). We justify the need for these tasks, de-scribe the algorithm for selecting them from a large cor-pus, give a statistical analysis of the data and presentbaseline whole-word hidden Markov model recognitionresults. The goal of the paper is to define a common dataset and to encourage other researchers to use it.1. IntroductionCurrently, one of the most challenging tasks in speechprocessing is the recognition of spontaneous conversa-tional speech. Compared to carefully read speech, con-versational speech has more variation in pronunciationand speaking rate as well as a higher degree of coartic-ulation. Another difficulty in studying spontaneous con-versational speech is that, although the core vocabulary ofeveryday speech may be just a few thousand words, thepossible vocabulary is typically in the tens or hundreds ofthousands of words: so there are a large number of tokensof low-frequency words.Researchers attempting to build better models of thecoarticulation and pronunciation variation present in spon-taneous speech are currently faced with the choice be-tween two unsatisfactory extremes: small vocabulary readspeech corpora such as OGI Numbers1 and (very) largevocabulary spontaneous speech corpora such as Switch-board [1] and the like.1.1. MotivationWorking with computationally expensive models (e.g. ar-ticulatory approaches such as [2, 3, 4, 5, 6]) presents prob-lems. Near-real-time and limited memory first-pass de-∗ Supported by EPSRC Advanced Research FellowshipGR/T04649/01. Work partially carried out whilst at the Univer-sity of Washington, funded by EPSRC grant GR/T12118/01.†Partially supported by NSF grant IIS-0093430 and an Intel Cor-poration Grant.1http://www.cslu.ogi.edu/corpora/numbers/index.htmlcoding on large vocabulary corpora using complex mod-els is currently impossible. On the other hand, workingwith small vocabulary corpora is quite feasible, but thesecorpora are usually not of spontaneous or conversationalspeech.It is useful to be able to decouple the problems of de-veloping a novel acoustic model from the problems of:constructing a lexicon; language modelling; decoding;dealing with words that are unseen in the training data.A carefully designed closed-vocabulary task could elimi-nate or avoid these problems.The standard solution is lattice rescoring; this is notentirely satisfactory. A large lattice does not sufficientlylimit computation. The low word error rate (WER) re-gion of search space represented by a small lattice maynot overlap with the low WER region of a novel model,thus preventing the novel model achieving a low WER byrescoring this part of search space. First-pass decoding ispreferable and makes error analysis easier (i.e. recogni-tion errors are entirely due to the novel model, not somecombination of novel model and lattice).An alternative solution, which we present here, is acorpus of spontaneous, conversational speech with a small,limited vocabulary. Of course, such a corpus does not ex-ist and could not be recorded. We introduce SVitchboard1, the Small Vocabulary Switchboard 1 database, whichis composed of selected utterances from the SwitchboardI corpus [1]. SVitchboard 1 contains a number of taskswith increasing vocabulary sizes: 10, 25, 50, 100, 250and 500 words.This paper will begin by describing the algorithm usedto construct the database in section 2. Section 3 definesa five-fold cross-validation procedure. We give statisticalinformation about the resulting corpus in section 4 andsection 5 gives baseline recognition results, which we in-vite other researchers to beat.2. Creating the corpusAll of the data in SVitchboard 1 are taken from the Switch-board 1 corpus of two-person telephone conversations [1].We present a simple algorithm for simultaneously find-ing a vocabulary of any given size and selecting the cor-responding in-vocabulary utterances from the larger cor-Task Partition Utterances Word Duration (hours)tokens Total Speech10A 1384 1617 0.67 0.20B 1275 1455 0.60 0.17C 1196 1389 0.56 0.16D 1446 1628 0.69 0.20E 1474 1703 0.70 0.20Total 6775 7792 3.22 0.9325A 1943 2698 0.95 0.29B 1887 2560 0.90 0.26C 1732 2359 0.83 0.25D 2078 2789 1.01 0.30E 2138 2918 1.04 0.31Total 9778 13324 4.74 1.4250A 2474 4228 1.24 0.39B 2392 3932 1.16 0.36C 2233 3789 1.10 0.34D 2594 4292 1.29 0.40E 2749 4673 1.37 0.43Total 12442 20914 6.16 1.93100A 2916 5814 1.51 0.51B 2794 5290 1.40 0.46C 2632 5237 1.34 0.45D 3059 5981 1.57 0.53E 3201 6289 1.64 0.55Total 14602 28611 7.47 2.48250A 3741 10400 2.10 0.81B 3681 10060 2.02 0.77C 3415 9336 1.88 0.71D 3927 10581 2.18 0.83E 4169 11573 2.32 0.89Total 18933 51950 10.50 4.01500A 4675 17948 2.92 1.30B 4673 17519 2.86 1.26C 4249 15857 2.60 1.13D 4871 18075 3.00 1.32E 5202 20021 3.23 1.43Total 23670 89420 14.62 6.44Table 1: Data set sizespus. The desirable properties of the limited-vocabularycorpus being created include:Closed vocabulary: there is a fixed, known vocabu-lary and that every utterance in both train and test setsonly contains words from within this vocabulary.Balance: all words should have examples in training,validation and test sets. If words (or other units) in the testset do not occur in the training set, then whole-word (orother unit) modelling is simply not possible. Whole wordmodelling is useful in articulatory models, for example,where one wishes to avoid phone-sized units altogether.Minimal number of low-frequency words: the longtail of singletons and low-frequency word types in thetypical Zipf-like distribution of a corpus like Switchboard1 is a problem for novel approaches.Maximum size: the maximum amount of data (num-ber of word tokens) should be selected.The first step in creating the limited vocabulary cor-pus was to divide each side of the long conversations ofSwitchboard 1 into shorter segments. The initial cuttingsused here were those published by Mississippi State Uni-versity [7]. To maximise the amount of in-vocabulary ut-terances available for selection, we further cut these seg-ments into smaller utterances at every silence longer than500ms. This algorithm is then used:sv vocabulary = 5 most common words in large corpusoov vocabulary = full vocabulary \ sv vocabularywhile |sv vocabulary| < target number of words dofor all word ∈ oov vocabulary donew vocabulary = sv vocabulary ∪ wordincoming utterances = all utterances that only containwords in new vocabularycountword = number of words in incoming utterancesend fornew word = arg maxwordcountwordsv vocabulary = sv vocabulary ∪ new wordoov vocabulary = oov vocabulary \ new wordend whileLabelled silences are always allowed, but do not con-tribute to countword. The algorithm incrementally addsone word at a time to the vocabulary. Larger vocabular-ies can be built by initialising sv vocabulary to a previ-ously found smaller vocabulary. We decided on vocab-ulary sizes of 10, 25, 50 ,100, 250 and 500 words (plusthe pseudo-word sil), which gives users a very simpletask to start with (e.g. for debugging their model) and asequence of increasingly difficult tasks to work through.The vocabulary of each task is a subset of all larger ones:e.g. all words in the 10 word task also appear in all thelarger tasks. The speech data for each task is likewise asubset of the data in all larger tasks. The 10 word vocabu-lary (in decreasing order of frequency) is: right, oh, okay,so, well, and, yes, really, I, the.2.1. Dealing with disfluencies and other problem wordsSome transcribed words or pseudo-words in Switchboard1 present problems for various reasons, including extremevariation in pronunciation or difficulty in writing pronun-ciations for them. Since SVitchboard 1 is designed to beuseful for novel acoustic modelling work, we decided tosimplify the corpus by excluding utterances containingany of these words: all word fragments (e.g. sim[ilar]-), words ending in a digit, uh, [noise], i-, yeah, [laughter],huh, hm, [laughter-*], uh-huh, um-hum hum, huh-uh, um.The tasks defined in this paper should be referred to as“SVitchboard 1: no filled pause condition”. A versionin which filled pauses are allowed may be constructed infuture.3. Cross-validation procedureThe speakers (and their speech data) are divided into non-overlapping subsets to create a 5-fold, speaker-independent,cross-validation scheme. Five partitions were constructed,denoted by the letters A to E, each of 108 speakers (parti-tions A–C) or 107 speakers (partitions D and E). For anygiven limited vocabulary task (e.g. the 50 word task), wetake only the in-vocabulary utterances from each parti-Subtask Train Validate Test1 ABC D E2 BCD E A3 CDE A B4 DEA B C5 EAB C DTable 2: Definition of the five-fold cross-validation setup.Numbers 1–5 denote the five subtasks, and letters A–Edenote the five partitions of the data.1 2 5 10 302000400060008000Length (words)5001 2 5 10 302000400060008000UtterancesLength (words)2501 2 5 10 3020004000600080001001 2 5 10 302000400060008000Utterances 501 2 5 10 302000400060008000251 2 5 10 302000400060008000Utterances 10Figure 1: Utterance length distributions (horizontal axisis logarithmically scaled).tion.Now we can use these partitions to define five sub-tasks using a jackknife procedure, as shown in table 2. Ineach subtask, three partitions from {A, B, C, D, E} areformed into a training set, with the remaining two setsbeing used as validation and test sets respectively.3.1. How to use the corpusEach vocabulary size corresponds to a task, which shouldbe referred to as, for example, “SVitchboard 1: no filledpause condition, 50 word task”. Within each task thereare five subtasks, numbered 1–5. Each subtask specifiesa particular arrangement of the five partitions (labelledA–E) into training, validation and testing sets.Across the 5 subtasks, each partition appears the samenumber of times in the training set, in the validation setand in the test set (table 2). Thus any inter-partition vari-ation in recognition difficulty is cancelled out if overallresults are reported across all 5 tasks (as in table 3).Each subtask is to be performed independently of theothers. For computationally expensive systems, the firstsubtask can be used alone. If training is particularly ex-pensive, then only the first partition of the training set canbe used (e.g. in subtask 1, train on A only, rather than A,B and C), but this should be done only as a last resort.Since one of the aims of this corpus is to facilitate direct1 250 5001234500Word types1 100 2501234250log 10(tokens)Word types1 50 10012341001 25 50123450log 10(tokens)1 10 251234251 5 10123410log 10(tokens)Figure 2: Word frequency distributions have a truncatedZipf shape.comparison of systems on identical data, the training setshould not be enhanced (e.g. by adding Fisher data).Where practical, it is preferable to perform a set offive independent experiments (e.g. as is done in section 5of this paper) and report results both by subtask and as anoverall word error rate per task. For systems employinga language model, this model must also be trained on thetraining data specified in table 2. That is, the languagemodel cannot be shared across all five experiments, un-less it is trained on data other than Switchboard 1 tran-scripts.Phonetically labelled data: partition E includes the12 speakers for which there is phonetically labelled datafrom the ICSI Switchboard Transcription Project [8], andthe remaining speakers were divided randomly. The pho-netically transcribed data could be used either as a reli-able reference for phone recognition (on either validationor test data) or as data to be used in co-training.Split conversations: if a language model is trainedon one side of a conversation and used to decode datataken from the other side of the conversation, this hasthe potential to give the language model an unfair advan-tage. This effect is reduced in SVitchboard 1, since low-frequency topic words (likely to be spoken by both speak-ers in a single conversations) are less likely to be in thelimited vocabularies. However, it should be noted that,in SVitchboard 1, it is common to find one side of a con-versation in the training set and the other other side in thecorresponding test set. It is impossible to avoid this, sinceany given speaker will have conversations with multipleother speakers (who will have conversations with furtherspeakers...); it is impossible to ensure both sides of allthose conversations appear within a single partition.Cross-talk: Note that a significant amount of cross-talk can be heard, especially during silence portions.4. Statistical analysis of the corpusThe sizes of the data sets can be seen in table 1. Utterancelength distributions are shown in figure 1 and the wordfrequency distributions are in figure 2. In all but the 500word task, every vocabulary word appears in every par-tition (A–E) and therefore in every train, validation andtest set. In the 500 word task, each partition has between1 and 4 words fewer than 500, but the missing wordsare different for every partition so therefore the subtasktraining sets (ABC, BCD, etc) all contain every vocabu-lary word. The lowest frequency word in each subtask(adding all 5 partitions together) has a frequency of 73,97, 55, 31, 16, 10 in the 10, 25, 50, 100, 250 and 500word tasks, respectively.5. Baseline resultsA simple baseline whole word HMM system was builtusing HTK [9]; the number of states was 3× the numberof phones for that word entry in the lexicon distributedwith [7]. Observation vectors were 12 MFCCs plus en-ergy, and their deltas and accelerations. Gaussian mix-ture distributions were trained using the HTK “mixing-up” procedure and the number of components was cho-sen per subtask to maximise validation set accuracy. Sim-ple bigram language models were constructed per subtaskusing the HTK program HLStats with a bigram countthreshold of 3; perplexities are shown in table 3. Theword insertion penalty and language model scaling factorwere optimised per subtask to maximise accuracy on thevalidation set. Results are shown in table 3.6. ConclusionSVitchboard 1 allows researchers to decouple the prob-lems of developing novel acoustic models from the var-ious problems of dealing with a large vocabulary. Fu-ture work will include publishing tasks with larger, butstill limited, vocabulary sizes and also tasks with disflu-encies (filled pauses, word fragments). Thanks are dueto Mississippi State University for their freely availableword alignments. SVitchboard 1 can be downloaded fromhttp://www.cstr.ed.ac.uk/research/projects/svitchboard.7. References[1] J. Godfrey, E. Holliman, and J. McDaniel, “SWITCH-BOARD: Telephone Speech Corpus for Research and De-velopment,” in Proceedings of the IEEE International Con-ference on Acoustics, Speech, and Signal Processing, SanFrancisco, California, March 1992, vol. 1, pp. 517–520.[2] L. Deng, G. Ramsay, and D. Sun, “Production models as astructural basis for automatic speech recognition,” SpeechCommunication, vol. 33, no. 2-3, pp. 93–111, Aug 1997.[3] M. Richardson, J. Bilmes, and C. Diorio, “Hidden-Articulator Markov Models for Speech Recognition,”Speech Communications, vol. 41, no. 2, October 2003.[4] Y. Zhang, Q. Diao, S. Huang, W. Hu, C. Bartels, andJ. Bilmes, “DBN based multi-stream models for speech,”in Proc. IEEE Intl. Conf. on Acoustics, Speech, and SignalProcessing, Hong Kong, China, April 2003.Task Subtask Perplexity WER(%)Val Test Val Test101 3.1 3.2 20.2 20.82 3.2 3.4 20.1 21.33 3.4 3.4 21.6 21.04 3.4 3.3 21.3 24.55 3.3 3.1 23.3 20.9overall 21.6251 5.0 5.2 35.6 34.92 5.2 5.3 35.5 35.93 5.2 5.4 35.0 37.24 5.4 5.3 35.4 37.95 5.2 4.9 37.6 35.4overall 36.2501 7.6 8.1 48.4 48.42 8.1 8.1 48.6 46.33 8.1 8.4 46.1 49.34 8.3 8.1 48.1 51.45 8.2 7.6 50.8 48.3overall 48.71001 11.4 11.5 57.9 56.82 11.4 11.7 56.4 55.13 11.7 11.7 55.1 58.84 11.6 11.6 55.6 57.45 11.8 11.5 57.5 57.2overall 57.02501 21.7 23.2 65.8 66.22 23.2 22.3 66.8 64.43 22.2 23.5 63.9 67.14 23.4 22.4 65.3 66.75 22.6 21.7 66.7 64.5overall 65.75001 38.4 39.5 69.8 70.82 39.4 38.5 70.0 67.93 38.1 39.7 67.6 70.04 39.4 38.0 69.2 69.75 38.2 37.9 70.1 68.9overall 69.5Table 3: Baseline LM perplexities, word error rates.[5] M. Wester, J. Frankel, and S. King, “Asynchronous ar-ticulatory feature recognition using dynamic Bayesian net-works,” in Proc. IEICI Beyond HMM Workshop, Kyoto,Dec. 2004.[6] K. Livescu and J. Glass, “Feature-based pronunciationmodeling for speech recognition,” in Proc. HLT/NAACL,Boston, Massachusetts, May 2004.[7] N. Ganapathiraju Deshmukh, A. Gleeson, A. Hamakera,and J. Picone, “Resegmentation of SWITCHBOARD,” inProceedings of the International Conference Spoken Lan-guage Processing, 1998, vol. 4, p. 1543.[8] S. Greenberg, J. Hollenback, and D. Ellis, “Insights intospoken language gleaned from phonetic transcription ofthe Switchboard corpus,” in In Proc ICSLP, Philadelphia,1996.[9] Steve Young, Gunnar Evermann, Dan Kershaw, GarethMoore, Julian Odell, Dave Ollason, Valtcho Valtchev, andPhil Woodland, HTK manual, Cambridge University Engi-neering Department, 2002.

Edinburgh Research Archive

SVitchboard 1: Small Vocabulary Tasks from Switchboard 1Simon King∗Centre for Speech Technology ResearchUniversity of Edinburgh, UKwww.cstr.ed.ac.ukChris Bartels, Jeff Bilmes†Department of Electrical EngineeringUniversity of Washington, USAssli.ee.washington.eduAbstractWe present a conversational telephone speech data setdesigned to support research on novel acoustic models.Small vocabulary tasks from 10 words up to 500 wordsare defined using subsets of the Switchboard-1 corpus;each task has a completely closed vocabulary (an OOVrate of 0%). We justify the need for these tasks, de-scribe the algorithm for selecting them from a large cor-pus, give a statistical analysis of the data and presentbaseline whole-word hidden Markov model recognitionresults. The goal of the paper is to define a common dataset and to encourage other researchers to use it.1. IntroductionCurrently, one of the most challenging tasks in speechprocessing is the recognition of spontaneous conversa-tional speech. Compared to carefully read speech, con-versational speech has more variation in pronunciationand speaking rate as well as a higher degree of coartic-ulation. Another difficulty in studying spontaneous con-versational speech is that, although the core vocabulary ofeveryday speech may be just a few thousand words, thepossible vocabulary is typically in the tens or hundreds ofthousands of words: so there are a large number of tokensof low-frequency words.Researchers attempting to build better models of thecoarticulation and pronunciation variation present in spon-taneous speech are currently faced with the choice be-tween two unsatisfactory extremes: small vocabulary readspeech corpora such as OGI Numbers1 and (very) largevocabulary spontaneous speech corpora such as Switch-board [1] and the like.1.1. MotivationWorking with computationally expensive models (e.g. ar-ticulatory approaches such as [2, 3, 4, 5, 6]) presents prob-lems. Near-real-time and limited memory first-pass de-∗ Supported by EPSRC Advanced Research FellowshipGR/T04649/01. Work partially carried out whilst at the Univer-sity of Washington, funded by EPSRC grant GR/T12118/01.† Partially supported by NSF grant IIS-0093430 and an Intel Cor-poration Grant.1http://www.cslu.ogi.edu/corpora/numbers/index.htmlcoding on large vocabulary corpora using complex mod-els is currently impossible. On the other hand, workingwith small vocabulary corpora is quite feasible, but thesecorpora are usually not of spontaneous or conversationalspeech.It is useful to be able to decouple the problems of de-veloping a novel acoustic model from the problems of:constructing a lexicon; language modelling; decoding;dealing with words that are unseen in the training data.A carefully designed closed-vocabulary task could elimi-nate or avoid these problems.The standard solution is lattice rescoring; this is notentirely satisfactory. A large lattice does not sufficientlylimit computation. The low word error rate (WER) re-gion of search space represented by a small lattice maynot overlap with the low WER region of a novel model,thus preventing the novel model achieving a low WER byrescoring this part of search space. First-pass decoding ispreferable and makes error analysis easier (i.e. recogni-tion errors are entirely due to the novel model, not somecombination of novel model and lattice).An alternative solution, which we present here, is acorpus of spontaneous, conversational speech with a small,limited vocabulary. Of course, such a corpus does not ex-ist and could not be recorded. We introduce SVitchboard1, the Small Vocabulary Switchboard 1 database, whichis composed of selected utterances from the SwitchboardI corpus [1]. SVitchboard 1 contains a number of taskswith increasing vocabulary sizes: 10, 25, 50, 100, 250and 500 words.This paper will begin by describing the algorithm usedto construct the database in section 2. Section 3 definesa five-fold cross-validation procedure. We give statisticalinformation about the resulting corpus in section 4 andsection 5 gives baseline recognition results, which we in-vite other researchers to beat.2. Creating the corpusAll of the data in SVitchboard 1 are taken from the Switch-board 1 corpus of two-person telephone conversations [1].We present a simple algorithm for simultaneously find-ing a vocabulary of any given size and selecting the cor-responding in-vocabulary utterances from the larger cor-Task Partition Utterances Word Duration (hours)tokens Total Speech10A 1384 1617 0.67 0.20B 1275 1455 0.60 0.17C 1196 1389 0.56 0.16D 1446 1628 0.69 0.20E 1474 1703 0.70 0.20Total 6775 7792 3.22 0.9325A 1943 2698 0.95 0.29B 1887 2560 0.90 0.26C 1732 2359 0.83 0.25D 2078 2789 1.01 0.30E 2138 2918 1.04 0.31Total 9778 13324 4.74 1.4250A 2474 4228 1.24 0.39B 2392 3932 1.16 0.36C 2233 3789 1.10 0.34D 2594 4292 1.29 0.40E 2749 4673 1.37 0.43Total 12442 20914 6.16 1.93100A 2916 5814 1.51 0.51B 2794 5290 1.40 0.46C 2632 5237 1.34 0.45D 3059 5981 1.57 0.53E 3201 6289 1.64 0.55Total 14602 28611 7.47 2.48250A 3741 10400 2.10 0.81B 3681 10060 2.02 0.77C 3415 9336 1.88 0.71D 3927 10581 2.18 0.83E 4169 11573 2.32 0.89Total 18933 51950 10.50 4.01500A 4675 17948 2.92 1.30B 4673 17519 2.86 1.26C 4249 15857 2.60 1.13D 4871 18075 3.00 1.32E 5202 20021 3.23 1.43Total 23670 89420 14.62 6.44Table 1: Data set sizespus. The desirable properties of the limited-vocabularycorpus being created include:Closed vocabulary: there is a fixed, known vocabu-lary and that every utterance in both train and test setsonly contains words from within this vocabulary.Balance: all words should have examples in training,validation and test sets. If words (or other units) in the testset do not occur in the training set, then whole-word (orother unit) modelling is simply not possible. Whole wordmodelling is useful in articulatory models, for example,where one wishes to avoid phone-sized units altogether.Minimal number of low-frequency words: the longtail of singletons and low-frequency word types in thetypical Zipf-like distribution of a corpus like Switchboard1 is a problem for novel approaches.Maximum size: the maximum amount of data (num-ber of word tokens) should be selected.The first step in creating the limited vocabulary cor-pus was to divide each side of the long conversations ofSwitchboard 1 into shorter segments. The initial cuttingsused here were those published by Mississippi State Uni-versity [7]. To maximise the amount of in-vocabulary ut-terances available for selection, we further cut these seg-ments into smaller utterances at every silence longer than500ms. This algorithm is then used:sv vocabulary = 5 most common words in large corpusoov vocabulary = full vocabulary \ sv vocabularywhile |sv vocabulary| < target number of words dofor all word ∈ oov vocabulary donew vocabulary = sv vocabulary ∪ wordincoming utterances = all utterances that only containwords in new vocabularycountword = number of words in incoming utterancesend fornew word = arg maxwordcountwordsv vocabulary = sv vocabulary ∪ new wordoov vocabulary = oov vocabulary \ new wordend whileLabelled silences are always allowed, but do not con-tribute to countword. The algorithm incrementally addsone word at a time to the vocabulary. Larger vocabular-ies can be built by initialising sv vocabulary to a previ-ously found smaller vocabulary. We decided on vocab-ulary sizes of 10, 25, 50 ,100, 250 and 500 words (plusthe pseudo-word sil), which gives users a very simpletask to start with (e.g. for debugging their model) and asequence of increasingly difficult tasks to work through.The vocabulary of each task is a subset of all larger ones:e.g. all words in the 10 word task also appear in all thelarger tasks. The speech data for each task is likewise asubset of the data in all larger tasks. The 10 word vocabu-lary (in decreasing order of frequency) is: right, oh, okay,so, well, and, yes, really, I, the.2.1. Dealing with disfluencies and other problem wordsSome transcribed words or pseudo-words in Switchboard1 present problems for various reasons, including extremevariation in pronunciation or difficulty in writing pronun-ciations for them. Since SVitchboard 1 is designed to beuseful for novel acoustic modelling work, we decided tosimplify the corpus by excluding utterances containingany of these words: all word fragments (e.g. sim[ilar]-), words ending in a digit, uh, [noise], i-, yeah, [laughter],huh, hm, [laughter-*], uh-huh, um-hum hum, huh-uh, um.The tasks defined in this paper should be referred to as“SVitchboard 1: no filled pause condition”. A versionin which filled pauses are allowed may be constructed infuture.3. Cross-validation procedureThe speakers (and their speech data) are divided into non-overlapping subsets to create a 5-fold, speaker-independent,cross-validation scheme. Five partitions were constructed,denoted by the letters A to E, each of 108 speakers (parti-tions A–C) or 107 speakers (partitions D and E). For anygiven limited vocabulary task (e.g. the 50 word task), wetake only the in-vocabulary utterances from each parti-Subtask Train Validate Test1 ABC D E2 BCD E A3 CDE A B4 DEA B C5 EAB C DTable 2: Definition of the five-fold cross-validation setup.Numbers 1–5 denote the five subtasks, and letters A–Edenote the five partitions of the data.1 2 5 10 302000400060008000Length (words)5001 2 5 10 302000400060008000UtterancesLength (words)2501 2 5 10 3020004000600080001001 2 5 10 302000400060008000Utterances 501 2 5 10 302000400060008000251 2 5 10 302000400060008000Utterances 10Figure 1: Utterance length distributions (horizontal axisis logarithmically scaled).tion.Now we can use these partitions to define five sub-tasks using a jackknife procedure, as shown in table 2. Ineach subtask, three partitions from {A, B, C, D, E} areformed into a training set, with the remaining two setsbeing used as validation and test sets respectively.3.1. How to use the corpusEach vocabulary size corresponds to a task, which shouldbe referred to as, for example, “SVitchboard 1: no filledpause condition, 50 word task”. Within each task thereare five subtasks, numbered 1–5. Each subtask specifiesa particular arrangement of the five partitions (labelledA–E) into training, validation and testing sets.Across the 5 subtasks, each partition appears the samenumber of times in the training set, in the validation setand in the test set (table 2). Thus any inter-partition vari-ation in recognition difficulty is cancelled out if overallresults are reported across all 5 tasks (as in table 3).Each subtask is to be performed independently of theothers. For computationally expensive systems, the firstsubtask can be used alone. If training is particularly ex-pensive, then only the first partition of the training set canbe used (e.g. in subtask 1, train on A only, rather than A,B and C), but this should be done only as a last resort.Since one of the aims of this corpus is to facilitate direct1 250 5001234500Word types1 100 2501234250log 10(tokens)Word types1 50 10012341001 25 50123450log 10(tokens)1 10 251234251 5 10123410log 10(tokens)Figure 2: Word frequency distributions have a truncatedZipf shape.comparison of systems on identical data, the training setshould not be enhanced (e.g. by adding Fisher data).Where practical, it is preferable to perform a set offive independent experiments (e.g. as is done in section 5of this paper) and report results both by subtask and as anoverall word error rate per task. For systems employinga language model, this model must also be trained on thetraining data specified in table 2. That is, the languagemodel cannot be shared across all five experiments, un-less it is trained on data other than Switchboard 1 tran-scripts.Phonetically labelled data: partition E includes the12 speakers for which there is phonetically labelled datafrom the ICSI Switchboard Transcription Project [8], andthe remaining speakers were divided randomly. The pho-netically transcribed data could be used either as a reli-able reference for phone recognition (on either validationor test data) or as data to be used in co-training.Split conversations: if a language model is trainedon one side of a conversation and used to decode datataken from the other side of the conversation, this hasthe potential to give the language model an unfair advan-tage. This effect is reduced in SVitchboard 1, since low-frequency topic words (likely to be spoken by both speak-ers in a single conversations) are less likely to be in thelimited vocabularies. However, it should be noted that,in SVitchboard 1, it is common to find one side of a con-versation in the training set and the other other side in thecorresponding test set. It is impossible to avoid this, sinceany given speaker will have conversations with multipleother speakers (who will have conversations with furtherspeakers...); it is impossible to ensure both sides of allthose conversations appear within a single partition.Cross-talk: Note that a significant amount of cross-talk can be heard, especially during silence portions.4. Statistical analysis of the corpusThe sizes of the data sets can be seen in table 1. Utterancelength distributions are shown in figure 1 and the wordfrequency distributions are in figure 2. In all but the 500word task, every vocabulary word appears in every par-tition (A–E) and therefore in every train, validation andtest set. In the 500 word task, each partition has between1 and 4 words fewer than 500, but the missing wordsare different for every partition so therefore the subtasktraining sets (ABC, BCD, etc) all contain every vocabu-lary word. The lowest frequency word in each subtask(adding all 5 partitions together) has a frequency of 73,97, 55, 31, 16, 10 in the 10, 25, 50, 100, 250 and 500word tasks, respectively.5. Baseline resultsA simple baseline whole word HMM system was builtusing HTK [9]; the number of states was 3× the numberof phones for that word entry in the lexicon distributedwith [7]. Observation vectors were 12 MFCCs plus en-ergy, and their deltas and accelerations. Gaussian mix-ture distributions were trained using the HTK “mixing-up” procedure and the number of components was cho-sen per subtask to maximise validation set accuracy. Sim-ple bigram language models were constructed per subtaskusing the HTK program HLStats with a bigram countthreshold of 3; perplexities are shown in table 3. Theword insertion penalty and language model scaling factorwere optimised per subtask to maximise accuracy on thevalidation set. Results are shown in table 3.6. ConclusionSVitchboard 1 allows researchers to decouple the prob-lems of developing novel acoustic models from the var-ious problems of dealing with a large vocabulary. Fu-ture work will include publishing tasks with larger, butstill limited, vocabulary sizes and also tasks with disflu-encies (filled pauses, word fragments). Thanks are dueto Mississippi State University for their freely availableword alignments. SVitchboard 1 can be downloaded fromhttp://www.cstr.ed.ac.uk/research/projects/svitchboard.7. References[1] J. Godfrey, E. Holliman, and J. McDaniel, “SWITCH-BOARD: Telephone Speech Corpus for Research and De-velopment,” in Proceedings of the IEEE International Con-ference on Acoustics, Speech, and Signal Processing, SanFrancisco, California, March 1992, vol. 1, pp. 517–520.[2] L. Deng, G. Ramsay, and D. Sun, “Production models as astructural basis for automatic speech recognition,” SpeechCommunication, vol. 33, no. 2-3, pp. 93–111, Aug 1997.[3] M. Richardson, J. Bilmes, and C. Diorio, “Hidden-Articulator Markov Models for Speech Recognition,”Speech Communications, vol. 41, no. 2, October 2003.[4] Y. Zhang, Q. Diao, S. Huang, W. Hu, C. Bartels, andJ. Bilmes, “DBN based multi-stream models for speech,”in Proc. IEEE Intl. Conf. on Acoustics, Speech, and SignalProcessing, Hong Kong, China, April 2003.Task Subtask Perplexity WER(%)Val Test Val Test101 3.1 3.2 20.2 20.82 3.2 3.4 20.1 21.33 3.4 3.4 21.6 21.04 3.4 3.3 21.3 24.55 3.3 3.1 23.3 20.9overall 21.6251 5.0 5.2 35.6 34.92 5.2 5.3 35.5 35.93 5.2 5.4 35.0 37.24 5.4 5.3 35.4 37.95 5.2 4.9 37.6 35.4overall 36.2501 7.6 8.1 48.4 48.42 8.1 8.1 48.6 46.33 8.1 8.4 46.1 49.34 8.3 8.1 48.1 51.45 8.2 7.6 50.8 48.3overall 48.71001 11.4 11.5 57.9 56.82 11.4 11.7 56.4 55.13 11.7 11.7 55.1 58.84 11.6 11.6 55.6 57.45 11.8 11.5 57.5 57.2overall 57.02501 21.7 23.2 65.8 66.22 23.2 22.3 66.8 64.43 22.2 23.5 63.9 67.14 23.4 22.4 65.3 66.75 22.6 21.7 66.7 64.5overall 65.75001 38.4 39.5 69.8 70.82 39.4 38.5 70.0 67.93 38.1 39.7 67.6 70.04 39.4 38.0 69.2 69.75 38.2 37.9 70.1 68.9overall 69.5Table 3: Baseline LM perplexities, word error rates.[5] M. Wester, J. Frankel, and S. King, “Asynchronous ar-ticulatory feature recognition using dynamic Bayesian net-works,” in Proc. IEICI Beyond HMM Workshop, Kyoto,Dec. 2004.[6] K. Livescu and J. Glass, “Feature-based pronunciationmodeling for speech recognition,” in Proc. HLT/NAACL,Boston, Massachusetts, May 2004.[7] N. Ganapathiraju Deshmukh, A. Gleeson, A. Hamakera,and J. Picone, “Resegmentation of SWITCHBOARD,” inProceedings of the International Conference Spoken Lan-guage Processing, 1998, vol. 4, p. 1543.[8] S. Greenberg, J. Hollenback, and D. Ellis, “Insights intospoken language gleaned from phonetic transcription ofthe Switchboard corpus,” in In Proc ICSLP, Philadelphia,1996.[9] Steve Young, Gunnar Evermann, Dan Kershaw, GarethMoore, Julian Odell, Dave Ollason, Valtcho Valtchev, andPhil Woodland, HTK manual, Cambridge University Engi-neering Department, 2002.

Svitchboard 1: Small vocabulary tasks from switchboard 1

https://www.pure.ed.ac.uk/ws/files/19879413/king_bartels_bilmes_svitchboard.pdf

Svitchboard 1: Small vocabulary tasks from switchboard 1

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Archive