Subword-based Indexing for a Minimal False Positive Rat

Heeren, Willemijn

Werff, Laurens van der

English

van der Werff, Laurens Bastiaan

Heeren, W.F.L.

University of Twente Research Information

Subword-based Indexing for a Minimal False Positive RateResearch ProposalLaurens van der WerffUniversity of TwentePO Box 217, NL-7500AEEnschede, The Netherlandslaurensw@ewi.utwente.nlWillemijn HeerenUniversity of TwentePO Box 217, NL-7500AEEnschede, The Netherlandsw.f.l.heeren@ewi.utwente.nl1. INTRODUCTIONThe aim of Spoken Document Retrieval (SDR) is to findspeech fragments that are relevant to user queries. Thisis usually accomplished by applying Information Retrieval(IR) methods to an automatic transcription of an audio col-lection. Although there are many different manners of gener-ating such a transcription and there are also many differenttechniques for retrieving documents, all popular techniquesshare the same mismatch: Automatic Speech Recognition(ASR) focuses on frequent terms, while IR typically exploitsrare events.A typical 65k ASR lexicon can describe >96% of the wordsspoken, depending on the subject and type of the speech.However, our own experiments on a set of Dutch ASR tran-scripts, a subset of the Spoken Dutch Corpus1 (CGN), showthat >80% of the unique terms that were uttered in thiscollection were left out in the lexicon, making these termsOut of Vocabulary (OOV). For a dictation-type task thisneed not be a problem, since the error rate is only affectedmarginally by the OOV-rate, but when the same transcriptis used for SDR, the consequences of the deletions may bemuch more severe: any queries using OOV terms will failon this collection. If queries typically use the most discrim-inative (read: rare) words in a collection, and >80% of therarest terms are omitted, the problem becomes obvious.Transcribing the audio in a manner that is not limited toa predefined lexicon may solve this issue. Attempts thathave been made so far include the use of phonemes andphoneme-groups (subwords) as basic recognition blocks in-stead of traditional word-sized units [6, 9]. There seem tobe two main disadvantages to such approaches: i) the per-plexity of the Language Model (LM) for these units is muchhigher than for the typical word-based LM, potentially lead-ing to more ASR errors, and ii) (near) homonyms and wordboundaries cannot be detected accurately. Both of these1http://lands.let.ru.nl/cgn/ehome.htmwill cause False Positive (FP) errors in word-spotting sce-narios. Despite these issues, subword units seem to be themost promising solution for handling OOV query terms.If an SDR system is thought of as a concatenation of ASR,IR and a User Interface (UI), then introducing subword unitswill have consequences for each of these components. Withinthe ASR, both the choice of subword units and the mannerin which they are generated have to be considered. For theIR engine, a hybrid index containing traditional words aswell as subword units provides new challenges for resolvingqueries, and finally the UI must be able to support the userin obtaining maximum benefit from subword-based strate-gies.The experiments proposed in this paper have three maingoals: i) determine the best way of creating a hybrid index,ii) find a criterion for automatically determining the bestretrieval strategy, and iii) estimate the expected reductionin search space that a user can be expected to achieve fromexpanding a query in order to reduce FPs.Section 2 will explain the choices that must be made whengenerating a transcription that is suitable for subword-basedindexing, while Section 3 will delve into the issues relatedto exploiting such an index in order to gain the most bene-fit from it. Section 4 explains how to measure the potentialcontribution of a user in making subword based retrieval fea-sible for large collections and Section 5 lists the experimentalsetup proposed to evaluate a hybrid SDR system.2. SUBWORD-BASED TRANSCRIPTIONAttempts made so far with respect to subword-based tran-scription for SDR vary from using a purely phonetic ap-proach to hybrid systems where both words and subwordunits are used in conjunction. Any approach based on pho-nemes inevitably introduces more recognition errors, butonly on in-vocabulary terms. The overall expected resulttherefore is: more errors on≈96% of words, but (potentially)better recognition on ≈80% of unique terms. For dictationpurposes, focus is on word error rate, for SDR term coverageis also important, making a subword-based approach poten-tially beneficial. A hybrid index allows for matching OOVquery-terms on subword units, while retaining the originalperformance on in-vocabulary (INV) query-terms by utiliz-ing the word-based transcription.Phonetic transcription systems typically use a word-basedspeech recognition engine. The word-based output is thenconverted into phonemes or, instead of using language mod-els and a lexicon to restrict the output to well-formed sen-tences, it is determined only by the acoustic match betweenspeech and models, thereby ignoring structural properties ofthe language. Both were examined in [5], where it was deter-mined, using Mean Average Precision (MAP) as a criterionand transcriptions with an error rate of around 45%, that aphonetic transcription that is produced through grapheme-to-phoneme conversion of a word-based transcription per-forms better in an SDR environment than one that is gen-erated directly.An alternative to a phonetic transcription is the use of largersubword units, i.e., word-fragments. This may allow forsome form of language modeling, while also simplifying thegeneration of an index. Especially when multiple transcrip-tion alternatives are generated (e.g., by using lattices), largerunits may provide some advantages. The combination ofsubword-sized units and a hybrid lattice transcription (con-taining both word and subword units) was put into practicein [9], which showed encouraging results for word-spottingperformance on OOV terms.In the experiments proposed here, word-based ASR latticeswill be used as a basis for generating the hybrid subword-/word-based index. Since most of the speech can be tran-scribed adequately by a word-based ASR system, making asubword transcription of all the data may not be beneficial.Ideally, only those areas that are likely to contain the termsthat are resolved by the IR component as a subword-basedquery, see Section 3, should be converted into a subwordrepresentation. Therefore, instead of making a full subword-based transcription of each utterance, only those areas of thelattice that show the most evidence for potential gain willbe converted, for example because of certain combinationsof acoustic and linguistic posterior probabilities [1].3. QUERIES AND THE HYBRID INDEXAn index that enables searching at both the acoustic level(phonemes or word-fragments) and the traditional word levelallows for some extra flexibility that should be exploited op-timally. Using the subword index only for OOV query terms,while using the word-based index for in-vocabulary (INV)terms is the approach chosen in [3]. FPs are only intro-duced for query terms that would previously have given noresults at all, while ‘normal’ performance can be maintainedfor all other query terms.However, as was shown in [9], there can be a benefit to us-ing subword based queries for INV terms as well, especiallywhen the language model (LM) that was used for generat-ing the transcription is a mismatch to the domain of thespeech. This can be explained through the fact that someLM probabilities may be incorrect due to insufficient train-ing data, leading to more recognition errors on such terms.This would most likely be the case for infrequent terms thatmay then become practically OOV, as was shown in [2].In our experiments we will attempt to find a criterion toautomatically divide the (potential) query terms into threegroups: 1. INV terms that are best served with traditionalword-based retrieval, 2. OOV terms that can only generateresults using subword-based retrieval, and 3. INV terms forwhich ASR performance is so poor that a combination ofsubword and word-based retrieval is expected to give thebest results.4. THE ROLE OF THE USERThe reason why FPs are so harmful can be understood fromthe fact that from a user perspective discarding non-relevantresults requires actually listening to audio fragments insteadof just glancing over pieces of text. The problem with FPsis generally amplified in settings with large corpora, such ascultural heritage collections. For instance, the ‘Radio Ri-jnmond’ archive that is kept at the Municipal Archives ofRotterdam spans approximately 20,000 hours. For an SDRsystem that has an FP-rate of 0.5 per hour (which in gen-eral is quite low) and an average amount of 20 true hits perOOV query term (provided the term was uttered at all inthe corpus), the average number of FPs would outnumberthe correct hits by 500:1, and then only if all of the ut-tered instances of the OOV query term were actually found!Reducing the FP-rate may improve the user experience forsmaller collections, but a purely subword-based retrieval isprobably not very useful for users of large (>100 hours) col-lections.However, users are quite capable of expanding the OOVquery with an INV term. This enables the IR engine to firstreduce the search space to those documents that contain theINV term and then perform the subword-based query ononly these documents. Key in such a scenario is the successthat can be expected from the reduction of the search spacethrough the use of an additional INV query term.In our experiments, all queries shall be formed from a com-bination of (at least) two terms, one of which should beresolved using word-based retrieval, while the other shouldbe done using the subword-based approach. This will en-able us to determine the average rate of reduction in searchspace that can be expected from using a strategically chosenadditional INV query-term. We will assume that the aver-age reduction from the ten best performing INV terms foreach OOV term is representative of what an average usercan expect to attain.5. PROPOSED EXPERIMENTSThe three main goals of the experiments are: i) determinethe best way of creating a hybrid index, variables will bethe amount of speech that is transcribed into subword unitsand the choice of the subword units themselves, ii) find acriterion for automatically determining the best retrievalstrategy for each query term in accordance with the groupsdescribed in Section 3, and iii) estimate the expected reduc-tion in search space that a user can expect to achieve fromadding an INV term to an OOV query.5.1 Experimental SetupExperiments will be carried out primarily on CGN, since thiscollection has been partially transcribed at the phonetic leveland fully at the word level, allowing for a direct evaluationof the lattice-derived subword transcript. Additionally, it isdivided into several types of speech, such as conversationaltelephone speech, broadcast news, sports comments, readspeech, etc., allowing for a comparison of results on thesedifferent types.Automatic transcription of the speech data will be performedusing an HMM-based ASR system [7] delivering output thatcontains word-based lattices. These will then be convertedinto a sausage structure [4] to enable easier conversion intoa hybrid index.5.2 EvaluationA typical Information Retrieval (IR) performance measuresuch as MAP is flawed when it is applied to SDR. The rea-son for this is that text-based IR and SDR find their chal-lenges at different parts of the lexical spectrum. Text-basedIR actually performs quite well when query terms are rareor highly discriminative, since even a simple word-spottingalgorithm is likely to return most relevant documents andproduce few non-relevant ones. This contrasts with SDR,where any OOV or rare term may produce no relevant doc-uments and/or large numbers of non-relevant ones (FPs)due to errors from the ASR stage. When this is combinedwith the knowledge that browsing through audio results is,for now at least, much more time-consuming than browsingthrough text results, it becomes clear that MAP is prob-ably too forgiving towards FPs. This becomes especiallyproblematic when looking at the subword-based approachesto dealing with OOV terms that are the subject of this re-search proposal. Additionally, MAP is highly dependent onthe ranking of the results and therefore on the calculatedrelevance of terms in each document. Having the mannerof score-calculation as an extra variable within these exper-iments is not desirable. Calculation of MAP also requiresground-truth judgments, severely limiting the queries thatcan be evaluated within such a context.Instead, performance will be evaluated through a word spot-ting task. This allows for a complete set of queries, con-taining every possible combination of INV and OOV queryterms. The Figure-of-Merit (FOM) and FP-rate can be usedto judge the impact of choice of subword, while the algorithmfor determining which fragments to convert into subwordscan be evaluated directly from the reference transcriptionand the list of query terms that are resolved through thesubword-based index.Evaluation of ASR output for use in SDR is typically car-ried out using traditional ASR performance measures, suchas Word Error Rate (WER) and Out-of-Vocabulary (OOV)rate. These measures are not ideally suited to the task evenfor a word-based transcription [8], but cannot be applied atall when the transcription is a (hybrid) lattice containingsubword units. WER and OOV rate will therefore be givenonly as an indication of word based 1-best ASR performancebut will not be used as an optimization criterion.Dividing all potential query terms into the three groupsmentioned in Section 3 can be done by simply measuringFOM/FP-rate for single word queries of each word in the ref-erence transcription and determining which strategy (wordor subword) performs best. Then an algorithm should beimplemented that can make such a division automatically,based on word or LM properties.It is then relatively straightforward to determine which (top-10) INV terms are the best ones to use for reducing thesearch space for the subword-based query terms. The aver-age reduction in search space from these terms will be takenas a measure of the potential success a user may achievefrom using an additional term based on his knowledge ofthe subject.6. CONCLUSIONA subword-based approach to SDR may seem like the bestway of handling the OOV issue, but the disadvantages asfor the user experience may very well be underestimated.Any system making use of a subword-based transcriptionshould at least ensure it is no more inconvenient than isstrictly necessary, while maintaining the potential benefits.In order to do this, it is important that not only the subword-based index itself is optimized, but that also the retrievalstrategy takes the properties of query terms into account.Optimization should be done based on performance that canbe achieved under practically relevant conditions, making itessential to incorporate a strategy that a user is likely toemploy when faced with unsatisfactory results.7. REFERENCES[1] G. Evermann and P. Woodland. Large vocabularydecoding and confidence estimation using wordposteriorprobabilities. In Proc. of IEEE Int. Conf. on Acoustics,Speech, and Signal Processing, volume 3, pages1655–1658, 2000.[2] G. Jones, K. Zhang, E. Newman, and A. M.Lam-Adenisa. Examining the contributions ofautomatic speech transcriptions and metadata sourcesfor searching spontaneous conversational speech. InProc. of the SIGIR 2007 workshop on SearchingSpontaneous Conversational Speech, pages 29–36, 2007.[3] B. Logan, P. Moreno, and O. Deshmukh. Word andsub-word indexing approaches for reducing the effectsof oov queries on spoken audio. In Proc. of the 2nd int.conf. on Human Language Technology Research, pages31–35, San Francisco, CA, USA, 2002. MorganKaufmann Publishers Inc.[4] L. Mangu, E. Brill, and A. Stolcke. Finding consensusin speech recognition: Word error minimization andother applications of confusion networks. ComputerSpeech and Language, 14(4):373–400, Oct 2000.[5] C. Ng, R. Wilkinson, and J. Zobel. Experiments inspoken document retrieval using phonetic n-grams.Speech Communication, 32(1–2):61–77, Sept. 2000.[6] K. Ng. Subword-based Approaches for Spoken DocumentRetrieval. PhD thesis, MIT, 2000.[7] B. Pellom. Sonic: The university of colorado continuousspeech recognizer. Technical report, University ofColorado, 2001.[8] L. van der Werff and W. Heeren. Evaluating ASRoutput for information retrieval. In Proc. of the SIGIR2007 workshop on Searching SpontaneousConversational Speech, pages 7–14, 2007.[9] P. Yu and F. Seide. A hybrid word / phoneme-basedapproach for improved vocabulary-independent searchin spontaneous speech. In Proc. ICSLP2004, pages293–296.

Subword-based Indexing for a Minimal False Positive Rate

NARCIS 

A hybrid word / phoneme-based approach for improved vocabulary-independent search in spontaneous speech.

Evaluating ASR output for information retrieval.

Examining the contributions of automatic speech transcriptions and metadata sources for searching spontaneous conversational speech.

Experiments in spoken document retrieval using phonetic n-grams.

Finding consensus in speech recognition: Word error minimization and other applications of confusion networks.

Large vocabulary decoding and conﬁdence estimation using wordposterior probabilities.

Sonic: The university of colorado continuous speech recognizer.

Subword-based Approaches for Spoken Document Retrieval.

Word and sub-word indexing approaches for reducing the eﬀects of oov queries on spoken audio.

http://purl.utwente.nl/publications/65238

Subword-based Indexing for a Minimal False Positive Rate

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information

NARCIS