This paper describes the issues and preliminary work involved

in the creation of an information retrieval system that will

manage the retrieval from collections composed of both speech

recognised and ordinary text documents. In previous work, it

has been shown that because of recognition errors, ordinary

documents are generally retrieved in preference to recognised

ones. Means of correcting or eliminating the observed bias is

the subject of this paper. Initial ideas and some preliminary

results are presented

Sanderson, M.

Shou, X.M.

English

White Rose Research Online

promoting access to White Rose research papers    White Rose Research Online   Universities of Leeds, Sheffield and York http://eprints.whiterose.ac.uk/   This is an author produced version of a paper published in Information Retrieval Techniques for Speech Applications.   White Rose Research Online URL for this paper: http://eprints.whiterose.ac.uk/4594/    Published paper Sanderson, M. and Shou, X.M. (2002) Speech and hand transcribed retrieval. In: Information Retrieval Techniques for Speech Applications : this book is based on the workshop "Information Retrieval Techniques for Speech Applications", held as part of the 24th Annual International ACM SIGIR Conference 2001. Lecture Notes In Computer Science (2273). Springer-Verlag , Berlin / Heidelberg, pp. 78-85. eprints@whiterose.ac.uk  Speech and Hand Transcribed RetrievalMark SandersonDepartment of Information StudiesUniversity of SheffieldWestern Bank, Sheffield, S10 2TN, UK+44 114 22 22648m.sanderson@shef.ac.ukXiao Mang ShouDepartment of Information StudiesUniversity of SheffieldWestern Bank, Sheffield, S10 2TN, UK+44 114 22 22675x.m.shou@shef.ac.ukABSTRACTThis paper describes the issues and preliminary work involvedin the creation of an information retrieval system that willmanage the retrieval from collections composed of both speechrecognised and ordinary text documents.  In previous work, ithas been shown that because of recognition errors, ordinarydocuments are generally retrieved in preference to recognisedones.  Means of correcting or eliminating the observed bias isthe subject of this paper.  Initial ideas and some preliminaryresults are presented.General TermsMeasurement, Experimentation.KeywordsInformation Retrieval, Spoken Document Retrieval, MixedCollections.1. IntroductionAt present, the main stream of information retrieval researchremains in homogenous data collections.  However, with theprogress of multimedia technologies, the trend for searchingheterogeneous data collections is growing.  The MIND(Multimedia International Digital Libraries) project [MIND01]that the authors are currently working with provides a typicalexample for this case.  The research in MIND addresses issuesthat arise when people have remote access to a great manyheterogeneous and distributed multimedia digital librariescontaining text, images, and audio data such as speechrecognised documents.  In order to cope effectively with suchmasses of knowledge, a subset of the digital libraries that mostlikely contains relevant documents must be identified, selected,and searched.  The results obtained from those resources needto be fused and merged into a single result.  Both the means ofresource selection and data fusion must be achieved withoutbias.When ranking documents in a collection in relation to a query,almost all of the weighting schemes and retrieval models usedassume that documents are created equal.  The presence orabsence of a query word or the number of times that wordoccurs within documents is assumed to have the samesignificance across the collection (once document length hasbeen taken into account).  If, however, a collection is composedof documents created in different ways, such as a collection ofordinary and speech recognised documents, or collections ofspeech recognised documents where the recognition accuracyvaries widely, the assumptions are likely to be false.In general, retrieval of spoken documents is viewed as asuccess [Garofolo99b], speech recognition (SR) systems areaccurate enough to allow retrieval to operate at an acceptablelevel.  Spoken document retrieval (SDR) is being used:internally within corporations ([Abberley98], [Renals99],[THISL01]), and as a Web search engine ([SpeechB00A],[SpeechB00B], [SpeechB01]).  Word error rates (WER) on theaudio documents being retrieved are still relatively high,however, because relevant documents generally contain eachquery term with a high term frequency (tf), as long as a few ofthe term occurrences are recognised, relevant documents willbe retrieved.  If, however, the collection is composed of bothspeech recognised and ordinary text documents, because the tfin the recognised documents is lower, the ordinary documentswill be retrieved in preference to the recognised.  This problemis also found within the collection of spoken documents, someaudio documents are likely to be recognised more accuratelythan others, those recognised well will contain query termswith higher tf than those recognised badly1.Within the framework of the MIND project, it is also assumedthat remote digital library providers will be either uninterestedor unwilling to provide detailed data on the content of theirlibraries, therefore, it is desirable to determine the informationindependently.  This paper proposes an approach for dealingwith this problem (for speech data) through the automaticidentification of spoken documents and subsequent estimationof the word error rate within those documents.  Once suchinformation has been gained, document retrieval scores will beadjusted to ensure a fairer ranking.  The rest of this paperdescribes past work in this area, followed by a description ofthe methods to be used in the system.  Some preliminary workis then outlined before finally concluding.2. Past workAlthough a great deal of research has been conducted in theretrieval of corrupted data, be it scanned text [Jones01], speech([Garofolo99b], [Abberley98]), or translated foreign language                                                            1 For such documents, there is also the problem of no queryterm occurrences being recognised at all.  Clearly animportant problem, which has been addressed in past work(e.g. the document expansion of Singhal [Singhal99] and useof multiple hypotheses by Siegler [Siegler97], it is notdiscussed here.documents [Franz98], relatively little work has investigated thenotion of varying levels of corruption in the collection(s) beingretrieved.  This is perhaps surprising, as it is quite reasonableto expect such variation.  Sanderson and Crestani brieflyaddressed the problem of retrieving from a collection composedof both ordinary and spoken documents [Sanderson98],reporting that using a conventional tf·idf weighting scheme,ordinary documents were retrieved in preference to spokendocuments.  Since then, research in the area has come from theCross Language Information Retrieval (CLIR) communitywhere retrieval from collections composed of documentswritten in different languages requires consideration oftranslation accuracy.  Poorly translated documents are lesslikely to be retrieved.  Effective methods to deal with thissituation have been devised by identifying the language of thedocuments and biasing ranking algorithms based on trainingdata derived from cross language test collections [Franz98].Reported as being effective, the only problem with thistechnique is the assumption that corruption from translationerror is consistent across documents of the same language, anassumption which is likely to be false with the corruption fromrecognition used in MIND.3. MethodsThe proposed means of providing a fair ranking when dealingwith collections of documents composed of either ordinary andspoken documents or spoken document recognised at varyinglevels of accuracy requires two components: a spoken documentidentification system; and a method to estimate word errorrates.  They are described here.3.1 Spoken document identificationWhen considering how to spot if a piece of text was written orautomatically recognised, one can view spoken documentidentification as a form of language identification([LangID01A], [LangID01B], which is used to identify the kindof language that a given document was written in.  Thoughtechniques applied in language identification such as training aword or character based n-gram model for recognised outputand ordinary text, etc. could also be used, some surface cues inthe document are likely to be able to identify whether or not itis the output of speech recognition.  For example, speechrecognisers generally do not insert punctuation in a finegranularity level and do not have clear boundary betweensentences and paragraphs (here only the raw speech recogniseddata with no human corrections is considered).  Therefore, byexamining the ratio of punctuation to non-punctuation tokens,and/or the ratio of capitalised to non-capitalised letters mayprovide sufficient clues for the identification task.  Anotherpossible solution could be to check whether a documentcontains common spelling mistakes as speech recognisersworking from a pre-defined vocabulary do not misspell words.Alternatively, a more sophisticated approach is to determine ifthe document contains any words that are outside therecogniser’s lexicon.  However, since different speechrecognisers have different lexicons and getting the lexicons ofthem can be difficult, this method cannot be generally applied.Table 1 shows some preliminary results by comparing thepunctuation and upper case words in 1804 speech recognisedand 115 hand transcribed files from TREC-9 collections.From the table, it shows a magnitude difference in thepercentage of punctuation in speech recognised and handtranscribed collections.  The punctuation within speechdocuments contain full stops and apostrophe marks onlywhereas hand transcriptions contains evenly distributed comma,hyphen, parenthesis and question marks as well as full stopsand apostrophes.  By further examining the speech recogniseddocuments, most of the full stops are not addressed to finish asentence but used as abbreviations such as H.I.V., U.N., etc.Though strictly they should not be counted as full stops, themagnitude difference in the punctuation percentage is notaffected by including them.The speech recognised documents used in the experiment arethe “.srt” format files where all words are in capital letters.This may not always be true as some other speech recognisedfiles may contain lower case letters only.  Therefore, ourexperiment does not cover all situations but extremely high orlow rate of upper case words can be used as an indication forspeech document identification.3.2 Word error rate estimationOnce a document has been identified as one generated by a SR,the next stage will be determining how much error is in thetranscript.  In speech recognition, the conventional method is tomeasure the Word Error Rate (WER) by comparing the outputof the recogniser to an accurate hand transcription.  In thesituation we anticipate such a transcript will not be available.In seeking an automatic method for estimating error rates, oneshould consider how speech recognisers determine the mostlikely sequence of words that were spoken.  In addition to theacoustic models that recognise phonemes, modern SR systemsuse language models derived from large corpora to determinethe most likely word sequence given a set of recognisedphonemes.  The models are applied over a small slidingwindow of uttered words (no more than four or five) and havethe effect of improving recognition quality.  However, evenwhen incorrectly recognising words, the models produce textsthat at the level of word pairs or triples make some form ofsense.  For example the word sequence “On world news tonightthis Wednesday...” is recognised by a poor speech recogniser as“on world is unlike his wins the...”: nonsensical, but each wordpairing “on world”, “world is”, “is unlike”, “unlike his”, “hiswins” & “wins the” are sensible when viewed on their own.TotalwordsTotalpunctuationPercent ofpunctuationTotal uppercase wordsPercent of uppercase wordsSpeech recognised files 9,117,820 360,359 0.04 9,117,820 1.00Hand transcriptions 822,760 198,151 0.24 59,401 0.07Table 1: Comparison of punctuation and upper case words in speech recognised and hand transcribed filesConsequently, searching a recognised text for unusual wordpairs or triples to estimate WER is unlikely to be fruitful.  It isbetter to examine attributes of documents beyond local context.The method we propose is based on the observation made inthe retrieval experiments of previous work by one of theauthors (i.e. [Sanderson98]): where ordinary documents werefound to be retrieved in preference to the speech recognised.The reason for this was the lower tf value of query terms in therecognised documents when compared to the hand transcribed.This was caused by the SR system failing to recognise alloccurrences of all the document words and replacing the mis-recognised words with others.  We postulate that, in general,the poorer the recognition accuracy of an audio sequence, thegreater the number of single occurrence words within thetranscription produced.To test this, a small initial experiment was conducted using theTREC SDR 1998 collection: a collection composed ofapproximately 3,000 short documents transcribed from USnews reports.  The average number of occurrences of all wordsin a document was computed and the average of this numberwas calculated across all documents of the collection.  Theresulting single value provided a simple measure of how oftenwords are repeated within collection documents.  Thanks toefforts by SDR track participants in 1998, transcripts from eightSR runs were made available to fellow participants along withthe hand transcription, all shared runs were used in theexperiment.  As reported by Garofolo et al [Garofolo99a],WERs on the runs varied greatly: the runs plus their error ratesare listed in table 2 ranked by WER.The average term occurrence measure (to) for each of thetranscripts along with the score for the hand transcript wascomputed.  The results are shown in table 3 with the transcriptsordered by score.  As can be seen even in the hand transcriptionwords are rarely used more than once, however the score withone notable exception (i.e. NIST B2) does appear to be rankingsystems by their WER.A closer examination of the NIST B2 transcripts indicated thatthe reason for the high term occurrence measure was due towords with low idf being repeated many times withindocuments.  Consequently the experiment was repeated butwith the 1,000 highest frequency words ignored whenmeasuring.  The results of this experiment are shown in table 4below.  (Note, the overall occurrence rate measure is lower forall transcripts, this is to be expected given the removal of themore frequent terms.)  Using a simple correlation test, theranking produced by the altered measure was closer to theRank System Description WER (%)1 LTT Hand transcription of audio data 0.02 CU HTK Cambridge University using HTK toolkit 24.63 Dragon University of Massachusetts & Dragon systems 29.54 ATT AT&T 31.05 NIST B1 NIST B1 system 33.86 Shef Sheffield University using Abbot system 35.67 NIST B2 NIST B2 system 47.58 DERA S2 Defence Evaluation and Research Agency, UK 61.39 DERA S1 Defence Evaluation and Research Agency, UK 66.0Table 2: Description and WER of SDR 1998 SR systems.Rank System to WER Rank1 LTT 1.28 12 CU HTK 1.21 23 ATT 1.19 44 NIST B1 1.19 55 NIST B2 1.18 76 Dragon 1.18 37 Shef 1.18 68 DERA S2 1.13 89 DERA S1 1.12 9Table 3: SR systems ranked by term occurrence measureRank System to WER Rank1 LTT 1.18 12 CU HTK 1.12 23 NIST B1 1.11 54 ATT 1.11 45 Dragon 1.10 36 Shef 1.09 67 NIST B2 1.07 78 DERA S2 1.06 89 DERA S1 1.05 9Table 4: Systems ranked by modified occurrence measureWER rate ranking in Garofolo et al’s paper.4. Conclusions and future workThis paper has described in part, the retrieval problem of theMIND project and described initial work tackling the importantproblem of retrieving from collections of documents withvarying levels of error within them.  A short experimentshowed that the proposed ideas are promising and deserving offurther investigation and it must be remembered that themethod illustrated here only provides a means of rankingtranscripts against each other.  What is desired is a means ofdetermining levels of error rate from a single transcript.  Howto achieve this, is our next challenge.5. AcknowledgementsThis work was funded by the EU 5th Framework project,MIND: IST-2000-26061 (mind.cs.strath.ac.uk).6. References[Abberley98] D. Abberley, S. Renals and G. Cook;“Retrieval of broadcast news documents withthe THISL system”; In Proceeding IEEEICASSP, pp 3781-3784; Seattle, 1998[Franz98] M. Franz, J.S. McCarley, S. Roukos;  “Ad hocand Multilingual Information Retrieval atIBM”; Proceeding of The Seventh TextREtrieval Conference (TREC 7); November 9-11, 1998; pp157-168[Garofolo99a] J.S. Garofolo, E.M. Voorhees, C.G.P. Auzanne,M. Stanford, B.A. Lund; “1998 TREC-7Spoken Document Retrieval Track Overviewand Results”, in Proceedings of the DARPABroadcast News Workshop, 1999[Garofolo99b] John S. Garofolo, Cedric G. P. Auzanne, EllenM. Voorhees;  “The TREC Spoken DocumentRetrieval Track: A Success Story”; TextRetrieval Conference (TREC) 8, E. Voorhees,Ed.; Gaithersburg, Maryland, USA; November16-19, 1999 [Jones01] G.J.F.Jones and M.Han;  “Retrieving ScannedDocuments from a Mixed-Media DocumentCollection”; Proceedings of the BCS-IRSGEuropean Colloquium on IR Research;Darmstadt, Germany, pp136-149, April 2001[LangID01A] Automatic Language IdentificationBibliography;http://speech.inesc.pt/~dcaseiro/html/bibliografia.html, last accessed: June 2001[LangID01B] Language Identification; http://translation-guide.com/language_identification.htm, lastaccessed: June 2001[MIND01] Resource Selection and Data Fusion forMultimedia International Digital Libraries;http://mind.cs.strath.ac.uk/, last accessed: June2001[Renals99] Steve Renals and Dave Abberley;  “The THISLSDR system at TREC-9”; Proceedings ofTREC-9;http://www.dcs.shef.ac.uk/~sjr/pubs/2001/trec9.html, last accessed: June 2001[Sanderson98]  Mark Sanderson and Fabio Crestani;  “Mixingand Merging for Spoken Document Retrieval”;Proceedings of the 2nd European Conference onDigital Libraries;  Heraklion, Greece,September 1998, pp397-407. Lecture Notes inComputer Science N. 1513, Springer Verlag,Berlin, Germany.[Siegler97] M.A. Siegler, M.J. Witbrock, S.T. Slattery, K.Seymore, R.E. Jones and A.G. Hauptmann;  “Experiments in Spoken Document Retrieval atCMU”; Proceeding of the 6th Text REtrievalConference (TREC 6); November 19-21, 1997;pp291-302[Singhal99] Amit Singhal, Fernando C. N. Pereira;“Document Expansion for Speech Retrieval”;SIGIR 1999; pp34-41[SpeechB00A] Jean-Manuel Van Thong, David Goddeau,Anna Litvinova, Beth Logan, Pedro Morenoand Michael Swain;  “SpeechBot: A SpeechRecognition based Audio Indexing System forthe Web”; International Conference onComputer-Assisted Information Retrieval,Recherche d'Informations Assistee parOrdinateur (RIAO2000); Paris, April 2000; pp106-115[SpeechB00B] Pedro Moreno, JM Van Thong, Beth Logan,Blair Fidler, Katrina Maffey,and MatthewMoores;  “SpeechBot: A Content-based SearchIndex for Multimedia on the Web”;  First IEEEPacific-Rim Conference on Multimedia, (IEEE-PCM 2000), 2000[SpeechB01] Compaq Speechbot;http://www.compaq.com/speechbot/, lastaccessed: June, 2001[THISL01] The THISL Spoken Document RetrievalProject;http://www.dcs.shef.ac.uk/spandh/projects/thisl/overview-oct98/, last accessed: June, 2001

Speech and hand transcribed retrieval

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

Crossref

White Rose Research Online