We describe the runs for our participation in the Search sub-task of the Search and Hyperlinking Task at MediaEval 2013. Our experiments investigate the aect of using information about speech segment boundaries and pauses

on the effectiveness of retrieving jump-in points within the retrieved segments. We segment all three available types of transcripts (automatic ones provided by LIMSI/Vocapia and LIUM, and manual subtitles provided by BBC) into fixed-length time units, and present the resulting runs using the original segment starts and using the potential jump-in points. Our method for adjustment of the jump-in points achieves higher scores for all LIMSI/Vocapia, LIUM, and subtitles based runs

Eskevich, Maria

Jones, Gareth J.F.

DCU Online Research Access Service

Time-based Segmentation and Use of Jump-in Pointsin DCU Search Runs at the Search and Hyperlinking Taskat MediaEval 2013Maria EskevichCNGL Centre for Global Intelligent ContentSchool of ComputingDublin City University, Dublin, Irelandmeskevich@computing.dcu.ieGareth J.F. JonesCNGL Centre for Global Intelligent ContentSchool of ComputingDublin City University, Dublin, Irelandgjones@computing.dcu.ieABSTRACTWe describe the runs for our participation in the Searchsub-task of the Search and Hyperlinking Task at MediaE-val 2013. Our experiments investigate the affect of usinginformation about speech segment boundaries and pauseson the effectiveness of retrieving jump-in points within theretrieved segments. We segment all three available typesof transcripts (automatic ones provided by LIMSI/Vocapiaand LIUM, and manual subtitles provided by BBC) intofixed-length time units, and present the resulting runs usingthe original segment starts and using the potential jump-inpoints. Our method for adjustment of the jump-in pointsachieves higher scores for all LIMSI/Vocapia, LIUM, andsubtitles based runs.1. INTRODUCTIONThe constant growth in the size and variability of digitalmultimedia content being stored requires the developmentof techniques that not only identify files containing relevantcontent, but also bring the user as close as possible to the be-ginning of the relevant passage within this file to maximizethe efficiency of information access. This starting point, re-ferred to as the jump-in point, cannot simply be related tothe locations of the words of interest being spoken, since theuser may need to listen to the whole utterance in which thewords were used, or slightly bigger passages, in order to getthe idea of the context. Thus we assume that these jump-inpoints should occur at the beginning of the speech segmentsor utterances, and might be expressed by a pause in thespeech signal. This idea underlies our experimental setup.We create one retrieval run for each fixed-length segmenta-tion unit, but present it in two ways for further comparison:with the initial boundaries of the segments, and with ad-justed jump-in points.2. DATASETANDEVALUATIONMETRICSThe Search and Hyperlinking Task at MediaEval 2013uses television broadcast data provided by BBC, and en-hanced with varying additional content such as automaticspeech recognition (ASR) transcripts [2]. The collectionconsists of circa 1260 hours of data that represent 6 weeksCopyright is held by the author/owner(s).MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, Spain.Table 1: Number of documentsTranscript Window Size (seconds)Type 60 90 180LIMSI 96 418 64 403 31 907LIUM 95 091 63 308 31 210Subtitles 82 220 54 698 26 742of broadcast content, including news programs, talk shows,episodes of TV series, etc. The 50 test set queries for theknown-item retrieval task were created during user studiesat the BBC [1].The task was evaluated using three metrics: mean recip-rocal rank (MRR) which scores the rank of the retrievedsegment containing relevant content, mean generalized av-erage precision (mGAP) which combines the rank of therelevant segment and distance to the ideal jump-in point atthe start of the relevant content within the segment [6], andmean average segment precision (MASP) which combinesthe rank of the relevant segment with (ir)relevant length ofthe segment.[3].3. RETRIEVAL FRAMEWORKAs the files in the collection vary in style and length, wedecided to segment all the content into fixed length units.For these experiments we chose three values for segmentlength: 60, 90, and 180 seconds. These time units were thesame for all types of transcripts. However, the transcriptsdo not always cover the spoken content in the same way:the ASR system might recognise some noise as words, orhumans who create the manual subtitle transcripts mightconsider certain parts of the video non relevant for tran-scription. This explains the difference in the number of doc-uments for diverse types of transcripts and time units givenin Table 1.At the segmentation stage we stored the information aboutpotential jump-in points within each segment in a separatefile. The LIMSI/Vocapia transcript contains speech seg-ments boundaries predicted by their system [4]; whereasthe LIUM transcript [7] has only time stamps for the wordsin the transcript; and manual subtitles have time stampsassigned on the utterance level. Thus for the official sub-mission to the task we used as potential jump-in pointsthe speech segments in the LIMSI transcript, pauses thatare longer than 0.5 seconds between words in case of LIUMtranscript and utterances in case of manual subtitles. Ad-Table 2: Metric results (window = 60 seconds)Run parameters MRR mGAP MASPTranscript Type Speech Segment Pause 60 90 180 60 90 180 60 90 180LIMSI – – 0.241 0.266 0.185 0.132 0.133 0.089 0.142 0.138 0.010LIMSI + – 0.250 0.295 0.240 0.151 0.164 0.132 0.132 0.146 0.124LIMSI* – + 0.258 0.305 0.240 0.150 0.153 0.135 0.139 0.145 0.124LIUM – – 0.265 0.298 0.205 0.124 0.152 0.094 0.140 0.169 0.103LIUM – + 0.284 0.317 0.254 0.146 0.163 0.114 0.138 0.173 0.126Subtitles – – 0.343 0.369 0.217 0.209 0.191 0.0.092 0.223 0.231 0.093Subtitles – + 0.365 0.376 0.280 0.211 0.221 0.154 0.212 0.220 0.116Table 3: MRR results with varying window size (window size = 60, 30, 10 seconds)Run parameters Unit Size = 60 sec Unit Size = 90 sec Unit Size = 180 secTranscript Type Speech Segment Pause metric window metric window metric window60 30 10 60 30 10 60 30 10LIMSI – – 0.241 0.169 0.090 0.266 0.195 0.059 0.185 0.107 0.041LIMSI – – 0.250 0.223 0.091 0.295 0.226 0.110 0.240 0.178 0.080LIMSI* – + 0.258 0.194 0.109 0.305 0.226 0.090 0.240 0.175 0.081LIUM – – 0.265 0.157 0.071 0.298 0.204 0.080 0.205 0.116 0.041LIUM – + 0.284 0.182 0.106 0.317 0.213 0.110 0.254 0.146 0.081Subtitles – – 0.343 0.273 0.144 0.369 0.255 0.096 0.217 0.113 0.042Subtitles – + 0.365 0.300 0.155 0.376 0.292 0.141 0.280 0.193 0.120ditionally we created an unofficial run that uses the pausesin the LIMSI transcript in order to be able to make a bettercomparison with the other types of transcript.We do not have access to details of the ASR transcrip-tion systems. However we can distinguish them by the sizeof the vocabulary they used for this collection which con-tain 36,815, 57,259, and 98,332 entries for LIMSI/Vocapia,LIUM, and subtitles respectively.For indexing and retrieval experiments we used the open-source Terrier Information Retrieval platform1 [5] with astandard language modelling method, with default lamdavalue equal to 0.15.4. RESULTS, CONCLUSIONS ANDFURTHERWORKTable 2 shows the results obtained for all three types oftranscript and different segmentation unit size. The lines forthe same transcript represent the same retrieval run, withthe second line (and the third one for LIMSI) represent-ing the enhanced result list. We highlight in bold the runsfor which the addition of the jump-in point information in-creases the effectiveness of the results. In the case of LIMSItranscript there is no consistency that indicates whether theuse of speech segments or pauses is preferable. This may becaused by the fact that sometimes these potential jump-inpoints coincide. For shorter segments the use of speech seg-ments has better mGAP scoring, meaning that the speechsegment based jump-in point brings the user closer to thebeginning of the relevant content.Table 3 shows the MRR results for varying window used inthe metric calculation. The LIUM and subtitles based runsshow the same trend of improvement of the score from theuse of pause information in calculating the jump-in point.These results allow us to argue that the simple time basedsegmentation results can be improved by use of pauses in-1http://www.terrier.orgformation contained in all types of transcripts.5. ACKNOWLEDGMENTSThis work was supported by Science Foundation Ireland(Grant 07/CE/I1142) as part of the Centre for Next Gener-ation Localisation (CNGL) project at DCU.6. REFERENCES[1] R. Aly, R. Ordelman, M. Eskevich, G. J. F. Jones, andS. Chen. Linking inside a video collection: what andhow to measure? In WWW (Companion Volume),pages 457–460, 2013.[2] M. Eskevich, G. J. Jones, S. Chen, R. Aly, andR. Ordelman. The Search and Hyperlinking Task atMediaEval 2013. In MediaEval 2013 Workshop,Barcelona, Spain, October 18-19 2013.[3] M. Eskevich, W. Magdy, and G. J. F. Jones. Newmetrics for meaningful evaluation of informallystructured speech retrieval. In Proceedings of ECIR2012, pages 170–181, 2012.[4] L. Lamel and J.-L. Gauvain. Speech processing foraudio indexing. In Advances in Natural LanguageProcessing (LNCS 5221), pages 4–15. Springer, 2008.[5] I. Ounis, G. Amati, V. Plachouras, B. He,C. Macdonald, and C. Lioma. Terrier: A HighPerformance and Scalable Information RetrievalPlatform. In Proceedings of ACM SIGIR’06 Workshopon Open Source Information Retrieval (OSIR 2006),2006.[6] P. Pecina, P. Hoffmannova, G. J. F. Jones, Y. Zhang,and D. W. Oard. Overview of the CLEF 2007cross-language speech retrieval track. In Proceedings ofCLEF 2007, pages 674–686. Springer, 2007.[7] A. Rousseau, F. Bougares, P. Dele´glise, H. Schwenk,and Y. Este`v. LIUM’s systems for the IWSLT 2011Speech Translation Tasks. In Proceedings of IWSLT2011, San Francisco, USA, 2011.

Time-based segmentation and use of jump-in points in DCU search runs at the search and hyperlinking task at MediaEval 2013

We describe the runs for our participation in the Search sub-task of the Search and Hyperlinking Task at MediaEval 2013. Our experiments investigate the aect of using information about speech segment boundaries and pauses\ud
on the effectiveness of retrieving jump-in points within the retrieved segments. We segment all three available types of transcripts (automatic ones provided by LIMSI/Vocapia and LIUM, and manual subtitles provided by BBC) into fixed-length time units, and present the resulting runs using the original segment starts and using the potential jump-in points. Our method for adjustment of the jump-in points achieves higher scores for all LIMSI/Vocapia, LIUM, and subtitles based runs

Name not available

We describe the runs for our participation in the Search sub-task of the Search and Hyperlinking Task at MediaEval 2013. Our experiments investigate the aect of using information about speech segment boundaries and pauses
on the effectiveness of retrieving jump-in points within the retrieved segments. We segment all three available types of transcripts (automatic ones provided by LIMSI/Vocapia and LIUM, and manual subtitles provided by BBC) into fixed-length time units, and present the resulting runs using the original segment starts and using the potential jump-in points. Our method for adjustment of the jump-in points achieves higher scores for all LIMSI/Vocapia, LIUM, and subtitles based runs

Irish Universities

http://doras.dcu.ie/20372/1/mediaeval2013_submission_72.pdf

Time-based segmentation and use of jump-in points in DCU search runs at the search and hyperlinking task at MediaEval 2013

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Name not available

Irish Universities