Information Retrieval is a supporting technique which underpins a broad range of content-based applications including retrieval, filtering, summarisation, browsing, classification, clustering, automatic linking, and others. Multimedia information retrieval (MMIR) represents those applications when applied to multimedia information such as image, video, music, etc. In this presentation and extended abstract we are primarily concerned with MMIR as applied to information in digital video format. We begin with a brief overview of large scale evaluations of IR tasks in areas such as text, image and music, just to illustrate that this phenomenon is not just restricted to MMIR on video. The main contribution, however, is a set of pointers and a summarisation of the work done as part of TRECVid, the annual benchmarking exercise for video retrieval tasks

G. Kazai

J.G. Fiscus

English

Smeaton, Alan F.

DCU Online Research Access Service

Large Scale Evaluations of MultimediaInformation Retrieval: The TRECVidExperienceAlan F. SmeatonAdaptive Information Cluster,Center for Digital Video Processing,Dublin City University,Glasnevin, Dublin 9, Ireland.Alan.Smeaton@dcu.ieAbstract. Information Retrieval is a supporting technique which un-derpins a broad range of content-based applications including retrieval,filtering, summarisation, browsing, classification, clustering, automaticlinking, and others. Multimedia information retrieval (MMIR) representsthose applications when applied to multimedia information such as im-age, video, music, etc. In this presentation and extended abstract weare primarily concerned with MMIR as applied to information in digitalvideo format. We begin with a brief overview of large scale evaluationsof IR tasks in areas such as text, image and music, just to illustrate thatthis phenomenon is not just restricted to MMIR on video. The maincontribution, however, is a set of pointers and a summarisation of thework done as part of TRECVid, the annual benchmarking exercise forvideo retrieval tasks.1 IntroductionThe broad area of Multimedia Information Retrieval (MMIR) represents an in-tersection of work across several disciplines; by the very definitions of multimediaand of information retrieval this is inescapable and we now draw on computer sci-ence, engineering, information science, mathematics, statistics, human-computerinteraction, networking, even hardware, and others in order to make progress inMMIR. However, the two areas which contribute most to current work in videoretrieval and also in image retrieval are image/video processing and informationretrieval. For MMIR, these two are mutually re-enforcing as in order to get us-able descriptions of large amounts of video and image content we need to be ableto analyse that video and image information, automatically; similarly, in orderto be able to build effective content-based retrieval systems1 for such informationwe need to use techniques developed in information retrieval research. Given the1 We define content-based retrieval systems to be those that support searching, brows-ing, summarisation, abstracting, (hyper-)linking, categorisation, clustering, and fil-tering . . . in fact any operations which work directly on image or video content.origins of MMIR, it is worth looking at how its parent disciplines have come toregard large-scale evaluations.For many years the main challenges driving the development of technologyto support video in digital form were to be able to capture video digitally, tostore it efficiently, to be able to package it on portable media, to send it overnetworks efficiently and then to render it on devices so people could then see it.Each of these tasks required the same essential ingredient in order to make themfeasible for all but toy examples, namely effective data compression. In the earlyand mid 1980s vendors developed image processing software without due regardto the benefits of using standard formats for encoding images and users of imageprocessing software had a real headache as proprietary formats predominatedand interoperability across image processing software was only a pipe dream.We can see the remnants of that situation even today with the proliferation ofimage coding standards still available.As computers became more powerful and manipulation of video in digitalformat on personal computers started to loom into the realms of possibility, theworld became much more conscious of the benefits of using a standard format forencoding media. As the standardisation of encoding formats became a commongoal in video processing, we became conscious of the importance of benchmark-ing and evaluation of video processing techniques, including compression, onstandard datasets. A good example of this in video compression is the fact thatthe mother and child and the table tennis videos are so well-known and well-usedas benchmarks for compression, that the people appearing in them are famous.2 Evaluation Benchmarking ActivitiesIn (text) information retrieval, the advantages of conducting retrieval experi-ments on a common dataset which was also being used by others has alwaysbeen a central part of the IR discipline and test collections consisting of docu-ments, queries or topics, and relevance assessments for those topics have beenaround since the earliest days. These tended to be small in size and not uni-versally available but there has always been reasonable comparability amongresearchers in terms of empirical research work. About 15 years we saw a scale-up on the size of the collections and more importantly a concerted effort atproviding relevance assessments on these large collections with the introductionof the first of the TREC (Text REtrieval Conference) exercises [1]. These havecontinued annually since 1991 and have spun off many “tracks” or variations in-cluding filtering, cross-lingual, non-English, web, high precision, high accuracy,genomic, question-answering, large scale and others. TREC, and its collections,evaluation measures and the sizes of its collections are now established in IR asthe baseline for empirical text-based IR.While TREC has had a huge impact on text information retrieval, its successin attracting researchers to the common benchmark has led to repeats of thebasic approach for other media and IR tasks.The INitiative for the Evaluation of XML Retrieval (INEX) [2] is an an-nual benchmarking exercise culminating in a workshop which examines retrievalperformance from highly-structured XML documents. The retrieval of XML el-ements using a variety of techniques tried in INEX has shown net improvementsin retrieval precision over its first few years. The music information retrievalcommunity who target content-based music retrieval based on melody, etc. arelaunching a TREC-like evaluation in 2005 called “MIREX”, the Music Informa-tion Retrieval Evaluation eXchange [3]. For the community of researchers whotarget content-based image retrieval the problem of creating a TREC-like evalu-ation is more difficult because so many CBIR researchers have been doing theirown thing for so long, and because the subjectivity of relevance judgments inimage retrieval is probably moreso than for any other media. However, a goodstarting point for evaluation of CBIR can be found in the Benchathlon effort [4].3 TRECVid: Evaluation of Video Retrieval TasksIn 2001 a new TREC track was introduced on the subject of video retrieval. Un-like previous tracks, the video track had more than one task and included shotboundary detection, feature detection and search as tasks. For the search task,the track followed the usual TREC mode of operation of gathering and distribut-ing (video) data, formulating search topics, accepting search submissions fromparticipating groups, pooling submitted results together and performing evalua-tion by comparing submissions against a ground truth of relevant shots derivedfrom manual assessment of pooled submissions. The additional, non-search taskshad a similar model for operation involving many of the phases used in search.In the first year the track had a small community of participants and thisgrew in the second and subsequent years. At the time of writing there are 63groups signed up for participation in 2005 and the data for 2005 (developmentand testing) is over 200 hours of video.In 2003 TRECVid separated from TREC because it was sufficiently differentto the main TREC activity, and it had enough participation to be independent,although the TREC and TRECVid workshops are co-located each year.The results obtained by participants in TRECVid each year are publishedin the TRECVid proceedings available online [5] and TRECVid activities havebeen published in a host of other places [6]. Overviews of the TRECVid activityhave also been published previously [7], [8] and rather than repeat that here Iwill simply summarise the achievements of each of the tasks to date.3.1 Shot Boundary DetectionThe shot boundary detection task basically involves automatically determiningboth hard and gradual transitions between shots. Within TRECVid shots arethe basic unit of information for search and feature detection and a commonshot boundary detection is made available for these other tasks but for the SBDtask a collection of approximately 5 hours is used, with manual ground truthestablished. The task is regarded as being one of the easier of the TRECVidtasks and has proven to be popular, and a good task with which to enter thearea of video retrieval for groups wishing to break into it.In 2004 we added performance speed as one of the metrics and this hasrevealed large differences in execution speed (from 1/30 to 3x real time) forapproximately the same levels of performance. In general we believe that hardcut detection seems more or less “solved but there is still room for improvementin the detection of gradual transitions. Another question we are planning toaddress in 2005 is how well do the approaches transfer to other sources/types ofvideo besides broadcast TV news ?3.2 Story SegmentationThe task here is to use audio/video and the automatic speech recognition (ASR)transcript (including transcription errors) to segment a broadcast into individualnews stories. This task is a more elaborate version of the task already tried withtranscript-only in the Topic Detection and Tracking activity [9]. In TRECVidwe have seen a wide range of approaches and of system complexity with thecombination of AV and ASR giving only a small gain for segmentation overASR only. Interestingly, most approaches are generic and not attuned to thepeculiarities of the TV broadcasters we used (ABC and CNN).Although this task ran for only 2 years (2003 and 2004) and the resultsimproved in the second year, the overall results obtained show there is stillfurther room for improvement.3.3 Feature DetectionThe automatic detection of features is potentially one of the real enablers forvideo retrieval. Being able to pre-process video to automatically detect a rangeof mid- and high-level semantic features would make video retrieval, and post-retrieval result clustering a powerful tool. The difficulty in achieving this is partlydetermining what makes a good set of features to target for automatic detection(in terms of complementarity and achievability), as well as then realising thosedetections. Because of its importance, feature detection has been present in eachof the years of TRECVid.The usual mode of operation for feature detection is to take a baseline ofannotated video, usually manually annotated, and to train some kind of classi-fier on this data. Support Vector machines (SVMs) and other machine learningapproaches have proved to be popular and feature-neutral systems are partic-ularly attractive since they can be re-applied to new features without a lot ofre-engineering needing just more and more training data. the supply of trainingdata for this task has proved a very large obstacle though we are fortunate thatin 2003 IBM coordiated an effort among TRECVid participants to annotate acorpus of video data which was then made available to all participants. In 2005we are repeating this annotation exercise and when we sough volunteers to assistwith this we had more than enough volunteer effort to annotate the whole set oftraining data (80 hours), twice, which is exactly what we are doing ! This trainingdata should be an invaluable resource for those developing feature detectors.3.4 SearchingIn TRECVid, search involves matching a multimedia expression of an informa-tion need against a corpus of video ad retrieving a ranked list of shots whichsatisfy the information need. the topic is expressed as a combination of text,example image(s) and example video clip(s). There are 3 types of searching fa-cilitated in TRECVid which vary in the amount of user intervention allowed inthe searching process. In the interactive search task the user is given a maximumof 15 minutes to use a search tool to find relevant shots; in the manual searchtask the user is allowed to formulate the topic as a query, once, and this is thensubmitted to the system which produces a ranked list of shots; in the automaticsearch task there is no user intervention at all and the topic is submitted to thesystem verbatim.After 4 years of the search task we are genuinely surprised by the amountof variation and creativity that participants introduce into the process. Thereare many interesting shot browsing interfaces which use the keyframes providedby TRECVid yet the text search which is run against the automatic speechtranscripts (ASR) continues to be the single most important modality for videoretrieval, being far more important than retrieval based on visual features. Thereis some use of high-level feature concepts which are made available from thefeature detection task (outdoors, people, water, . . .), and a lot of use of low-levelfeatures (color, edges, texture, . . .) to enable query by visual similarity againstthe keyframes. Many groups have managed to use browsing in the temporalneighborhood of an already found shot, leveraging the fact that a story aboutflooding will likely have several shots of flood waters and thus local contextbrowsing is useful for shots about flood waters. Some groups also use outsideresources to enhance their text search, resources such as Google and WordNetand while most groups use positive relevance feedback, not so many use negativerelevance feedback.At the end of a 2-year cycle of evaluation of shot retrieval from TV newswe can say that the evaluation has stabilized but in 2004 we did not get anygiant leap forward in systems which suggests we have reached a plateau. Yetagainst this we continue attract more and more groups, each of which brings innew approaches and techniques the best of which are then picked up by others.Furthermore we continue to gain insights and improvements, even in the bettersystems. Most impressive of all are the range of demonstrations of these systemswhich when viewed as a collective are at, or near to, the state of te art in videoretrieval3.5 TRECVid in 2005TRECVid 2005 is likely to see growth in the number of participants (63 groupssigned up for 2005 whereas just over 30 completed at least 1 task in 2004), inthe amount of data (nearly 220 hours of video), and in the complexity of thevideo (166 hrs of English, Chinese and Arabic news from Nov 2004 with ASRand machine translation to English as well as 50 hours of “rushes” from BBCand further data from NASA educational programs). The specific tasks to beincluded in 2005 are:– Shot boundary determination, which is the same as previously except on theNASA data– Low-level feature extraction (mostly camera motion)– High-level feature extraction using 10 semantic features, some of which area repeat from 2004– Search (interactive, manual, and automatic) on the broadcast TV news in 3languages, plus a new pilot task on BBC rushesThe search tasks in 2005 are by far the most challenging to date. The noiseintroduced by using far more broadcast sources as opposed to the CNN/ABCpair used previously, as well as the noise introduced by the machine translationof Chinese and Arabic to English will pose real challenges, especially given theway text has continually shown to be the most important of modes for retrieval.The other challenge is in the task of providing search on the BBC rushes data.Rushes are the raw video clips which are then edited and post-produced into TVprograms and the rushes contain lots and lots of redundancy both in the repeatedre-filming of a given shot as well as the lingering of the camera before andafter the “action” component of the shot. Most interestingly, there is generallyno dialogue in rushes video, so there is no ASR and no basis for using text,meaning that the search on rushes will have to be video-only. Definitely, thiswill be challenging !4 ConclusionIn this paper we have provided a short taster of the work undertaken each yearas part of the TRECVid evaluation exercise as well as some pointers to otherbenchmarking exercises in related fields in IR. TRECVid continues to grow fromstrength to strength and is probably the single most influential activity in thearea of video retrieval. This is something we have to be cautious about, however,and we must be aware of the danger of the field wrapping itself around TRECVidand its tasks rather than having TRECVid as something which should supportthe development of the field of video information retrieval.AcknowledgmentsThe author wishes to acknowledge the considerable input of Paul Over (NIST)and Wessel Kraaij (TNO) who, along with the author, coordinate TRECVideach year. The author is partly-supported by Science Foundation Ireland undergrant 03/IN.3/I361.References1. The TREC Website. Available at http://trec.nist.gov Last visited April 2005.2. Kazai, G., Lalmas, M. and de Vries, A.P. The Overlap Problem in Content-OrientedXML Retrieval Evaluation. In: Proceedings of the 27th annual international confer-ence on Research and development in information retrieval, SIGIR ’04, pp. 72–79,Sheffield, United Kingdom, ACM Press, 2004.3. MIREX 2005: 2nd Annual Music Information Retrieval Evaluation eXchange. Avail-able at http://www.music-ir.org/evaluation/MIREX/ Last visited April 2005.4. The Benchathlon Network: Home of CBIR Benchmarking Available athttp://www.benchathlon.net/ Last visited April 2005.5. TREC Video Retrieval Evaluation Online Proceedings. Available at http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html Last visited April 2005.6. TREC Video Retrieval Evaluation Bibliography. Available at http://www-nlpir.nist.gov/projects/t01v/trecvid.bibliography.txt Last visited April 2005.7. Smeaton, A.F., Kraaij, W. and Over, P. The TREC Video Retrieval Evaluation(TRECVID): A Case Study and Status Report. In Proceedings of RIAO 2004 - Cou-pling Approaches, Coupling Media and Coupling Languages for Information Retrieval,Avignon, France, 26-28 April 2004.8. Kraaij, W., Smeaton, A. F. , Over, P. and Arlandis, J. TRECVID 2004 - AnOverview. In Proceedings of TRECVid 2004, National Institute for Standards andTechnology http://www-nlpir.nist.gov/projects/tvpubs/tvpapers04/tv4overview.pdfLast visited April 2005.9. J.G. Fiscus and G.R. Doddington. Topic Detection and Tracking EvaluationOverview. In: Topic Detection and Tracking: Event-Based Information Organization,Kluwer Academic Publishers, Norwell, MA, USA, pp. 17–31, 2002.

2nd Annual Music Information Retrieval Evaluation eXchange. Available at http://www.music-ir.org/evaluation/MIREX/ Last visited

An Overview.

Retrieval Evaluation Bibliography. Available at http://wwwnlpir.nist.gov/projects/t01v/trecvid.bibliography.txt Last visited

Retrieval Evaluation Online Proceedings. Available at http://wwwnlpir.nist.gov/projects/tvpubs/tv.pubs.org.html Last visited

The Benchathlon Network: Home of CBIR Benchmarking Available at http://www.benchathlon.net/ Last visited

The Overlap Problem in Content-Oriented XML Retrieval Evaluation. In:

The TREC Video Retrieval Evaluation (TRECVID): A Case Study and Status Report.

The TREC Website. Available at http://trec.nist.gov Last visited

Topic Detection and Tracking Evaluation Overview. In: Topic Detection and Tracking: Event-Based Information Organization,

Large scale evaluations of multimedia information retrieval: the TRECVid experience

Name not available

Crossref

Large Scale Evaluations of Multimedia Information Retrieval: The TRECVid Experience

Irish Universities

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.134.522

Large scale evaluations of multimedia information retrieval: the TRECVid experience

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Name not available

Crossref

Irish Universities