There are several different approaches to video retrieval which vary in sophistication, and in the level of their deployment. Some are well-known, others are not yet within our reach for any kind of large volumes of video. In particular, object-based video retrieval, where an object from within a video is used for retrieval, is often particularly desirable from a searcher's perspective. In this paper we introduce Fischlar-Simpsons, a system providing retrieval from an archive of video using any combination of text searching, keyframe image matching, shot-level browsing, as well as object-based retrieval. The system is driven by user feedback and interaction rather than having the conventional search/browse/search metaphor and the purpose of the system is to explore how users can use detected objects in a shot as part of a retrieval task

Browne, Paul

Smeaton, Alan F.

English

Crossref

Video retrieval using dialogue, keyframe similarity and video objects

Name not available

VIDEO RETRIEVAL USING DIALOGUE, KEYFRAME SIMILARITY AND VIDEOOBJECTSPaul Browne1 and Alan F. Smeaton1,2Centre for Digital Video Processing1 and Adaptive Information Cluster 2Dublin City University,Glasnevin, Dublin 9, Ireland.ABSTRACTThere are several different approaches to video retrieval wh-ich vary in sophistication, and in the level of their deploy-ment. Some are well-known, others are not yet within ourreach for any kind of large volumes of video. In particular,object-based video retrieval, where an object from withina video is used for retrieval, is often particularly desirablefrom a searcher’s perspective. In this paper we introduceF I´SCHLA´R-SIMPSONS, a system providing retrieval froman archive of video using any combination of text search-ing, keyframe image matching, shot-level browsing, as wellas object-based retrieval. The system is driven by user feed-back and interaction rather than having the conventionalsearch/browse/search metaphor and the purpose of the sys-tem is to explore how users can use detected objects in ashot as part of a retrieval task.1. OVERVIEW OF VIDEO RETRIEVALCurrent approaches to content-based retrieval of video canbe broadly divided into those which support text match-ing against dialogue, image matching against keyframes, orwhich detect concept features in order to filter shots. Thisobservation is based on the annual TRECVid exercise [4]which, for each of the last four years, has benchmarked theeffectiveness of video retrieval tasks on a shared video cor-pus using a metrics-based approach. During 2004, over 30research groups took part in this retrieval exercise so thedifferent approaches taken in TRECVid represent a broadsweep of research in the field.The first approach to video retrieval, text matching, mat-ches a user’s text query against text obtained from videosuch as from closed captions or from recognized speech,or from video OCR. Conventional text-based informationretrieval approaches can be used and term weighting ap-proaches such as BM25 or IDF have been applied with suc-cess [4]. Text-based video retrieval works well where theThe support of the Informatics Directorate of Enterprise Ireland isgratefully acknowledged. This work was partially supported by ScienceFoundation Ireland under grant No. 03/IN.3/I361.focus of the search is the item under discussion in the videobut when the focus of the search is the visual content itselfthen the second approach to video retrieval, image match-ing, is more suitable.The simplest kind of image matching uses low-level im-age features such as global or local colour, texture, edgesor shapes [5]. In video retrieval, a single keyframe is usu-ally chosen as the representative image for a shot and of-ten it is the middle frame from a shot that is chosen. Us-ing this approach it is not surprising that matching a user’squery image(s) against keyframes using low-level featurescan be successful only if the user’s information need canbe captured in a still image and the keyframe is genuinelyrepresentative of the shot. Searches for shots of a “rocketlaunch” for example, with a query image consisting of ablaze of flames coming from a rocket against a blue skybackground will be visually similar to keyframes taken fromrocket launch footage, so there are also cases when this ap-proach to retrieval works well.The automatic detection of semantic concepts in videorepresents a third approach to video retrieval. This involvesmanually annotating a large corpus of shots as either havingor not having feature “X” and then using a machine learningalgorithm to build an automatic classifier for this. Examplesof such features used in TRECVid [4] are “indoor”, “out-door”, “buildings”, etc. Building concept feature detectorsis a difficult task because the definition of a set of conceptsbroad enough to cover a large range of information needsyet discriminative enough to be useful in user searching isan open question and the task of labeling enough positiveand negative examples of a concept and then learning a clas-sifier is something that requires much effort. Nevertheless,if a concept feature detector is available and has been ap-plied to a corpus of video with a high level of accuracy thenit can certainly be useful in a user’s search. The problem isthat there are only a small number of such classifiers builtand many have a level of accuracy which is less than thatrequired for reliable video retrieval.With a range of search modalities available for videoretrieval, and each working best only on certain types of0-7803-9134-9/05/$20.00 ©2005 IEEEsearch, clearly the best overall approach is to make as manyof these as possible available to a user and let the user usesome or all of them, in some weighted combination. A re-cent approach by Yan et al. [6] has been to assign a userssearch to a pre-defined type and for each the optimal com-bination of retrieval tools such as those outlined above willhave been learned. This appears to be accepted as the bestoverall approach to video shot retrieval.Retrieval based on the presence, or absence, of givenobjects in the video has not received much attention to date.Sometimes we want to retrieve video shots based on anidentifiable object in the video such as a car, a dog, the Eif-fel Tower or a motorbike. In such cases we could hope thatthe dialogue will mention what is on-screen but we cannotrely on this. We could hope that reliably detected and use-ful features will help narrow down the corpus so we canbrowse the remainder but we cannot rely on this nor can werely on matching an entire example image containing ourdesired object against a keyframe based on colour, textureor edges. A quantum leap in video retrieval would be madeif a searcher could specify one or more visual objects, seg-mented from their backgrounds, and use those as part of thequery. In the next section we shall see how such object de-tection and segmentation can be done.2. OBJECT DETECTION, SEGMENTATION ANDTRACKING IN “THE SIMPSONS”Despite much focus and attention, fully automatic object de-tection, segmentation and tracking across video is not yetachievable for natural video and available systems requireuser involvement and are thus semi-automatic at best. Whatthis means in practice is that such systems are assisted bya human who draws a rough boundary around an object ofinterest and the system will detect and segment this object,separating it from its background, and track it temporallythrough the shot [1].Object detection from natural video needs to be invari-ant to scale and to rotation. Part of the object may be oc-cluded either by another object or by the object itself, andvariations of the same object type may generate very dif-ferent object images, such as a car object for example. De-tection will also have to deal with noise such as edges fromother objects in the image and/or errors in the edge detectionprocess itself. Finally, images and video frames are 2D rep-resentations of a 3D world without depth information andfor a symmetrical object like a vase it doesn’t matter whatangle we look at it from as the shape will always be thesame, but most objects are not like this which means a carobject, for example, can have very many different shapesdepending on the angle of view.Our approach to segmentation is to use object templateswhere a number of templates of the object to be detected areFig. 1. Screenshot to illustrate template matchingpre-defined and shape matching is used during the detectionto compare each image against the available templates [1].This performs well on animated content where there are afinite and limited number of views of objects, objects donot turn and the camera usually has fixed positions.We automatically detected the presence of the majorcharacters appearing on-screen in a collection of Simpsonsepisodes by extracting all yellow coloured objects from shotkeyframes (all Simpsons characters are yellow) and match-ing these extracted objects against a number of previouslyconstructed templates for each character. In the detectionprocess each template was compared against shape bound-aries in the comparison image using a shape matching algo-rithm which generates a number of comparison scores andany scores above a pre-defined threshold were regarded as apositive match. Colour plays an important part in the shapematching process, as positive matches require yellow areasto match the templates. All other colours are ignored andthis reduces false detection.Figure 1 is a screenshot taken from an interface to thetemplate-based object detection application we developed[1]. At the bottom of the screen we can see 5 members of theSimpsons family and their representative templates. At thetop left of the screen we can see the keyframe image that isbeing compared while the top right shows positive templatematches for Homer, Lisa and Marge. The overall accuracyof our template-matching detection needs to be high in orderfor subsequent searching to be reliable. To measure thiswe used a development corpus of 12 episodes (4.8 hours,6,525 shots) and manually tagged each shot which had anyof our target characters (objects). Table 1 gives a list of the10 characters who were detected automatically using thisapproach, their occurrence in the development corpus, andTable 1. Automatic object detection in the Simpsons corpusof 6,525 shotsCharacter Occurrences Recall PrecisionHomer 2,066 (32%) 0.36 0.98Marge 1,343 (21%) 0.32 0.87Bart 1,361 (21%) 0.49 0.98Lisa 647 (10%) 0.57 0.93Maggie 321 (5%) 0.31 0.88Mr. Burns 269 (4%) 0.47 1.0Mr. Smithers 175 (2.6%) 0.46 0.98Abe Simpson 83 (1.3%) 0.27 0.91Principal Skinner 246 (4%) 0.8 1.0Moe 43 (0.7%) 0.61 0.78Avg. (weighted) 0.421 0.947the accuracy of detection measured against our manuallytagged ground truth using precision and recall. The resultsof our tests show that precision at 0.947 is high for almostall characters, but recall at 0.421 is low, indicating we aremissing many character occurrences. These could be due tothe character not being present or perhaps occluded in thekeyframe but present in some other part of the shot. While atfirst glance this may seem unacceptable, we should note thatthe presence of the minor characters in the series (i.e. notBart, Homer or Marge) tends to occur in clusters or scenesso even if a given character occurrence is missed, it is likelythat occurrences in neighbouring shots will be detected andpicked up by the user when browsing the the video.3. THE F I´SCHLA´R-SIMPSONS VIDEORETRIEVAL SYSTEM3.1. Video RetrievalBecause it is visual in nature, video information is intrin-sically browsable and invites a browsing-style interactionfrom users seeking information. A typical search interac-tion involves a search generating a ranked list of documents,which the user previews and chooses some document toread and this continues until the user decides to re-formulatethe query and the process re-iterates. In our work we use aninteraction metaphor in which a user’s initial query gener-ates a ranked list of shots, the user browses these, exam-ines one in more detail and judges the shot as one of theset of relevant (RS) or the set of non-relevant (RS) shots orjudges some object in the shot as one of the set of relevant(RO) or the set of non-relevant (RO) objects. Based on thisjudgment the shots are then dynamically re-ranked and theuser is given a re-ranking to browse and choose another shotand make a second shot or object relevance judgment. Allshot or object relevance judgments, plus the initial query,are used to re-rank shots and the process of dynamicallyre-ranking after every judgment continues until the user issatisfied.3.2. Ostensive Relevance FeedbackIn the interaction scenario outlined above, the system usesa number of shot and/or object relevance judgments to gen-erate a shot ranking. In our work we use ostensive rele-vance feedback [3], where the importance of a shot or ob-ject relevance judgment is a function of the “age” of thejudgment with more recent judgments having more impor-tance and “age” being the relative order of shot judgmentsas received by the system, measured in terms of the user’scurrent search session. The premise behind this is that it cancapture the shifting nature of a user’s information need, es-pecially in a highly browsing-intensive application such asvideo searching. A number of decay functions for weight-ing older shots by age have been used in our work includinglinear and logarithmic and for the purposes of the work de-scribed here we use a linear decay. In the case where a userhas viewed a number of shots and found k relevant shots,l non-relevant shots, m relevant objects and n non-relevantobjects, an unseen shot Cu is ranked by the function Sc de-fined as:Sc(Cu) = S(Cu, Q) +k∑i=1S(Cu, Xreli ) ODi −l∑j=1S(Cu, Xnonj ) ODj+m∑i=1S(Cu, Y reli ) ODi −n∑j=1S(Cu, Y nonj )ODjwhere S(S1, S2) is a function which measures the similaritybetween two shots S1 and S2, Q is the original query (inter-preted as shot) and Xreli ∈ RS , Xnonj ∈ RS , Y reli ∈ ROand Y nonj ∈ RO. The weights ODi and ODj are the osten-sive decays of the previously judged shots and objects andare scaled linearly between 0 and 1.3.3. Video Shot SimilarityComputing a similarity S(S1, S2) between two shots is doneusing text-text, image-image and object-object metrics. Fortext, we use common stopwords weighted by their inversedocument frequency, the number of shots in the corpus (N )divided by the number of those for which the stopword oc-curs (ni). This is defined as Stext(S1, S2) =∑W1 ×W2where Wx = log2(N/nx).For image similarity we use the sum of the absolute dif-ference in feature values for a range of low-level image fea-tures including average colour over 9 frame regions, me-dian colour over each of 9 frame regions, an 18-bin huehistogram for 4 frame regions and a 16-bin edge histogramfor 4 frame regions. These absolute differences in featurevalues are divided by their absolute sums and can be con-sidered as a relative Manhattan distance.Simg(S1, S2) =k∑i−1[|xi,j − xk,j ||xi,j + xk,j | ]The final component of our shot matching, using objects,uses a set of pre-defined static weights as additions or sub-tractions from the shot-shot similarity, depending on thepresence of common objects in the set of relevant (RO) ornon-relevant (RO) objects respectively. Finally, the three el-ements of our shot similarity, Stext(), Simg , and object sim-ilarity, are linearly combined with the overall shot similarityformula incorporating the ostensive decay weights.3.4. The F I´SCHLA´R-SIMPSONS Video Retrieval SystemThe TV show “The Simpsons” now has over 340 episodes(136 hours of 5.6 days of content) and for our video corpuswe used 52 of these (20.8 hours or 20,529 shots) taken fromseasons 2 and 3, and themed DVDs covering seasons 4 to12. These were transcoded into MPEG-1 for image anal-ysis, storage and playback. For closed captions the DVDsubtitle text was stored as a series of images and thereforeoptical character recognition (OCR) was required to convertthese into machine-readable text.In preparing our video content we first applied a shotboundary detection algorithm to this corpus [2] and for eachshot we identified a keyframe. We then built a video re-trieval system to support the search and shot/object feed-back interaction described earlier. Once a user’s search isunderway it is important to be able to view the context of aranked shot. In many cases a query topic might have rele-vant shots near the same location of a ranked shot, thereforeit is important to be able to view the shot context (shots thatcame before and after a ranked result). The Shot Contextscreen (Figure 2) shows the context (preceding 5 shots andfollowing 5 shots) of a given shot shown in the centre of thescreen and shows how detected objects are presented to theuser.4. SUMMARYThe key interest in the system developed here is to explorehow useful object selection can be as part of video shot re-trieval which we do by combing object-retrieval with text-based and image-based shot retrieval under a single osten-sive relevance feedback framework. Our future work willinvolve evaluating this system with real users in a searchtask environment to gauge exactly what contribution object-based searching can make.Fig. 2. Shot context screen from F I´SCHLA´R-SIMPSONS5. REFERENCES[1] T. Adamek and N. O’Connor. “Efficient Contour-basedShape Representation and Matching,” MIR 2003 - 5thACM SIGMM International Workshop on MultimediaInformation Retrieval, Berkeley, Ca, 7 Nov. 2003.[2] P. Browne, A.F. Smeaton, N. Murphy, N. O’Connor, S.Marlow and C. Berrut. “Evaluating and CombiningDigital Video Shot Boundary Detection Algorithms,”Proc. of the Irish Machine Vision and Image ProcessingConference, Belfast, N. Ireland, 31 Aug.-2 Sept. 2000.[3] I. Campbell, “Interactive evaluation of the OstensiveModel using a new test collection of images with mul-tiple relevance assessments,” Journal of InformationRetrieval, 1, pages 85-112, 2000.[4] A.F. Smeaton, W. Kraaij, and P. Over. “The TRECVideo Retrieval Evaluation: A Case Study and StatusReport,” RIAO 2004 - Coupling Approaches, CouplingMedia and Coupling Languages for Information Re-trieval, Avignon, France, 26-28 Apr. 2004.[5] A.W.M. Smeulders, M. Worring, S. Santini, A. Guptaand R. Jain. “Content-Based Image Retrieval at the Endof the Early Years,” IEEE Trans. Pattern Anal. Mach.Intell. 2000, 22(12), pp. 1349-1380.[6] R. Yan, J. Yang. and A.G. Hauptmann. “LearningQuery-Class Dependent Weights in Automatic VideoRetrieval,” Proc. of the ACM Multimedia Conference,New York, Oct. 2004, pp.548-555, ACM Press.

There are several different approaches to video retrieval which vary in sophistication, and in the level of their deployment. Some are well-known, others are not yet within our reach for any kind of large volumes of video. In particular, object-based video retrieval, where an object from within a video is used for retrieval, is often particularly desirable from a searcher’s perspective. In this paper we introduce FÍSCHLÁR-SIMPSONS, a system providing retrieval from an archive of video using any combination of text searching, keyframe image matching, shot-level browsing, as well as object-based retrieval. The system is driven by user feedback and interaction rather than having the conventional search/browse/search metaphor and the purpose of the system is to explore how users can use detected objects in a shot as part of a retrieval task. 1. OVERVIEW OF VIDEO RETRIEVAL Current approaches to content-based retrieval of video can be broadly divided into those which support text matching against dialogue, image matching against keyframes, or which detect concept features in order to filter shots. This observation is based on the annual TRECVid exercise [4] which, for each of the last four years, has benchmarked the effectiveness of video retrieval tasks on a shared video corpus using a metrics-based approach. During 2004, over 30 research groups took part in this retrieval exercise so the different approaches taken in TRECVid represent a broad sweep of research in the field. The first approach to video retrieval, text matching, matches a user’s text query against text obtained from video such as from closed captions or from recognized speech, or from video OCR. Conventional text-based information retrieval approaches can be used and term weighting approaches such as BM25 or IDF have been applied with success [4]. Text-based video retrieval works well where th

Paul Browne

Alan F. Smeaton

CiteSeerX

Video Retrieval Using Dialogue, Keyframe Similarity and Video Objects

There are several different approaches to video retrieval which vary in sophistication, and in the level of their deployment. Some are well-known, others are not yet within our reach for any kind of large volumes of video. In particular, object-based video retrieval, where an object from within a video is used for retrieval, is often particularly desirable from a searcher\u27s perspective. In this paper we introduce Fischlar-Simpsons, a system providing retrieval from an archive of video using any combination of text searching, keyframe image matching, shot-level browsing, as well as object-based retrieval. The system is driven by user feedback and interaction rather than having the conventional search/browse/search metaphor and the purpose of the system is to explore how users can use detected objects in a shot as part of a retrieval task

Irish Universities

DCU Online Research Access Service

Content-Based Image Retrieval at the End of the Early Years,”

Efﬁcient Contour-based Shape Representation and

Evaluating and Combining Digital Video Shot Boundary Detection Algorithms,”

Interactive evaluation of the Ostensive Model using a new test collection of images with multiple relevance assessments,”

Learning Query-Class Dependent Weights in Automatic Video Retrieval,”

The TREC Video Retrieval Evaluation: A Case Study and Status Report,” RIAO

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.66.5656

Video retrieval using dialogue, keyframe similarity and video objects

Abstract

Similar works

Full text

Available Versions

Crossref

Name not available

CiteSeerX

Irish Universities

DCU Online Research Access Service