There is an increasing emphasis on including  semantic concept detection as part of video retrieval.  This represents a modality for retrieval quite different from metadata-based and keyframe similarity-based approaches.  One of the premises on which the success of this is based, is that good quality detection is available in order to guarantee retrieval quality. But how good does the feature

detection actually need to be?  Is it possible to achieve good retrieval quality, even with poor quality concept detection and if so then what is the 'tipping point' below which detection accuracy proves not to be beneficial? In

this paper we explore this question using a collection of rushes video where we artificially vary the quality of detection of semantic features and we study the impact on the  resulting retrieval. Our results show that the impact of improving or degrading performance of concept detectors is not directly reflected  as retrieval performance and this raises interesting questions about how accurate concept

detection really needs to be

A.F. Smeaton

A.G. Hauptmann

C.G.M. Snoek

M. Wang

M.G. Christel

P. Toharia

X.Y. Wei

English

There is an increasing emphasis on including  semantic concept detection as part of video retrieval.  This represents a modality for retrieval quite different from metadata-based and keyframe similarity-based approaches.  One of the premises on which the success of this is based, is that good quality detection is available in order to guarantee retrieval quality. But how good does the feature\ud
detection actually need to be?  Is it possible to achieve good retrieval quality, even with poor quality concept detection and if so then what is the 'tipping point' below which detection accuracy proves not to be beneficial? In\ud
this paper we explore this question using a collection of rushes video where we artificially vary the quality of detection of semantic features and we study the impact on the  resulting retrieval. Our results show that the impact of improving or degrading performance of concept detectors is not directly reflected  as retrieval performance and this raises interesting questions about how accurate concept\ud
detection really needs to be

Toharia, Pablo

Robles, Oscar D.

Smeaton, Alan F.

Rodríguez, Ángel

Name not available

Measuring the Inﬂuence of Concept Detection onVideo RetrievalPablo Toharia1, Oscar D. Robles1, Alan F. Smeaton2, and Ángel Rodríguez31 Dpto. de Arquitectura y Tecnología de Computadores, Ciencias de la Computacióne Inteligencia Artiﬁcial,U. Rey Juan Carlos, C/ Tulipán, s/n. 28933 Móstoles. Madrid. Spain.{pablo.toharia,oscardavid.robles}@urjc.es,2 CLARITY: Center for Sensor Web Technologies, Dublin City University,Glasnevin, Dublin 9, Irelandalan.smeaton@dcu.ie3 Dpto. de Tecnología Fotónica, U. Politécnica de Madrid,Campus de Montegancedo s/n, 28660 Boadilla del Monte, Madrid, Spainarodri@fi.upm.esAbstract. There is an increasing emphasis on including semantic con-cept detection as part of video retrieval. This represents a modality forretrieval quite diﬀerent from metadata-based and keyframe similarity-based approaches. One of the premises on which the success of this isbased, is that good quality detection is available in order to guarantee re-trieval quality. But how good does the feature detection actually need tobe? Is it possible to achieve good retrieval quality, even with poor qualityconcept detection and if so then what is the tipping point below whichdetection accuracy proves not to be beneﬁcial? In this paper we explorethis question using a collection of rushes video where we artiﬁcially varythe quality of detection of semantic features and we study the impacton the resulting retrieval. Our results show that the impact of improvingor degrading performance of concept detectors is not directly reﬂectedas retrieval performance and this raises interesting questions about howaccurate concept detection really needs to be.1 Introduction and BackgroundThe automatic detection of semantic concepts from video is opening up a com-pletely new modality for supporting content-based operations like search, sum-marisation, and directed browsing. This approach to managing content compli-ments using video metadata and using keyframe similarity and is being enabledby improvements in the accuracy, and the number of, such detectors or classi-ﬁers by many research groups. This can be seen in the recent development inactivities such as TRECVid [1] where it is now realised that retrieval systemsbased on low-level features like colour and texture do not succeed in describinghigh-level concepts as a human would do.Various authors are now making eﬀorts on optimizing automatic detectionof semantic concepts for use in applications such as retrieval. However, it is not2clear what is the real impact of improving the accuracy of the detection process,i.e. whether a signiﬁcant improvement in the performance of detection will yieldbetter quality retrieval. There have been some previous studies of the eﬃciencyof using concepts in retrieval [24]. Recently, Snoek et al. [5] analyzed whetherincreasing the number of concept detectors as well as their combination wouldimprove the performance of retrieval and found that it does.Wang and Hua study how to improve the performance of combining videoconcept detectors when dealing with a large number of them by following aBottom-Up Incremental Fusion (BUIF) approach [6], but they do not deal withthe issue of assessing detectors' real inﬂuence in retrieval. Thus it appears thereis no work studying the relationship between the quality of detectors and re-trieval performance. The work here explores the relationship between conceptdetection performance and content-based retrieval and to examine whether im-proving detection will yield an improvement at the retrieval stage, or whetherthis is worth the eﬀort.2 Materials and MethodsWe now detail how we set up an experimental environment for video retrievalusing semantic concepts. Controlled noise in concept detection is introduced so asto improve or worsen it, allowing performance of retrieval to be measured. Section3 presents experiments together with the analysis and conclusions reached.2.1 Concept DetectionThe ﬁrst step is to set up a system to extract concepts from shots. In our workwe used the TRECVid [7] 2006 rushes collection of 27 hours which gave rise toapproximately 2,900 shots. The concepts selected to work with are deﬁned fromwithin LSCOM-Lite, a reduced version of the 449 Large Scale Concept Ontologyfor Multimedia [8] annotated concepts that form LSCOM.The concept detection process is broken into several steps. First, a prepro-cessing stage extracts keyframes that represent the video content. These are thenﬁltered in order to discard shots such as calibration charts, black frames and soon. We then extract low-level features for these keyframes which are then used asthe input to the 39 classiﬁers. More details of the keyframe extraction and ﬁlter-ing stages can be found in [9]. Finally, Support Vector Machines (SVM) providedby Dublin City University from our high level feature detection submission inTRECVid 2006 are used, using low-level primitive features like colour and tex-ture, extracted by the AceToolbox [10]. The concept classiﬁers each provide acertainty value Ci ∈ [−1, 1] that each of the shots' keyframes in the originalvideo contains each of the concepts and we use these as baseline examples of theaccuracy of a real implementation of concept detection.3Fig. 1: Retrieval example using weighted concepts.2.2 Interactive concept-based retrieval engineAn interactive video retrieval system is used in order to test the relationshipbetween the quality of detected concepts and retrieval performance. This allowsa user to select which of the available concepts should be used in retrieval, as wellas ﬁxing Wi weights for each of the concepts. These are positive if the conceptis relevant for the query, and if its absence is relevant to the query it will benegative, else it will be 0. The retrieval engine will assign a value scorei for eachshot so a sorted list of results can be presented to the user. Assuming there areN concepts the following is how we obtain a score for each shot:shoti = {Ci1, Ci2, . . . , CiN} , Cij ∈ [−1, 1] (1)scorei =∑Ni=1Wi · CiN, Wij ∈ [−1, 1] (2)As was previously stated, other approaches to combining concept features inretrieval are possible, such as proposed by Wang and Hua [6] or by Snoek andWorring [5], but in our present work we were not interested in addressing thedetector fusion method. Figure 1 shows a retrieval result based on 8 conceptsselected by the user. On the left side, 8 sliding bars allow a user to adjust weightsfor each concept and a visualization of the top-ranked shots is also shown.2.3 Degradation and improvement of concept detectionPerforming an artiﬁcial degradation or improvement of concept detection qual-ity can be achieved by introducing noise into the concept detector output, sothe certainty value is increased or decreased as needed. However, rather thandepend on the accuracy of automatic detection only, the existence of a groundtruth allows us to faithfully simulate improvement and degradation of concept4Table 1: Concepts used in experiments.Concept DescriptionBuilding Shots of an exterior of a buildingCar Shots of a carCrowd Shots depicting a crowdOutdoor Shots of Outdoor locationsPerson Shots depicting a person. The face may be partially visibleRoad Shots depicting a roadSky Shots depicting skyVegetation Shots depicting natural or artiﬁcial greenery, vegetation woods, etc.detection. To obtain this, a manual process of double annotation of each of theconcepts over the whole collection was performed.To vary detection quality, a percentage P of shots from the collection arerandomly selected and their certainty degree is modiﬁed for each detector. Toimprove performance, a value A is added to the certainty value of shots fromthe ground truth in which the concept is known to be present. If a shot does notcontain the concept, the value A will be subtracted from the certainty value. Incase of degrading the detectors' performance, the process is reversed.In measuring the impact of concept detection on retrieval, we use an oineretrieval proces. We use the keyframes of the shots selected to initiate a low-level retrieval process using the low-level image characteristics used as input toconcept recognition, to perform keyframe similarity. This generates a content-based ranking of shots for each topic. A concept-based retrieval ranking is alsogenerated using the weights selected by the users and degrading/upgrading theperformance of the detectors accordingly. The results of both retrieval rankingsare normalized and combined in a 50:50 ratio to give the ﬁnal retrieval output.While this may seem like diluting the impact of concept retrieval, and conceptdetection accuracy, it reﬂects the true way in which video retrieval is carried outin practice. Retrieval performance is evaluated using Mean Average Precision(MAP) over the set of topics and thus by varying the parameters A and P , achange in retrieval MAP should be obtained for each concept.For our experiments we concentrated on a subset of concepts from LSCOM-Lite, shown in Table 1, chosen because they occur throughout the whole videodataset whereas others occur much less frequently. Our experiments are carriedout in two parts, an online part working with non-expert users who performiterative retrieval, and an automated oine part using results from the userretrieval and performing more exhaustive tests varying concept detection quality.This is shown in Figure 2.2.4 Experimental methodologyOur experimental methodology is as follows. In the ﬁrst stage a user searches forshots given a topic using the interactive system and the concept-based retrievalengine described earlier. Topics have been constructed in such a way that theyrequire iterations of the retrieval system to reﬁne and adjust topic weights untilthey are optimal, and that they use concepts both in a positive or negative way.Topics are shown in Table 2, along with their number of relevant shots.5RETRIEVALOPERATIONCONCEPTRETRIEVALSELECTION OFRELEVANTSWEIGHT CONCEPTSELECTIONRESULTSOnline tasksRELEVANTSHOTSOffline tasksCONCEPTRETRIEVALLOW-LEVELRETRIEVAL+RESULTSNOISEINTRODUCTIONFig. 2: Experimental framework.Table 2: Search topics.Topic Description: Find shots containing . . . " # rel. shots1 . . . open-sea views 332 . . . 2 or more people with plants in an urban area 2433 . . . desert-like landscapes 554 . . . village settlements on the coast 735 . . . 2 or more people interacting in a natural environment 916 . . . a person talking to an audience inside a building 397 . . . people sailing 42Table 3: Use of concepts in Topics.Building Car Crowd Outdoor Person Road Sky VegetationTopic 1: 6(-) 6(-) 3(-) 9(+) 2(-) 5(-) 7(+) 5(-)Topic 2: 9(+) 4(+) 9(+) 2(+) 7(+) 6(+) 2(-) 9(+)Topic 3: 7(-) 5(-) 3(-) 9(+) 3(+) 4(-) 9(+) 9(-)Topic 4: 8(+) 3(+) 2(+) 9(+) 3(+) 5(-) 8(+) 5(+)Topic 5: 5(-) 3(-) 8(+) 9(+) 8(+) 3(-) 4(+) 8(+)Topic 6: 5(+) 4(-) 9(+) 9(-) 7(+) 6(-) 6(-) 3(-)Topic 7: 6(-) 5(-) 5(+) 9(+) 8(+) 7(-) 9(+) 5(+)Topics will use available concepts in positive or negative ways, depending onthe subject matter. Topic 6 can be associated with negative weighting of theconcept Outdoor, since the aim is that action takes place inside a building.Table 3 shows the ways that the set of 9 users use topics in positive or negativeways. For example for Topic 2, 4 users used the concept Car in a positive wayand 2 used Sky in a negative way. Some aspects of some topics may not beaddressable in query formulation with the available concepts and while this mayseem a limiting factor, it is also representative of a real world search where therewill never be enough appropriate concepts for the variety of user search topics.For our experiments, 9 users without any professional experience of searchingwere recruited. Each user was given an introduction to the system, and the 7topics were presented in a rotating order to avoid bias. Each user adjusted con-cept weights for each of the topics and a retrieval operation was performed with6Table 4: MAP variation average for retrieval introducing controlled noise intodetector performance.(a) DegradationP/A -0.1 -0.3 -0.510% -0.10% -0.74% -1.23%30% -0.67% -2.61% -4.25%50% -0.92% -3.42% -5.69%(b) ImprovementP/A +0.1 +0.3 +0.510% 0.10% 0.19% 0.53%30% 0.45% 1.20% 2.16%50% 0.81% 2.12% 3.98%the user marking relevant shots or adjusting concept weights and performing anew search. Once the sets of relevant shots had been identiﬁed, we can calculateretrieval rankings based on combined weighted concept-based and content-basedtechniques and calculate MAP retrieval performance by measuring against anexhaustive manual assessment of shot relevance, our ground truth for retrieval.We can examine the eﬀect of detector quality on retrieval performance by intro-ducing noise into the output of the concept detectors as described in section 2.3.Each variation on the parameters that results in degraded or improved detec-tors gives a new list of ranked shots which can be evaluated against the groundtruth, and MAP calculated. Combining the diﬀerent options available, we havea total of 9 users, each running 7 queries with improvements and degradationson 8 concepts, to be evaluated.3 Results and Discussion3.1 Performance of retrievalTable 4 shows the average MAP percentage variations when we degrade or im-prove the quality of the underlying concept detection above or below the levelof concept detection performance obtained from the automatic DCU conceptdetection. Thus we use the real performance ﬁgures as a baseline and vary de-tection quality above and below this. The MAP performance using unmodiﬁedconcept detection performance is 0.0254.What these results tell us, for example, is that when we degrade concept de-tection performance for all concepts by reducing the certainty value for detectionby 0.5 (on a scale of -1 to 1) for 50% of the shots, we get a net drop in MAPperformance for retrieval of only 5.69% (bolded entry in Table 4).Table 5 collects the average Coeﬃcient of Variation values considering theresults achieved by all users and among all topics. Coeﬃcient of Variation valuesare more stable across users rather than across topics, but the worst cases appearfor the lower values of P and A variables because the average variations are verylow (Table 4). This can be due to the user interaction with the retrieval engineand to the random controlled noise introduced.3.2 Detector performance versus the retrieval taskFigure 3 shows MAP variations when ﬁxing one of the parameters, either A orP , for both detection performance and for concept retrieval performance. Thex-axis depicts A or P values for improvement (represented as positive valuesof the scale) or degradation (negative values). The y-axis shows the variation7Table 5: Avg. Coeﬃcients of Variation considering responses by users, all topics.(a) Per user.DegradationP/A -0.1 -0.3 -0.510% 5.277 0.638 0.72330% 1.156 0.327 0.35050% 1.299 0.453 0.414ImprovementP/A +0.1 +0.3 +0.510% 2.454 8.337 5.07830% 1.960 1.764 1.28550% 1.081 1.105 0.929(b) Per topic.DegradationP/A -0.1 -0.3 -0.510% 8.361 1.794 1.83530% 1.881 1.182 1.23150% 1.620 1.195 1.214ImprovementP/A +0.1 +0.3 +0.510% 6.052 12.117 7.20530% 2.511 2.523 2.04350% 1.372 1.276 0.986MAPvariation(%)−100−50 0 50 100 150 200 250−0.6 −0.4 −0.2  0  0.2  0.4  0.6% de variación del MAPCantidad de degradación/mejora (A)P=10%P=30%P=50%Degradation (-)/improvement (+)(a) Detectors varying AMAPvariation(%)−6−5−4−3−2−1 0 1 2 3 4−0.6 −0.4 −0.2  0  0.2  0.4  0.6% de variación del MAPCantidad de degradación/mejora (A)P=10%P=30%P=50%Degradation (-)/improvement (+)(b) Retrieval varying AMAPvariation(%)−100−50 0 50 100 150 200 250−60 −40 −20  0  20  40  60% de variación del MAP% de tomas (P)A=0,1A=0,3A=0,5Degradation (-)/improvement (+)(c) Detectors varying PMAPvariation(%)−6−5−4−3−2−1 0 1 2 3 4−60 −40 −20  0  20  40  60% de variación del MAP% de tomas (P)A=0,1A=0,3A=0,5Degradation (-)/improvement (+)(d) Retrieval varying PFig. 3: MAP variations for detection and retrieval, varying A and P .of the MAP in percentages. The curves show similar trends for both A and Ptransformations for all the tests shown. However, Figures 3(a) and 3(b) (A asparameter) show diﬀerent range values for positive (improvement) and negative(degradation) intervals, being the variation most noticeable in the improvementtransformation. On the other hand, both the tendency and the interval of Figures3(c) and 3(d) (P as parameter) are very similar. Overall, however, we can saythat the impact of detection accuracy is far less pronounced than we wouldexpect, indicating that even poor detection accuracy provides useful retrieval.84 ConclusionsWe have implemented a methodology to analyze the impact of concept detec-tion accuracy on video retrieval on a collection of rushes video. We found thateven poor quality detection can yield good retrieval and that as the quality ofdetection improves the quality of retrieval does not rise accordingly. While thismay appear as just an interesting exercise and the results do depend on theset of concepts used, it does represent the state of the art in using conceptsin retrieval, as shown in TRECVid, where it is shown that exploiting the de-pendencies among concepts is non-existent. For future work we plan to furtherinvestigate how detection performance is impacted when semantic dependenciesamong concepts (e.g. Outdoor/Building and Person/Crowd) and this willintegrate concept ontologies into our work. Other work will be to extend thenumber of concepts to see if similar results are obtained for concepts which donot occur as frequently in the video as the ones used here.Acknowledgments This work has been partially funded by the the Spanish Min-istry of Education and Science (grant TIN2007-67188). BBC video was provided byTRECVid [1]. AS was funded by Science Foundation Ireland under grant 07/CE/I1147.References1. Smeaton, A.F., Over, P., Kraaij, W.: High-Level Feature Detection from Video inTRECVid: a 5-Year Retrospective of Achievements. In Divakaran, A., ed.: Multi-media Content Analysis, Theory and Applic. Springer Verlag, Berlin (2009) 1511742. Hauptmann, A.G., Yan, R., Lin, W.H., Christel, M.G., Wactlar, H.D.: Can high-level concepts ﬁll the semantic gap in video retrieval? a case study with broadcastnews. IEEE Transactions on Multimedia 9(5) (2007) 9589663. Christel, M.G., Hauptmann, A.G.: The use and utility of high-level semantic featuresin video retrieval. In Leow, W.K., Lew, M.S., Chua, T.S., Ma, W.Y., Chaisorn, L.,Bakker, E.M., eds.: CIVR. Volume 3568 of LNCS, Springer (2005) 1341444. Wei, X.Y., Ngo, C.W.: Fusing semantics, observability, reliability and diversity ofconcept detectors for video search. In: Proc. MM '08, NY, USA, ACM (2008) 81905. Snoek, C.G.M., Worring, M.: Are concept detector lexicons eﬀective for videosearch? In: ICME, IEEE (2007) 196619696. Wang, M., Hua, X.S.: Study on the combination of video concept detectors. In:Proc. MM '08 NY, USA, ACM (2008) 6476507. Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and trecvid. In: Proc.MIR '06, NY, USA, ACM Press (2006) 3213308. Naphade, M.R., Kennedy, L., Kender, J.R., Chang, S.F., Smith, J.R., Over, P.,Hauptmann, A.: A light scale concept ontology for multimedia understanding forTRECVID 2005. Tech. Rep. RC23612, IBM T.J. Watson Research Center (2005)9. Toharia, P., Robles, O.D., Pastor, L., Rodríguez, A.: Combining activity and tem-poral coherence with low-level information for summarization of video rushes. In:Proc. TVS '08: NY, USA, ACM (Oct. 2008) 707410. O'Connor, N., Cooke, E., le Borgne, H., Blighe, M., Adamek., T.: The acetoolbox:Low-level audiovisual feature extraction for retrieval and classiﬁcation. In Proc.EWIMT'05 (2005)

Measuring the influence of concept detection on video retrieval

There is an increasing emphasis on including  semantic concept detection as part of video retrieval.  This represents a modality for retrieval quite different from metadata-based and keyframe similarity-based approaches.  One of the premises on which the success of this is based, is that good quality detection is available in order to guarantee retrieval quality. But how good does the feature
detection actually need to be?  Is it possible to achieve good retrieval quality, even with poor quality concept detection and if so then what is the \u27tipping point\u27 below which detection accuracy proves not to be beneficial? In
this paper we explore this question using a collection of rushes video where we artificially vary the quality of detection of semantic features and we study the impact on the  resulting retrieval. Our results show that the impact of improving or degrading performance of concept detectors is not directly reflected  as retrieval performance and this raises interesting questions about how accurate concept
detection really needs to be

Measuring the influence of concept detection on video retrieval

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

DCU Online Research Access Service

Crossref