Ekenel, H. K.

Gao, H.

Stiefelhagen, Rainer

English

KITopen

Universität Karlsruhe (TH) at TRECVID 2008Hazım Kemal Ekenel, Hua Gao, Rainer StiefelhagenComputer Science Department, Universität Karlsruhe (TH)Am Fasanengarten 5, Karlsruhe 76131, Germany{ekenel,hua.gao,stiefel}@ira.uka.deWeb page: http://isl.ira.uka.de/cvhciAbstract—In this paper, we present the system developedby the Interactive Systems Labs at Universität Karlsruhe forthe TRECVID 2008 evaluation. It is the second time that weparticipate in the TRECVID evaluation this year. Last year, themain goal of our first participation was to develop a commonsoftware framework for multimedia processing and to buildbaseline systems for the shot boundary detection and high levelfeature (HLF) extraction tasks. The evaluation task for shotboundary detection was dropped this year, hence we focusedour work on HLF extraction only. The color, texture and shapefeatures were used as low-level features as before, while textualfeature was not used. We used support vector machine (SVM)classifier for low-level feature classification. The parameters forSVMs were optimized in a grid-search scheme. A simple weightedsum fusion was applied to extract the high-level features, wherethe weights were calculated for each category and each low-levelfeature. More training data was used and evaluation results wereimproved compared to the last year’s evaluation results.I. INTRODUCTIONIt was the second time that we participated in the TRECVIDevaluation this year. Last year, the main goal of our firstparticipation was to develop a common software frameworkfor multimedia processing and to build baseline systems forthe shot boundary detection and high level feature (HLF) ex-traction tasks. The evaluation task for shot boundary detectionwas dropped this year, hence we focused our work on HLFextraction only. Unfortunately, the evaluation results on ourHLF extraction system from last year were not as good aswe expected, they were below the average. So, this year,we focused our work on finding implementation problems ofthe old system and parameter optimization for training SVM-based classifiers.The paper is organized as follows: The old HLF detec-tion system from the last year is described in Section II.In Section III, we explain the modifications we have donefor this year’s evaluation. The experimental results on theTRECVID 2007 and 2008 data are presented in Section IV.The conclusions are given in Section V.II. OVERVIEWOur baseline system for HLF detection was composed ofseveral successive processing steps. In the first step, the key-frames of the master shots in the videos were represented bylow-level feature vectors, which describe the color, shape andtextual content of the frames. In the second step, these featureswere normalized and fed into a set of SVM classifies in orderto determine whether a category exists in a key-frame or not.We trained SVM classifier for each low-level feature descriptorand category. In the third step, the scores that are returned bythe SVM classifiers were normalized and combined to generatethe final decision and the shots were ranked according to thefused scores.Both visual cues and textual cues were used as low-levelfeatures. For the visual cues, we employed the features similarto MPEG-7 visual descriptors, including color histogram [10],grid color moments [9], color correlogram [5], co-occurrencetexture [2], wavelet texture [2] and edge histogram [4]. Inaddition to the color-based and texture-based features, thegeometric blur [1] was also used as a shape-based visual cue.Relative term frequency-based textual features were investi-gated as an additional descriptor which were obtained fromthe machine translated automatic speech recognition output.Details about the design and implementation of these featuresare given in [4].We split the development data of TRECVID 2007 intothree portions. The first part was used for low-level featurelearning, the second was for validation of the learned clas-sifiers and learning the high-level fusion (for learning-basedfusion methods), and finally the third part was used for thevalidation of high-level fusion. Through permutation, a 6-foldcross validation was performed on these three portions of data.The classifiers from the fold that achieved the highest meanaverage precision (MAP) were chosen as the final classifiers.We considered several high-level fusion schemes last yearto combine the outputs from low-level feature classifiers. Twoof which are learning-based fusion approaches that are trainedwith the scores returned by the low-level feature classifiers. Weused neural networks and SVMs, respectively, for this kind ofhigh-level fusion. Simple fusion rules were utilized as well tocombine the low-level classifiers, which include the sum rule,min-max rule, voting rule and the weighted version of the sumrules. Correlation among the categories was also considered asa simple semantical analysis between the categories.Last year, we submitted totally 6 runs for the final evalua-tion. However, all the results were below the average. The bestrun was generated using the fusion scheme of sum rule and theachieved infAP was 2.3%. There were several reasons for thelow performance compared to other systems using the similarlow-level features and classification scheme, according to ouranalysis in [4] two of which were the fixed assignment ofweights for a certain low-level feature extractor and parameteroptimization for SVMs.III. REVISED SYSTEMWe investigated the old system intensively to find theproblems that cause the poor performance. We started fromextracting the low-level features. Since the textual cues didnot help in the old system, we dropped this feature. Only thesum rule was used since it outperformed the other fusion ap-proaches and it was also widely applied in many other systems.The major revision steps of the old system is described in thefollowing subsections.A. Low-level featuresLast year, the color feature vector of grid color momentswas calculated in HSV space and the grid size was 3 × 3.We followed the implementation in [2] where the grid size is5×5 to obtain finer local color features. The first order (mean),the second (variance) and the third order (skewness) colormoments were calculated in each local block in the image andthe Lab color space was used instead of HSV. The dimensionof this feature was 9× 5× 5 = 225.B. Low-level feature classificationAccording to [3], comparing the histogram-based featurevectors with the L1 metric or the χ2 metric outperforms the L2metric, because the histograms are discrete densities. We chosethe L1 distance for its simplicity and outstanding performancein [3]. The L1 distance is defined as:dL1(x, x′) =∑i|xi − x′i|,which gives the Laplacian RBF kernel:KLaplacian(x, x′) = e− dL1(x,x′)σ2 .We trained the SVMs for the histogram-based features such asthe color histogram, edge histogram and the color correlogramwith the Laplacian RBF kernel, while the remaining featureswere trained with the Gaussian RBF kernel. The LaplacianRBF kernel was implemented as a user defined kernel in theemployed SVM-light [6] implementation.The parameters for the SVMs were optimized by conductinga coarse grid search with 3-fold cross validation.C. FusionThe output of the SVM-based classifiers were actually thedistance of the feature vector to the learned hyper-plane. Toavoid the dominance of some outputs, we normalized thedistance scores with a sigmoid function [8]:Snorm =11 + e−Sraw.Where Snorm is the normalized score and Sraw is the rawscore from the SVM classifier.We only considered the weighted sum rule for combiningthe low-level classifiers. In the old system, a global weight wasassigned to each low-level feature, which was not reasonablebecause a specific feature classifier may be more suitable forsome categories but not for the others. Thus we assignedweights for each feature and each category according to theaverage precision (AP) score obtained by the experimentson the validation data. The fused score is then computed asfollows:Sfusion =∑Ni=1 wiSi∑Ni=1 wi.Where Sfusion is the fusion score, Si is the score to be fused,wi is the corresponding weight of score Si.D. Training dataThe positive and negative training samples in the develop-ment data were quite unbalanced. Some categories, such asthe concept “Flag-US” in TRECVID 2007, only contains 12positive samples while the number of the negative samples isover 20000. We set the SVM parameter for controlling the costmargin to the number of negative samples in the training setdivided by the number of positive samples [7]. However, SVMclassifiers with this parameter setting return too few positivereturns which leads to low recall.To avoid this problem, we down-sampled the negativesamples in the training set, while all the positive samples werekept. The down-sampling followed the method in [11]. Fora given category, the number of the negative samples waslimited according the number of the positive samples. Thetraining samples were then not heavily unbalanced for somecategories and less support vectors were generated during theSVM training, which accelerated both training and evaluation.To obtain the weights for each low-level classifier andthe parameters for the SVMs, we conducted a 3-fold crossvalidation. The development data was split into two partitions,since the sum rule fusion is not learning-based fusion, wedo not have to split the data to learn the scores. 2/3 of thestratified training samples were selected for training and theremaining samples were used as a validation set. However,all the samples were trained again using the optimized SVMparameters to provide enough positive training samples.E. RankingSince we got very few positive returns last year, somenegative returns were also added in the ranking list. The lengthof the ranking list was set according to the priori ratio of thepositive samples in the development data. This was the majorreason which cause the poor results of the old system. Withdown-sampling of the negative samples we got reasonablenumber of positive returns in the revised system, and theranking lists were limited to 2000 according to the requirementfor evaluation without any other limitations.IV. EXPERIMENTAL RESULTSAfter conducting grid search with 3-fold cross validation,the parameters for the SVMs were coarsely optimized (witha coarse step-size because of time limit). The parameter fortrading-off between training error and margin was set to 1.0,and the parameter gamma (γ) for RBF kernel was set to0.05. We evaluated the revised system on the 2007 TRECVIDtesting data with ground-truth annotation provided by NIST. 0 0.2 0.4 0.6 0.8 1 0  0.2  0.4  0.6  0.8  1PrecisionRecallFig. 1. Interpolated recall-precision. Estimated using 50% sample (e.g.,estimated precision = 2 * actual from sample) 0 0.2 0.4 0.6 0.8 11 3 5 6 10 12 17 23 24 26 27 28 29 30 32 33 35 36 38 39infAPFeature numberFig. 2. The inferred average precision (infAP) scores for all evaluatedcategories. Estimated using 50% sample (e.g., estimated precision = 2 * actualfrom sample)The mean infAP score was 6.3%, which was above theaverage. The interpolated recall-precision diagram is shownin Figure 1 and the infAP scores for all evaluated categoriesare plotted in Figure 2. With the priori ratio limitation, thesystem got a mean infAP score of 3.9%, which means thatthe limitation discarded lots of hits and thus degraded theperformance. The number of the positive samples also playsan important role for the final performance. For example,the group NII-ISM (National Institute of Informatics, TheInstitute of Statistical Mathematics) [8] used the similar low-level features and fusion scheme and got similar infAP score(6.6%) if only the development data of TRECVID 2007 isused. But the score improved to 10.1% when they trained theirsystem using more data including TRECVID 2005, 2006 and2007.Last year, totally 36 concepts were used for HLF detectiontask, while only 20 categories were evaluated. In this year,however, 20 concepts were selected and all were evaluatedwith the same criteria as before. The task was more objectdetection oriented and 15 of the 20 features were annotatedwith bounding boxes in the key-frames for training. Due toscarcity of time, we did not build another system to utilizethe object annotation for object detection. We evaluated ourrevised system on the data provided for this year and submittedtwo runs. However, they were quite similar runs. The secondRun Mean infAP DescriptionUKA1 3.0% Weighted sum fusionUKA2 3.8% Weighted sum fusion with unconfidence filtering,TABLE IHIGH-LEVEL FEATURE EXTRACTION RESULTSConcept UKA1 UKA2 Median MaxClassroom 0.001 0.003 0.008 0.152Bridge 0.006 0.010 0.004 0.117Emergency Vehicle 0.000 0.000 0.003 0.065Dog 0.057 0.094 0.101 0.275Kitchen 0.004 0.009 0.010 0.165Airplane flying 0.028 0.034 0.029 0.278Two people 0.032 0.030 0.050 0.174Bus 0.002 0.001 0.004 0.119Driver 0.035 0.059 0.046 0.324Cityscape 0.034 0.053 0.059 0.258Harbor 0.001 0.005 0.008 0.182Telephone 0.007 0.006 0.011 0.136Street 0.055 0.080 0.113 0.413Demonstration Or Protest 0.001 0.002 0.013 0.233Hand 0.014 0.020 0.095 0.377Mountain 0.032 0.037 0.041 0.246Nighttime 0.129 0.130 0.102 0.323Boat Ship 0.100 0.113 0.093 0.394Flower 0.048 0.062 0.058 0.161Singing 0.025 0.014 0.014 0.258Mean 0.030 0.038 0.043 0.232TABLE IIRESULTS FOR EACH CONCEPTrun filtered out some relatively less confident returns. Theresults are listed in Table I. The inferred average precisions foreach concept are listed in Table II. The columns UKA1 andUKA2 list the infAP scores of our two submitted runs. Themedian and maximum infAP scores are listed in the fourth andfifth columns, respectively. Our results are close to median, butthey are still far from the best system.V. CONCLUSIONSeveral modifications were made based on the old systemfrom last year. We evaluated the revised system on the 2007TRECVID data with ground-truth annotation and the resultswere improved. From our experiments and the report fromthe other groups, we concluded that training with more dataimproves the performance. However, since the evaluation inthis year is more object detection oriented, we should haveutilized the provided object annotations to improve our resultsfurther.REFERENCES[1] A. C. Berg, T. L. Berg, and J. Malik, ”Shape matching and objectrecognition using low distortion correspondences”, Proc. IEEE. Conf.Computer Vision and Pattern Recognition, 2005.[2] M. Campbell, A. Haubold, and S. Ebadollahi et al., ”IBM ResearchTRECVID-2006 Video Retrieval System”, NIST TRECVID Workshop,Gaithersburg, USA, Nov. 2006.[3] O. Chapelle, P. Haffner, and V. Vapnik, ”SVMs for Histogram-BasedImage Classification”, IEEE Trans. on Neural Networks, vol. 10, pp.1055-1064, 1999.[4] H. K. Ekenel, M. Fischer, H. Gao, K. Kilgour, J. S. Marcos, and R.Stiefelhagen, ”Universität Karlsruhe (TH) at TRECVID 2007”, NISTTRECVID Workshop, Gaithersburg, USA, Nov. 2007.[5] J. Huang, S. R. Kumar, M. Mitra, W.-J. Zhu, and R. Zabih, ”Imageindexing using color correlograms”, Proc. IEEE Conf. Computer Visionand Pattern Recognition, pp. 762-768, San Juan, 1997.[6] T. Joachims, ”Making large-Scale SVM Learning Practical. Advances inKernel Methods - Support Vector Learning, B. Schölkopf and C. Burgesand A. Smola (ed.), MIT-Press, 1999.[7] K. Morik, P. Brockhausen, T. Joachims, ”Combining statistical learningwith a knowledge-based approach - A case study in intensive caremonitoring”, Proc. 16th Int’l Conf. on Machine Learning, 1999.[8] D.-D. Le, S. Satoh, and T. Matsui, ”NII-ISM, Japan at TRECVID 2007:High level Feature Extraction”, NIST TRECVID Workshop, Gaithersburg,USA, Nov. 2007.[9] M. Stricker and M. Orengo, ”Similarity of color images”, Proc. SPIEStorage and Retrieval for Image and Video Databases, vol. 2420, pp.381-392, San Jose, USA, Feb, 1995.[10] M. J. Swain and D. H. Ballard, ”Color Indexing”, Int. Journal ofComputer Vision, vol. 7, no. 1, pp. 11-32, 1991.[11] A. Yanagawa, S.-F. Chang, L. Kennedy, and W. Hsu, ”Columbia Univer-sity’s Baseline Detectors for 374 LSCOM Semantic Visual Concepts”,Columbia University ADVENT Technical Report #222-2006-8, March,2007.

Universität Karlsruhe (TH) at TRECVID 2008

https://publikationen.bibliothek.kit.edu/1000026424/2027006

Universität Karlsruhe (TH) at TRECVID 2008

Abstract

Similar works

Full text

Available Versions

KITopen