We describe our fourth participation, that includes two high-level feature extraction runs, and one manual search run, to the TRECVID video retrieval evaluation. All of these runs have used a system trained on the common development collection. Only visual information, consisting of color, texture and edge-based low-level features, was used

Aksoy, C.

Aksoy, S.

Aydin, E.

Duygulu P.

G̈unaydin, D.

Hadimli, K.

Koç L.

Olgun, Y.

Orhan, C.

Yakin G.

Bilkent University Institutional Repository

Bilkent University at TRECVID 2007S. Aksoy, P. Duygulu,C. Aksoy, E. Aydın, D. Günaydın, K. Hadımlı,L. Koç, Y. Olgun, C. Orhan, G. YakınRETINA Vision and Learning GroupDepartment of Computer EngineeringBilkent UniversityBilkent, 06800, Ankara, TurkeyAbstractWe describe our fourth participation, that includes two high-level featureextraction runs, and one manual search run, to the TRECVID video retrievalevaluation. All of these runs have used a system trained on the commondevelopment collection. Only visual information, consisting of color, textureand edge-based low-level features, was used.1 IntroductionThis is the fourth participation of the RETINA Vision and Learning Groupat Bilkent University to TRECVID. The team that participated to TRECVIDincluded eight undergraduate students supervised by two faculty members.We have developed a system for automatic classification and indexing ofvideo archives as part of undergraduate research projects. This paper sum-marizes the approaches we have taken in two high-level feature extractionruns, and one manual search run we have submitted this year.2 PreprocessingIn all of the runs, we have used the shot boundaries, keyframes, and manualannotations provided with the TRECVID 2007 data.We model spatial content of images using grids and partition the 352×240 video frames into 5 rows and 7 columns. Then, low-level features basedon color, texture and edge are computed individually on each grid cell andalso on the entire image. The features used are the histograms and statistics(mean and standard deviation) of RGB, HSV and LUV values of the corre-sponding pixels as the color features and the statistics of the Gabor waveletresponses of the pixels at 3 different scales and 4 different orientations as thetexture features. Histograms of the gradient orientation values of the Cannyedge detector outputs are used as the edge features. Orientation values aredivided into bins with increments of 45 degrees and an extra bin is usedto store the number of non-edge pixels. Individual components of each fea-ture vector are normalized to unit variance to approximately equalize rangesof the features and make them have approximately the same effect in thecomputation of similarity.3 High-Level Feature ExtractionWe have developed two methods for high level feature extraction. The firstone (Bilkent1) is a generic classifier that uses the low-level features describedwith the k-nearest neighbor rule. First, we performed experiments usingdifferent feature combinations and different k values on the developmentdata. We have empirically decided to use tiled HSV histograms and Canny-based edge orientation histogram features. The k value was also chosen as51. The examples provided with the common annotation were used to trainthe classifier for all high-level features.The second type of detectors developed by Bilkent University (Bilkent2)exploit both color and spatial information using a bag-of-regions represen-tation [1]. The shot classification problem has two critical components: rep-resenting shots and learning models for semantic categories using these rep-resentations. Shot representation is achieved by segmenting keyframes intoregions and forming a region codebook that is used to construct histogramsof individual regions and region pairs. Learning process is implemented bytraining Bayesian classifiers using these representations.The first step is the partitioning of keyframes into regions. After ex-perimenting with several segmentation algorithms, we decided to use thek-means with connectivity constraint algorithm [2]. After an image is seg-mented into several regions, each region is modeled using the multivariatehistogram of the HSV values of its pixels with 8 bins used for the H channeland 3 bins for each of S and V channels, resulting in a 72-dimensional featurevector. Then, a codebook of region types is constructed using the k-meansalgorithm for vector quantization. The number of codewords (k) was set to1000 empirically. The output of this step is a discrete type label assignedto each region.Color information can be very useful in discriminating objects/regions inan image if they have very distinct colors. However, just like any other low-level features, color cannot distinguish conceptually different objects/regionsif they fall to nearby locations in the feature space. An important elementof image understanding is the spatial information. For example, finding aregion with dominant blue color (that may be water) and a neighboringbeige region (that may be sand) with another blue region (that may besky) above them can increase the possibility of being a coast image for thatimage. Furthermore, two images with similar regions can have very differentinterpretations if the regions have different spatial arrangements. Hence,spatial information can be used to resolve ambiguities in image classification.Different methods have been proposed to model region spatial relation-ships. However, it becomes a combinatorial problem if one tries to model allpossible relationships between regions in an image. Therefore, we decidedto use only the vertical relationship of “above-below” because it arguablyprovides a better characterization of the content. For example, flipping aphotograph horizontally does not usually alter its semantics but flipping itvertically or rotating it greatly perturb its perception. To determine thevertical relative position of two regions, we use their projections on bothaxes. If there is an overlap between the projections on the x-axis, their pro-jections on the y-axis are compared. If they have no overlap on the y-axisor if the overlap is less than 50 percent of the area of the smaller region, weconclude that the one with a greater centroid ordinate is above the otherone. If these overlap criteria are not met, it is concluded that no significantvertical relative arrangement exists between these two regions. The result ofthis step is a list of region pairs that satisfy the “above-below” relationshipfor each image.After each region is assigned a type label and the pairwise spatial re-lationships are computed, each image is represented as a “bag-of-regions”.We consider two settings for this bag-of-regions representation: 1) each re-gion is regarded separately and a “bag of individual regions” representationis generated, and 2) regions that satisfy the above-below relationship aregrouped together and a “bag of region pairs” representation is constructed.Finally, these two representations are used separately to train Bayesian clas-sifiers. Given the positive examples for each semantic concept (high-levelfeature), using multinomial density models, the probability values neededby the Bayesian decision rule are computed using the maximum likelihoodestimates.(a)(b)Figure 1: Manual search results using low-level color and edge-based fea-tures, and number of faces (a) for a query on finding people in crowd (topic214), and (b) for a query on finding three or more people at a table (topic209).4 SearchThe manual run used only low-level visual features that were also used inthe high-level feature extraction task. The number of faces in the shots werealso used as an additional information. The shots were sorted according totheir distance to the query shot using these features. Example queries areshown in Figure 1.5 ConclusionsOur participation to TRECVID consisted of two high-level feature extrac-tion runs, and one manual search run this year. We are currently workingon extending our system with new low-level features, classifiers and novelmethods for their multi-modal fusion.References[1] D. Gokalp and S. Aksoy. Scene classification using bag-of-regions rep-resentations. In Proceedings of IEEE Conference on Computer Visionand Pattern Recognition, Beyond Patches Workshop, Minneapolis, Min-nesota, June 23, 2007.[2] I. Kompatsiaris and M. G. Strintzis. Spatiotemporal segmentationand tracking of objects for visualization of videoconference image se-quences. IEEE Transactions on Circuits and Systems for Video Tech-nology, 10(8):1388–1402, December 2000.

Bilkent University at TRECVID 2007

http://repository.bilkent.edu.tr/bitstream/11693/27128/1/Bilkent%20University%20at%20TRECVID%202007.pdf

Bilkent University at TRECVID 2007

Abstract

Similar works

Full text

Available Versions

Bilkent University Institutional Repository