International audienceAutomatic semantic classification of image databases is very useful for users searching and browsing, but it is at the same time a very challenging research problem as well. Local features based image classification is one of the key issues to bridge the semantic gap in order to detect concepts. This paper proposes a framework for incorporating contextual information into the concept detection process. The proposed method combines local and global classifiers with stacking, using SVM.We studied the impact of topologic and semantic contexts in concept detection performance and proposed solutions to handle the large amount of dimensions involved in classified data. We conducted experiments on TRECVIDï¿½04 subset with 48104 images and 5 concepts. We found that the use of context yields a significant improvement both for the topologic and semantic contexts

Ayache, Stéphane

Quénot, Georges

Satoh, Shin'Ichi

Hal - Université Grenoble Alpes

Context-based conceptual image indexingSte´phane Ayache, Georges Que´not, Shin’Ichi SatohTo cite this version:Ste´phane Ayache, Georges Que´not, Shin’Ichi Satoh. Context-based conceptual image indexing.IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2006,Toulouse, 2006. <hal-00953903>HAL Id: hal-00953903https://hal.inria.fr/hal-00953903Submitted on 3 Mar 2014HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.CONTEXT-BASED CONCEPTUAL IMAGE INDEXINGSte´phane Ayache and Georges Que´not∗CLIPS-IMAGBP 5338041 Grenoble Cedex 9, FranceShin’ichi SatohNational Institute of Informatics2-1-2 Hitotsubashi, Chiyoda-kuTokyo 101-8430, JapanABSTRACTAutomatic semantic classification of image databases isvery useful for users searching and browsing, but it is at thesame time a very challenging research problem as well. Lo-cal features based image classification is one of the key issuesto bridge the semantic gap in order to detect concepts. Thispaper proposes a framework for incorporating contextual in-formation into the concept detection process. The proposedmethod combines local and global classifiers with stacking,using SVM. We studied the impact of topologic and semanticcontexts in concept detection performance and proposed so-lutions to handle the large amount of dimensions involved inclassified data. We conducted experiments on TRECVID’04subset with 48104 images and 5 concepts. We found that theuse of context yields a significant improvement both for thetopologic and semantic contexts.1. INTRODUCTIONIn order to retrieve images from huge digital databases, userscannot specify their needs with low-level features, but withconcepts which are much more understandable. New issues inContent-Based Image Indexing (CBII) field are arising for thereduction of this semantic gap. In order to improve conceptdetection, many approaches take into account the context. Todo so, some approaches fuse local and global descriptors [? ?] combined using boosted classifiers [? ].Other define context as spatial relationships between objectswithin an image [? ? ? ] using probabilistic frameworks.However, in order to deal with a large amount of local de-scriptors and simplify computation, such approaches first de-tect points of interest and then assume them independent.In their work, [? ? ? ] handle semantic relationships [? ]between concepts using Stacked Classifiers [? ? ]. They firstclassify intermediate concepts, and learn their relationships inthe context of a higher level concept by a second-level classi-fier. In TRECVID’04 experiments, [? ] used a basis of 22 in-termediate concepts. By adding the 10 TRECVID’s concepts,they learned semantic context of 32 concepts with Stacking.∗Georges Que´not is researcher at CNRSIn this study, we propose and evaluate a new framework forincorporating contextual information into image indexing pro-cess. We assume that both semantic and local contexts can in-crease the accuracy of a classifier. We incorporate both localand inter-concept information in the decision process, in orderto increase accuracy of classification. However, it is difficultto merge many information while managing the curse of di-mensionality problem. Thus, we study an ”hybrid” approachwhich tries to merge contextual information with few data.We evaluate and validate our approach on five high-level fea-tures of TRECVID 2003 and 2004 Corpus.2. MULTI-LEVEL FRAMEWORKOur framework uses several local and global classifiers andarrange them into networks, using stacking, in order to catchcontextual information. The idea is that the correlation be-tween the input (low-level features) and the output (concepts)is too weak to be efficiently recovered by a single “flat” clas-sifier even if the low-level features have been carefully cho-sen. Combining classifiers in a multi-level framework al-lows extraction of intermediate-level data from low-level fea-tures and other classifiers. Detection of concepts from theseintermediate-level data is expected to improve the overall per-formance (both correctness and computation time) of the sys-tem. Figure ?? shows an overall architecture of our frame-work, and how classifiers are combined.Classifiers from each level bridge a small part of the seman-tic gap and are expected to do it well because the correlationbetween their inputs and their outputs is expected to be bet-ter than the correlation between the inputs and the outputsof a single classifier that would bridge alone the whole se-mantic gap. Also, not only the levels are cascaded but manyintermediate data are computed in parallel and the outputs ofall of them are combined as inputs to the next level. Thisframework is able to derive high-level concepts from other in-termediate concepts, but also reinforce intermediate conceptsdetection. Experiments of the present work will focus of thesecond point.The objective of the present work is to validate our assump-tions and to quantify the benefits that can be obtained fromFig. 1. Overall architecture and experimented networkscontextual information. We conducted several experimentswith various networks of classifiers.Figure ?? shows the multi-layer framework, the first layer(Patch Classifier, on the left) computes scores of concepts de-tection at patch-level. Then higher level layers assign scoresfor the whole images. The first layer is basically the only onethat handles low-level features. The higher layers handle out-put of lower level layers i.e. intermediate data.3. LOW-LEVEL FEATURES EXTRACTIONAs we want to handle topologic context, we need to computelow-level features for parts of the image. In order to computelocal features, many approaches have been proposed. Whileautomatic and a priori segmented regions are too far from se-mantic meaning of image, we decided to split images intopatches. By doing so, we should be very far from semantic,but with such granularity one patch is more likely to containa single concept.The low-level features extractor’s process first splits imageinto overlapped patches. Basically, we used 260 overlappedpatches of 32 × 32 pixels (in 352 × 240 images). And then,extracts 9 moments color (3 means + 6 co-variances), Gaborwavelets for texture (3 scales × 8 orientations), and coordi-nates of patches.Those choices have been made for a baseline system. Themain goal here is to explore the use of context for conceptindexing. We want to study and evaluate various ways of do-ing it by combining classifiers into networks. In further work,we plan to enrich and optimize the set and characteristics oflow level features, especially for video content indexing. Cur-rently, we expect to obtain representative results from the cur-rent set of low-level features.4. USE OF THE TOPOLOGIC CONTEXTThe idea behind the use of topologic context is that the confi-dence (or score) for a single patch (and for the whole image)could be computed more accurately by taking into accountthe confidences obtained for other patches in the image forthe same concept.We studied three network organizations to evaluate the effectof using the topologic context in concept detection at the im-age level. The first one is a baseline in which no context (ei-ther topologic or semantic) is used. The second one uses thetopologic context in a flat (single layer) way while the thirduses the topologic context in a serial (two layers) way.In this part, we consider concepts independently one fromanother. Concept classifiers are trained independently fromeach other whatever their levels. In the following, N willbe the number of concepts considered, P will be the numberof patches used (260 in our experiments) and F will be thenumber of low-level feature vectors components (35 in ourexperiments).4.1. Baseline, no context, one level (1)In order to evaluate the patch level alone, we define an imagescore based on the patch confidence values. To do so, we sim-ply compute the average of all of the patch confidence scores.This baseline is very basic it does not take into account anyspatial or semantic context. We have here N classifiers, eachwith F inputs and 1 output. Each of them is callled P timeson a given image and the P output values are averaged.4.2. Topologic context, flat, one level (2)The “flat” network directly computes scores at the image levelfrom feature vectors. We have here N classifiers, each withF ×P inputs and 1 output. Each of them is callled only onceon a given image and the single output value is taken as theimage score. This network organization is not very scalableand requires a lot of training data and training times becauseof the large number of inputs of the classifiers.4.3. Topologic context, serial, two levels (3)The “serial” network is similar to the baseline one. The dif-ference is that the scores at the image level are computed by asecond level of classifiers instead of averaging. We have hereN level 1 classifiers, each with F inputs and 1 output andN level 2 classifiers, each with P inputs and 1 output. Eachlevel 1 classifier is called P times on a given image and itsP output values are passed to the corresponding level 2 clas-sifier which is callled only once. Topologic context is takeninto account by concatenating patches confidence value in avector.5. USE OF TOPOLOGIC AND SEMANTICCONTEXTSThe idea behind the use of semantic context is that the confi-dence (or score) for a single concept could be computed moreaccurately by taking into account the confidences obtained forother concepts in the same image.We studied three other network organizations to evaluate theeffect of using additionnally the semantic context in conceptdetection at the image level. We still include outputs fromthe patch level, but we do so using the outputs related to allother concepts for the detection of any given concept. We arenow considering concepts as related one to each other (and nolonger independently one from another).5.1. Topologic and Semantic contexts, sequential, threelevels (4)The fourth network simply take the output of the third one(topologic context, serial, two levels) and adds a third levelthat uses the scores computed for all concepts to re-evaluatethe score of a given concept. We have additionnally here Nlevel 3 classifiers, each with N inputs and 1 output. Eachlevel 3 classifier is callled only once on a given image.5.2. Topologic and Semantic contexts, parallel, two levels(5)The fifth network is similar to the previous version exceptthat the last two levels have been flattened and merged into asingle classifier. The difference is similar to the difference be-tween the serial and flat versions of the networks that use onlythe topologic context. We have here N level 1 classifiers,each with F inputs and 1 output and N level 2 classifiers,each with N × P inputs and 1 output. All level 1 classifiersare callled P times on a given image and their N × P out-put values are passed to the corresponding level 2 classifierwhich is called only once.5.3. Topologic and Semantic contexts, parallel, three lev-els (6)The previous network suffers from the same limitation as theother flattened version it is not very scalable and requires alot of training data and training times because of the largenumber of inputs of the classifiers. The flattening, however,permits to use the topologic and semantic information in par-allel and in a correlated way. The sequential organization, onthe contrary, though making use of both information does itin a non-correlated way.The sixth network organization try to keep both contexts cor-related (though less coupled) while avoiding the curse of di-mensionality problem. The N × P number of inputs is re-placed by N+P . The architecture is a kind of hybrid betweenthe two previous ones. It is the same as in the sequential casebut P inputs are added to the classifiers f the last level. TheseP input comes directly from the output of the first level but forthe corresponding concept only (instead as the output from allP patches times all N like in the flattened case).6. EXPERIMENTSWe conducted several experiments in TRECVID’04 Collabo-rative Annotation Corpus, in order to study our framework’sbehavior. We used the trec eval tool and TRECVID pro-tocol, i.e. return a ranked list of 2000 top images. TRECVID-’04 Collaborative Annotation Corpus contains 48104 key fra-mes. We split it into 50% training set and 50% test set.We focus on 5 intermediate concepts which can be extractedas patch-level: building, sky, greenery, skin faceand studio setting background. We choose thembecause of their semantics relationships. building, sky,greenery are closer than others. Additionally,skin faceandstudio setting background occur often together.We used SVM classifier with RBF Kernel, because it hasshown good classification results in many fields, especiallyin CBIR [GC04]. We use cross validation for parameter se-lection, using grid search tool to select the best combinationof parameters C and gamma (out of 110).In order to obtain the training set, we extracted patches fromannotated regions, it is easy to get many patches by perform-ing overlapped patches. Annotating whole images is harderas annotators must observe each one.We collected many positive samples for patches annota-tion, and defined experimentally a threshold for maximumnumbers of positive samples. We found that 2048 positivesamples is a good compromise to obtain good accuracy withsmaller training time. Also, we found that twice negativesamples is a good compromise. Finally, we randomly choosenegative samples. The Table ?? shows the number of positiveimage examples, for each concept.Table ?? shows the relative performance and training timefor the detection of five concepts and for the six considerednetwork organizations. As expected, the flattened version re-quires much higher training time. For the presented times, weadded the training times of each intermediate levels, it includethe cross-validation time. Also, the cross-validation processcan be performed in parallel [? ], we used 11 3Ghz Pentium4”Hyperthreaded” processors, 1Go RAM each. The reportedresults are the corresponding for one single processor.The use of topologic context improves the performance overthe baseline and combined with the semantic context improvesit even further. The performance of the three-level sequentialclassifier is poorer than the two-level serial one. This maybe due to the lack of information of his final level classifier,build. sky stud. green. skin. all timeBaseline, no context, one level (1) 0.3406 0.1608 0.4087 0.6258 0.1580 0.3388 396Topologic context, flat, one level (2) 0.1927 0.5453 0.8905 0.4623 0.3421 0.4866 836Topologic context, serial, two levels (3) 0.3077 0.4331 0.7675 0.7207 0.4562 0.5370 418Topo. and Semantic, sequential, three levels (4) 0.2823 0.4036 0.6498 0.7233 0.4391 0.4996 459Topo. and Semantic, parallel, two levels (5) 0.4230 0.5606 0.9106 0.7283 0.4280 0.6101 484Topo. and Semantic, parallel, three levels (6) 0.3381 0.4639 0.8444 0.6812 0.4424 0.5540 451Nb positives images examples 383 1583 429 712 895Table 1. Comparative performance of network organizations: mean average precision (MAP) for five concepts, mean of MAPS,and corresponding training times (in minutes)which have N (currently 5) inputs only. This may changewhen a much higher number of concepts are used.For the networks which use both topologic and semantic con-texts, the hybrid version has an intermediate performance be-tween the sequential and parallel flattened versions. The two-level version has the better performance as it merge more in-formation. However, it does not scale well with the numberof concepts while the hybrid version suffers much less fromthis limitation and should performs better with more concepts.Also, by comparing second and fifth networks results, wecan conclude that dimensionality reduction induced by ourapproach is really significant, in term of both accuracy andcomputational time.7. CONCLUSIONWe presented a framework for incorporate semantic and topo-logic contexts in CBII. We compared several networks andshowed that both contextual information improves conceptsdetection. We proposed an hybrid network which is promis-ing for further scalable experiments. Then, we found that di-mensionality reduction induced by our framework, providesbetter accuracy in shorter computation time than a flattenedclassifier. This result is also very useful for further scalableexperiments. Finally, we plan to evaluate how our frameworkcan derive higher-level semantic concepts.8. ACKNOWLEDGMENTSThis work has been spported by the ISERE CNRS ASIA-STIC project and the Video Indexing INPG BQR project.References[] K. Murphy, A. Torralba, D. Eaton and W. Freeman Object de-tection and localization using local and global features. SicilyWorkshop on Object Recognition, 2005. Lecture Notes inComputer Science (submitted)[] A Garg, S Agarwal, T.S. Huang Fusion of Global and LocalInformation for Object Detection. In 16th International Con-ference on Pattern Recognition (ICPR’02) - Volume 3, 2002.[] J. Amores, N. Sebe, P. Radeva, T. Gevers and A. SmeuldersBoosting Contextual Information in Content-Based Image Re-trieval. In MIR, 2004.[] A. Singhal, J. Luo, and W. Zhu Probabilistic spatial contextmodels for scene content understanding. In CVPR, 2003.[] A. Torralba, K. Murphy, and W. Freeman Contextual modelsfor object detection using boosted random felds. In Advancesin Neural Info. Proc. Systems, 2004.[] P. Carbonetto, N. d. Freitas, and K. Barnard. A statistical modelfor general contextual object recognition. In ECCV, 2004.[] G. Iyengar and H.J. Nock Discriminative model fusion for se-mantic concept detection and annotation in video. MULTI-MEDIA ’03: Proceedings of the eleventh ACM internationalconference on Multimedia, 2003.[] M. Naphade. On supervision and statistical learning for se-mantic multimedia analysis. Journal of Visual Communicationand Image Representation, 15(3):348369, 2004.[] C.G.M. Snoek, M. Worring, J.M. Geusebroek, D.C. Koelmaand F.J. Seinstra The MediaMill TRECVID 2004 SemanticVideo Search Engine. InTRECVID Workshop, 2004.[] M. Fink P. Perona Mutual boosting for contextual influence. InAdvances in Neural Info. Proc. Systems, 2003.[] D.H. Wolpert Stacked Generalization. Neural Networks, Vol.5, pp. 241-259, Pergamon Press.[] D. A. Lisin, M. A. Mattar, M B. BlMark C. Benfield andE. G. Learned-Miller Combining Local and Global Image Fea-tures for Object Class Recognition In CVPR, 2005.[] P. Howarth and S. Rueger. Evaluation of Texture Features forContent-Based Image Retrieval. In P. Enser et al. (Eds.): CIVR2004, LNCS 3115, pp.326-334, 2004.[] P.H. Gosselin and M. Cord. A Comparison of Active Classifi-cation Methods for Content-Based Image Retrieval. Int. Work-shop on Computer Vision Meets Database, CVDB, 2004.[] C. Chang and C Lin LIBSVM: a library for sup-port vector machines, 2001. Software available athttp://www.csie.ntu.edu.tw/ cjlin/libsvm

Context-based conceptual image indexing

Abstract

Similar works

Full text

Available Versions

Hal - Université Grenoble Alpes