Lab ImageCLEF: Photo Annotation - Conference website: http://clef2011.eu/International audienceWe describe in this paper the different approaches tested for the Photo Annotation task for CLEF 2011. We experimented state of the art techniques, by proposing late fusions of several classifiers trained on several features extracted from the images. The classifiers are SVMs and the late fusion is a simple addition of classification probabilities coming from the SVMs. The results obtained place our runs in the middle of the pack, with our best visual-based MAP at 0.337 We also integrated of Flickr human annotations, leading to a large increase of the MAP with a value of 0.377

Albatal, Rami

Mulhem, Philippe

Quénot, Georges

Safadi, Bahjat

INRIA a CCSD electronic archive server

HAL Id: hal-00762711https://hal.archives-ouvertes.fr/hal-00762711Submitted on 7 Dec 2012HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.LIG-MRIM at Image Photo Annotation Task inImageCLEF 2011Rami Albatal, Bahjat Safadi, Georges Quénot, Philippe MulhemTo cite this version:Rami Albatal, Bahjat Safadi, Georges Quénot, Philippe Mulhem. LIG-MRIM at Image Photo Annota-tion Task in ImageCLEF 2011. CLEF 2011 - Conference on Multilingual and Multimodal InformationAccess Evaluation, Sep 2011, Amsterdam, Netherlands. 8p. ￿hal-00762711￿LIG-MRIM at Image Photo Annotation task inImageCLEF 2011Rami Albatal1, Bahjat Safadi2, Georges Quénot3, and Philippe Mulhem31 LIG-UPMF2 LIG-UJF3 LIG-CNRS385 av. de la bibliothèque, 38041 Grenoble CedexRami.Albatal@imag.frAbstract. We describe in this paper the different approaches tested forthe Photo Annotation task for CLEF 2011. We experimented state of theart techniques, by proposing late fusions of several classifiers trained onseveral features extracted from the images. The classifiers are SVMs andthe late fusion is a simple addition of classification probabilities comingfrom the SVMs. The results obtained place our runs in the middle of thepack, with our best visual-based MAP at 0.337 We also integrated ofFlickr human annotations, leading to a large increase of the MAP witha value of 0.377.1 IntroductionThis paper aims at describing the proposal and results of the LIG-MRIMresearch group at the Photo Annotation task for CLEF 2011. The pro-posal of the group focused mainly on applying a late fusion on multiplelearners based on SVM. We also experimented some processes to reducethe feature space dimensions, and we made use of a simple integrationwith Flickr tags. The findings according to the official evaluations confirmthat: late fusion of multiple features lead to good result, that dimensionreduction on few features is an interesting direction to focus on, and thata simple integration of human assigned tags improves results.The corpus [4] for this year is composed of a training set of 8,000 imagesand the test set is 10,000 images large. The image annotation is a multi-label classification process, where the 99 labels go from image elements(like Flowers), to feelings generated by the images (like scary). The im-ages are possibly associated with EXIF data, as well as with Flickr tagsprovided by human. The main evaluation is MAP-based, and we focushere only on this measure to evaluate our runs.The outline of this paper is the following. In section 2, we begin describethe visual feature extracted and their representation. Section 3 presentsthe processing applied on Flickr tags. In section 4, we focus on the clas-sification applied on the extraction, as on the fusion processed betweenthe different learners results. In section 5, we list our results, and weconclude in section 6.22 Extraction and representation of visualfeaturesWe focus here the feature extracted, as well some PCA-based dimen-sion reduction on some features. The features considered cover most thecommon feature we find the in literature.2.1 Simple featuresThe features that were extracted are color-based as well as texture based.Some features are extracted globally from the whole image, and othersare extracted from image regions, before being aggregated to representone image. In the following, we give an identifier for each feature beforeexplaining the extracted feature. Such identifiers will be reused in section5.Global features– h3d 64: normalized RGB Histogram. Such color-based histogram is64 dimensions large, using a simple 4 x 4 x 4 subsampling respectivelythe R, G and B colors components;– gab 40: normalized Gabor transform [2]. For this texture-based fea-ture, we select 8 orientations at 5 scales, leading to a 40 dimensionsspace for these histograms;– hg 104: this feature results in a simple concatenation of the two rep-resentations above (h3d64 and gab40), generating a 104 dimensionsspace for the histograms.Local features The local features extracted are SIFT-like. They areextracted for regions of the images, resulting from dense sampling ofharris-laplace region of interest detection. Each of these features are rep-resented as bag of visual word, similarly to [1]; the visual vocabularyis generated using a Kmeans algorithm on a sample of the features ex-tracted from the training set;– opp sift har 1000 and opp sift har 4000: opponent sift features withHarris-Laplace region of interest detector, generated using Koen Vande Sande’s software [5]. Two representations are considered: one of1000 and one of 4000 dimensions.– opp sift dense 1000: features similar to above, except that the re-gions or obtained by dense sampling every 8 pixels of the images. Thebag of word representation generates 1000 dimensional histograms;– rgSift har 4000: rgSIFT features are extracted based on regions ob-tained by the Harris-Laplace detector. The same tool than above isused to generate the 1000 dimensions histograms;– rgSift dense 4000: the rgSIFT are extracted with dense samplingselection. The size of the histograms is 4000 dimensions;All the features described above are based on 1 nearest neighbor as-signment for the generation of the bag of visual words histograms. Asdescribed in [6], softer assignments may be used. We experiment thoseon opponent sift features:3– opp sift har unc 1000 and opp sift har unc 4000: opponent sift fea-tures extracted from region generated by Harris Laplace detectors,with soft assignment. The space dimensions are respectively here1000 and 4000;– opp sift dense unc 1000: opponent sift features extracted from densesampling, with soft assignment. The space dimensions are 1000.2.2 Dimension reduction on features representationsAs shown by [3], some space dimension reduction do not necessarilydegrades the results, and has a large advantage during the learning phase.That is why we applied PCA-based dimension reduction on some ofthe large spaces defined in the previous subsection. First, to modify thevalues in histograms bin we apply a power law normalization, similarto [3], so that the normalized value vnorm for each bin of the histogramsis: vnorm = vα, with v the initial value of the bin, and α a float numberdepending on the collection. On the normalized histograms, we reducethe dimensions to a fixed number by using Principal Component Analysis(PCA). The resulting features are generated using the same a power lawnormalization with α = 0.500 or α = 0.450 (according to the pw in theidentifier) and PCA reduction to 400 dimensions, leading to:– rgsift har 4000 pw0.500p400: from rgsift har 4000;– rgsift dense 4000 pw0.500p400: from rgsift har 4000;– opp sift har 1000 pw0.450 p400 and opp sift har 4000 pw0.450 p400:from respectively opp sift har 1000 and opp sift har 4000;– opp sift dense 1000 pw0.450 p400: from opp sift dense 1000;– opp sift har unc 1000 pw0.450 p400: from opp sift har unc 1000;– opp sift dense unc 1000 pw0.450 p400: from opp sift dense unc 1000;For the “low dimensional” features h3D64, gab40 and hg104, similar tech-niques lead to h3d 64 pw0.250 32, gab 40 pw0.500 20 and hg 104 pw0.375 54,when considering reducing the dimensions by a half.3 Extraction and representation of Flickr tagsAs Image annotation collection is an excerpt from Flickr, the humangenerated tags are available. We know that such manually input tags arenot always easy to process (typos, jokes, etc.), but we propose a simpleway to handle some of them. First, for each image, we split the tagsinto words, and we apply a Porter stemmer in a way to group similarwords into classes. In a second step, if one stemmed tag equals one ofthe 99 stemmed labels, then the label is selected for the image. Theresulting representation is a 99 dimensions binary vector, with 1 if thelabel describes the image and 0 otherwise.4 Classification4.1 Visual onlyAll the classification processes on the visual features use Multiple-SVMclassifiers based on Radial Basis Funcion (RBF) kernels, since it was4Fig. 1. Global classification process for visual featuresproved to be a good solution for data imbalance problems. Such problemsoccur for many labels in the collection under consideration here. So, foreach label, we get positive and negative samples that are used as inputfor the learning of the Support Vectors.During the classification, the image representation is input to the SVMusing each model, and a binary classification is processed. We assumehere that each classifier outputs a probability of classification in [0,1].The final score for each label is then the average of each individual scorefrom each classifier of the label, as shown in figure 1.4.2 Visual + Flickr tagsFor integrating Flickr tags and visual elements, we also use a late fusionapproach. In this case the visual classification result for each label isfused using a max with the label value for the image according to theFlickr tags processing described earlier. The overall process is describedin figure 2.5 Validation set ResultsWe present the MAP results obtained on a validation set. Our trainingset is composed of 2/3rd of the official training set, generated randomlywith a post processing ensuring a similar distribution of the tags that onthe official training set. This last point is important, especially for the5Fig. 2. Global classification process for visual + Flickr featureslabels that have only few samples (like skateboard with only 12 positivesamples). The validation set in composed of the 1/3rd remaining imagesof the official training set.The table 1 presents the results obtained feature by feature for eachvisual feature listed in section 2.1. This table shows that all the SIFT-based features with hard assignment behave consistently, with MAP val-ues between 0.246 and 0.258. The soft assignment opp sift har unc 1000outperforms slightly the hard assignments, but only marginally. We no-tice also that the hg 104 features behave surprisingly well compared toSIFT-like features.The table 2 focuses on the results obtained when considering the dimen-sion reduction process depicted in part 2.2. In the last column of thistable, we list the percentage of increase compared to the original (i.e.,not reduced) features. This table shows that the reduction of dimen-sion proposed always outperforms the original features. This result isespecially visible with the opponent sift features with strict assignment.In any cases, the dimension reduction seems effective for harris laplacefeatures, and less for dense sampling-based features. For the “low diman-sional” features, we notice also a large imrpovement with one one halfreduction, leading to very good results for hg 104 pw0.375 54, which hasonlyt 54 dimensions.The last table, 3 of this section deals with the results obtained afterfusing the results, according to the explanations of section 4. We chosethree fusions, which correspond to the configuration of the official runsubmitted:6Table 1. MAP results on the validation setDescriptor identifier MAPh3d 64 0.186gab 40 0.213hg 104 0.243opp sift har 1000 0.252opp sift har 4000 0.253opp sift dense 1000 0.255rgSift har 4000 0.246rgSift dense 4000 0.258opp sift har unc 1000 0.262opp sift dense unc 1000 0.255Table 2. MAP results on the validation set for reduced feature representationsDescriptor identifier MAP (increase vs. no reduction)opp sift har 1000 pw0.450 p400 0.273 (+ 8.33%))opp sift har 4000 pw0.450 p400 0.282 (+ 11.46%)opp sift dense 1000 pw0.450 p400 0.267 (+ 4.71%)rgsift har 4000 pw0.500p400 0.264 (+ 7.3%)rgsift dense 4000 pw0.500p400 0.270 (+ 4.7%)opp sift har unc 1000 pw0.450 p400 0.280 (+ 6.9%)opp sift dense unc 1000 pw0.450 p400 0.267 (+ 1.9%)h3d 64 pw0.250 32 0.211 (+ 13.44%))gab 40 pw0.500 20 0.215 (+ 0.94%))hg 104 pw0.375 54 0.259 (+ 6.58%))– msvm: the late fusion of all the 20 visual features considered earlierin the paper;– msvm tags: the late fusion of the visual scores and the Flickr tagsscores;– msvw two desc: the late fusion of the two best features according thetable 2, but considering two different kinds of regions for the features(i.e. one Harris-Laplace based, and one dense sampling based) to en-sure variability in the fused results: opp sift har unc 1000pw0.450p400and opp sift dense 1000pw0.450p400.The conclusions drawn from this table is that the fusion always outper-forms each f its components (such result is well known in the community).We see here that Flickr tags integration, even is the processing is quitestraightforward, leads to an important increase of the results.These three configurations are the ones used for the official submissions.7Table 3. MAP results on the validation set for three late fusionsDescriptor identifier MAP (increase vs. best visual feature in the fusion)msvm 0.314 (+ 11.35%)msvm tags 0.357 (+ 26.60%)msvw two desc 0.297 (+ 6.07%)6 Official ResultsWe present here the official MAP results obtained from our runs in 4.This table shows also in the last column the rank obtained in comparablelists (i.e., list of visual results for msvm and msvw two desc, and list ofmulti-modal results for msvm tags). The results obtained place our bestvisual run, msvm with a MAP of 0.336, in the first tier of the list, andabove the average and the median values respectively of 0.289 and 0.323.For the multimodal run, msvm tags with a MAP of 0.378, the rank isabove the middle, and also above the average and the median values ofrespectively 0.370 and 0.371 .Table 4. Official MAP results for the submitted MRIM runsDescriptor identifier Official Id MAP rank (in comparable list)msvm 1308318230664 0.336 15/46msvm tags 1308226825708 0.378 11/25msvw two desc 1308318529187 0.324 23/467 ConclusionThis paper presented the worjk of the LIG-MRIM team for the PhotoAnnotation task for CLEF 2011. We used a large set of 20 features, withor without strict bin assignment, dimension reductions, with and withoutintegrating Flickr tags. The results obtained place our run in the firsttier for the visual runs, and in the first half for the multimedia runs.In the future, we will focus on dimension reductions to find out whatreductions are useful.AknowledgementsThis work was partly supported by: a) the Quaero Programme, fundedby OSEO, French State agency for innovation and b) the Région RhonesAlpes (projet LIMA).8References1. G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray. Visualcategorization with bags of keypoints. In In Workshop on StatisticalLearning in Computer Vision, ECCV, pages 1–22, 2004.2. B. S. Manjunath and W. Y. Ma. Texture features for browsing andretrieval of image data. IEEE Trans. Pattern Anal. Mach. Intell.,18:837–842, August 1996.3. T. Mensink, G. Csurka, F. Perronnin, J. Sánchez, and J. Verbeek. Learand xrces participation to visual concept detection task - imageclef2010. In Working Notes for the CLEF 2010 Workshop, page 48, sep2010.4. S. Nowak, K. Nagel, and J. Liebetrau. The clef 2011 photo annotationand concept-based retrieval tasks. In CLEF 2011 working notes, 2011.5. K. E. A. van de Sande, T. Gevers, and C. G. M. Snoek. Evaluatingcolor descriptors for object and scene recognition. IEEE Transactionson Pattern Analysis and Machine Intelligence, 32(9):1582–1596, 2010.6. J. C. van Gemert, C. J. Veenman, A. W. Smeulders, and J.-M. Geuse-broek. Visual word ambiguity. IEEE Transactions on Pattern Anal-ysis and Machine Intelligence, 32:1271–1283, 2010.

LIG-MRIM at Image Photo Annotation Task in ImageCLEF 2011

Hal - Université Grenoble Alpes

https://hal.archives-ouvertes.fr/hal-00762711/document

LIG-MRIM at Image Photo Annotation Task in ImageCLEF 2011

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

Hal - Université Grenoble Alpes