Searching images based on descriptions of image attributes is an intuitive process that can be easily understood by humans and recently made feasible by a few promising works in both the computer vision and multimedia communities. In this report, we describe some experiments of image retrieval methods that utilize weak attributes

Chang, Shih-Fu

Ji, Rongrong

Tsai, Ming-Hen

Ye, Guangnan

Yu, Xinnan

English

Columbia University Academic Commons

Columbia University Computer Science Department Technical Report # CUCS 005-12 (2012)Experiments of Image Retrieval Using Weak AttributesFelix X. Yuy, Rongrong Jiy, Ming-Hen Tsaix, Guangnan Yey, Shih-Fu ChangxColumbia University, New York, NY 10027yfyuxinnan, rrji, yegng@ee.columbia.edu xfminghen, sfchangg@cs.columbia.eduAbstractSearching images based on descriptions of image attributes is an intuitive processthat can be easily understood by humans and recently made feasible by a few promisingworks in both the computer vision and multimedia communities [4,7,9,11]. In thisreport, we describe some experiments of image retrieval methods that utilize weakattributes [11].1 IntroductionSearching images based on descriptions of image attributes is an intuitive processthat can be easily understood by humans and recently made feasible by a few promisingworks in both the computer vision and multimedia communities [4,7,9,11]. In thisreport, we describe some experiments of image retrieval methods that utilize weakattributes [11]. This technical report includes experimental results and discussions inaddition to those in [11]. The experiments are done on the Labeled Faces in the Wild(LFW) [3], a-PASCAL/a-Yahoo [1], and a-TRECVID [11] datasets. For LFW, we haveselected a small number of weak attributes, to visualize the learnt dependency model.For a-PASCAL/a-Yahoo and a-TRECVID, we evaluate the contribution of dierenttypes of weak attributes. In this report, our focus is evaluation of attribute basedretrieval, instead of recognition or verication.2 Labeled Faces in the Wild (LFW)We implemented and tested the weak-attribute based image retrieval method de-scribed in [11]. The rst evaluation is on the Labeled Faces in the Wild (LFW) dataset[3], which contains 9,992 images with manual annotations of 27 query attributes, in-cluding \Asian", \Beard", \Bald", \Gray Hair", etc. Following the setting of [9], we1Figure 1: Learntw for LFW dataset with sparsity k = 40 (best viewed in color). Vertical labels arequery attributes, and horizontal labels are weak attributes. Red, blue and white represent positive,negative and no dependency respectively. High intensity means high dependency level. The sparsepattern of weak attributes results in faster training/testing and less overtting. This learned matrixis semantically plausible for most dependency patterns. For instance, people \Wearing Lipstick"(query) is unlikely to be \Male" (weak), and \Kid" (query) is highly related to \Child" (weak).Note that not all mappings are semantically meaningful, due to the fact that the weak attributes,though with some specic names, can be \weak", i.e. inaccurate, in terms of semantic meanings.randomly choose 50% of this dataset for training and the rest for testing. For visu-alization, the weak attributes for this dataset contain only attribute classier scoresfrom [5] (scores of 73 attribute classiers designed for human faces) and latent variablesproduced by the graphical model1.In order to get the baseline results of individual classiers (direct classiers cor-responding to the query attributes), we have omitted three attributes: \Long Hair",\Short Hair" and \No Beard" which are not covered by the classier scores from[5]. Figure 1 shows the learnt dependency model w using our proposed method withsparsity k = 40. For visualization, we only show the weak attributes selected by single-attribute queries, i.e. each query attribute is uniquely mapped to 40 weak attributesonly. Note the sparse pattern considered in our model is query specic, rather thana xed one across dierent queries. For example, for a single query \Asian", the s-elected weak attributes might be \Asian" and \White", while for a double attributequery \Asian"+\Woman", the selected weak attributes might be \Asian" and \Male".In both learning and prediction processes involving the dependency model, only theselected weak attributes will be considered for each multi-attribute query.Figure 2 shows the comparisons of our method to several existing approaches, in-cluding TagProp [2], Reverse Multi-Label Learning (RMLL) [8], Multi-Attribute based1The value of latent variables are acquired by inferencing on the unsupervised graph, conditioned on theweak attributes. We have transformed the conditional marginal distribution of latent variables back to thereal interval [ 1; 1] by linear mapping.2Figure 2: Retrieval performance on LFW dataset. The rst three results are copied from [9], inwhich dierent individual classiers are used. The last three results are based on our implementa-tion.Ranking and Retrieval (MARR) [9], individual classier scores from [5], and our im-plementation of MARR based on individual classier scores. Our method outperformsall the other competing methods consistently for all types of queries including single,double and triple attributes. It is interesting to note that in this experiment, theweak attributes are actually not \weak", in the sense that even individual classiersoutperform TagProp, RMLL and MARR reported in [9], in which dierent individualclassiers are used. The weak attributes are weak in the sense that they are trainedfrom other sources, therefore are not directly related to the specic dataset at hand.Closely following [9], our implementation of MARR only slightly outperforms individ-ual classiers: the stronger baseline reduces the amount of improvement that can beobtained by utilizing dependency information within query attributes only.3 a-PASCAL and a-Yahoo2The dataset a-PASCAL [1] contains 12,695 images (6,340 for training and 6,355 fortesting). Each image is assigned one of the 20 object class labels: people, bird, cat,etc. Each image also has 64 query attribute labels, such as \Round", \Head", \Torso",\Label", \Feather" etc. The a-Yahoo dataset is collected for 12 object categories fromthe Yahoo images search engine. Each image in a-Yahoo is described by the same set2The rst few paragraphs summarizing the datasets and weak attributes are copied from [11]. In the endof the section, we add analysis and discussions about the contributions by dierent types of weak attributes.3Figure 3: Retrieval performance on a-PASCAL dataset. This gure is copied from [11]. Left:AUC comparison based on optimal sparsity (k = 400). The rst three results are copied from[9], under the same congurations compared to ours. Right: AUC of our approach with varyingsparsity.of 64 attributes, but with dierent category labels compared to a-PASCAL, includingwolf, zebra, goat, donkey, monkey etc.Following the setting of [1], we use the pre-dened training images of a-PASCAL astraining set, and test on pre-dened test images of a-PASCAL and a-Yahoo respectively.We use the feature provided in [1]: 9,751-dimensional features of color, texture, visualwords, and edges to train individual classiers. Following [11], other weak attributesinclude: Scores from Classemes semantic classiers [10]: 2,659 classiers trained on imagesreturned by search engines of corresponding query words/phrases; Discriminative attributes [1], which are trained using linear SVM by randomlyselecting 1-3 categories as positive, and 1-3 categories as negative; Random image distances: the distance of each image to some randomly selectedimages based on the 9,751-dimensional feature vector; Latent variables, as detailed in Section 2.This nally results in 5,000 weak attributes for each image.Figure 3 shows performance evaluation results using the a-PASCAL benchmark, incomparison with the state-of-the-art approaches in [2,8,9]. Figure 4 shows the perfor-mance of weak attributes on the a-Yahoo benchmark compared to individual classiersand MARR. From both of them, the weak attribute approach has demonstrated sub-stantial performance gain compared to other methods. Detailed analysis of the twogures can be found in [11].4Figure 4: Retrieval performance on a-Yahoo dataset. This gure is copied from [11]. Left: AUCcomparison based on optimal sparsity (k = 200). Right: AUC of our approach with varyingsparsity.To evaluate the contribution of heterogeneous types of weak attributes in the de-pendency model, we conduct experiments to compute the percentage of weights inthe dependency model coming from dierent weak attribute types. Table 1 shows theresults. For a-PASCAL, about 90% of the weights are from the weak attributes (exclud-ing individual query attribute classier scores). For a-Yahoo (cross-dataset retrieval,the training and test set are under dierent statistics), this value is even higher. Itmeans that for both datasets, weak attributes play a key role. And for a-Yahoo, the in-dividual query attribute classier scores have a much worse generalization power (3.23%) compared to a-PASCAL (10.49%), due to cross-dataset issues. We now discuss thecontributions from dierent types of weak attributes.Classemes. It is encouraging to notice that, for both a-PASCAL and a-Yahoo,Classemes [10] contribute considerably in the dependency model. Classemes representa class of weak but semantically meaningful attributes trained on some other datasources. Given extensive emerging research on image attributes and classication, thiskind of classiers are becoming widely available, and can be readily utilized in the weakattribute framework to improve the performance. What is more encouraging is thatClassemes are trained automatically from results of image search engines. Therefore,even in the original weak attribute training process, it requires no human labelingburdens.Discriminative attributes. The contribution of discriminative attributes is muchmore substantial for a-PASCAL compared to a-Yahoo. The reason is that categoriesof a-Yahoo are very dierent from those of a-PASCAL, from which the discriminativeattributes are trained. Thus this kind of attributes have far less description poweron a-Yahoo. Generally speaking, the more relevant the weak attributes to the query5Query Att Classier \Classemes" Disriminative Att Othersa-PASCAL 10.49%(64) 25.87%(2659) 48.73%(1350) 14.91%(927)a-Yahoo 3.23%(64) 74.30%(2659) 17.23%(1350) 5.24%(927)a-TRECVID 20.09%(126) 70.83%(2659) 3.84%(2000) 5.24%(1215)Table 1: Contributions of dierent types of weak attributes. The table shows percentage ofweights in the dependency model coming from dierent weak attribute types. The number inbrackets represents the number of weak attributes in the corresponding type. In our case, \others"include random image distances and latent variables.attributes, the more weights will be associated with them.Others (latent variables and random distances in our case). For both a-PASCALand a-Yahoo, other weak attributes do not contribute much in the dependency model.However, based on our experiments, discarding them does have a negative impact onthe nal performance. So likely they do contribute to some queries. The advantageof the proposed weak attribute based retrieval framework is that it can incorporateall kinds of weak attributes, even random distances, while automatically selecting asubset that are potentially useful.We have also conducted experiments to visualize the dependency model similar toFigure 1. Ideally, the weak attributes in helping to answer a specic query shouldbe semantically related to the query (the relation can be either positive or negative).For example, a query describing a person is likely to be mapped positively to weakattributes including classiers scores related to person, template instances related toperson images, or latent variables representing person related weak attribute groupsetc. Also it should be negatively mapped to some weak attributes describing scene,object, texture etc. without person. Similar to Figure 1, the visualization has demon-strated some relations with semantically meaningful explanations. However, manyelements in the learnt dependency model still cannot be explained. The reason is thata large number of weak attributes are weak recognizing semantic meanings. This is notsurprising, because learning classiers for semantically meaningful attributes remainsdicult. Fortunately, the proposed weak attribute approach does not require weakattributes to be semantically plausible.4 a-TRECVIDThe last evaluation is done on a-TRECVID [11]. It contains 126 uniquely labeledquery attributes, and 6,000 weak attributes, of 0.26 million images (video frames). Theindividual attribute classiers are trained using bag-of-words SIFT features under thespatial pyramid conguration [6]. Following the setting of Section 3, weak attributesinclude individual classier scores, Classemes, discriminative attributes, distance to6Figure 5: Retrieval performance on a-TRECVID dataset, with the varying training size. Thisgure is copied from [11]. From left to right: performance of single, double and triple attributequeries.randomly selected images, and latent variables. Dierent from a-PASCAL dataset, wetreat images from the same video as belonging to the same category. Figure 5 showsthe performance of our approach comparing to individual classiers and MARR.From Table 1, we have found that about 80% of the weights are from the weakattributes (excluding individual query attribute classier scores). This clearly demon-strates the contribution of weak attributes. The analysis of dierent types of weakattributes follows Section 3. It is interesting to notice that discriminative attributesdo not contribute much for a-TRECVID. The possible reason is that weak attributestrained by the specic way are not directly related to the query attributes.5 ConclusionWe evaluate performance of weak attributes based retrieval on LFW, a-PASCAL/a-Yahoo and a-TRECVID datasets. The learnt dependency model is visualized on LFW.We also analyze the contributions of heterogeneous types of weak attributes. The ex-periments verify the unique advantages of using weak attributes in the image retrievalframework [11]. It is encouraging to nd that weak but semantically meaningful clas-siers, e.g. Classemes [10], make strong contributions in our experiments.Acknowledgement We would like to thank Dr. Michele Merler for sharing the LFWlabels, Dr. Behjat Siddiquie and Dr. Neeraj Kumar for helpful discussions.7References[1] H. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by theirattributes. In CVPR, 2009. 1, 3, 4[2] M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid. Discriminative metriclearning in nearest neighbor models for image auto-annotation. In ICCV, 2009.2, 4[3] G. Huang, M. Ramesh, T. Berg, and E. Learned-Miller. Labeled faces in thewild: A database for studying face recognition in unconstrained environments.University of Massachusetts, Amherst, Technical Report 07-49, 2007. 1[4] N. Kumar, A. Berg, P. Belhumeur, and S. Nayar. A search engine for largecollections of images with faces. In ECCV, 2008. 1[5] N. Kumar, A. Berg, P. Belhumeur, and S. Nayar. Attribute and simile classiersfor face verication. In CVPR, 2009. 2, 3[6] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramidmatching for recognizing natural scene categories. In CVPR, 2006. 6[7] A. Natsev, A. Haubold, J. Tesic, L. Xie, and R. Yan. Semantic concept-basedquery expansion and re-ranking for multimedia retrieval. In ACM Multimedia,2007. 1[8] J. Petterson and T. Caetano. Reverse multi-label learning. In NIPS, 2010. 2, 4[9] B. Siddiquie, R. Feris, and L. Davis. Image ranking and retrieval based on multi-attribute queries. In CVPR, 2011. 1, 3, 4[10] L. Torresani, M. Szummer, and A. Fitzgibbon. Ecient object category recogni-tion using classemes. In ECCV, 2010. 4, 5, 7[11] F. Yu, R. Ji, M.-H. Tsai, G. Ye, and S.-F. Chang. Weak attributes for large-scaleimage retrieval. In CVPR, 2012. 1, 3, 4, 5, 6, 78

Experiments of Image Retrieval Using Weak Attributes

https://academiccommons.columbia.edu/doi/10.7916/D8XW4SW2/download

Experiments of Image Retrieval Using Weak Attributes

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons