Manno-Kovács, Andrea

SZTAKI Publication Repository

Tartalom alapu kepi visszakereses kiemelt iranyhisztogram hasznalataval ?Manno-Kovacs AndreaElosztott Esemenyek Elemzese KutatolaboratoriumMTA SZTAKI, Budapestandrea.manno-kovacs@sztaki.mta.huAbstract. A tartalom alapu kepi visszakereses napjaink kiemeltkutatasi terulete, mivel a rendelkezesre allo digitalis kepi adatmennyisege ugrasszer}uen n}o. A letez}o vazlat (,,sketch") alapukepi visszakeres}o (SBIR) rendszerek a valos felveteleken kevesbehatekonyak, mivel az osszetett hatter informacio miatt a kepi lerokeltorzulnak. A torzulas csokkentesere ez a cikk els}o lepeskent egyel}ofeldolgozast javasol, ami kepes automatikusan kiemelni a hangsulyos(,,salient") kepreszleteket. Ezutan a kiemelt reszeken a ModiedHarris for Edges and Corners (MHEC) kulcspont keres}o altalvisszaadott ponthalmazt alapul veve, a kep lerasara a kulcspontokkoruli iranyinformaciot hasznaljuk fel, feleptve a kepet jellemz}oSalient Orientation Histogram-ot (SOH). A javasolt SBIR rendszer avisszakereses mellett kepes a visszaadott kep szemgentalasara is, akepen lev}o salient objektum korvonalanak megkeresesevel. A modszer arandom internetes kepeket tartalmazo THUR15000 adatbazison kerulttesztelesre, a javasolt eljarast az egyeb szakirodalmi modszerekkelosszehasonltva a visszakeresesi es az objektum detektalasi eredmenyekis igen biztatoak.1 IntroductionContent-aware image retrieval is a very important topic nowadays with theconstantly increasing amount of digital image data. Outline sketches haverecently been shown to be more comfortable for retrieval than a complete image,as sketch based image retrieval (SBIR) expects simpler descriptors resulting infaster comparison and retrieval.Descriptors can be grouped into global and local types. While the formerincludes information of the whole image [1], the latter concentrates only ona small image part [2]. Recently published SBIR systems [3, 4] employs local? Eredeti publikacio: A. Manno-Kovacs: "Content based Image Retrieval usingSalient Orientation Histogram", in Proc. of IEEE International Conference onImage Processing, Phoenix, Arizona, USA, 2016. A munkat a European SpaceAgency (ESA) Hungary Industry Incentive Scheme tamogatta az OWETIS projektkereteben.2 Manno-Kovacs A.features, as global ones are not handling ane variations well, and the fact thatne details of the drawing are often missing.Existing SBIR systems are mainly tested on image databases withoutsignicant background information. However, randomly selected internet imagesoften contain a lot of background data with varying texture and color, whichcan inuence the image descriptors and make the comparisons more challenging.To avoid this, a preprocessing step can help to distinguish between foregroundand background, which increases the importance of saliency detection [4{7].However, the dimension of a salient area description can still be very high, thusfurther reduction is needed. Interest point detectors, like Harris [8] emphasizerelevant structures in the image. Thus, if the local descriptors are calculated atinterest point locations, the extracted salient region information can be reducedwhile retaining their relevance. Modied Harris for Edges and Corners (MHEC)was proposed earlier by the author [9] for ecient image segmentation, andthe method's strong ability for object detection was also shown previously [10],supporting its capability of holding ecient structure and content informationfor image comparisons and retrieval.Orientation as a descriptor has already been introduced in earlier SBIRsystems [1], moreover many improvements of the Histogram of OrientedGradients (HoG) [11] were published over the past years [12], [3]. The originalHoG calculated the histogram for the whole image. Improved adaptations ofHoG for SBIR systems are mostly using Canny edge maps [12, 3] with orientationhistograms calculated on pixels of the Canny edge map [12] or randomized pixels[3]. Following this technique, the background texture may create false edges inthe Canny edge map and the keypoint selection could include background hits.Both of them may cause the distortion of the orientation histogram and reducedretrieval accuracy.In this paper, an improved SBIR with object detection is presented fornatural images. First, the salient region is located by a texture distinctivenessmap [5], followed by an MHEC feature point detection step to extract featuresfrom relevant areas. Moreover, instead of the Canny edge map, an improvededge map is introduced for orientation calculation and object detection. Asalient orientation histogram (SOH) is built using the salient keypoints andthe improved edge map. The proposed method is tested on the THUR15000database [13] for image retrieval and detection purposes.2 Salient keypoint detection2.1 Texture distinctivenessStatistical texture distinctiveness model was introduced in [5] and it useda rotational-invariant neighborhood-based textural representation to learnrepresentative texture atoms for sparse texture model. Statistical texturedistinctiveness measures the uniqueness of the atoms and the relationshipsbetween them, by constructing a weighted graph model. The S(x; y) textureTartalom alapu kepi visszakereses kiemelt irany hisztogrammokal 3(a) (b) (c) (d) (e) (f)Fig. 1. Sample images for illustrating the proposed SBIR system: (a) the original image;(b) S texture distinctiveness map; (c) Rmod improved edge map; (d) MHECS pointset; (e) Canny edge map and (f) the detection result achieved by DHVFC.distinctiveness map (see Fig. 1(b)) quanties the expected relative distinctivenessof each texture, incorporating high-level aspects, such as spatial location ofregions relative to the image center. For detailed description, see [5].The S map is thresholded with Otsu's method [14] to extract highlydistinctive image parts. By selecting the region of maximal area, an automaticSROI is initialized. In Figure 1(b) the SROI areas can be well identied in bothsamples.2.2 MHEC interest point set and improved edge mapIf the aim is to emphasize object contours, the Modied Harris for Edges andCorners (MHEC) detector is shown to be an ecient tool [15, 10]. The methodadapts the Harris matrix and uses its 1 and 2 eigenvalues in the followingmodication of the R characteristic function [15]:Rmod = max(1; 2): (1)The introduced modication has the ability to emphasize edge and cornerregions in a balanced manner. MHEC interest points (pi) are selected as thelocal maxima of the Rmod. The point set is able to represent points-of-interestinside the salient SROI region, therefore, the importance of the extracted pixelsis two-fold: while the S ensures the distinctiveness of the selected texture;MHEC points represent potential object contours inside the SROI area. Featuresextracted for the MHEC points are able to describe the corresponding objectmore eciently. The point subset inside the SROI area is marked by MHECS ,samples are given in Fig. 1(d).Besides the MHEC point set, the calculated Rmod map emphasizes theobject contours, therefore it can be applied for gradient calculation and for theextraction of the orientation histogram. By using the Rmod map, a more specic4 Manno-Kovacs A.contour map is obtained, than other traditional edge maps (e.g. Canny) used inprevious SBIR systems [3].The examples of Figure 1 illustrate the main contribution of the paper: whilethe Canny edge maps in earlier SBIR systems often include false edges which canseverely distort the orientation histograms (Fig. 1(e)), the improved edge map(Fig. 1(c)) together with the salient point set (Fig. 1(d)) is able to sample themost relevant pixels of the image and extract essential orientation information.For earlier methods, the parallel edges of the background in the buttery imageand the presence of other objects in the dog image may inuence the orientationstatistics, leading to distorted histograms.3 Image retrieval and segmentationThe members of the MHECS point set include the relevant feature points ofthe most distinctive image area. Therefore, the extracted salient orientationhistogram (SOH) for the MHECS point set can give an improved objectdescription. To obtain the SOH, the local gradient orientation density [16] isanalyzed around a feature point to nd the main direction. Instead of histogrambinning [11], the exact main direction is calculated for the members of thepoint set and each count is also weighted by the gradient magnitude at thepixel. Let the gradient vector be denoted by rgRi with krgRi k magnitude and'ri orientation for the ith point, where R upper index refers to the gradientcalculation on the Rmod improved edge map. Then, if Wn(i) denes the n  nneighborhood of the point (where n depends on the resolution), the weighteddensity of 'ri is as follows:i(') =1NiXr2Wn(i)1h krgRr k  k'  'rrh; (2)with Ni =Pr2Wn(i) krgRr k and k(:) kernel function with h bandwidthparameter. From this orientation histogram, the main orientation for ith featurepoint is dened as: 'i = argmax'2[ 90;+90] fig.After calculating the dominant direction for all K feature points, the salientorientation histogram (SOH) function # is:#(') =1KKXi=1Hi('); (3)where Hi(') is a logical function: 1, if 'i = '; 0 otherwise.If the SOH is calculated for both the query object and the members ofthe test database (see Fig. 2), the distance of the computed SOHs has to bemeasured. The detailed study of [12] showed that for orientation histograms, the2 metric gives the best results. However, since then the Quadratic-Chi (QC)histogram distance family was published [17], introducing an improved techniqueTartalom alapu kepi visszakereses kiemelt irany hisztogrammokal 5Query image QC = 0.3571 QC = 1.0819Fig. 2. Image comparison based on #(') SOH function (Eq. 3) calculated for keypointsmarked by red; and the QC distance metric (Eq. 4). Smaller QC value indicate moresimilar image to the query image.for histogram comparison. The QC histogram distance for P and Q histogramsis dened as:QCAm(P;Q) =vuutXijPi  Qi(Pc(Pc +Qc)Aci)m(Pj  Qj)(Pc(Pc +Qc)Acj)mAij ;(4)with an A non-negative symmetric bounded bin-similarity matrix and an mnormalization factor.The cross-bin 2-like normalization of the QC method reduces the eect oflarge bins and it is less sensitive to light changes and shape deformations than2. The retrieval tests of the author have also conrmed QC's superiority over2 metric, thus QC was applied in the evaluation process. After calculating theQC distance metric for each member of the database, the closer the distance,the more relevant image in the database, which is shown in Figure 2.Another advantage of the improved Rmod map (Eq. 1) is its applicability inthe external force of parametric active contours for object contour detection,shown in the previous work of the author [10]. It can be applied for segmentingthe retrieved images in an automatic system. As a novelty, this paper proposesto integrate the texture distinctiveness information (SROI) through the MHECSpoint set to achieve an automatic initialization step. As a next step, the activecontour is initialized as the convex hull of the MHECS point set. Finally, applyingthe Directional Harris based Vector Field Convolution (DHVFC) [10] parametricactive contour method, an iterative detection is performed (see Fig. 1(f)).6 Manno-Kovacs A.Fig. 3. Best retrieval results of dierent methods for keyword "buttery" (Flickr) andquery image in the second row (SHOG [3] and proposed SOH). First row: Flickr; secondrow: query image and SHOG; third row: the proposed SOH.TPR(%)Buttery Coee mug Dog jump Girae Plane AverageT50 T100 T50 T100 T50 T100 T50 T100 T50 T100 T50 T100Flickr 28 28 58 51 56 55 30 25 44 48 41.4 43.2SHoG [3] 36 40 82 78 74 73 18 18 90 91 60.0 60.0Proposed 48 50 82 82 84 87 54 59 54 64 64.4 68.4Table 1. True positive retrievals for THUR15000 keywords.4 ExperimentsThe THUR15000 database was introduced in [13] as the largest datasetfor content based image retrieval using randomly selected internet images,downloaded from Flickr for 5 keywords: "buttery", "coee mug", "dog jump","girae" and "plane". The database also contains salient regions marked at pixelaccuracy, for images where such region exists. Therefore, it can also be used forevaluating the performance of segmentation methods. In the rst part of theevaluation the database is used to evaluate the SOH descriptor, then the shapebased image segmentation is tested.In the rst part, the proposed SOH based SBIR system is compared to Flickrand the SHOG based SBIR [3] (retrieval results are given in [13]). Table 1 showsthe retrieval results for the best 50 and the best 100 retrieved images for eachcategory. An image is considered as true positive if it contains a target objectspecied by the keywords. According to the results, the proposed SOH-basedretrieval looks promising, the achieved true positive ratio is the highest in almostall categories and the method reached the highest average relevance on the wholeTHUR15000. The best retrieval images for all the three methods are shown inFigure 4.In the second part, the images with pixel accurate ground truth aresegmented, by applying the proposed automatic saliency-based initialization.The segmentation method is compared to other state-of-the-art techniques: FT[7], SEG [6], RC [4] and GS [13]. To weigh precision more than recall (advisedin [7]), the F is calculated with 2 = 0:3:F =(1 + 2)PrecisionRecall2  Precision+Recall : (5)Tartalom alapu kepi visszakereses kiemelt irany hisztogrammokal 7FT SEG RC GS Prop.Aver. F 0.51 0.53 0.56 0.68 0.72Table 2. Average F values of segmentation methods.Figure 4 shows that the proposed automatic detection technique is able toperform at high accuracy, achieving the highest performance in the majorityof the keywords and for the average F (Table 2) on the whole database.(Segmentation samples are presented in Figure 1(f)).Fig. 4. F evaluation results for THUR15K database for dierent segmentationmethods: FT [7], SEG [6], RC [4], GS [13] and the proposed method.5 ConclusionIn this paper, a novel SBIR system is introduced, using a salient keypoint basedorientation histogram (SOH). The proposed method rst extracts the salientimage region based on texture distinctiveness, followed by a Modied Harrisfor Edges and Corners (MHEC) interest point detection. This way the most8 Manno-Kovacs A.relevant pixels of the image are selected to build an orientation histogram onan improved edge map, instead of applying Canny edge map like earlier SBIRsystems. The edge map is also adapted for segmentation. Overall, the proposeddescriptor achieves high performance on the THUR15000 dataset, and it alsoprovides an ecient object detection method. Future work will investigate theimproved integration of saliency in SBIR systems.References1. Chalechale, A., Naghdy, G., Mertins, A.: Sketch-based image matching usingangular partitioning. Systems, Man and Cybernetics, Part A: Systems andHumans, IEEE Transactions on 35 (2005) 28{412. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proc. ofInternational Conf. on Computer Vision. (1999) 1150{11573. Eitz, M., Hildebrand, K., Boubekeur, T., Alexa, M.: Sketch-based image retrieval:Benchmark and bag-of-features descriptors. Visualization and Computer Graphics,IEEE Transactions on 17 (2011) 1624{16364. Cheng, M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.: Global contrast based salientregion detection. Pattern Analysis and Machine Intelligence, IEEE Transactionson 37 (2015) 569{5825. Scharfenberger, C., Wong, A., Fergani, K., Zelek, J.S., Clausi, D., et al.: Statisticaltextural distinctiveness for salient region detection in natural images. In: ComputerVision and Pattern Recognition (CVPR), 2013 IEEE Conference on. (2013)979{9866. Rahtu, E., Kannala, J., Salo, M., Heikkila, J.: Segmenting salient objects fromimages and videos. In: European Conference on Computer Vision. Springer (2010)366{3797. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned SalientRegion Detection. In: IEEE International Conference on Computer Vision andPattern Recognition (CVPR 2009). (2009) 1597 { 16048. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proc. of the4th Alvey Vision Conf. (1988) 147{1519. Kovacs, A., Sziranyi, T.: Improved Harris feature point set for orientation sensitiveurban area detection in aerial images. IEEE Geoscience and Remote SensingLetters (in press) 10 (2013) 796{80010. Manno-Kovacs, A.: Direction selective vector eld convolution for contourdetection. In: IEEE International Conference on Image Processing (ICIP). (2014)4722{4726 "Top10%" award.11. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In:Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE ComputerSociety Conference on. Volume 1. (2005) 886{89312. Hu, R., Barnard, M., Collomosse, J.: Gradient eld descriptor for sketchbased retrieval and localization. In: Image Processing (ICIP), 2010 17th IEEEInternational Conference on, IEEE (2010) 1025{102813. Cheng, M.M., Mitra, N., Huang, X., Hu, S.M.: Salientshape: group saliency inimage collections. The Visual Computer 30 (2014) 443{45314. Otsu, N.: A threshold selection method from gray-level histograms. IEEETransactions on Systems, Man and Cybernetics 9 (1979) 62{66Tartalom alapu kepi visszakereses kiemelt irany hisztogrammokal 915. Kovacs, A., Sziranyi, T.: Harris function based active contour external force forimage segmentation. Pattern Recognition Letters 33 (2012) 1180{118716. Kumar, S., Hebert, M.: Man-made structure detection in natural images usinga causal multiscale random eld. In: Proc. IEEE Conf. Comput. Vision PatternRecogn. (2003) 119{12617. Pele, O., Werman, M.: The quadratic-chi histogram distance family. In:Proceedings of the 11th European Conference on Computer Vision: Part II,Springer-Verlag (2010) 749{762

Tartalom alapú képi visszakeresés kiemelt irány hisztogram használatával

Abstract

Similar works

Full text

Available Versions

SZTAKI Publication Repository