Spatiograms were introduced as a generalisation of the commonly used histogram, providing the flexibility of adding spatial context information to the feature distribution information of a histogram. The originally proposed spatiogram comparison measure has significant disadvantages that we detail here. We propose an improved measure based on deriving the Bhattacharyya coefficient for an infinite number of spatial-feature bins. Its advantages over the previous measure and over histogram-based matching are demonstrated in object tracking scenarios

O'Connor, Noel E.

Smeaton, Alan F.

Ó Conaire, Ciarán

English

Name not available

AN IMPROVED SPATIOGRAM SIMILARITY MEASUREFOR ROBUST OBJECT LOCALISATIONC. O´ Conaire, N. E. O’Connor, A. F. SmeatonAdaptive Information Cluster, Dublin City University, IrelandABSTRACTSpatiograms were introduced as a generalisation of the com-monly used histogram, providing the ?exibility of adding spa-tial context information to the feature distribution informationof a histogram. The originally proposed spatiogram com-parison measure has signi?cant disadvantages that we detailhere. We propose an improved measure based on deriving theBhattacharyya coef?cient for an in?nite number of spatial-feature bins. Its advantages over the previous measure andover histogram-based matching are demonstrated in objecttracking scenarios.Index Terms— Image matching, Image pattern recogni-tion, Tracking1. INTRODUCTIONHistograms have been widely used as descriptors for imageregions. All spatial information is discarded and only featureoccurrence counts are retained. Spatial information may beimportant for some applications, such as object tracking, anda descriptor that retains some spatial information is desirable.Spatiograms [1] are one such descriptor. As well as objecttracking, they have also been shown to be useful in imageclustering and data fusion. In this paper, we derive a new spa-tiogram similarity measure that is superior to the previouslyproposed measure, and provide theoretical and practical ex-amples to justify its use.This paper is organised as follows: In section 2, we reviewrelated work on spatiograms and other feature-spatial distri-bution models; section 3 contains a brief overview of spa-tiograms and the drawbacks of using the original similaritymeasure; we derive our new measure in section 4; in section5, we demonstrate experimentally the advantages of our newmeasure over existing measures and give our conclusions insection 6.This material is based on works supported by Science Foundation Ire-land under Grant No. 03/IN.3/I361 and sponsored by a scholarship from theIrish Research Council for Science, Engineering and Technology (IRCSET):Funded by the National Development Plan. The authors would also like toexpress their gratitude to Mitsubishi Electric Research Labs (MERL) for theircontribution to this work.2. RELATED RESEARCHMany descriptors have been proposed for object modellingin order to provide robust tracking in video sequences. His-tograms [2, 3, 4] have commonly been used for tracking, asthey discard spatial information and are therefore insensitiveto small object pose changes. At the other extreme, imagetemplates [5], which impose rigid spatial contraints on fea-ture layouts, have also been used. In [6], Elgammal et al. pro-vide a parametric feature-spatial distribution model for ob-ject modelling. A spatial kernel bandwidth parameter mustbe supplied, to tradeoff between rigid and no spatial con-straints. Spatiograms [1] generalise the histogram to allowhigher-order spatial moments to be part of the descriptor. Spa-tiograms have been shown to be useful in head tracking [1],in fusing features of different modalities for multi-spectral ob-ject tracking [7] and in image clustering for retrieval [8].3. REVIEW OF SPATIOGRAMS3.1. Spatiogram modelsAs well as histograms of feature occurrences, spatiogramsalso capture higher-order spatial moments. A 2nd-order spa-tiogram model of an object is identical to a histogram of itsfeatures, except that it also stores additional spatial informa-tion, namely the mean and covariance of the spatial positionof all pixels that fall into each histogram bin.To compute a normalised histogram for a target image ofN pixels, we write the histogram bin count, nb, of bin b as:nb = CN∑i=1δib (1)where δib = 1 if the ith pixel falls in the bth bin and δib = 0otherwise. The normalising constant, C, ensures than all binssum to one (i.e.∑Bi=1 ni = 1). For 2nd-order spatiograms,the spatial mean, μb, and covariance, Σb, also need to be com-puted for each bin:μb =1∑Nj=1 δjbN∑i=1xiδib (2)I ­ 10691­4244­0728­1/07/$20.00 ©2007 IEEE ICASSP 2007Σb =1∑Nj=1 δjbN∑i=1(xi − μb)(xi − μb)T δib (3)where xi = [x,y]T is the spatial position of pixel i. To com-pare regions of different sizes, it is necessary to map the spa-tial coordinates to the same scale; for example, by mappingcoordinates into the [−1,+1] range. Additionally, in order toensure that each Σb is invertible, we force them to be diago-nal, and set a minimum variance value to one pixel.3.2. Comparing SpatiogramsTo compare two spatiograms, S = {n, μ,Σ} and S′ ={n′ , μ′ ,Σ′}, each with B bins, the following similarity mea-sure is used in [1]:ρ(S, S′) =B∑b=1ψb√nbn′b (4)where ψb is the spatial similarity measure, given by:ψb = ηexp{−12(μb − μ′b)T Σˆ−1b (μb − μ′b)}(5)where η is the Gaussian normalisation term, and Σˆ−1b = (Σ−1b+ (Σ′b)−1), so that the distance between the spatial means isnormalised to the average of the two Mahalanobis distances.We will term this similarity measure the original measure.3.3. Disadvantages of the original measureThe original similarity measure has two main disadvantages.Firstly, it is not tolerant of small spatial changes of the fea-tures. This is clear from the Σˆb term, whose value is lessthan either of the individual variances. This means that smallspatial changes are heavily punished by the original measure.Secondly, a good similarity measure should have the prop-erty that if an image region spatiogram is compared to itself,the measure should return its maximal value. This is not trueof the original measure. With N(x;μ,Σ) representing a nor-malised Gaussian evaluated at x, we write the similarity be-tween a spatiogram and itself:ρ(S, S) =B∑b=1√n2b[N(μb;μb, (2Σ−1b )−1)]=B∑b=1nb2π|(1/2)Σb|1/2 =B∑b=1nbπ|Σb|1/2This shows that comparing a spatiogram to itself does notequal a constant. Indeed, with the original similarity measure,it is possible for a patch which is different from the targetpatch to be a better match to the target, than the target itself!This is because the normalisation constant of the original sim-ilarity measure adds more weight to spatially-tighter featureclusters. This leads to a non-smooth similarity surface, whichoften has many spiked peaks. We overcome this dif?culty byderiving a new similarity measure in the next section.4. PROPOSED MEASURETo derive our new similarity measure, we leverage the factthat we can convert the 2nd order spatiogram back to a his-togram, adding an extra dimension of space. For bin b, wedivide its contents, nb, over an in?nite number of spatial bins,nb,k, where k is an integer ranging from −∞ to +∞. Weexpress this as:nb,k =nbφb(kΔw)Δw∑+∞i=−∞ φb(iΔw)Δw(6)where Δw is the spatial size of each bin and φb is a nor-malised Gaussian with the mean and covariance of bin b. Sincewe can now (theoretically) create a histogram from any spa-tiogram, we can now compare spatiograms using the Bhat-tacharyya coef?cient [2]. This has a relationship with theprobability of Bayes error [9], and therefore is more similar toa probability than equation (4). Given two spatigrams (con-verted to histograms), nb,k and n′b,k, we compare them usingthe Bhattacharyya coef?cient as follows:ρ(n, n′) =B∑b=1+∞∑k=−∞√nb,kn′b,k=B∑b=1+∞∑k=−∞√√√√( nbφb(kΔw)Δw∑+∞i=−∞ φb(iΔw)Δw)√√√√( n′bφ′b(kΔw)Δw∑+∞i=−∞ φ′b(iΔw)Δw)As Δw → 0, the denominators of both fractions disappear,since φb and φ′b are both normalised Gaussians, therefore:+∞∑i=−∞φb(iΔw)Δw ≈∫ +∞−∞φb(x) dx = 1This gives:ρ(n, n′) =B∑b=1√nbn′b∫ +∞−∞√φb(x)φ′b(x) dx (7)This can be simpli?ed further by noting that the product oftwo Gaussians is Gaussian, and also that the square-root of aGaussian is Gaussian. The resulting Gaussians are not nec-essarily normalised, however, and therefore do not usuallyintegrate to one. We are left with constant terms which canbe thought of as weights for each bin comparison. Given that√N(x; a,A) = qN(x; a, 2A), with q = 2(2π)m/4|A|1/4I ­ 1070for m dimensions, and N(x; a,A)N(x; b,B) = zN(x; c, C)with z = N(a; b, A + B), we can simplify equation (7) toproduce our compact new measure:ρ(n, n′) =B∑b=1√nbn′b∫ +∞−∞√zbφˆb(x) dx=B∑b=1√nbn′b√zb∫ +∞−∞√φˆb(x) dx=B∑b=1√nbn′b√zb∫ +∞−∞qbˆˆφb(x) dx=B∑b=1√nbn′b [qb√zb]=B∑b=1√nbn′b[qb√N(μb;μ′b,Σb + Σ′b)]=B∑b=1√nbn′b[qbQbN(μb;μ′b, 2(Σb + Σ′b))]where φˆb andˆˆφb are normalised Gaussians. Terms qb and Qbare given byqb = 2√2π∣∣∣Σb + Σ′b∣∣∣1/4Qb = 2√2π∣∣∣(Σ−1b + (Σ′b)−1)−1∣∣∣1/4Noting that qbQb = 8π|ΣbΣ′b|1/4, our ?nal similarity mea-sure becomes:ρ =B∑b=1√nbn′b[8π|ΣbΣ′b|1/4N(μb;μ′b, 2(Σb + Σ′b))](8)4.1. Analysis of the new measureFirstly, comparing equations (4) and (8), of the old and newmeasures respectively, it is clear that the new measure is moretolerant of small spatial changes, as the covariance is equalto twice the sum of individual covariances. Secondly, If wecompare two identical spatiograms using the new measure,we obtain:ρ =B∑b=1√n2b8π|Σb|1/2N(μb;μb, 4Σb)=B∑b=18πnb|Σb|1/22π|4Σb|(1/2) =B∑b=1nb = 1This shows that any spatiogram compared to itself will alwaysreceive a similarity score of 1 using the new measure, whichis its maximal value.5. EXPERIMENTAL RESULTSThe plot in ?gure 1 shows the effect of noise on our pro-posed spatiogram similarity measure, compared to the orig-inal measure and to histogram similarity. The original simi-larity score is normalised by dividing by its maximum valueso that it reaches a value of one at zero noise. The image weused (top right corner of ?gure 1) was selected as it containedpixel clusters with different variances. Adding Gaussian noisecauses a sharp decrease in the similarity score of the originalmeasure, due to its intolerance of small spatial changes. His-togram based matching is quite insensitive to noise and evenreturns a relatively high matching score when the original im-age is lost in the noise signal. Our proposed measure has alinear response within a large window of added noise.Figure 2 shows the similarity surfaces generated by (c)theoriginal measure, (d)histogram matching and (e)our proposedmeasure for two tracking examples, corresponding to local-ising a rigid highly-textured object (a book) and a non-rigidhuman object (a football player). In the top row of ?g 2(c)the original measure produces a spiked similarity surface and?nds the best match on the ball, instead of the player. Thisis because it weights bins with low variances higher, such asthe tightly clustered white pixels of the ball. Its surface in ?g2(c), bottom row, is also not very smooth, but it correctly lo-calises the book. Figure 2(d) shows the histogram similaritysurfaces. Since it contains no spatial information, it providespoor object localisation, as can be seen by its poor lock on thebook in ?g 2(b). Our proposed measure’s surfaces, shown in?g 2(e), are smooth, as in histogram matching, but also havegood object localisation, indicated by the tighter peaks.Our ?nal experiment in ?gure 3 shows results from typicalobject-tracking scenarios. For this we use the multi-featurespatiogram-based mean-shift tracking framework describedin our previous work [7]. We compare the original measure toour proposed measure for person tracking (top row) and headtracking (bottom row) using thermal infrared and colour pixelfeatures. The original measure (shown in red) fails quicklywhen the tracked object moves, due to its narrow similaritysurface peak. The mean-shift procedure generally fails whenan object moves faster than the peak width. The proposedmeasure (shown in yellow) successfully tracks both objects.Videos online at: http://www.eeng.dcu.ie/˜oconaire/icassp076. CONCLUSIONSIn this work, we have detailed the limitations of the originalspatiogram comparison measure and have proposed an im-proved measure, derived from the Bhattacharyya coef?cient.Its advantages, both in more accurate object localisation andin giving a smooth similarity surface, were demonstrated the-oretically and on real video data. Future work will exam-ine how to update object spatiogram models during tracking,while minimising model drift.I ­ 1071(a) (b) (c) (d) (e)Fig. 2. (a) Target objects, (b) Search areas with best matches shown: Original measure (blue), Histogram matching (red),Proposed measure (green). Similarity surfaces: (c) Original measure, (d) Histogram matching and (e) Proposed spatiogrammeasureFig. 1. The effects on the similarity measures of addingGaussian noise to a target image (top right): Histogram sim-ilarity (blue), original spatiogram similarity measure (green),proposed measure (red). Added noise RMS shown on x-axis.7. REFERENCES[1] S. T. Birch?eld and S. Rangarajan, “Spatiograms versus histogramsfor region-based tracking,” in IEEE Computer Society Conference onComputer Vision and Pattern Recognition (CVPR), June 2005, pp. 1158–1163.[2] D. Comaniciu, V. Ramesh, and P. Meer, “Real-time tracking of non-rigidobjects using mean shift,” in IEEE Conference on Computer Vision andPattern Recognition (CVPR), 2000, pp. 142–151.[3] R. T. Collins, “Mean-shift blob tracking through scale space,” in IEEEConference on Computer Vision and Pattern Recognition, 2003, vol. 2,pp. 234–240.[4] Z. Zivkovic and B. Krose, “An em-like algorithm for color-histogram-based object tracking,” in IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition (CVPR), 2004.[5] I. Matthews, T. Ishikawa, and S. Baker, “The template update problem,”(a) (b)Fig. 3. Multi-feature mean-shift tracking results using:(in red) Original measure, (in yellow) Proposed Measure.(a)Visible images and (b)Infrared images.IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.26, no. 6, June 2004.[6] A. Elgammal, R. Duraiswami, and L. S. Davis, “Probabilistic trackingin joint feature-spatial spaces,” in IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2003.[7] C. O´ Conaire, N. E. O’Connor, and A. Smeaton, “Thermo-visual featurefusion for object tracking using multiple spatiogram trackers,” MachineVision and Applications (submitted to), 2006.[8] C. O´ Conaire, N. E. O’Connor, A. Smeaton, and G. J. F. Jones, “Organ-ising a daily visual diary using multi-feature clustering,” in Proc. of 19thannual Symposium on Electronic Imaging, 2007.[9] K. Tumer and J. Ghosh, “Estimating the bayes error rate through clas-si?er combining,” in In Proceedings of the International Conference onPattern Recognition, Vienna, 1996, pp. 695–699.I ­ 1072

An improved spatiogram similarity measure for robust object localisation

DCU Online Research Access Service

An em-like algorithm for color-histogrambased object tracking,”

Mean-shift blob tracking through scale space,”

Real-time tracking of non-rigid objects using mean shift,”

Spatiograms versus histograms for region-based tracking,”

The template update problem,”

Crossref

An Improved Spatiogram Similarity Measure for Robust Object Localisation

Spatiograms were introduced as a generalisation of the com-monly used histogram, providing the ƀexibility of adding spa-tial context information to the feature distribution information of a histogram. The originally proposed spatiogram com-parison measure has signiſcant disadvantages that we detail here. We propose an improved measure based on deriving the Bhattacharyya coefſcient for an inſnite number of spatial-feature bins. Its advantages over the previous measure and over histogram-based matching are demonstrated in object tracking scenarios. Index Terms — Image matching, Image pattern recogni-tion, Trackin

An improved spatiogram similarity measure for robust object localisation

Abstract

Similar works

Full text

Available Versions

Name not available

DCU Online Research Access Service

Crossref

CiteSeerX

Irish Universities