Bottom-up visual saliency can be computed through information theoretic models but existing methods face significant computational challenges. Whilst nonparametric methods suffer from the curse of dimensionality problem and are computationally expensive, parametric approaches have the difficulty of determining the shape parameters of the distribution models. This paper makes two contributions to information theoretic based visual saliency models. First, we formulate visual saliency as center surround conditional entropy which gives a direct and intuitive interpretation of the center surround mechanism under the information theoretic framework. Second, and more importantly, we introduce a fast nonparametric multidimensional entropy estimation solution to make information theoretic-based saliency models computationally tractable and practicable in realtime applications. We present experimental results on publicly available eyetracking image databases to demonstrate that the proposed method is competitive to state of the art

Ang, Li-minn

Ngo, A. C.

Qiu, G.

Seng, K. P.

Underwood, G.

English

Research Online @ ECU

Edith Cowan University Research Online ECU Publications 2012 1-1-2012 Visual Saliency Based on Fast Nonparametric Multidimensional Entropy Estimation Ngo G. Qiu G. Underwood Li-minn Ang Edith Cowan University K. P. Seng Follow this and additional works at: https://ro.ecu.edu.au/ecuworks2012  Part of the Engineering Commons 10.1109/ICASSP.2012.6288129 This is an Author's Accepted Manuscript of: Ngo, A., Qiu, G., Underwood, G., Ang, L. K., & Seng, K. (2012). Visual Saliency Based on Fast Nonparametric Multidimensional Entropy Estimation. Proceedings of 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). (pp. 1305-1308). Kyoto, Japan. IEEE. Available here © 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This Conference Proceeding is posted at Research Online. https://ro.ecu.edu.au/ecuworks2012/264 VISUAL SALIENCY BASED ON FAST NONPARAMETRIC MULTIDIMENSIONALENTROPY ESTIMATIONAnh Cat Le Ngo1,2 Guoping Qiu1 Geoff Underwood3 Li-Minn Ang2 Kah Phooi Seng21 School of Computer Science, University of Nottingham, Jubilee Campus, UK2 Faculty of Engineering, University of Nottingham, Malaysia Campus, Malaysia3 School of Psychology, University of Nottingham, Unipark Campus, UKABSTRACTBottom-up visual saliency can be computed through infor-mation theoretic models but existing methods face signifi-cant computational challenges. Whilst nonparametric meth-ods suffer from the curse of dimensionality problem and arecomputationally expensive, parametric approaches have thedifficulty of determining the shape parameters of the distri-bution models. This paper makes two contributions to infor-mation theoretic based visual saliency models. First, we for-mulate visual saliency as center surround conditional entropywhich gives a direct and intuitive interpretation of the centersurround mechanism under the information theoretic frame-work. Second, and more importantly, we introduce a fastnonparametric multidimensional entropy estimation solutionto make information theoretic-based saliency models compu-tationally tractable and practicable in realtime applications.We present experimental results on publicly available eye-tracking image databases to demonstrate that the proposedmethod is competitive to state of the art.Index Terms— visual saliency, conditional entropy, k-d tree, information theory, multidimensional entropy estima-tion.1. INTRODUCTIONIn recent years, there has been increasing interest in the ap-plication of the visual saliency mechanism to visual signalprocessing problems. A predominant theory of computa-tional visual saliency is the center-surround mechanism thatis ubiquitously found in the early stages of biological vision[1]. Center-surround saliency models in the time domain [2],frequency domain[3, 4], and information domain [5, 6, 7]have been proposed. Radically, Judd et al.[8] proposed thatsaliency maps can be learned directly from training samplesby machine learning methods instead of the center-surroundmechanism.Center-surround methods in the information domain arecomputationally most challenging because of the curse of di-mensionality problem. For instance, both the self-information[6] and the mutual information [7] approaches involve esti-mating probability density functions in very high dimensionalspaces with limited samples. A few work-around solutionshave been proposed, by projecting the data onto lower di-mensional spaces through independent component analysis(ICA) [6], discrete cosine transform (DCT) [9], and Walsh-Hadamard Transform (WHT) [10] or by modeling informa-tion quantity as a parametric Generalized Gaussian Distribu-tion (GGD) [7]. However, projection methods are compu-tationally intensive and estimating the shape parameters ofGGD is hard as the authors of [7] have already pointed out.In this paper, the center surround principle of visualsaliency is directly formulated as the conditional entropy ofthe center given its surrounds. A major contribution of thispaper is a fast nonparametric multidimensional entropy esti-mation solution that overcomes the curse of dimensionalityproblem and computational complexity issue of informationdomain visual saliency models thus making information-based saliency models computationally tractable and prac-ticable in real-time applications. We present experimentalresults on two publicly available eye-tracking still imagedatabases [6, 8] to demonstrate the effectiveness of the pro-posed method and compare it with existing techniques.2. SALIENCY BASED ON CENTER SURROUNDCONDITIONAL ENTROPYLet Ic(x, y) be an image patch at location (x, y) and Isr(x, y)its surrounding regions. The conditional entropy of the centergiven its surround can be defined as H(Ic(x, y)|Isr(x, y)) =H(Ic(x, y), Isr(x, y)) − H(Isr(x, y)) or in terms of joint andmarginal probabilitiesH =∑Ic(x,y)∈IIsr(x,y)∈Ip(Ic(x, y), Isr(x, y))logp(Isr(x, y))p(Ic(x, y), Isr(x, y))(1)where H is short for H(Ic(x, y)|Isr(x, y)). The conditionalentropy H(Ic(x, y)|Isr(x, y)) can be understood in a numberof ways. From a coding or information theory’s perspective,it will take H(Ic(x, y), Isr(x, y)) bits to code the center andits surrounds together, but if we knew the surround Isr(x, y)already, we will have gained H(Isr(x, y)) bits of information,and the conditional entropy measures the remaining bits nec-essary for coding the center. From an uncertainty or infor-mativeness point of view, the conditional entropy measuresthe remaining uncertainty of the center once its surrounds areknown, or the amount of information of the center given theknowledge of its surrounds. We can use the conditional en-tropy as a measure of saliency, i.e.S(x, y) = H(Ic(x, y)|Isr(x, y)) (2)The definition of saliency in equation (2) and (1) is consis-tent with a number of definitions in the literature includingself-information [6], surprise [11] and decision theoreticsaliency [7]. The self-information saliency of [6] measuresthe self-information of Ic(x, y) in the context of its surrounds,−log{p(Ic(x, y))}. If Ic(x, y) is a common patch within theimage, then p(Ic(x, y)) is large, −log{p(Ic(x, y))} will besmall, hence the saliency will be small. S(x, y) in (2) hasthe same property, that is, if the center and its surrounds arevery similar, then S(x, y) will be small and vice versa. Thesurprise measure of [11] can be re-written as ( S is short forS(Ic(x, y), Isr(x, y))S =∑Ic(x,y)∈IIsr(x,y)∈Ip(Isr(x, y))logp(Isr(x, y))p(Isr(x, y)|Ic(x, y)) (3)Here, the surrounds Isr(x, y)) can be interpreted as the modelor background information and the center Ic(x, y) as the newobservation data. Again, the surprise measure will be smallwhen the center and surround are similar and large when theyare different. The decision theoretic discriminant saliency of[7] boils down to the computation of mutual information be-tween the center and its surround, while mutual informationand conditional entropy are related as follows.MI(Ic(x, y), Isr(x, y)) = H(Ic(x, y))−H(Ic(x, y)|Isr(x, y))(4)MI(Ic(x, y), Isr(x, y)) is deduced amount of uncertainty forthe center Ic(x, y) if its surrounds Isr(x, y) are known. MIcan be interpreted as how much similarity surround and centerdata has, therefore it is consistent with conditional entropy. Alarge mutual information means significant overlap betweencenter and surround information hence the saliency is small,so is the conditional entropy. All these information measure-ments involve the estimation of probability density functionsin very high dimensional spaces with limited data samples,a very challenging problem. In practice, various simplifica-tion processes have to be used, e.g., [6] employed indepen-dent component analysis (ICA) and [7] assumed a parametricGeneralized Gaussian Distribution (GGD) model. In the nextsection, we introduce a fast non-parametric method.3. FAST NONPARAMETRIC ESTIMATION OFCENTER SURROUND CONDITIONAL ENTROPYVisual data have excessive amount of information, but onlysome attracts attention at early stage. Itti et al.[2] used low-Fig. 1. Medium Band Filter Flow Chartlevel features of intensity, colour and orientation at multi-resolution to build several conspicuity maps and combinethem linearly to form a saliency map. In the discriminantsaliency map approach, Gao et al.[7] extracted and mod-eled band-pass features by Wavelet/Gabor Filters and usedparametric GGD to estimate the mutual information betweenthe center and surround. In this paper, we use mid-bandfrequency features which have been shown to allow thebest prediction of attention globally [12]. Figure 1 showsa step-by-step illustration of mid-band filtering. Firstly, a9/7 Cohen-Daubechies-Feauveau (CDF) wavelet[13] decom-poses an image by three levels. Then, all level 1 componentsand level 3 low-low frequency component are removed. Fi-nally, the remaining components are converted back to time-domain by the inverse of the 9/7 CDF wavelet to form themid-band image. The mid-band image is divided into NxNpatches (8x8 patches are utilized in this paper). The saliencyof each center patch C, is computed as the conditional en-tropy of C given four of its surrounding patches (N, S, W,and E) asS(C) = H(C|(N,S,W,E))= H(C,N,S,W,E)−H(N,S,W,E) (5)Estimating the two joint entropies on the right-hand side of(5) is challenging because of the high dimensionality of thedata. To get round the problem, we take a similar approach as[14] and treat the coordinate locations of the pixels as randomvariables and approximate (5) asS(C) = H(c(x, y), n(x, y), s(x, y), w(x, y), e(x, y))− H(n(x, y), s(x, y), w(x, y), e(x, y)) (6)where c(x, y), n(x, y), s(x, y), w(x, y), e(x, y) are respectivelypixels from the C,N,S,W,E patches at the same referencelocation (x,y). We treat the problem as drawing samples from(x,y) in order to approximate the conditional entropy. Withthe formulation of (6), we can now simplify the problem asestimating the entropies in the 4-D and 5-D spaces with a totalof 8x8 = 64 samples. We use a technique similar to [15] toachieve fast implementation of (6). The technique is based ona k-d tree style approach to partitioning the input data spaceΩ ∈ <D into A = {Aj |j = 1, 2, . . . ,m} with Ai⋂Aj = ∅if i 6= j and ⋃j Aj = Ω. Let nj be the number of samplesin the cell Aj , V(Aj) the volume of cell Aj , the total numberof samples N, then the multidimensional joint entropy can beestimated asHˆ =m∑j=1njNlog(NnjV (Aj))(7)Fig. 2. From left to right: Image Sample, ITT, AIM, ENT andMIT saliency mapsThe computational complexity of the algorithm is Θ (DNlogN)and the space complexity is Θ(DN). For the algorithm towork, the sample size has to satisfy N ≥ 2D. Our setting,N = 64 and D = 5 or D = 4, 25 = 32 and 24 = 16, thereforemeets the samples size requirement of the algorithm.4. EXPERIMENTAL RESULTSWe evaluate the new conditional entropy based saliencymethod (from now on referred to as ENT) on publicly avail-able eye tracking databases of Bruce and Tsotos [6] and Juddet al.[8], and compare it with a number of saliency estima-tion methods in the literature including, Itti and Koch (ITT)[2], spectral residual saliency (SRS) [3], Information Maxi-mization (AIM)[6], and discriminant saliency (DIS) [7]. Fig.2 shows the saliency maps generated by different methodsof a sample image. It is seen that the visual appearance ofthese saliency maps are quite similar. To compare the perfor-mances of different methods quantitatively, we use Tatler’snumeric measurement [16]. Saliency maps are treated asbinary classifiers to discriminate fixation points versus non-fixation points. Threshold for classifying fixation points arenot fixed but systematically changed from minimum to maxi-mum of saliency values to generate ROC curves. ROC curvesof various methods on Bruce’s database [17] are shown inFig. 3. Area under the ROC curves (AUC) have been usedby a number of authors to give quantitative comparison ofsaliency computation methods and table 1 shows the AUCvalues of five different methods.Table 1. Area Under Curve (AUC) for different methods.Methods ITT [2] AIM [6] New ENT DIS[7] SRS[3]AUC 0.70947 0.73873 0.78167 0.76940 0.75434The ROC curves show that the new ENT method gener-ally performs better than AIM and ITT methods, and the per-formances are reconfirmed by the area under curve (AUC)results in table 1. In the table, the AUC result of DIS saliencymethod was performed on the same database by the originalauthors and taken directly from [7]. These AUC results showthat ENT methods also performs better or at least as well asthe DIS method.Fig. 3. ROC of ITT, AIM, ENT and SRS methodsFig. 4. Inter-subject ROC of ITT, AIM, ENT, and SRSROC curves and AUC values are useful for comparingdifferent computational saliency approaches, but they do notshow relationships between these methods and eye trackingdata. Inter-subject ROC curves proposed by Harel et al.[18]help to show performances of human visual system versusthat of computational saliency methods. Fig. 4 shows theInter-subject ROC curves of ENT against a few other meth-ods for the database of [6]. This plot clearly demonstrates thatour new ENT technique has outperformed current state-of-art saliency methods and displayed good matching with eye-tracking data.Table 2. Time Consumption of Saliency MethodsMethods ITT AIM ENT SRSTime (s) 1.2488 66.2673 0.93094 0.33654Information-theoretic saliency methods such as AIM hasdrawbacks due to their intensive computational requirementsand are unsuitable for real-time applications. The proposedmethod can overcome this issue. Table 2 shows the compu-tational speeds of several techniques. It is seen that ENT isover 70 times faster than AIM and 1.3 times faster than ITT.Though it is slower than SRS, ENT can better match eye-fixation data than SRS. All experiments are done in MATLABon a 2.33 GHz Intel Core 2 Duo computer running Linux.The ENT method was further tested on the database cre-ated by Judd et al.[8]. Quantitative results in table 3 onceagain show that our method compares very well against otherstate of the art methods.Table 3. Area Under Curve (AUC)Methods ITT AIM ENT MIT[8]AUC 0.74940 0.71165 0.78157 0.688455. CONCLUDING REMARKSIn this paper, we have formulated center-surround bottom-up visual saliency as conditional entropy and presented a fastnonparametric multidimensional entropy estimation solutionto overcome the inherent curse of dimensionality in conven-tional nonparametric approaches and difficulty of determin-ing shapes of distribution in parametric approaches of infor-mation domain visual saliency models. We have shown thatthe new method is not only computationally efficient but alsoachieves state of the art performances on publicly availableeye tracking databases.6. REFERENCES[1] D.H. Hubel and T.N. Wiesel, “Receptive fields and func-tional architecture into nonstriate visual areas (18 and19) of the cat.,” Journal of neurophysiology, vol. 28, pp.229–89, Mar. 1965.[2] L. Itti, C. Koch, and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis,” IEEETransactions on pattern analysis and machine intelli-gence, vol. 20, no. 11, pp. 1254–1259, 1998.[3] X. Hou and L. Zhang, “Saliency detection: A spectralresidual approach,” in IEEE Conference on ComputerVision and Pattern Recognition (CVPR07). IEEE Com-puter Society. 2007, number 800, pp. 1–8, Citeseer.[4] C.L. Guo, Q. Ma, and L.M. Zhang, “Spatio-temporalSaliency detection using phase spectrum of quaternionfourier transform,” 2008 IEEE Conference on ComputerVision and Pattern Recognition, pp. 1–8, 2008.[5] T. Kadir and M. Brady, “Saliency, scale and image de-scription,” International Journal of Computer Vision,vol. 45, no. 2, pp. 83–105, 2001.[6] N. Bruce and J. Tsotsos, “Saliency based on informa-tion maximization,” Advances in neural informationprocessing systems, vol. 18, pp. 155, 2006.[7] D. Gao, V. Mahadevan, and N. Vasconcelos, “Thediscriminant center-surround hypothesis for bottom-upsaliency,” Advances in Neural Information ProcessingSystems, vol. 20, pp. 1–8, 2007.[8] T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learn-ing to predict where humans look,” 2009 IEEE 12th In-ternational Conference on Computer Vision, pp. 2106–2113, Sept. 2009.[9] G. Qiu, X. Gu, Z. Chen, Q. Chen, and C. Wang, “Aninformation theoretic model of spatiotemporal visualsaliency.,” in Proc. of ICME2007, 2007, pp. 1806–1809.[10] L. Zheng, G. Qiu, J. Huang, and H. Fu, “Salient covari-ance for near-duplicate image and video detection,” inProc. of ICIP2011, 2011, pp. 2585–2588.[11] L. Itti and P. Baldi, “Bayesian surprise attracts humanattention,” Advances in neural information processingsystems, vol. 18, pp. 547, 2006.[12] F. Urban, B. Follet, C. Chamaret, O. Meur, and T. Bac-cino, “Medium Spatial Frequencies, a Strong Predictorof Salience,” Cognitive Computation, Nov. 2010.[13] A. Cohen, I. Daubechies, and J.C. Feauveau, “Biorthog-onal bases of compactly supported wavelets,” Commu-nications on Pure and Applied Mathematics, vol. 45, no.5, pp. 485–560, June 1992.[14] P. Viola and W.M. Wells III, “Alignment by maximiza-tion of mutual information,” in ICCV. 1995, vol. 24,p. 16, Published by the IEEE Computer Society.[15] D. Stowell and M.D. Plumbley, “Fast MultidimensionalEntropy Estimation by k-d Partitioning,” Signal Pro-cessing Letters, IEEE, vol. 16, no. 6, pp. 537–540, 2009.[16] B.W. Tatler, R.J. Baddeley, and I.D. Gilchrist, “Vi-sual correlates of fixation selection: effects of scale andtime.,” Vision research, vol. 45, no. 5, pp. 643–59, Mar.2005.[17] N. Bruce and J.K. Tsotsos, “Saliency , attention , and vi-sual search : An information theoretic approach,” Jour-nal of Vision, vol. 9, pp. 1–24, 2009.[18] J. Harel, C. Koch, and P. Perona, “Graph-based visualsaliency,” Advances in neural information processingsystems, vol. 19, pp. 545, 2007.

Visual Saliency Based on Fast Nonparametric Multidimensional Entropy Estimation

Abstract

Similar works

Full text

Available Versions

Research Online @ ECU

Crossref

Deakin Research Online