Identifying Regions of Interest (ROIs) in images has been shown an effective way to enhance the performance of Content Based Image Retrieval (CBIR). Most existing ROI identification methods are based on salience detection, and the identified ROIs may not be the regions that users are really interested in. While manual selection of ROIs can directly reflect users’ interests, it puts extra cognitive overhead to users. To alleviate these limitations, in this paper, we propose a novel eye-tracking based method to detect ROIs for CBIR, in an unobtrusive way. Experimental results have demonstrated that our model performed effectively compared with various state of the art methods

Chen, Yongqiang

Hou, Yuexian

Ren, Qingtao

Song, Dawei

Zhang, Peng

English

Open Research Online (The Open University)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsDetecting regions of interest using eye tracking forCBIRConference or Workshop ItemHow to cite:Ren, Qingtao; Chen, Yongqiang; Zhang, Peng; Song, Dawei and Hou, Yuexian (2015). Detecting regions ofinterest using eye tracking for CBIR. In: SIGIR2015 Workshop on Neuro-Physiological Methods in IR Research, 13Aug 2015, Santiago, Chile.For guidance on citations see FAQs.c© 2015 The AuthorsVersion: Version of RecordLink(s) to article on publisher’s website:https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxuZXVyb2lyMjAxNXxneDo1NWQ3NGJkMjNjYTU2Y2ZlCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukDetecting Regions of Interest using Eye Tracking for CBIRQingtao Ren1, Yongqiang Chen1, Peng Zhang1, Dawei Song1,2, Yuexian Hou1,1Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, China2Department of Computing and Communications, The Open University, United Kingdom{qtren, cyq, pzhang}@tju.edu.cn, dawei.song2010@gmail.com, yxhou@tju.edu.cnABSTRACTIdentifying Regions of Interest (ROIs) in images has beenshown an effective way to enhance the performance of Con-tent Based Image Retrieval (CBIR). Most existing ROI i-dentification methods are based on salience detection, andthe identified ROIs may not be the regions that users arereally interested in. While manual selection of ROIs candirectly reflect users’ interests, it puts extra cognitive over-head to users. To alleviate these limitations, in this pa-per, we propose a novel eye-tracking based method to de-tect ROIs for CBIR, in an unobtrusive way. Experimentalresults have demonstrated that our model performed effec-tively compared with various state of the art methods.Keywords: CBIR, Eye Tracking, ROI1. INTRODUCTIONIdentifying Regions of Interest (ROIs) to improve Content-Based Image Retrieval (CBIR) has recently drawn increasingattention [5]. Salience detection is a process of simulatinghuman’s detection of fixation points that a user would focuson at the first glance of an image by using various algo-rithms. Most existing ROI identification methods focus onsalience detection and simply assume the detected ROIs aretruly of users’ interest. Such approach, however, neglectsthe fact that the salient regions of an image may not neces-sarily be what users are really interested in. Allowing usersto manually select ROIs could solve this problems to someextent, but it brings a substantial cognitive burden to users.Recently, advanced eye tracking technologies have beenapplied in various fields such as human computer interac-tion, for its convenience of capturing, in an unobtrusive andnatural way, users’ eye movements that largely reflect theusers’ interests. Eye tracking in IR has been studied to mod-el user’s implicit relevance feedback and to improve searchexperience. It has been shown by Rayner et al. [3] thatdifferent eye movement behaviors reveal different levels ofuser engagement during document reading. Cole et al. [2]investigated the relationships between eye movement pat-Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. To copy otherwise, or republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee.(C) 2015 Copyright is held by authors.NeuroIR’15, August 13, 2015, Santiago, Chile..terns and different tasks, and utilized eye tracking data toexamine users’ information acquisition strategies. To ourbest knowledge, most existing studies focus on text retrievalrather than CBIR. There is also a lack of study on integera-tion of eye-tracking data into realtime retrieval models.In this paper, we propose to combine image segmentationand eye tracking to automatically detect ROIs, which arethen used to improve CBIR. Our hypothesis is that userstend to spend more time fixating on particular parts of theimages they are interested in. This is consistent with variouswidely used assumptions in implicit relevance feedback, e.g.,the SAT criteria (i.e., the documents clicked and viewed forabove a certain amount of time are assumed relevant).2. METHODOLOGYFormally, an image consists of N segmented regions:R = {r1, r2, . . . , rN} (1)where ri is the ith segmented region. Eye tracking data col-lected when users interact with the images, particularly thefixations on segmented regions, are used to identify ROIs. Afixation refers to a time of more than 250ms duration whenthe user is gazing at one point [1]. We select fixation dura-tion as the main feature, by assuming a longer fixation canbetter reflect a user’s interest. For each region ri, an impor-tance value Ci is defined based on the relative gaze durationdi on ri. Ci reflects the degree of the user’s interest on theregion ri, and can be calculated as Ci = di/∑Ni=1 di where∑Ni=1 Ci = 1. Its value will be 0 if there is no fixation onthe region.Consequently, an image is represented by a set of regions,some of which may be identified as ROIs. The color andtexture features are extracted to represent regions. Specif-ically, a feature set including an 11x11x11 HSV color his-togram and a 1x41 texture histogram of the edge map, isused to characterize a region. For similarity measuremen-t between two images, the Euclidean distance between twofeature vectors, denoted as dis(·, ·), is measured, and thenthe similarity between two images is defined as the minimumdistance of regions. This allows retrieval of similar imagesat a finer semantic level according to the weights of differentROIs. Given a query image q, the ranking score Score(q, d)for an image d can be computed as follows:Score(q, d) = mini∈[1,|Rq|],j∈[1,|Rd|]{dis((1+Cqi )rqi , (1+Cdj )rdi )}(2)0 20 40 60 80 1000.050.10.150.20.250.30.350.4The number of return imageAverage semantic similarityAsian architecture  Eye tracking basedAttention basedAttention object basedGlobal based(a)0 20 40 60 80 1000.050.10.150.20.250.3The number of return imageAverage semantic similarityVehicle  Eye tracking basedAttention basedAttention object basedGlobal based(b)0 20 40 60 80 10000.050.10.150.20.25The number of return imageAverage semantic similarityPeople  Eye tracking basedAttention basedAttention object basedGlobal based(c)0 20 40 60 80 10000.050.10.150.20.250.30.35The number of return imageAverage semantic similarityAnimals  Eye tracking basedAttention basedAttention object basedGlobal based(d)Figure 1: Average semantic similarity scores along with the number of returned imageswhere Cqi and Cdj are the importance values of the segmentedregions rqi and rdj in q and d, respectively. |Rq| and |Rd|) arethe numbers of segmented regions in q and d respectively.2.1 Experimental SetupOur experiments are conducted on the public Hemmer col-or image database, which contains more than 7000 imageswith 8 categories, i.e., people, Asian architecture, animals,landscapes, tools, vehicles, fruits, and arts. All images havea resolution of 1920*1200 pixels in the 32-bit color mode. Inthis paper, we select 4 categories (Asian architecture, vehi-cle, people and animals), each of which contains 100 images.Each image in a category will be issued into the system asa query example to search over the whole collection.In order to test the eye tracking based retrieval method,we first need to collect users’ eye movement data and iden-tify ROIs. This was done through a controlled task-baseduser study. We adopted a Tobii TX300 eye tracker in auser-friendly environment with a high accuracy (0.5 degree)at 300Hz sample rate. The freedom of head movement is30x22x30 cm. Fixation is a widely used eye movement fea-ture, which can be easily extracted by the Tobii Studio soft-ware. We asked users to complete 4 different retrieval tasksdesigned according to the 4 selected categories. Taking the“vehicle” category as an example, we requested users to gothrough all the 100 images to find vehicles and used the eyetracker to record all the fixations falling into different re-gions to identify ROIs. The more fixations a ROI contains,the higher weight (i.e., Ci) of the ROI will have. Once theROIs are identified, they will be used to in the similaritymeasurement as described in the previous section.We evaluate the retrieval performance by computing theaverage semantic similarity over all search rounds. Each im-age in the dataset has 20-30 keywords on average to describewhat objects it contains. We define the semantic similari-ty S between a query image and a retrieved image as theratio of matched keywords between their keyword descrip-tions: S =2PM + N, where M and N are the numbers ofkeywords that the two images contain respectively, and P isthe number of matched keywords.Three baselines for comparison are: 1) Global featurebased approach: retrieval based on the Euclidean distanceof the global color and texture histograms of two images; 2)Attention based approach: retrieval based on the manuallyextracted salience objects (as ROIs) [4]; 3) Attention objectbased approach: Instead of using all the objects extractedin 2), Wang et al. used the first popped-out object only inthe attention-driven image retrieval strategy [4].2.2 Result AnalysisThe performance of our proposed image retrieval modeland the three baselines on different categories of query im-ages are shown in Figure 2. We can see that for “AsianArchitecture” and “People” image classes, our eye-trackingbased method performs better than the others, in terms ofaverage semantic similarity. For the other two image class-es “Vehicle” and “Animals”, our method is better when thenumber of returned images remains small (20 or less). Theresults demonstrate the robustness of our method in achiev-ing better retrieval performance.3. CONCLUSIONSIn this paper, we have presented a novel fixation-basedmethod to detect regions of interest (ROIs) for content-based image retrieval. Compared with traditional methods,our proposed eye tracking based method generates encour-aging retrieval performance without causing extra burden tousers during the search process. In the future, we will val-idate our method in larger datasets and take into accountmore eye movement features such as pupil diameters, regres-sions and saccade trajectories.4. ACKNOWLEDGEMENTThis work is funded in part by the Chinese National Pro-gram on Key Basic Research Project (973 Program, grantno. 2013CB329304 and 2014CB744604), the Chinese 863Program (grant no. 2015AA015403), and the Natural Sci-ence Foundation of China (grant no. 61272265 and 61402324).5. REFERENCES[1] G. Buscher, A. Dengel, R. Biedert, and L. V. Elst. Attentivedocuments: Eye tracking as implicit feedback forinformation retrieval and beyond. ACM Transactions onInteractive Intelligent Systems (TiiS), 1(2):9, 2012.[2] M. J. Cole, J. Gwizdka, C. Liu, R. Bierig, N. J. Belkin, andX. Zhang. Task and user effects on reading patterns ininformation search. Interacting with Computers,23(4):346–362, 2011.[3] K. Rayner. Eye movements in reading and informationprocessing: 20 years of research. Psychological bulletin,124(3):372, 1998.[4] W. Xiangyang, H. Fengli, and Y. Hongying. A novelregions-of-interest based image retrieval using multiplefeatures. In Multi-Media Modelling Conference Proceedings,2006 12th International, pages 4–pp. IEEE, 2006.[5] Y. Zhang, H. Fu, Z. Liang, Z. Chi, and D. Feng. Eyemovement as an interaction mechanism for relevancefeedback in a content-based image retrieval system. InProceedings of the 2010 Symposium on Eye-TrackingResearch & Applications, pages 37–40. ACM, 2010.

Detecting regions of interest using eye tracking for CBIR

Qingtao Ren

Yongqiang Chen

Peng Zhang

Dawei Song

Yuexian Hou

CiteSeerX

Detecting Regions of Interest using Eye Tracking for CBIR

http://oro.open.ac.uk/44134/1/08-Detecting%20Regions%20of%20Interest%20using%20Eye%20Tracking%20for%20CBIR.pdf

Detecting regions of interest using eye tracking for CBIR

Abstract

Similar works

Full text

Available Versions

Open Research Online (The Open University)

CiteSeerX