Agrafiotis, D

Bull, DR

Canagarajah, CN

Davies, SJC

English

Crossref

A gaze prediction technique for open signed video content using a track before detect algorithm

This paper proposes a gaze prediction model for open signed video content. A face detection algorithm is used to locate faces across each frame in both profile and frontal orientations. A grid-based likelihood ratio track before detect routine is used to predict the orientation of the signer's head, which allows the gaze location to be localised to either the signer or the inset. The face detections are then used to narrow down the gaze prediction further. The gaze predictor is able to predict the results of an eye tracking study with up to 95% accuracy, and an average accuracy of over 80%.This paper proposes a gaze prediction model for open signed video content. A face detection algorithm is used to locate faces across each frame in both profile and frontal orientations. A grid-based likelihood ratio track before detect routine is used to predict the orientation of the signer's head, which allows the gaze location to be localised to either the signer or the inset. The face detections are then used to narrow down the gaze prediction further. The gaze predictor is able to predict the results of an eye tracking study with up to 95% accuracy, and an average accuracy of over 80

Explore Bristol Research

                          Davies, S. J. C., Agrafiotis, D., Canagarajah, C. N., & Bull, D. R. (2008). Agaze prediction technique for open signed video content using a track beforedetect algorithm. In 15th IEEE International Conference on ImageProcessing, 2008 (ICIP 2008). (pp. 705 - 708). Institute of Electrical andElectronics Engineers (IEEE). 10.1109/ICIP.2008.4711852Link to published version (if available):10.1109/ICIP.2008.4711852Link to publication record in Explore Bristol ResearchPDF-documentUniversity of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only the publishedversion using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/about/ebr-terms.htmlTake down policyExplore Bristol Research is a digital archive and the intention is that deposited content should not beremoved. However, if you believe that this version of the work breaches copyright law please contactopen-access@bristol.ac.uk and include the following information in your message:• Your contact details• Bibliographic details for the item, including a URL• An outline of the nature of the complaintOn receipt of your message the Open Access Team will immediately investigate your claim, make aninitial judgement of the validity of the claim and, where appropriate, withdraw the item in questionfrom public view.A GAZE PREDICTION TECHNIQUE FOR OPEN SIGNED VIDEO CONTENT USING ATRACK BEFORE DETECT ALGORITHMS.J.C. Davies, D. Agraﬁotis, C.N. Canagarajah and D.R. BullDepartment of Electrical and Electronic EngineeringUniversity of Bristol, Bristol, BS8 1UB, UKe-mail: sam.davies@bris.ac.ukABSTRACTThis paper proposes a gaze prediction model for open signedvideo content. A face detection algorithm is used to locatefaces across each frame in both proﬁle and frontal orienta-tions. A grid-based likelihood ratio track before detect rou-tine is used to predict the orientation of the signer’s head,which allows the gaze location to be localised to either thesigner or the inset. The face detections are then used to nar-row down the gaze prediction further. The gaze predictor isable to predict the results of an eye tracking study with up to95% accuracy, and an average accuracy of over 80%.Index Terms— Video contexts, video coding, gaze track-ing1. INTRODUCTIONOpen sign language is the term used to describe video withan in vision signer and is used as an alternative to subtitles,which are often inadequate at conveying concecpts, such asemotion. Figure 1 shows a sample frame from open signedvideo footage.Fig. 1: A sample frame from one of the ‘holby’ open signedvideo sequencesOpen sign language is an excellent candidate for percep-tual video coding due to the well-deﬁned nature of the gazepatterns, as shown in [1], [2], [3] and [4]. Previous work hasbeen done on analysing the gaze patterns of sign language invarious forms, including videoconferencing [4] and open signvideo [2], and both went on to code video with perceptuallyvarying quality. It was shown that within the sign languagecontext it is possible to make coding gains without loss ofintelligibility or perceived quality.In order to improve on these techniques we look at im-proving the gaze prediction routine, speciﬁcally for open signlanugage. Previous work into gaze prediction has often in-volved the concept of saliency [5], where low-level featuresof the video (such as colour contrast, intensity, orientationetc.) are combined into a saliency map, and this is then usedto plot predicted gaze patterns. Due to the bottom-up natureof this scheme it has been shown to be inadequate for manyspeciﬁed contexts, sign language included [2].An eye tracking study was carried out (see Section 2)which shows that the orientation of the signer’s head is an ex-cellent cue as to whether or not the viewers would be lookingat the signer or the inset video. Therefore we ﬁrst propose agrid-based likelihood ratio tracking system to predict the ori-entation of a face (i.e. frontal or proﬁle), before utilising thistracker to construct a gaze prediction routine for open signedvideo. The gaze predictor outputs the most likely locationsof an eye ﬁxation for each frame. These locations can beused for various applications including to modulate a variablequality coder, or add region-based error protection.This paper is organised as follows: Section 2 details aneye tracking study carried out with open sign language mate-rial, including, in Section 2.1, some observations of the col-lected data. Section 3 introduces the proposed gaze predic-tion technique, with the face detector being detailed in Sec-tion 3.1, the track before detect routine in Section 3.2, and thegaze predictor in Section 3.3. Section 4 demonstrates resultsfrom both the tracker and the gaze predictor, before the paperis concluded in Section 5, including a discussion on furtherwork.2. EYE TRACKING STUDYFive participants, each of whom is ﬂuent in British SignLanguage (BSL), were shown a random selection of opensigned clips at standard deﬁnition (720×576, 25fps). Therewere 30 clips in total, sourced from 3 separate programmesoriginally broadcast by the British Broadcasting Corporation(BBC). Eye tracking was carried out using a tobii x50 eyetracker system, with the video displayed on a widescreen705978-1-4244-1764-3/08/$25.00 ©2008 IEEE ICIP 2008Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on October 6, 2009 at 07:42 from IEEE Xplore.  Restrictions apply. plasma television, setup according to the ITU recommendedguidelines [6]. This resulted in an eye gaze location, perparticipant, for every frame.2.1. Eye Tracking Results and AnalysisPrevious work [2] has shown that in open signed materialthere are two especially important regions - the signer’s faceand, to a lesser extent, the inset showing the original video.As a tool to help verify results, the source video frameswere manually classiﬁed, in 2 categories - ‘signer signing?’(yes or no) and ‘signer’s facial orientation’ (frontal, proﬁle orunknown). In over 95% of the 7000 frames classiﬁed, whenthe face was frontal then the signer was signing. This sug-gests that the orientation of the signer’s face is a good cue forwhether or not signing is taking in any given frame.We also investigate whether or not the orientation of thehead is a good measure for whether or not the viewers arelooking at the signer. Equation 1 shows how to calculate theproportion of eye gaze locations which are correctly predictedby the facial orientation of the signer, pcorrect. Here a gaze lo-cation is ‘correct’ if it is on the signer when the face is frontalor is not on the signer when the face is proﬁle.pcorrect =|Fpro ∩ S|+ |Ffro ∩ S||S ∪ S|(1)Ffro and Fpro are the sets of gaze locations when the sign-ers face is frontal and proﬁle respectively, and S is the set ofgaze locations in which the viewer is ﬁxated on the signer.Using the previously described ground truth data, the pro-portion of accurately ‘predicted’ locations is found to be 80%.This implies that a predictor based on the orientation of theface of the signer would yield impressive results for gaze lo-cation prediction. Therefore a model for the orientation of theface of the signer is proposed in Section 3, which uses face de-tections and a tracker in an attempt to discover the orientationof the head of the signer. Although based on the orientationof the signer’s head, this model is ultimately attempting topredict whether a viewer will be ﬁxating on the signer or theinset for a given frame.3. PROPOSED TECHNIQUE3.1. Face DetectionThe proposed gaze detector is based very heavily on face lo-cations in a given frame for predicting the eye gaze position.We utilise the face detector proposed by Viola and Jones [7],which uses a cascade framework and Haar-like features to de-tect faces. The detector works on an individual frame basis,and detects faces of varying sizes based on a set of similarfeatures. The detector is run with both frontal and proﬁle fea-ture cascades, so that 2 sets of detections are made. Eachdetection is speciﬁed by a box, within which a face has beendetected. Figure 2 shows a sample frame and its associatedface detections.(a) Frame from a ‘holby’ sequence, with correctly de-tected faces(b) Frame from a ‘meaning’ sequence, demonstratingan error with the face detectionFig. 2: Output from the face detector with the solid line rep-resenting a frontal detection, and the hashed line a proﬁle de-tectionAlthough the detector can make correct detections (Figure2(a)), it can also make incorrect detections, such as the proﬁledetection on the signer’s face in ﬁgure 2(b). Therefore it is notpossible to simply rely on the output of the face detector. Thismotivates the introduction of a tracker, which will decide howlikely a face of a given orientation is at a given time.3.2. Face TrackerWe use a grid based likelihood ratio tracker [8, pages 10.25 -10.31] in order to detect not only the location of a face, butwhether or not a face of a particular orientation (frontal orproﬁle) exists at a given time. This will result in a surfacewhich represents the likelihood ratio of a face existing at agiven point versus no face being present in the frame at all.We deﬁne states s ∈ S as triples (sx, sy, sd), which can bethought of as multiple grid-planes, each representing a track-ing direction, sd ∈ D. Here we choose |D| to be 4, repre-senting perpendicular directions in the 2-dimensional plane(up, down, left and right). Measurements ξk ∈ Ξ representthe locations of detected faces in frame k as detected by theface detector from Section 3.1, and can be expressed as a co-ordinate pair (ξx,k, ξy,k). The likelihood ratio for frame k iswritten Λk(s), ∀s ∈ S. The likelihood ratio recursion beginswith a motion update, shown in Equation 2.Λ−k (s|sd) = R(Πk(sd), sd) ∗fd|fd|(2)706Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on October 6, 2009 at 07:42 from IEEE Xplore.  Restrictions apply. where ∗ represents a convolution, Λ−k (s|sd) is the motionupdate likelihood ratio for a given direction plane sd andR(H, v) rotates the plane H by vπ2 , v ∈ {0, 1, 2, 3}. Πk(d)is the result of leaking likelihood ratio between directionalplanes, shown in Equation 3:Πk(d) =∑i∈DΛk−1(s|sd = i) · l(i, d) (3)l(i, d) is the leak coefﬁcient from direction plane i to planed. In our implementation we choose a rotationally symmetricfunction such that l(i, d) = 0.5 · δβ,0 + 0.1 · δβ,1, where δi,jis the kronecker delta and β = |i − d| (mod |D|). d, i ∈{0, 1, 2, 3} in this work, since |D| = 4. fd is a vector of ﬁltercoefﬁcients deﬁned in Equation 4:fd(i) ={1 0 ≤ i < cde−τ(i−c) cd ≤ i < n(4)fd(i) is the ith coefﬁcient of the motion prediction ﬁlter oflength n. The ﬁlter is designed such that pixel movementsup to distance cd are equally likely, after which the likelihooddies away exponentially. Values of cd = 3 for horizontaldirection planes, and cd = 1 for the vertical planes were used,along with a decay factor of τ = 0.25.The next step of the recursion involves calculating thelikelihood ratio Lk(ξ|sx, sy) for frame k, measurement ξ,given the states deﬁned by sx and sy (the measurementsare independent of direction). Equation 5 shows how this iscalculated.Lk(ξ|sx, sy) = e−[(ξx−sx)2+(ξy−sy)2]/2σ2 (5)ξ is a detection at point (ξx, ξy), and σ2 is the variance of the2-dimensional Gaussian.Finally the likelihood ratio surface is updated with thenew information as detailed in Equation 6.ln Λk(s) = lnΛ−k (s) + lnLk(ξk|s) for s ∈ S (6)This recursive system is repeated through the frames ofthe video and a likelihood ratio surface evolves. Since we areinterested in the likelihood ratio of a detection at any pointin the image plane, we marginalise over the different direc-tional planes to form a 2-dimensional likelihood ratio surface,Λ˜k(x, y), as shown in Equation 7.ln Λ˜k(x, y) =∑sd∈Dln Λk(s|sd) (7)3.3. Gaze PredictorWe propose a gaze predictor which is based on the face de-tections from Section 3.1 and the likelihood ratio tracker fromSection 3.2. The tracker is used to decide whether or not thegaze is concentrated on the signer or the inset. Two sepa-rate trackers are run on a predeﬁned bounding box around thesigner’s face simultaneously - one tracking frontal face views,the other proﬁle. The output of each of these trackers is a like-lihood ratio surface for each frame, Λ˜frok for the frontal trackerand Λ˜prok for the proﬁle tracker for frame k. Comaparison ofthe maxima of these surfaces allows a prediction of whetheror not the viewer will be looking at the signer or the inset.Equation 8 describes the prediction process, with d being aconstant.Gksign ={1 if max (ln Λ˜frok )−max (ln Λ˜prok ) > d0 otherwise (8)The gaze predictor speciﬁes that if Gksign = 1, i.e. thesigner is signing, then the gaze prediction will be at the loca-tion of the maximum of Λ˜frok , i.e. the tracked location of thehead of the signer. If Gksign = 0 then the viewer will be look-ing at the inset, and hence the predictor returns this region.4. RESULTSWe use the eye tracking data collected as detailed in Section2 to verify both the face tracker and gaze predictor.4.1. Face Tracking ResultsThe face tracker is used to predict whether or not the viewerswill be concentrating on the signer or elsewhere for any givenframe. To this end we investigate the accuracy of Gksign. Wedeﬁne Gksign = 1 to be correct if the viewer is looking at thesigner’s face for frame k. From this we calculate recall, whichis a measure of the proportion of false negatives recorded. Inthis scenario we are primarily concerned in minimising thefalse negatives, since it is more important to correctly predictﬁxations on the signer than it is to predict them on the inset.Equations 9 and 10 deﬁne precision and recall respectively,and Figure 3 shows how they variy with d (from Equation 8)for a set of 30 sequences.Precision = Number CorrectNumber Correct + False Positives (9)Recall = Number CorrectNumber Correct + False Negatives (10)As can be seen in Figure 3 it is possible to attain a 95%recall value with d = −5. This is the value of d used in thiswork, as it yields high recall, but a suitably small number offalse positive detections (the precision is over 80%).707Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on October 6, 2009 at 07:42 from IEEE Xplore.  Restrictions apply. −20 −15 −10 −5 0 5 1000.10.20.30.40.50.60.70.80.91d  RecallPrecisionFig. 3: Average Precision and Recall for ‘meaning’ se-quences, varying with d from Equation 84.2. Gaze Prediction ResultsIn order to assess the accuracy of the gaze predictor presentedin Section 3.3 the results of the eye tracking study are com-pared to the gaze predictor. Since the gaze predictor returnsboxes of high gaze likelihood, either the face of the signer orthe inset, we say an eye track result has been correctly pre-dicted if it falls within the predicted region for a given frame.From this we calculate a percentage of the eye tracking resultsthat were correctly predicted by the gaze predictor.1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300  102030405060708090100Clip NumberAccuracy (%)(a) Accuracy results predicting gaze location as either signeror inset1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300  102030405060708090100Clip NumberAccuracy (%)(b) Accuracy results when localising the ‘inset’ predictions toface detectionsFig. 4: Percentage of eye track locations correctly predictedby the gaze tracker for 30 different clips. Including the meanand a standard deviation either side of the mean.Figure 4(a) shows the individual results for 30 differentvideo sequences. The results vary from 70% to 95%, with themean being 86%. These results, however, only localise thegaze prediction to either the signer’s face or the inset video. Inorder to attempt to further localise the gaze prediction withinthe inset we use the face detections found in Section 3.1 -deﬁning an eye gaze location within the inset as predictedcorrectly if it is within the bounding box of a face detection.Figure 4(b) shows the results for the same 30 clips as before,which vary from 60% to 95%, with the mean being 79%.5. CONCLUSIONS AND FUTURE WORKIn this paper a gaze prediction technique based on face detec-tion and tracking has been described. A grid-based likelihoodratio tracker has been developed which will allow decisionsregarding the orientation of a face (frontal or proﬁle) to bemade using detections from an error-prone detector. It hasbeen shown that the face tracker can achieve a recall of over95% when comparing the output with gaze locations froman eye tracking study. The gaze prediction routine has beenshown to achieve an average of over 80% correct gaze loca-tions, when compared with the same eye tracking study.Improvements could be made to this system by introduc-ing face tracking across the entire frame, as opposed to mainlyaround the signer, however this would result in a correspond-ing increase in computational complexity. Introducing thecues demonstrated in [2] could also see an improvement inperformance. The gaze predictions can be used for a vari-ety of applications, including variable quality coding and er-ror protection. These applications require a subjective qualitystudy to be undertaken to investigate the effects noticed byviewers.6. REFERENCES[1] L. J Muir and I. E. G Richardson, “Perception of sign language andits application to visual communications for deaf people,” J. Deaf Stud.Deaf Educ., vol. 10, no. 4, pp. 390–401, 2005.[2] S. J. C Davies, D Agraﬁotis, C. N Canagarajah, and D Bull, “Percep-tually optimised coding of open signed video based on gaze prediction,”Electronics Letters, vol. 39, no. 21, pp. 1135–1136, October 2007.[3] D Agraﬁotis, N Canagarajah, D Bull, and M Dye, “Perceptually opti-mised sign language video coding based on eye tracking analysis,” Elec-tronics Letters, vol. 39, no. 24, pp. 1703–1705, 27 Nov. 2003.[4] D Argraﬁotis, N Canagarajah, D Bull, H Twyford, J Kyle, andJ Chung How, “Optimised sign language video coding based on eye-tracking analysis,” Visual Communications and Image Processing, Proc.of SPIE, vol. 5150, pp. 1244–1252, 2003.[5] L Itti, “Automatic foveation for video compression using a neurobiolog-ical model of visual attention,” IEEE Trans. on Image Processing, vol.13, no. 10, pp. 1304–1318, Oct 2004.[6] I. T Union, “Recommendation itu-r bt.500-11: Methodology for thesubjective assessment of the quality of television pictures,” 2002.[7] P Viola and M. J Jones, “Robust real-time face detection,” InternationalJournal of Computer Vision, vol. 57, no. 2, pp. 137–154, May 2004.[8] D. L Hall and J Llinas, Eds., Handbook of Multisensor Data Fusion,CRC Press, 2001.7. ACKNOWLEDGEMENTSThank you to Henry Knowles for his assistance with designing theface tracker. This work was funded by the British Broadcasting Cor-poration (BBC).708Authorized licensed use limited to: UNIVERSITY OF BRISTOL. Downloaded on October 6, 2009 at 07:42 from IEEE Xplore.  Restrictions apply. 

Automatic foveation for video compression using a neurobiological model of visual attention,”

Eds., Handbook of Multisensor Data Fusion,

Optimised sign language video coding based on eyetracking analysis,”

Perception of sign language and its application to visual communications for deaf people,”

Perceptually optimised coding of open signed video based on gaze prediction,”

Perceptually optimised sign language video coding based on eye tracking analysis,”

Recommendation itu-r bt.500-11: Methodology for the subjective assessment of the quality of television pictures,”

Robust real-time face detection,”

https://research-information.bris.ac.uk/files/3017216/Davies_IEEE_ICIP_2008.pdf

A gaze prediction technique for open signed video content using a track before detect algorithm

Abstract

Similar works

Full text

Available Versions

Crossref

Explore Bristol Research