We present ongoing work to harness biological approaches to achieving highly
efficient egocentric perception by combining the space-variant imaging
architecture of the mammalian retina with Deep Learning methods. By
pre-processing images collected by means of eye-tracking glasses to control the
fixation locations of a software retina model, we demonstrate that we can
reduce the input to a DCNN by a factor of 3, reduce the required number of
training epochs and obtain over 98% classification rates when training and
validating the system on a database of over 26,000 images of 9 object classes.Comment: Accepted for: EPIC Workshop at the European Conference on Computer
  Vision, ECCV201

Hristozova, Nina

Ozimek, Piotr

Siebert, Jan Paul

English

arXiv

We present ongoing work to harness biological approaches to achieving highly efficient egocentric perception by combining the space- variant imaging architecture of the mammalian retina with Deep Learn- ing methods. By pre-processing images collected by means of eye-tracking glasses to control the fixation locations of a software retina model, we demonstrate that we can reduce the input to a DCNN by a factor of 3, reduce the required number of training epochs and obtain over 98% clas- sification rates when training and validating the system on a database of over 26,000 images of 9 object classes

Enlighten

Hristozova, N., Ozimek, P. and Siebert, J. P. (2018) Efficient Egocentric Visual Perception Combining Eye-tracking, a Software Retina and Deep Learning. Third International Workshop on Egocentric Perception, Interaction and Computing (EPIC@ECCV18), Munich, Germany, 09 Sep 2018. There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it. http://eprints.gla.ac.uk/183354/ Deposited on: 3 April 2019 Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Efficient Egocentric Visual PerceptionCombining Eye-tracking, a Software Retina andDeep LearningNina Hristozova, Piotr Ozimek, and Jan Paul SiebertUniversity of Glasgow, School of Computing Science, Scotland, UK.paul.siebert@glasgow.ac.ukhttp://www.dcs.gla.ac.ukAbstract. We present ongoing work to harness biological approaches toachieving highly efficient egocentric perception by combining the space-variant imaging architecture of the mammalian retina with Deep Learn-ing methods. By pre-processing images collected by means of eye-trackingglasses to control the fixation locations of a software retina model, wedemonstrate that we can reduce the input to a DCNN by a factor of 3,reduce the required number of training epochs and obtain over 98% clas-sification rates when training and validating the system on a databaseof over 26,000 images of 9 object classes.Keywords: Data Efficiency, Deep Learning, Retina, Foveated Vision,Biological Vision, Egocentric Perception, Robot Vision, Visual Cortex1 IntroductionIn this paper we report the results of an experiment [3] that combines a high-resolution 50K node software retina [2,6,5] with a custom designed DL architec-ture (based on DeepFix[4]) coupled to an image stream collected by Tobii Pro2 eye-tracking glasses [1] (Figure 1) worn by a human observer. Our objectiveis to allow a human operator to collect appropriate training data for a softwareretina-based egocentric perception[10,5] system simply by looking at objects.These objects may then be recognised in images collected by a human observerusing eye tracking glasses, or a machine observer equipped with a saliency modelto direct visual gaze.The space-variant sampling within the retina, as illustrated in Figure 2, af-fords almost two orders of magnitude data reduction to the brain. Furthermore,the pathway from the retina to area 1 in the visual cortex (V1) computes a formof complex-log conformal mapping that affords a degree of scale and rotationinvariance to the appearance of fixated objects[6,7,8]. We previously reportedthat this mapping can reduce the input visual data rate by a factor of ˜×7 for a4K node retina and ˜×16.7 for a 50K node retina. A corresponding network sizereduction of ˜40% and ˜83% is obtained respectively, however, at the expense ofan F1 classification score reduction of 0.86% to 0.80% as reported at [6,5].2 Nina Hristozova, Piotr Ozimek, Jan Paul SiebertIn this paper our objective is to demonstrate that we can achieve state-of-the-art recognition performance using our high-resolution retina implementationwhile also achieving efficiency gains. In addition, we wanted to investigate thepotential to adopt a human observer for directing our software retina’s gaze tothereby construct a truly egocentric perception system suitable for both humansand robots.2 ApproachOur processing pipeline comprises four stages: image capture, fixation clusterextraction, retina transformation and DCNN processing. Following image col-lection using the Tobii glasses, described below, the images for the observationsof each object are composited, in order to allow the individual fixations associ-ated with each observation (fixation) to be overlaid on a single reference image.This approximate alignment was initially achieved by means of SIFT descriptormatching and extraction of the inter-image homographies, however, we discov-ered that simple head stabilisation was sufficient to achieve the required imageregistration. K -means clustering is then applied to these co-referenced fixationlocations, where K has been set to 1% of the number of fixations in the ob-servations for the current object class. This both reduces the number of fixatedtraining images to manageable numbers and also selects locally coherent clustersof fixations, whose convex hulls are used to locate the software retina within theinput image, as shown in Figure 1. The smaller cortical images produced by theretina are then input to the DCNN for both training and inference purposes.Fig. 1: Left, Tobii Pro2 Eye Tracker Glasses; Right, Fixation clustering followinghomography alignment3 Data collectionA custom interface was developed to allow an operator to control the acquisitionof images using the Tobii Pro 2 eye tracking glasses. The two observers whoparticipated in this experiment were instructed to look at locations on the surfaceof each object which seem particularly salient, or diagnostic of each object’sidentity, when collecting images. In order to obtain usable image data, it wasnecessary to stabilise observer’s head by resting their chin on a desk surfacewhile observing each object using the Tobii glasses and also by their consciouslyminimising any head movement.A data set of over 26,000 images was collected using the Tobii glasses, splitinto three categories: Training, Validation and Test. Each of these categoriesEPIC Workshop @ ECCV2018, 9 September, Munich, Germany. 3contains 9 object classes: Eggs, Gnocchi, Juice, Ling, Milk, Rice, Strep, VitCand Yogurt. Each of the data categories comprises the following proportion ofthe total data: Training 80%, Validation 18% and Test 2%.Fig. 2: Left-Right: Retina Sampling, Fixation Crop Image, Corresponding Cor-tical Image, Example Cortical Images of the 9 Object Classes Captured4 Building a DCNN classifierInspired by Kruthiventi’s DeepFix [4] network we developed a hand-optimisedDCNN architecture comprising seven convolutional layers(CLs), each followedby a max pooling layer. The input shape to the first convolutional layer was(399, 752, 3). All seven CLs have a rectifier activation function. The first twoCLs have been configured with 32 convolution filters, the second two CLs with64 convolution filters, the fifth CL comprises 128 filters while the final two CLscomprise 256 filters. In all seven convolution layers the filters are 3x3 in size andeach of these layers is coupled by a 2x2 ReLU max pooling function. Thereafter,the output of the last pooling layer has been flattened prior to being coupledto three fully connected(FC) layers, each comprising 132 nodes and a final fullyconnected layer configured with nodes corresponding to the number of outputclasses, in this case 9. Each FC layer had a rectifier activation function. While theactivation function of the output layer was set to softmax, to provide accuracyand loss values ranging from 0 to 1. The classifier was optimised using stochasticgradient descent and categorical cross entropy was used to compute the lossfunction. Drop-out was set to 50% and only applied between the first two fullyconnected layers.5 Cortical and Fixation Crop image DCNN validationIn order to compare the performance obtained when pre-processing images usingthe software retina, as opposed to classifying standard images, two DCNN modelswere trained: the first with fixation crop images of size 926x926px and the secondwith the cortical images of size 399x752px. Examples of fixation crop and corticalimages are given Figure 2. These images were also normalised prior to being inputto the DCNN.The cortical image classification DCNN model was trained using 270 stepsper epoch (total number of images/batch size), where the batch size was setto 64 and the training and validation data sets comprised 21,310 and 4,800images respectively. This model required 55 minutes processing time to execute4 Nina Hristozova, Piotr Ozimek, Jan Paul Siebert18 epochs and produced 98% validation accuracy. Figure 3 shows the accuracyand loss respectively. 6 seconds of processing time were required for this modelto classify the data in its test set, resulting in an average accuracy of 98.2%.As illustrated in Figure 2, the fixation crop of an original image contains theretina’s field of view, but retains the full image resolution. In order to benchmarkthe performance of the cortical image DCNN classifier, a DCNN model wastrained using the full-resolution fixation crop images. In this case 1217 steps perepoch were used (again total number of images/ batch size) where the batchsize is set to 16 and the training and validation data sets comprised 19,485and 4,390 images respectively. The batch size had to be reduced to 16 from 64used for the cortical image DCNN, because the increased numbers of pixels inthe fixation crop images invoked a Tensorflow memory exhaustion error at anylarger batch size. The resulting accuracy and loss are shown on Figure 3. Thismodel required 2 hours and 30 minutes to execute 18 epochs and produced 99%validation accuracy. 12 seconds of processing time were required for this modelto classify the data in its test set, resulting in an average accuracy of 99.5%.Fig. 3: Left-Right:Cortical image accuracy, Cortical image loss, Fixation Cropimage accuracy, Fixation Crop image loss, all versus training epoch.In both the cortical image and fixation crop classification experiments, thevalidation accuracy obtained for each classifier is close to the corresponding train-ing accuracy result. Given the limited range of observations when the observer’shead is constrained to be stationary, the captured images will be correspondinglysimilar. However, due to the non-linear nature of the retina transformation, thecortical images continue to exhibit significant variation in appearance.6 ConclusionsIn this work we have demonstrated the viability of using human fixations to drivesoftware retina image sampling and subsequent training using our own DCNNmodel. Excellent classification performance on a database of 9 object classes anda factor of 3 reduction in the size of the input network has also been obtained(˜ ×16.7 visual data reduction). However two key challenges remain: Firstly, inorder to benefit fully from the data reduction afforded by the retina mapping,we must implement a custom network layer which is connected directly to theretina output samples, as opposed to generating a cortical image which is a muchless efficient representation. Secondly, the current experiment required the headof the observer to be constrained in order to be able to obtain sufficiently over-lapped images to undertake fixation cluster analysis. An auxiliary network isEPIC Workshop @ ECCV2018, 9 September, Munich, Germany. 5under investigation to both predict potentially diagnostic fixation points on thesurface an object being learned and also to undertake (soft) object segmentationbased on views of the same object taken from arbitrary and partially occludedviewpoints (e.g. when holding an object and inspecting it) to facilitate uncon-strained egocentric data collection and classification by means of eye trackingglasses.7 AddendumSubsequent to preparing this paper, our recent investigations [9] have now re-vealed that simply subsampling the cortical image can produce a network in-put data reduction of ×11.5 without compromising performance when under-taking the eyetracking-based classification task described above. Furthermore,by adopting a scattered datapoint gridding algorithm, developed for astronomypurposes [11], we have been able to produce cortical images which yield both anetwork input data reuction of ×10.8, and also a modest increase in classificationaccuracy.References1. AB, T.: Tobii pro glasses 2: Users manual (Nov 2017),https://www.tobiipro.com/siteassets/tobii-pro/user-manuals/tobii-pro-glasses-2-user-manual.pdf/?v=1.762. Balasuriya, S.: A Computational Model of Space-Variant Vision Based on a Self-Organized Artifical Retina Tesselation. Ph.D. thesis, Department of ComputingScience, University of Glasgow, Scotland (March 2006)3. Hristozova, N.: Using eye-tracking glasses for training dcnns (March 2018)4. Kruthiventi, S.S.S., Ayush, K., Babu, R.V.: Deepfix: A fully convolutional neuralnetwork for predicting human eye fixations. IEEE Transactions on Image Process-ing 26(9), 4446–4456 (Sept 2017). https://doi.org/10.1109/TIP.2017.27106205. Ozimek, P., Balog, L., Wong, R., Esparon, T., Siebert, J.P.: Egocentric Percep-tion using a Biologically Inspired Software Retina Integrated with a Deep CNN.In: ICCV 2017 Workshop on Egocentric Perception, Interaction and Computing(October 2017)6. Ozimek, P., Siebert, J.: Integrating a Non-Uniformly Sampled Software Retina witha Deep CNN Model. In: BMVC 2017 Workshop on Deep Learning On IrregularDomains (September 2017)7. Schwartz, E.L.: Spatial mapping in the primate sensory projection: Analytic struc-ture and relevance to perception. Biological Cybernetics 25(4), 181–194 (1977).https://doi.org/10.1007/BF01885636, http://dx.doi.org/10.1007/BF018856368. Schwartz, E.L.: Computational anatomy and functional architecture of striate cor-tex: A spatial mapping approach to perceptual coding. Vision Research 20(8),645 – 669 (1980). https://doi.org/http://dx.doi.org/10.1016/0042-6989(80)90090-5, http://www.sciencedirect.com/science/article/pii/00426989809009059. Shaikh, H.A.: A Data Efficient Retino-Cortical Image Transformation Mapping. Master’s thesis, School of Computing Science, University of Glasgow, Glasgow,Scotland UK (2018)6 Nina Hristozova, Piotr Ozimek, Jan Paul Siebert10. Siebert, J., Schmidt, A., Aragon-Camarasa, G., Hockings, N., Wang, X., Cockshott,W.P.: A Software Retina for Egocentric & Robotic Vision Applications on MobilePlatforms. In: ECCV 2016 Workshop on Egocentric Perception, Interaction andComputing (October 2016)11. Winkel, B., Lenz, D., Flo¨er, L.: Cygrid: A fast cython-powered convolution-basedgridding module for python. A&A 591, A12 (2016). https://doi.org/10.1051/0004-6361/201628475, https://doi.org/10.1051/0004-6361/201628475

Efficient Egocentric Visual Perception Combining Eye-tracking, a Software Retina and Deep Learning

Abstract

Similar works

Full text

Available Versions

Enlighten

Enlighten: Publications