We present an approach to efficient visual sensing and perception based on a non-uniformly sampled, biologically inspired, software retina that when combined with a DCNN classifier has enabled megapixel-sized camera input images to be processed in a single pass, while maintaining state-of-the recognition performance

Aragon-Camarasa, Gerardo

Balog, Lorinc

Hristozova, Nina

Ozimek, Piotr

Siebert, Jan Paul

English

Enlighten

      Siebert, J. P., Ozimek, P., Balog, L., Hristozova, N. and Aragon-Camarasa, G. (2018) Smart Visual Sensing Using a Software Retina Model. IROS2018 Workshop: Unconventional Sensing and Processing for Robotic Visual Perception, at 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 05 Oct 2018.     There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it.    http://eprints.gla.ac.uk/183359/              Deposited on: 8 April 2019                       Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Smart Visual Sensing Using a Software Retina ModelJan P. Siebert1,2 and Piotr Ozimek1and Lorinc Balog1and Nina Hristozova1and Gerardo Aragon-Camarasa1Abstract— We present an approach to efficient visual sensingand perception based on a non-uniformly sampled, biologicallyinspired, software retina that when combined with a DCNNclassifier has enabled megapixel-sized camera input images tobe processed in a single pass, while maintaining state-of-therecognition performance.I. INTRODUCTIONA key issue in designing robotics systems is the costof an integrated camera sensor that meets the band-width/processing requirement for many advanced roboticsapplications. Lightweight visual sensing is especially impor-tant for many applications, such as SLAM in autonomousaerial vehicles, or for wearable camera devices intendedfor egocentric perception applications. Even in conventionalrobotics tasks such as, grasping and manipulation, both thesheer visual data rate to be processed in real-time andthe need for data efficiency when using Deep Learningtechnology present significant challenges. As DL networksbecome ever larger, more sophisticated, and correspondinglymore computationally expensive to train, the need for dataefficiency is becoming accordingly more critical.To address the above issues, we have been investigatingbiologically motivated foveated vision algorithms based onthe visual processing architectures found in mammals. Thisevolutionary development appears to reduce visual load byaround two orders of magnitude. Based on this observation,we have developed a foveated visual architecture that im-plements a functional model of the retina-cortex mappingFig.1 Right. In our prior research, this retina model servedto produce feature vectors that were matched/classified usingconventional methods[1]. Our software retina and mappinghas been adapted to serve as a data-reducing/normalising pre-processor for classification and interpretation by means ofDeep Convolutional Neural Nets (DCNN), [2], [3], allowingmegapixel-sized camera input images to be processed in asingle pass.Fig. 1. Left: Gaussian receptive fields on top of a retina tessellation, takenfrom [1]. Centre: The 4196 node tessellation used in this paper. Right: Aschematic view of the retino-coprtical mapping, taken from [4].*This work was supported by the EU funded Human Brain Project andalso the Innovate UK funded iSee project.1School of Computing Science, University of Glasgow, Scotland, UK.2Correspondence: paul.siebert@glasgow.ac.ukII. APPROACHThe retina model employed in this paper was originallydeveloped by Balasuriya [1] (and later optimised by Ozimek[2]) who investigated developing a visual architecture thatintegrates feature extraction and gaze control based on a self-organised software retina. To generate the retina tessellationwithout local discontinuities, distortions or other artefacts, aself-similar neural network is used to define retina samplinglocations as described by Clippingdale & Wilson [5], asillustrated in Fig.1 Left. This method relies on a network ofN nodes jointly undergoing random translations to producea tessellation with a near-uniform dense foveal region thatseamlessly transitions into a sparse periphery. Each node inthe resultant tessellation defines the location of a receptivefield’s centre. Each receptive field has a Gaussian responseprofile, the standard deviation of which scales linearly asan (inverse) function of local node density and this in turnscales inversely with eccentricity, visualised in Fig.1 Centre.This scaling balances between introducing aliasing at thesparsely sampled peripheries and super-Nyquist sampling atthe densely sampled foveal region [2].Fig. 2. An example of a transformed Brown Bear image: Left Corteximage, Centre: Retina Back-projection, Right: input image.The values sampled by the receptive fields are then storedin an imagevector, comprising a one-dimensional array ofintensity values which are input to the remainder of his visualprocessing chain and are also used to feed the processingpipeline in this work. To be compatible with current DCNNvisual processing networks, which require regular imageinput matrices, we create a cortical image, Fig. 2 Left, byprojecting the imagevector intensities via Gaussian kernelscentred on the polar-transformed retina sampling locations.The cortical image mapping should ideally be conformal,i.e. preserve local angles and maintain a fairly uniform recep-tive field density, while preserving local information capturedby the retina without introducing any artefacts. These criteriamust be satisfied to enable the convolution kernels of DCNNsto extract features from the resultant cortical image. WhileSwartz [4] reports modelling the retino-cortical mappingusing a modified log-polar transform, we obtained a goodmapping experimentally via a modified polar transform thatconfers a degree of scale and rotation invariance in additionto a substantial degree of visual data reduction. It is alsopossible to invert the image vector via back-projection toproduce an equivalent retina image, shown in Fig.2 Centre,c.f. the full-resolution input field-of-view in Fig. 2 Right.III. RECENT WORKWe reported a first pilot study [2] demonstrating a func-tional retina-integrated DCNN implementation which con-firmed that DCNNs can be trained to recognise objects incortical space and yield efficiency gains: Using a 4K noderetina the method reduced the visual data by ˜×7, the inputdata to the CNN by 40% and the number of CNN trainingepochs by 36%. In this case the gains came at the expenseof a slightly reduced F1 score of 0.80 when applying thefull retino-cortical transform in a 4-way classification task,compared to an identical network yielding an F1 score of0.86 for the same data set, but trained and classified usingfull-resolution images.Having demonstrated the basic viability of the retino-cortical mapping-DCNN approach, we implemented a 50Knode high-resolution retina capable of sampling a 930×930pixel input image to produce a cortical image of around150K pixels, yielding a visual data reduction of ˜×16.7 anda network input reduction of ˜×5.8[3]. A GPU acceleratedversion of this 50K node retina[6] generates a cortical imagein ˜13ms when executing on an NVIDIA GTX1080TI GPU,running CUDA C codes.By pre-processing images collected by means of Tobii Pro2 eye-tracking glasses [7], to control the fixation locationsof a software retina model, we were able to demonstrate [8]that we can reduce the input to a custom designed DCNNarchitecture (based on DeepFix[9]) by ×3 (subsequentlyimproved to ×10.8 [10]), reduce the required training time by×3.8 and obtain over 98% classification rate when trainingand validating the system on a database of over 26,000images of 9 object classes comprising common supermarketproducts. In this experiment, our objective was to allow ahuman operator to collect appropriate training data for afoveated egocentric perception[11], [3], [12] system simplyby looking at objects. These objects may then be recognisedin images collected by a human observer using eye trackingglasses, or a machine observer equipped with a saliencymodel to direct visual gaze.We have also demonstrated the 4K node retina runningon an Apple iPhone[13]. This comparatively small retinasamples a patch in an image captured by the iPhone’s cameraand SIFT descriptors are extracted from the cortical imageto direct the next retinal fixation location of the next inconjunction with a simple inhibition of return algorithm. TheiPhone can therefore serve as an autonomous cortical imagecapture device for tasks such as object appearance learning.The GPU accelerated implementation of the 50K noderetina has been used to demonstrate real-time gaze controlin a target tracking task[14]. In this experiment we trained aretina pre-processed DCNN pipeline with example trackingdata based on a centroid colour tracker following an orangecoloured target set against a dark background. The DCNNsystem learned to drive the Baxter Research Robot’s wristcamera using pan-tilt signals learned from the cortical imagesinput to a custom DCNN we designed. Accordingly ourDCNN was able to regress, directly in real time, from corticalspace, the appropriate pan-tilt activation to allow the robot’swrist camera to track the coloured target.Finally, in order to improve the overall data efficiencyof the retina-DCNN combination, we have implemented agamut of colour and intensity retinal ganglion P-cells[15]based on a model described by [16]. These include Red-Green & Blue-Yellow difference cells, and Red-Green &Blue-Yellow single and double opponent cells. We have alsoimplemented the classical intensity difference of Gaussianresponse cells. These have been structured to generate thepairs of response outputs corresponding to rectified +ve &-ve responses. As consequence these cells appear to exhibitthe first stages of figure-ground separation in their responsesto appropriate intensity and colour differential inputs and weanticipate this will simplify learning segmentation and objectboundaries.IV. WORK IN PROGRESSOur research group is currently tackling the fundamentalissue of how best to couple the retina directly to the DCNNto obtain the maximum data efficiency, avoiding the needto construct a cortical image by interpolation. Large-scalevalidation of the 50K node retina is also under investigation,based on the EPIC Kitchens dataset [17], as are methodsfor efficiently producing very large retinas able to processimages of the order of 25Mpx. This effort is combinedwith integrating the retina approach with the latest DCNNarchitectures for robot vision, such as Deep6DPose [18]. Ourintention is to process images from a robot on-wrist camera,such as the provided by the Baxter Research Robot, or an ex-perimental in-palm camera mounted on the Smart GraspingSystem, developed by The Shadow Robot Company. We arealso porting the retina to the Android smartphone platformand produce self-contained smartphone-based retina sensorsfor robotics and egocentric perception applications and aconvenient method for capturing training data by non-expertusers. We are constructing interfaces to commercial high-resolution pan-tilt security cameras to allow these systemsto serve within active vision systems using retina processingand cortex-based gaze control algorithms. To manage thedata generated by these retina-supported camera systems, weare developing tools to allow editing and formatting prior totraining DCNN systems.V. CONCLUSIONSWe have demonstrated that it is possible to make sub-stantial data efficiency gains in terms of training compu-tation, network sizes and inference rates by pre-processingimages using a biologically inspired retina-cortex mappingthat affords both visual data reduction and also a degreeof scale and rotation invariance. Our ongoing investigationsaim to realise fully the potential gains for this approach,its integration within mainstream robot vision DCNNs andunderpin practical low-cost visual sensors for autonomoussystems.REFERENCES[1] Balasuriya, S.: A Computational Model of Space-Variant VisionBased on a Self-Organized Artifical Retina Tesselation. PhD thesis,Department of Computing Science, University of Glasgow, Scotland(March 2006)[2] Ozimek, P., Siebert, J.: Integrating a Non-Uniformly Sampled Soft-ware Retina with a Deep CNN Model. In: BMVC 2017 Workshop onDeep Learning On Irregular Domains. (September 2017)[3] Ozimek, P., Balog, L., Wong, R., Esparon, T., Siebert, J.P.: EgocentricPerception using a Biologically Inspired Software Retina Integratedwith a Deep CNN. In: ICCV 2017 Workshop on Egocentric Percep-tion, Interaction and Computing. (October 2017)[4] Schwartz, E.L.: Spatial mapping in the primate sensory projection:Analytic structure and relevance to perception. Biological Cybernetics25(4) (1977) 181–194[5] Clippingdale, S., Wilson, R.: Self-similar neural networks based on akohonen learning rule. Neural Networks 9(5) (1996) 747–763[6] Balog, L.: A GPU accelerated software retina. Master’s thesis, Schoolof Computing Science, University of Glasgow, Glasgow, Scotland UK(2017)[7] AB, T.: Tobii pro glasses 2: Users manual (Nov 2017)[8] Hristozova, N.: Dissertation using eye-tracking glasses for trainingdcnns (March 2018)[9] Kruthiventi, S.S.S., Ayush, K., Babu, R.V.: Deepfix: A fully con-volutional neural network for predicting human eye fixations. IEEETransactions on Image Processing 26(9) (Sept 2017) 4446–4456[10] Shaikh, H.A.: A Data Efficient Retino-Cortical Image TransformationMapping . Master’s thesis, School of Computing Science, Universityof Glasgow, Glasgow, Scotland UK (2018)[11] Siebert, J., Schmidt, A., Aragon-Camarasa, G., Hockings, N., Wang,X., Cockshott, W.P.: A Software Retina for Egocentric & RoboticVision Applications on Mobile Platforms. In: ECCV 2016 Workshopon Egocentric Perception, Interaction and Computing. (October 2016)[12] Hristozova, N., Ozimek, P., Siebert, J.P.: Effcient Egocentric VisualPerception Combining Eye-tracking, a Software Retina and DeepLearning. In: ECCV 2018 Workshop on Egocentric Perception,Interaction and Computing. (September 2018)[13] Wong, R.: A Smartphone Software Retina. Master’s thesis, Schoolof Computing Science, University of Glasgow, Glasgow, Scotland UK(2017)[14] Boyd, L.: A retina-based vision system for motion control of thebaxter robot (March 2018)[15] Esparon, T.: An investigation of a software retina for robot vision.Master’s thesis, School of Computing Science, University of Glasgow,Glasgow, Scotland UK (2017)[16] Gao, S., Yang, K., Li, C., Li, Y.: Color constancy using double-opponency. IEEE Transactions on Pattern Analysis & MachineIntelligence 37(10) (Oct. 2015) 1973–1985[17] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A.,Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray,M.: Scaling egocentric vision: The epic-kitchens dataset. In: EuropeanConference on Computer Vision (ECCV). (2018)[18] Do, T., Cai, M., Pham, T., Reid, I.D.: Deep-6dpose: Recovering 6dobject pose from a single RGB image. CoRR abs/1802.10367 (2018)

Smart Visual Sensing Using a Software Retina Model

http://eprints.gla.ac.uk/183359/1/183359.pdf

Smart Visual Sensing Using a Software Retina Model

Abstract

Similar works

Full text

Available Versions

Enlighten