We presented the concept of of a software retina, capable

of significant visual data reduction in combination with

scale and rotation invariance, for applications in egocentric

and robot vision at the first EPIC workshop in Amsterdam

[9]. Our method is based on the mammalian retino-cortical

transform: a mapping between a pseudo-randomly tessellated

retina model (used to sample an input image) and a

CNN. The aim of this first pilot study is to demonstrate a

functional retina-integrated CNN implementation and this

produced the following results: a network using the full

retino-cortical transform yielded an F1 score of 0.80 on a

test set during a 4-way classification task, while an identical

network not using the proposed method yielded an F1

score of 0.86 on the same task. On a 40K node retina the

method reduced the visual data bye×7, the input data to the

CNN by 40% and the number of CNN training epochs by

36%. These results demonstrate the viability of our method

and hint at the potential of exploiting functional traits of

natural vision systems in CNNs. In addition, to the above

study, we present further recent developments in porting

the retina to an Apple iPhone, an implementation in CUDA

C for NVIDIA GPU platforms and extensions of the retina

model we have adopted

Balog, Lorinc

Esparon, Tom

Ozimek, Piotr

Siebert, J. Paul

Wong, Ryan

English

Enlighten

      Ozimek, P., Balog, L., Wong, R., Esparon, T. and Siebert, J. P. (2017) Egocentric Perception using a Biologically Inspired Software Retina Integrated with a Deep CNN. International Conference on Computer Vision 2017, ICCV 2017, Second International Workshop on Egocentric Perception, Interaction and Computing, Lido Venice, Italy, 29 Oct 2017.     There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it.    http://eprints.gla.ac.uk/148802/            Deposited on: 27 September 2017                       Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Egocentric Perception using a Biologically Inspired Software Retina Integratedwith a Deep CNNPiotr Ozimek Lorinc Balog Ryan Wong Tom EsparonJ. Paul SiebertSchool of Computing ScienceUniversity of Glasgow, Lilybank Gardens, Scotland, UK G12 8RZpaul.siebert@glasgow.ac.ukAbstractWe presented the concept of of a software retina, capa-ble of significant visual data reduction in combination withscale and rotation invariance, for applications in egocentricand robot vision at the first EPIC workshop in Amsterdam[9]. Our method is based on the mammalian retino-corticaltransform: a mapping between a pseudo-randomly tessel-lated retina model (used to sample an input image) and aCNN. The aim of this first pilot study is to demonstrate afunctional retina-integrated CNN implementation and thisproduced the following results: a network using the fullretino-cortical transform yielded an F1 score of 0.80 on atest set during a 4-way classification task, while an identi-cal network not using the proposed method yielded an F1score of 0.86 on the same task. On a 40K node retina themethod reduced the visual data by˜×7, the input data to theCNN by 40% and the number of CNN training epochs by36%. These results demonstrate the viability of our methodand hint at the potential of exploiting functional traits ofnatural vision systems in CNNs. In addition, to the abovestudy, we present further recent developments in portingthe retina to an Apple iPhone, an implementation in CUDAC for NVIDIA GPU platforms and extensions of the retinamodel we have adopted.1. IntroductionThis paper updates progress in developing and imple-menting the concept of a combining a software retina witha DCNN for efficient visual processing as detailed in [7].We argue that a key design issue for egocentric and robotvision is the ability to perform advanced computer visionwhile using only lightweight processing platforms. Thisrequirement implies a need for a smart camera capable oftransferring low bandwidth visual data streams and data ef-ficient Deep CNN processing. Accordingly, as we notedat the EIPC Workshop in 2016, there is currently muchwork going on to adapt smartphones to provide completeportable vision systems, as the relatively low-cost and ubiq-uitously available smartphone is so exquisitely integratedby having camera(s), inertial sensing, sound input/outputand excellent wireless connectivity. Mass market produc-tion makes this a very low-cost platform and manufacturersfrom quadrotor drone suppliers to childrens toys, such as theMeccanoid robot, employ a smartphone to provide a visionsystem/control system.The key challenge of this work we proposed in [9] ishow to transform the irregularly distributed retina samplesto a matrix format that can be processed within conventionalCNN environments and then to devise a CNN architecturewhich is compatible with this input. We demonstrate thatour software retina-integrated CNN formulation is capableof learning object classes, affords a reduction in networksize and also requires fewer training epochs to achieve aclassification performance similar to that of a standard CNNformulation. We now outline a retina implementation run-ning on the CPU, a GPU accelerated version and also a firstdemonstration e on an Apple iPhone.We summarise a first study [7] into improving the ef-ficiency of image analysis using CNNs by pre-processingusing a software-based retina model, similar in structure tothose of mammals and humans, to substantially reduce thevisual input data size to the CNN and also simplify its learn-ing requirements. Our retina model samples the input imageusing Gaussian receptive fields located on a space-variant(foveated) pseudo-random sampling tessellation generatedby annealing. This data is then spatially transformed us-ing a polar transform to generate a cortical map, similar tothat observed in the human visual system. This mappingsplits the visual field into two hemifields, as observed inthe brain, and these are projected into a regular image suit-able for processing by a Keras CNN model we formulated.This mapping not only reduces the visual data by a factor1of ˜×7 for a 4K node retina (˜ ×16.7 for a 50K node retina),it also affords a degree of input image scale and rotation in-variance and therefore has the potential to both reduce thenetwork size substantially and also simplify its learning re-quirements.2. Balasuriya’s RetinaFigure 1. Left: Gaussian receptive fields on top of a retina tessel-lation, taken from [1]. Centre: The 4196 node tessellation used inthis paper. Right: A backprojected retinal image.The retina model that has been employed in this paperwas developed by Balasuriya [1] whose work investigatesthe generation, sampling function, feature extraction andgaze control mechanism of a self-organized software retina.To generate the retina tessellation without local discon-tinuities, distortions or other artefacts Balasuriya employsa self-similar neural network as described by Clippingdale& Wilson [2]. This method relies on a network of N nodesjointly undergoing random translations to produce a tessel-lation with a near-uniform dense foveal region that seam-lessly transitions into a sparse periphery. Each node inthe resultant tessellation defines the location of a recep-tive field’s centre. The receptive fields somewhat followthe biological retina’s architecture; they all have a Gaus-sian response profile the standard deviation of which scaleslinearly as a function of local node density, which in turnscales with eccentricity. This scaling balances between in-troducing aliasing at the sparsely sampled peripheries andsuper-Nyquist sampling at the densely sampled foveal re-gion.The values sampled by the receptive fields are thenstored in an imagevector, which is a one-dimensional arrayof intensity values which supply the remainder of his visualprocessing chain and are also used to feed the processingpipeline in this work.3. The Retino-Cortical TransformThe core idea behind cortical images is to first map thereceptive field centres onto a new space and then projectthe associated imagevector intensities via Gaussian kernelscentred on these locations, i.e. perform a forward warp.The approach taken eliminates the possibility of holes inthe mapping as the size of the Gaussian projections can beincreased to compensate.The cortical images should ideally be conformal, i.e.preserve local angles and maintain a fairly uniform recep-tive field density while preserving local information cap-tured by the retina without introducing any artefacts. Thesecriteria must be satisfied to enable the convolution kernelsof CNNs to extract features from the resultant cortical im-age. The literature reports sampling points at an adjustedlog-polar space are the most appropriate retinocortical map-ping, as it is believed they are employed in the primate vi-sual cortex [8]. It is mathematically a plausible mapping forfoveated images as it stretches out the fovea and compressesthe peripheral field; it is also a conformal mapping.4. 4K Node Retina ValidationFigure 2. An example Brown Bear image from each of the threesubsets (A, B and C).A dataset suitable for training retina-integrated CNNs(RI-CNNs) was created by selecting and pre-processing theappropriate classes from ImageNet [3]. To prevent the clas-sification task from being trivial, each object class in thedataset should share a subset of its visual features with atleast one other class, meaning that the classes should besomewhat similar to each other.In order to evaluate each part of the proposed retino-cortical transform in isolation, three validation subsets havebeen constructed: subset A is made up of cortical images(Fig. 2, left), subset B is retinal backprojected images (Fig.2, centre) and subset C consists of the conventional images,masked with the retinal lens (Fig. 2, right).Figure 3. Left: Per class and per split fixation image counts. Thenumbers are consistent across all 3 subsets of the dataset. Right:The CNN architecture used in this paper.The object categories selected for the classification taskare Basketball Hoop, Brown Bear, Keyboard and Racoon.The similarities between Brown Bear and Racoon (furryanimal), Basketball Hoop and Keyboard (synthetic ob-ject with a grid-like key feature) helped ensure that theclassification task is not trivial. The class objects werecropped out from their original images using the bound-ing boxes provided in ImageNet [3]. The resultant im-ages passed automatic selection that ensured the imageswere not too small (width, height > 75, 75) or too long(1/3 < width/height < 3), and were then processed byappropriate parts of the retina pipeline to produce the threesubsets.Figure 4. Left: A retinal backprojection image. Right: the equiv-alent saliency map. Note the inhibition near the foveal region.The image locations fixated on by the retina were se-lected by a gaze control system that is both simple and suf-ficient to meet the needs of this study. The algorithm main-tained a saliency map driven by SIFT features that was in-hibited at locations of past fixations (Figure 4, right). In or-der to correct a large imbalance between the class frequen-cies the number of retina fixations was varied per class. Thefinal image counts in the dataset can be seen in Figure 3.5. Results and DiscussionIn order to evaluate in isolation the performance contri-butions of the retinal subsampling mechanism and the corti-cal image representation to the overall pipeline, three CNNswere trained using Keras 2.0.2, each with the same architec-ture but each using a different subset of the dataset built inthe previous section. The CNN architecture used (Figure 3)was chosen by trialling various architectures to maximisetheir performance over the cortical image dataset. A rela-tively simple architecture was chosen in accordance the ob-jectives of this study.Adam [6] was employed for training optimisation incombination with categorical cross-entropy as the loss func-tion. Early stopping callbacks (used to monitor improve-ments in validation accuracy) were employed to prevent un-productive training. L2 regularisation of strength λ = 0.02was applied to the internal fully connected layers to preventover-fitting, however that value could have been increasedas the model still displayed signs of over-fitting. The keyfigures from training are:• Network EVAL-A, using (96x179) cortical images,reached its peak performance (validation loss= 0.605,validation accuracy=82.26%) after 16 epochs.• Network EVAL-B, using (168x168) retinal backpro-jected images, reached its peak performance (valida-tion loss= 0.493, validation accuracy=86.14%) after21 epochs.• Network EVAL-C, using (168x168) conventional im-ages, reached its peak performance (validation loss=0.488, validation accuracy=87.51%) after 25 epochs.The results from evaluating the networks against the testset (Figure 5) show that both applying the full retinocor-tical transform and the retinal subsampling led to a smalldecrease in the CNNs’ performance. The network trainedon conventional images performed the best, with an aver-age F1 score of 0.86; the network trained on retinal imageslanded an F1 score of 0.84 while the cortical images net-work had an F1 score of 0.80 showing that remapping theimage from the retinal to the cortical space was the mostdamaging aspect of the retinocortical transform. As seen inthe matrices in Figure 5, the majority of the networks’ con-fusion is between the classes sharing similar key features.Although the retina has reduced classification performance,the gap between the performance scores of the different net-works is modest, the required training epochs have been re-duced significantly and the network EVAL-A has success-fully demonstrated the learning capacity of convolutionalneural networks for images in the cortical view.6. Recent WorkSubsequent to validating the 4K node retina, as describedabove, we have implemented a 50K node retina capable ofsampling a 930× 930 pixel input image to produce a corti-cal image of around 150K pixels, yielding a visual data re-duction of ˜×16.7 and a network input reduction of ˜×5.8.This 50K node retina generates a cortical image in ˜13mswhen executing on an NVIDIA GTX1080TI GPU, runningCUDA C codes.We have also demonstrated the 4K node retina runningon an Apple iPhone. This comparatively small retina sam-ples a patch in an image captured by iPhone’s camera andSIFT descriptors are extracted from the cortical image todirect the next retinal fixation location of the next in con-junction with a simple inhibition of return algorithm. TheiPhone can therefore serve as an autonomous cortical imagecapture device for tasks such as object appearance learning.Our next objective is to couple the retina input to a mobileTensorflow model, trained using the iPhone retina, to sup-port egocentric perception processing tasks.Finally, in order to improve the overall data efficiencyof the retina-DCNN combination, we have implementeda gamut of colour and intensity retinal ganglion P-cells.These include RG & BY difference cells, and RG & BY sin-gle and double opponent cells. We have also implementedthe classical intensity difference of Gaussian response cells.Figure 5. Confusion matrices and different performance metrics of the three CNNs evaluated against the appropriate test sets.These have been structured to generate the pairs of responseoutputs corresponding to rectified +ve & -ve responses. Asconsequence these cells appear to exhibit the first stages onfigure-ground separation in there responses to appropriateintensity and colour differential inputs and we anticipatethis will simplify learning segmentation and object bound-aries.7. Conclusions & Further WorkThis work has presented a pilot study into a novelmethod for pre-processing images provided to CNNs. Themethod draws inspiration from the mammalian visual sys-tem by imitating the retinocortical transform to reduce thenetworks’ memory requirements as well as affording a de-gree of scale and rotation invariance. The contributions ofour work comprise: a specific retina model, a coarsely op-timised cortical transform formulation and an image classi-fication dataset comprising three subsets designed to probethe impact of the spatial sampling and transformation com-ponents of the pipeline. Evaluating a CNN architecture onthe three data subsets has shown that the performance ofthe retina-integrated CNN is comparable to that of a CNNworking with conventional images, while image data reduc-tion, network size reduction and learning simplification hasbeen confirmed. To the best of the authors’ knowledge noprior attempts have been made at integrating a similar pro-cess to CNNs.This work has laid the groundwork for further investiga-tions into integrating the retinocortical transform with con-volutional neural networks. The authors propose to developa custom non-shared CNN layer which is fed directly bythe retina image vector, appropriately transformed into a 2Dpolar retinotopic mapping. We then propose to investigatemore elaborate CNN architectures which are best suited forretinocortical transformed input. Locally connected con-volution layers, as well as streams of parallel convolutionseach processing a separate portion of the cortical image, areboth relevant features of CNN architectures that appear tobe worth investigating.Our most recent work includes modelling the known reti-nal ganglion cells, GPU and iPhone retina implementations,and a high-resolution 50K node retina. Our currently inves-tigations include the wider range of low level computationsthat are essential to high-level visual reasoning tasks re-lated to edge detection, motion and prediction [4] and moresophisticated gaze control algorithms, potentially based onlearning and generating target specific saliency maps, as inHong et a. [5]. We are also working towards more exten-sive training datasets based on both static and video imagerycaptured using the iPhone retina system and also a cameramounted on a robot arm to support scene exploration andhand-eye visual serving.References[1] S. Balasuriya. A Computational Model of Space-Variant Vi-sion Based on a Self-Organized Artifical Retina Tesselation.PhD thesis, Department of Computing Science, University ofGlasgow, March 2006. 2[2] S. Clippingdale and R. Wilson. Self-similar neural networksbased on a kohonen learning rule. Neural Networks, 9(5):747–763, 1996. 2[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. InComputer Vision and Pattern Recognition, 2009. CVPR 2009.IEEE Conference on, pages 248–255. IEEE, 2009. 2, 3[4] T. Gollisch and M. Meister. Eye smarter than scientists be-lieved: neural computations in circuits of the retina. Neuron,65(2):150–164, 2010. 4[5] S. Hong, T. You, S. Kwak, and B. Han. Online tracking bylearning discriminative saliency map with convolutional neu-ral network. In ICML, pages 597–606, 2015. 4[6] D. Kingma and J. Ba. Adam: A method for stochastic opti-mization. arXiv preprint arXiv:1412.6980, 2014. 3[7] P. Ozimek and J. Siebert. Integrating a Non-Uniformly Sam-pled Software Retina with a Deep CNN Model. In BMVC2017 Workshop on Deep Learning On Irregular Domains,September 2017. 1[8] E. L. Schwartz. Spatial mapping in the primate sensory pro-jection: Analytic structure and relevance to perception. Bio-logical Cybernetics, 25(4):181–194, 1977. 2[9] J. Siebert, A. Schmidt, G. Aragon-Camarasa, N. Hockings,X. Wang, and W. P. Cockshott. A Software Retina for Ego-centric & Robotic Vision Applications on Mobile Platforms.In ECCV 2016 Workshop on Egocentric Perception, Interac-tion and Computing, October 2016. 1

Egocentric Perception using a Biologically Inspired Software Retina Integrated

with a Deep CNN

Enlighten: Publications

  
 
 
 
 
Ozimek, P., Balog, L., Wong, R., Esparon, T. and Siebert, J. P. (2017) Egocentric 
Perception using a Biologically Inspired Software Retina Integrated with a Deep CNN. 
International Conference on Computer Vision 2017, ICCV 2017, Second International 
Workshop on Egocentric Perception, Interaction and Computing, Lido Venice, Italy, 29 
Oct 2017. 
 
   
There may be differences between this version and the published version. You are 
advised to consult the publisher’s version if you wish to cite from it. 
 
 
 
http://eprints.gla.ac.uk/148802/ 
     
 
 
 
 
 
 
Deposited on: 27 September 2017 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Enlighten – Research publications by members of the University of Glasgow 
http://eprints.gla.ac.uk 
Egocentric Perception using a Biologically Inspired Software Retina Integrated
with a Deep CNN
Piotr Ozimek Lorinc Balog Ryan Wong Tom Esparon
J. Paul Siebert
School of Computing Science
University of Glasgow, Lilybank Gardens, Scotland, UK G12 8RZ
paul.siebert@glasgow.ac.uk
Abstract
We presented the concept of of a software retina, capa-
ble of significant visual data reduction in combination with
scale and rotation invariance, for applications in egocentric
and robot vision at the first EPIC workshop in Amsterdam
[9]. Our method is based on the mammalian retino-cortical
transform: a mapping between a pseudo-randomly tessel-
lated retina model (used to sample an input image) and a
CNN. The aim of this first pilot study is to demonstrate a
functional retina-integrated CNN implementation and this
produced the following results: a network using the full
retino-cortical transform yielded an F1 score of 0.80 on a
test set during a 4-way classification task, while an identi-
cal network not using the proposed method yielded an F1
score of 0.86 on the same task. On a 40K node retina the
method reduced the visual data by˜×7, the input data to the
CNN by 40% and the number of CNN training epochs by
36%. These results demonstrate the viability of our method
and hint at the potential of exploiting functional traits of
natural vision systems in CNNs. In addition, to the above
study, we present further recent developments in porting
the retina to an Apple iPhone, an implementation in CUDA
C for NVIDIA GPU platforms and extensions of the retina
model we have adopted.
1. Introduction
This paper updates progress in developing and imple-
menting the concept of a combining a software retina with
a DCNN for efficient visual processing as detailed in [7].
We argue that a key design issue for egocentric and robot
vision is the ability to perform advanced computer vision
while using only lightweight processing platforms. This
requirement implies a need for a smart camera capable of
transferring low bandwidth visual data streams and data ef-
ficient Deep CNN processing. Accordingly, as we noted
at the EIPC Workshop in 2016, there is currently much
work going on to adapt smartphones to provide complete
portable vision systems, as the relatively low-cost and ubiq-
uitously available smartphone is so exquisitely integrated
by having camera(s), inertial sensing, sound input/output
and excellent wireless connectivity. Mass market produc-
tion makes this a very low-cost platform and manufacturers
from quadrotor drone suppliers to childrens toys, such as the
Meccanoid robot, employ a smartphone to provide a vision
system/control system.
The key challenge of this work we proposed in [9] is
how to transform the irregularly distributed retina samples
to a matrix format that can be processed within conventional
CNN environments and then to devise a CNN architecture
which is compatible with this input. We demonstrate that
our software retina-integrated CNN formulation is capable
of learning object classes, affords a reduction in network
size and also requires fewer training epochs to achieve a
classification performance similar to that of a standard CNN
formulation. We now outline a retina implementation run-
ning on the CPU, a GPU accelerated version and also a first
demonstration e on an Apple iPhone.
We summarise a first study [7] into improving the ef-
ficiency of image analysis using CNNs by pre-processing
using a software-based retina model, similar in structure to
those of mammals and humans, to substantially reduce the
visual input data size to the CNN and also simplify its learn-
ing requirements. Our retina model samples the input image
using Gaussian receptive fields located on a space-variant
(foveated) pseudo-random sampling tessellation generated
by annealing. This data is then spatially transformed us-
ing a polar transform to generate a cortical map, similar to
that observed in the human visual system. This mapping
splits the visual field into two hemifields, as observed in
the brain, and these are projected into a regular image suit-
able for processing by a Keras CNN model we formulated.
This mapping not only reduces the visual data by a factor
1
of ˜×7 for a 4K node retina (˜ ×16.7 for a 50K node retina),
it also affords a degree of input image scale and rotation in-
variance and therefore has the potential to both reduce the
network size substantially and also simplify its learning re-
quirements.
2. Balasuriya’s Retina
Figure 1. Left: Gaussian receptive fields on top of a retina tessel-
lation, taken from [1]. Centre: The 4196 node tessellation used in
this paper. Right: A backprojected retinal image.
The retina model that has been employed in this paper
was developed by Balasuriya [1] whose work investigates
the generation, sampling function, feature extraction and
gaze control mechanism of a self-organized software retina.
To generate the retina tessellation without local discon-
tinuities, distortions or other artefacts Balasuriya employs
a self-similar neural network as described by Clippingdale
& Wilson [2]. This method relies on a network of N nodes
jointly undergoing random translations to produce a tessel-
lation with a near-uniform dense foveal region that seam-
lessly transitions into a sparse periphery. Each node in
the resultant tessellation defines the location of a recep-
tive field’s centre. The receptive fields somewhat follow
the biological retina’s architecture; they all have a Gaus-
sian response profile the standard deviation of which scales
linearly as a function of local node density, which in turn
scales with eccentricity. This scaling balances between in-
troducing aliasing at the sparsely sampled peripheries and
super-Nyquist sampling at the densely sampled foveal re-
gion.
The values sampled by the receptive fields are then
stored in an imagevector, which is a one-dimensional array
of intensity values which supply the remainder of his visual
processing chain and are also used to feed the processing
pipeline in this work.
3. The Retino-Cortical Transform
The core idea behind cortical images is to first map the
receptive field centres onto a new space and then project
the associated imagevector intensities via Gaussian kernels
centred on these locations, i.e. perform a forward warp.
The approach taken eliminates the possibility of holes in
the mapping as the size of the Gaussian projections can be
increased to compensate.
The cortical images should ideally be conformal, i.e.
preserve local angles and maintain a fairly uniform recep-
tive field density while preserving local information cap-
tured by the retina without introducing any artefacts. These
criteria must be satisfied to enable the convolution kernels
of CNNs to extract features from the resultant cortical im-
age. The literature reports sampling points at an adjusted
log-polar space are the most appropriate retinocortical map-
ping, as it is believed they are employed in the primate vi-
sual cortex [8]. It is mathematically a plausible mapping for
foveated images as it stretches out the fovea and compresses
the peripheral field; it is also a conformal mapping.
4. 4K Node Retina Validation
Figure 2. An example Brown Bear image from each of the three
subsets (A, B and C).
A dataset suitable for training retina-integrated CNNs
(RI-CNNs) was created by selecting and pre-processing the
appropriate classes from ImageNet [3]. To prevent the clas-
sification task from being trivial, each object class in the
dataset should share a subset of its visual features with at
least one other class, meaning that the classes should be
somewhat similar to each other.
In order to evaluate each part of the proposed retino-
cortical transform in isolation, three validation subsets have
been constructed: subset A is made up of cortical images
(Fig. 2, left), subset B is retinal backprojected images (Fig.
2, centre) and subset C consists of the conventional images,
masked with the retinal lens (Fig. 2, right).
Figure 3. Left: Per class and per split fixation image counts. The
numbers are consistent across all 3 subsets of the dataset. Right:
The CNN architecture used in this paper.
The object categories selected for the classification task
are Basketball Hoop, Brown Bear, Keyboard and Racoon.
The similarities between Brown Bear and Racoon (furry
animal), Basketball Hoop and Keyboard (synthetic ob-
ject with a grid-like key feature) helped ensure that the
classification task is not trivial. The class objects were
cropped out from their original images using the bound-
ing boxes provided in ImageNet [3]. The resultant im-
ages passed automatic selection that ensured the images
were not too small (width, height > 75, 75) or too long
(1/3 < width/height < 3), and were then processed by
appropriate parts of the retina pipeline to produce the three
subsets.
Figure 4. Left: A retinal backprojection image. Right: the equiv-
alent saliency map. Note the inhibition near the foveal region.
The image locations fixated on by the retina were se-
lected by a gaze control system that is both simple and suf-
ficient to meet the needs of this study. The algorithm main-
tained a saliency map driven by SIFT features that was in-
hibited at locations of past fixations (Figure 4, right). In or-
der to correct a large imbalance between the class frequen-
cies the number of retina fixations was varied per class. The
final image counts in the dataset can be seen in Figure 3.
5. Results and Discussion
In order to evaluate in isolation the performance contri-
butions of the retinal subsampling mechanism and the corti-
cal image representation to the overall pipeline, three CNNs
were trained using Keras 2.0.2, each with the same architec-
ture but each using a different subset of the dataset built in
the previous section. The CNN architecture used (Figure 3)
was chosen by trialling various architectures to maximise
their performance over the cortical image dataset. A rela-
tively simple architecture was chosen in accordance the ob-
jectives of this study.
Adam [6] was employed for training optimisation in
combination with categorical cross-entropy as the loss func-
tion. Early stopping callbacks (used to monitor improve-
ments in validation accuracy) were employed to prevent un-
productive training. L2 regularisation of strength λ = 0.02
was applied to the internal fully connected layers to prevent
over-fitting, however that value could have been increased
as the model still displayed signs of over-fitting. The key
figures from training are:
• Network EVAL-A, using (96x179) cortical images,
reached its peak performance (validation loss= 0.605,
validation accuracy=82.26%) after 16 epochs.
• Network EVAL-B, using (168x168) retinal backpro-
jected images, reached its peak performance (valida-
tion loss= 0.493, validation accuracy=86.14%) after
21 epochs.
• Network EVAL-C, using (168x168) conventional im-
ages, reached its peak performance (validation loss=
0.488, validation accuracy=87.51%) after 25 epochs.
The results from evaluating the networks against the test
set (Figure 5) show that both applying the full retinocor-
tical transform and the retinal subsampling led to a small
decrease in the CNNs’ performance. The network trained
on conventional images performed the best, with an aver-
age F1 score of 0.86; the network trained on retinal images
landed an F1 score of 0.84 while the cortical images net-
work had an F1 score of 0.80 showing that remapping the
image from the retinal to the cortical space was the most
damaging aspect of the retinocortical transform. As seen in
the matrices in Figure 5, the majority of the networks’ con-
fusion is between the classes sharing similar key features.
Although the retina has reduced classification performance,
the gap between the performance scores of the different net-
works is modest, the required training epochs have been re-
duced significantly and the network EVAL-A has success-
fully demonstrated the learning capacity of convolutional
neural networks for images in the cortical view.
6. Recent Work
Subsequent to validating the 4K node retina, as described
above, we have implemented a 50K node retina capable of
sampling a 930× 930 pixel input image to produce a corti-
cal image of around 150K pixels, yielding a visual data re-
duction of ˜×16.7 and a network input reduction of ˜×5.8.
This 50K node retina generates a cortical image in ˜13ms
when executing on an NVIDIA GTX1080TI GPU, running
CUDA C codes.
We have also demonstrated the 4K node retina running
on an Apple iPhone. This comparatively small retina sam-
ples a patch in an image captured by iPhone’s camera and
SIFT descriptors are extracted from the cortical image to
direct the next retinal fixation location of the next in con-
junction with a simple inhibition of return algorithm. The
iPhone can therefore serve as an autonomous cortical image
capture device for tasks such as object appearance learning.
Our next objective is to couple the retina input to a mobile
Tensorflow model, trained using the iPhone retina, to sup-
port egocentric perception processing tasks.
Finally, in order to improve the overall data efficiency
of the retina-DCNN combination, we have implemented
a gamut of colour and intensity retinal ganglion P-cells.
These include RG & BY difference cells, and RG & BY sin-
gle and double opponent cells. We have also implemented
the classical intensity difference of Gaussian response cells.
Figure 5. Confusion matrices and different performance metrics of the three CNNs evaluated against the appropriate test sets.
These have been structured to generate the pairs of response
outputs corresponding to rectified +ve & -ve responses. As
consequence these cells appear to exhibit the first stages on
figure-ground separation in there responses to appropriate
intensity and colour differential inputs and we anticipate
this will simplify learning segmentation and object bound-
aries.
7. Conclusions & Further Work
This work has presented a pilot study into a novel
method for pre-processing images provided to CNNs. The
method draws inspiration from the mammalian visual sys-
tem by imitating the retinocortical transform to reduce the
networks’ memory requirements as well as affording a de-
gree of scale and rotation invariance. The contributions of
our work comprise: a specific retina model, a coarsely op-
timised cortical transform formulation and an image classi-
fication dataset comprising three subsets designed to probe
the impact of the spatial sampling and transformation com-
ponents of the pipeline. Evaluating a CNN architecture on
the three data subsets has shown that the performance of
the retina-integrated CNN is comparable to that of a CNN
working with conventional images, while image data reduc-
tion, network size reduction and learning simplification has
been confirmed. To the best of the authors’ knowledge no
prior attempts have been made at integrating a similar pro-
cess to CNNs.
This work has laid the groundwork for further investiga-
tions into integrating the retinocortical transform with con-
volutional neural networks. The authors propose to develop
a custom non-shared CNN layer which is fed directly by
the retina image vector, appropriately transformed into a 2D
polar retinotopic mapping. We then propose to investigate
more elaborate CNN architectures which are best suited for
retinocortical transformed input. Locally connected con-
volution layers, as well as streams of parallel convolutions
each processing a separate portion of the cortical image, are
both relevant features of CNN architectures that appear to
be worth investigating.
Our most recent work includes modelling the known reti-
nal ganglion cells, GPU and iPhone retina implementations,
and a high-resolution 50K node retina. Our currently inves-
tigations include the wider range of low level computations
that are essential to high-level visual reasoning tasks re-
lated to edge detection, motion and prediction [4] and more
sophisticated gaze control algorithms, potentially based on
learning and generating target specific saliency maps, as in
Hong et a. [5]. We are also working towards more exten-
sive training datasets based on both static and video imagery
captured using the iPhone retina system and also a camera
mounted on a robot arm to support scene exploration and
hand-eye visual serving.
References
[1] S. Balasuriya. A Computational Model of Space-Variant Vi-
sion Based on a Self-Organized Artifical Retina Tesselation.
PhD thesis, Department of Computing Science, University of
Glasgow, March 2006. 2
[2] S. Clippingdale and R. Wilson. Self-similar neural networks
based on a kohonen learning rule. Neural Networks, 9(5):747–
763, 1996. 2
[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-
Fei. Imagenet: A large-scale hierarchical image database. In
Computer Vision and Pattern Recognition, 2009. CVPR 2009.
IEEE Conference on, pages 248–255. IEEE, 2009. 2, 3
[4] T. Gollisch and M. Meister. Eye smarter than scientists be-
lieved: neural computations in circuits of the retina. Neuron,
65(2):150–164, 2010. 4
[5] S. Hong, T. You, S. Kwak, and B. Han. Online tracking by
learning discriminative saliency map with convolutional neu-
ral network. In ICML, pages 597–606, 2015. 4
[6] D. Kingma and J. Ba. Adam: A method for stochastic opti-
mization. arXiv preprint arXiv:1412.6980, 2014. 3
[7] P. Ozimek and J. Siebert. Integrating a Non-Uniformly Sam-
pled Software Retina with a Deep CNN Model. In BMVC
2017 Workshop on Deep Learning On Irregular Domains,
September 2017. 1
[8] E. L. Schwartz. Spatial mapping in the primate sensory pro-
jection: Analytic structure and relevance to perception. Bio-
logical Cybernetics, 25(4):181–194, 1977. 2
[9] J. Siebert, A. Schmidt, G. Aragon-Camarasa, N. Hockings,
X. Wang, and W. P. Cockshott. A Software Retina for Ego-
centric & Robotic Vision Applications on Mobile Platforms.
In ECCV 2016 Workshop on Egocentric Perception, Interac-
tion and Computing, October 2016. 1


http://eprints.gla.ac.uk/148802/13/148802.pdf

Egocentric Perception using a Biologically Inspired Software Retina Integrated with a Deep CNN

Abstract

Similar works

Full text

Available Versions

Enlighten

Enlighten: Publications