The work described in this paper is inspired by SpikeNET, a system

developed to test the feasibility of using rank-order codes in modelling largescale

networks of asynchronously spiking neurons. The rank-order code theory

proposed by Thorpe concerns the encoding of information by a population of

spiking neurons in the primate visual system. The theory proposes using the order

of firing across a network of asynchronously firing spiking neurons as a neural

code for information transmission. In this paper we aim to measure the perceptual

similarity between the image input to a model retina, based on that originally

designed and developed by VanRullen and Thorpe, and an image reconstructed

from the rank-order encoding of the input image. We use an objective metric

originally proposed by Petrovic to estimate perceptual edge preservation in image

fusion which, after minor modifcations, is very much suited to our purpose. The

results show that typically 75% of the edge information of the input stimulus is

retained in the reconstructed image, and we show how the available information

increases with successive spikes in the rank-order code

Furber, Steve

Sen Bhattacharya, Basabdatta

English

University of Lincoln Institutional Repository

Information Recovery from Rank-Order EncodedImagesBasabdatta Sen1 and Steve Furber11School of Computer ScienceUniversity of ManchesterOxford RoadManchester M13 9PL UKAbstract. The work described in this paper is inspired by SpikeNET, a systemdeveloped to test the feasibility of using rank-order codes in modelling large-scale networks of asynchronously spiking neurons. The rank-order code theoryproposed by Thorpe concerns the encoding of information by a population ofspiking neurons in the primate visual system. The theory proposes using the orderof firing across a network of asynchronously firing spiking neurons as a neuralcode for information transmission. In this paper we aim to measure the perceptualsimilarity between the image input to a model retina, based on that originallydesigned and developed by VanRullen and Thorpe, and an image reconstructedfrom the rank-order encoding of the input image. We use an objective metricoriginally proposed by Petrovic to estimate perceptual edge preservation in imagefusion which, after minor modifications, is very much suited to our purpose. Theresults show that typically 75% of the edge information of the input stimulus isretained in the reconstructed image, and we show how the available informationincreases with successive spikes in the rank-order code.1 IntroductionHow does a population of retinal ganglion cells encode visual information into se-quences of action potentials? The ring-rate code theory proposed by Adrian [1] saysthat a population of neurons encode information entirely in the frequencies of ringof the individual neurons. If we imagine ourselves to be at the receiving end of thesespikes, then we need at least two spikes from a neuron to determine its ring frequency.More recently, experiments have shown that at each synaptic stage of the Human VisualSystem (HVS), a neuron has about 10msec to propagate information by ring a spike.Most cortical neurons have a ring rate of below 100 spikes per second. Thus, in a timewindow of 10msec, a neuron can re at the most a single spike, or it may not re at all[25]. These ndings question the plausibility of the rate-code theory as applied to theHVS [6].The rank-order code theory proposed by Thorpe [7] overcomes the timing con-straints mentioned above. The hypothesis is that the input stimulus applied to a popu-lation of neurons is encoded with an intensity-to-delay transformation function. Whenapplied to a spiking neural model of the retina, it is observed that the perceptually im-portant parts of the input stimulus are well reproduced by the time only the rst 1% ofthe neurons have red their rst spikes [8].At this point, we propose to quantify the above results by measuring the perceptualsimilarity between the input image and an image reconstructed from the rank-order en-coding of the input. An algorithm proposed by Petrovic [9] gives an objective measurefor the preservation of edges in a fused image with respect to two parent images. Inthis paper we describe how we have used a slightly modied version of the algorithmto measure the perceptually important edges that are preserved in a rank-order encodedvisual input with respect to the input itself.2 Rank-Order Codes: Definition and PerformanceThe rank-order code theory proposes that the latency of a spike red by a neuron willbe inversely proportional to the applied stimulus strength. The exact latency at which aneuron res is not critical here. Rather, it is the rank-order of the rst spike generated byeach neuron in a population that is important. Thus, the change in overall luminance andcontrast of an input stimulus will not change the relative order of ring of the neurons,although there will be a change in the ring latency of each, resulting in an automaticnormalization of the inputs [7].2.1 Rank-Order Encoding of an ImageTo test the performance of rank-order codes, Van Rullen and Thorpe built a modelretina [8]. The centre-surround structure of the retinal ganglion cell receptive elds arerepresented by Difference of Gaussian (DoG) functions, with the width of the surroundthree times that of the centre [10], as shown in Fig. 1(a). On-centre and off-centre DoGlters at eight scales are used to simulate the different sizes of ganglion cells in theretina, and are shown in Fig. 1(b)(c). The input image is ltered using this set of sixteen(a)positive Gaussiannegative GaussianDifference of Gaussian(b) (c)Fig. 1. (a) The one-dimensional DoG function. (b) On-Centre Off Surround and (c) Off-CentreOn-Surround DoG functions.DoG lters. The sampling resolution of ltering decreases with increasing scale of theDoG lter. The result of ltering is a set of sixteen matrices containing the convolutioncoefcients. A coefcient value models the activation level of a neuron which drivesit towards the ring threshold. The largest coefcient value corresponds to the neuronwhich is the rst in the population to spike. The coefcients are arranged in descendingorder to rank the neurons according to their latency of ring their rst spike. What wethen have is the input image encoded as rank-ordered coefcients [8].2.2 Decoding the Rank-Ordered Data: Stimulus ReconstructionWhen a population of neurons re, we only know the order in which the neurons re, notthe stimulus values that drove them above threshold. True rank-order encoding meansthat we must throw away the true coefcients and adopt a generic activation level valuecorresponding to the rank of ring of a neuron. In other words, a neuron is weightedaccording to its order of ring, and the weight is xed for a certain rank across all in-put images. This is done by using a look-up table (LUT) for the average coefcientvalue corresponding to a certain rank. The coefcient value corresponding to each rankis generated by averaging the true coefcients of ltering at that particular rank for aset of images. VanRullen and Thorpe [8] generated the table using three thousand im-ages. In our simulations, we generated a similar LUT using an array of thirty imagesof resolution 256× 256. The average coefcient values are plotted against rank on alog-log scale in Fig. 2, which shows that they closely follow a power law. The plot istted with an equation of the form y = C×x−a where a' 0.63 and C' 100. To test the(a)100 101 102 103 104 10510−210−1100101102Rank of the coefficientsLUT for coefficient value corresponding to a rank(b)100 101 102 103 104 10510−210−1100101102Rank of the coefficientsA Power Law fit for the LUTLUTfit for LUTFig. 2. (a) A normalised LUT plot on the logarithmic scale. (b) Power Law fit for the LUT.performance of rank-order codes, VanRullen and Thorpe reconstructed the original in-put stimulus using rank-ordered coefcients read from the LUT. They observed that bythe time the rst 1% of the spikes have arrived, the subject of the reconstructed pictureis fairly recognisable. The reconstructed pictures from our emulation of VanRullen’smodel are shown in Fig. 3.3 An Objective Metric for the Performance of Rank-Order CodesWe see in Fig. 3 that there is a distinct perceptual difference between the reconstructedand original pictures. We wish to nd a measure for the proportion of the information(a) (b) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10100200300400500600700(c) (d) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10500100015002000(e) (f) (i) (j)(g) (h) (k) (l)Fig. 3. (a),(c) Original images and (b),(d) their respective histograms. (e)–(h) Reconstruction ofthe image in (a) using 1%, 5%, 10% and 50% of the LUT values. (i)–(l) Similar data for the imagein (c).in the input image that is transmitted by the rank-order encoding, and to estimate theway that this information builds up through the arrival of successive spikes from aneural population. Petrovic et al. [9] have developed an algorithm for measuring theperceptual loss suffered by an image during the fusion of two parent images, which wehave adapted to our purpose. The owchart of the algorithm as used in our applicationis shown in Fig. 4.As seen in Fig. 5(d), the histogram of the reconstructed picture has a greater spreadthan that of the original Fig. 5(b). To overcome this discrepancy, both the original andthe reconstructed pictures are normalised to a common mean (0.5) and standard devia-tion (0.16) before providing them as inputs to the algorithm. The two normalised imagesare then passed through a Sobel rst-order differentiator to detect the magnitude and di-rection of all of the edges in each. From this data, we derive the contrast ratio (delstr)and difference in orientation (deldir) of the two pictures at each pixel position. Thislinear data is then scaled to conform to the non-linear behaviour of the HVS as denedby the psychometric function Q = K/(1 + exp−d(x+s)) where x = {delstr,deldir}, andK is a constant such that for optimal values of d and s decided by results of subjectiveevaluation [9], Q = 1. The method described above gives a measure of the perceptual in-formation that has been preserved in the reconstructed image with respect to the originalimage, both for edge strength (Qstr) and orientation (Qdir), and is shown in Fig. 6(g).The mean Q =√Qstr×Qdir is then importance-weighted with the edge-strength valuesof the original picture (Worig). This is expressed as a normalized sum to give a singleOriginal Image   Reconstructed    ImageQvalueNormalise to mean 0.5         andstandard deviation 0.16Sobel OperatorEdge StrengthEdge OrientationReconstructedEdge StrengthEdge OrientationOriginal Ratio Differencedel_strdel_dir   Human   Visual    System Non-LinearityQdirQstrQ = sqrt(Qdir X Qstr)Importance weighted withrespect to the Edge Strengthmatrix of the original imageFig. 4. The Edge Preservation Estimation Algorithm(a) (b) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10100200300400500600700800900(c) (d) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10100200300400500600700800(e) (f) (g) (h)Fig. 5. (a) Original and (c) Reconstructed images along with (b), (d) their respective histograms.(e), (g) Edge Magnitude and (f), (h) Edge Orientation of the original and reconstructed imagesrespectively.measure Qvalue = ∑(Worig×Q)/∑Worig for the performance. We apply the algorithm il-lustrated in Fig. 4 to the images shown in Fig. 6(a)(e) and their reconstructions. Plots ofthe information build-up (Qvalue) with the arrival of each spike are shown in Fig. 6(f). Itis observed that, on average, the reconstruction using the LUT retrieves upto 75% of theinformation contained in the original image. Again, of the total information retrievedby the algorithm, more than 90% is obtained by the time 15%  20% of the neuronshave red their rst spikes .4 ConclusionIn this work we rst discussed our replication of Thorpe’s work of simulating the visualprocessing in the retina using rank-order codes. We then wanted a means to measure(a) (b) (c) (d) (e)(f)0 0.5 1 1.5 2 2.5 3 3.5x 10400.10.20.30.40.50.60.70.80.91Rank of the coefficientsPerceptual Edge Information recovered during stimulus reconstructionimage(a)image(b)image(c)image(d)image(e)(g)0 0.2 0.4 0.6 0.8 100.20.40.60.81Percentage variation in edge strength and orientationPerceptual Edge InformationQstrQdirFig. 6. (a)–(e): Five images for which the rate of information retrieval from their rank-order en-coding is plotted in (f); (g) Perceptual Edge Preservation measure.the performance of the rank-order codes in terms of the preservation of the perceptu-ally important information of the original image. For this, we used the objective edgepreservation estimation algorithm devised by Petrovic et al. Our results indicate thatmore than 90% of the information that the model is capable of coding can be retrievedvery early on in the process, by the time only 20% of the neurons have red their rstspikes.References1. Adrian, E.D.: Basis of sensation. Hafner Publishing Company, London (1928)2. Thorpe, S., Imbert, M.: Biological constraints on connectionist models. In: Connectionismin perspective, 63-92, Elsevier, Amsterdam (1989)3. Thorpe, S., Fize, D., Marlot, C.: Speed of processing in human visual system. Nature 381,520–522 (1996)4. Fabre-Thorpe, M., Delorme, A., Merlot, C., Thorpe, S.: A limit to the speed of processing inultra-rapid visual categorisation of novel scenes. J. Cog. Neurosc. 13, 171–180 (2001)5. Thorpe, S., Gautrais, J.: Rapid visual processing using spike asynchrony. In: Advances inneural information processing systems, 901–907 (1997)6. Gautrais, J., Thorpe, S.: Rate coding versus temporal order coding: a theoretical approach.Biosystems 48, 57–65 (1998)7. Thorpe, S., Gautrais, J.: Rank order coding. In: Computational Neuroscience: Trends in re-search, 113–119 (1998)8. Van Rullen, R., Thorpe, S.: Rate coding versus temporal order coding: what the retinal gan-glion cells tell the visual cortex. Neural Comp. 13, 1255–1283 (2001)9. Petrovic, V., Xydeas, C.: Objective evaluation of signal-level image fusion performance.Optical Engineering 44(8) (2005)10. Field, D.J.: What is the goal of sensory coding? Neural Comp. 6, 559–601 (1994)

Basis of sensation.

Biological constraints on connectionist models. In: Connectionism in perspective, 63-92, Elsevier,

Rank order coding. In: Computational Neuroscience: Trends in research,

Rapid visual processing using spike asynchrony. In: Advances in neural information processing systems,

Rate coding versus temporal order coding: a theoretical approach.

Rate coding versus temporal order coding: what the retinal ganglion cells tell the visual cortex.

S.: A limit to the speed of processing in ultra-rapid visual categorisation of novel scenes.

Speed of processing in human visual system.

What is the goal of sensory coding?

Xydeas, C.: Objective evaluation of signal-level image fusion performance.

Information recovery from rank-order encoded images

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.108.6269

Information recovery from rank-order encoded images

Abstract

Similar works

Full text

Available Versions

University of Lincoln Institutional Repository