Bryan, Scotney

Coleman, SA

Jing, Min

McGinnity, TM

English

Fast image processing is a key element in achieving real-time image and video analysis. The spiral addressing scheme has been an efficient tool for hexagonal image processing (HIP), whereby the image pixel indices are stored in a one-dimensional vector that enables fast processing. Unlike HIP, which requires a complex resampling scheme, we present a novel “squiral” (square spiral) image processing (SIP) framework that provides a spiral addressing scheme for direct application to standard square pixel-based images. A SIP-based non-overlapping convolution technique is developed by simulating the eye tremor phenomenon of the human visual system to accelerate computation in feature extraction. Furthermore, we deploy the proposed simulated eye tremor technique on a sequence of video frames. The preliminary results based on two action video clips demonstrate the potential of the SIP-based eye tremor model to facilitate fast video processing

Ulster University's Research Portal

BIOLOGICALLY MOTIVATED SPIRAL ARCHITECTURE FOR FAST VIDEO PROCESSINGMin Jing1∗, Sonya Coleman1, Bryan Scotney21Intelligent Systems Research Centre2School of Computing and Information EngineeringUniversity of UlsterUnited KingdomMartin McGinnitySchool of Science and TechnologyNottingham Trent UniversityUnited KingdomABSTRACTFast image processing is a key element in achieving real-timeimage and video analysis. The spiral addressing scheme [10]has been an efficient tool for hexagonal image processing(HIP), whereby the image pixel indices are stored in a one-dimensional vector that enables fast processing. Unlike HIP,which requires a complex resampling scheme, we present anovel “squiral” (square spiral) image processing (SIP) frame-work that provides a spiral addressing scheme for direct ap-plication to standard square pixel-based images. A SIP-basednon-overlapping convolution technique is developed by simu-lating the eye tremor phenomenon of the human visual systemto accelerate computation in feature extraction. Furthermore,we deploy the proposed simulated eye tremor technique on asequence of video frames. The preliminary results based ontwo action video clips demonstrate the potential of the SIP-based eye tremor model to facilitate fast video processing.Index Terms— spiral image processing, spiral address-ing, eye tremor, non-overlapping convolution, video process-ing1. INTRODUCTIONReal-time data processing is a challenging task, particularlywhen handling large-scale image and video data from socialmedia. Recently fast image processing based on a hexagonalimage processing (HIP) framework [5, 7] has attracted atten-tion [1,2,4,9]. By incorporating a spiral architecture [10], theHIP image pixel indices can be stored in a one-dimensionalvector that enables fast processing. However, the computa-tional advantages of HIP are undermined significantly by theadditional time and effort required for conversion of standardimage data to a HIP environment, as existing hardware forimage capture and display are based predominantly on tradi-tional rectangular pixels.In this work, we introduce a novel “squiral” (square spi-ral) image processing (SIP) framework that develops a novelspiral addressing scheme for standard square pixel-basedimages. Hence, unlike HIP, conversion of (standard two-dimensional) pixel indeces to the SIP addressing scheme can∗This work is supported by FP7 project SLANDAIL.be achieved easily using an existing lattice with a Cartesiancoordinate system. Further, the approach can be used toimplement efficiently existing image processing operatorsdesigned for standard rectangular pixel-based images; thereis also no need to design special image processing operators,as was the case for HIP [2].Standard feature extraction is usually executed via con-volution, where typically a gradient-based operator is appliedto a pixel and its neighbours. The traditional approach to fea-ture extraction based on overlapping convolution operatorsdoes not closely represent the human visual system. Fur-thermore, the human visual system does not process singlestatic images, but instead a series of temporal images thatare slightly off-set due to involuntary eye movements. Theuse of eye tremor, rhythmic oscillations of the eye, for imageprocessing was first exploited in [8] to mimic the human vi-sual system and potentially reduce the heavy computationalburden of standard convolution. To more closely mimic thehuman visual system, a biologically inspired framework wasproposed [9] by modelling eye tremor for HIP using the one-dimensional addressing scheme of the spiral architecture.Inspired by [9], we developed a SIP-based non-overlappingconvolution technique by simulating the eye tremor phe-nomenon to facilitate fast computation. For application, wedeploy the proposed approach to a sequence of video framesto investigate its potential for fast video processing.We first explain the spiral addressing scheme for squarepixel-based images and conversion to SIP from the standardCartesian 2D addressing scheme. We then demonstrate thedevelopment of SIP-based non-overlapping convolution. Inexperiments the SIP-based eye tremor model is applied tovideo frame sequences representing both very small (almostimperceptible) and perceptible degrees of movement betweensuccessive frames. The preliminary results demonstrate thepotential of the proposed method for efficient video process-ing.2. SPIRAL IMAGE PROCESSING2.1. Spiral Addressing SchemeThe proposed spiral scheme for square pixel-based images isinspired by the spiral addressing for HIP [10]. Similar to HIP,Fig. 1. The spiral addressing scheme for layer-2 SIP.Fig. 2. One-dimensional address values for a SIP image.the SIP image originates at the centre of a square image andspirals out using one-dimensional indexing. As illustrated inFig. 1, a layer-1 SIP cluster consists of nine pixels (ie, ninelayer-0 SIP clusters); higher order layers are generated re-cursively, i.e., a layer-2 SIP cluster consists of nine layer-1SIP clusters, comprising 81 pixels in total. The SIP imageis stored in a one-dimensional vector based on the spiral ad-dressing scheme as illustrated (in part) in Fig. 2.2.2. Conversion to SIP from a Square ImageUnlike HIP conversion which requires pre-processing using aresampling scheme to match the location and value of pointsin an original rectangular pixel-based image to location andvalue in a hexagonal lattice [7], SIP conversion can use di-rectly the original lattice of a square image. For a rectangularimage with size 𝑀 × 𝑁 , the number of SIP layers 𝜆 can befound by 𝜆 = (𝑙𝑜𝑔𝑀 + 𝑙𝑜𝑔𝑁)/𝑙𝑜𝑔9; then the length of the(one-dimensional) SIP image is 9𝜆. We can adapt the spiraladdressing scheme for HIP [10] to the SIP case, and thus aSIP address can be represented as:𝑎𝑛𝑎𝑛−1...𝑎1 =𝑛∑𝑖=1𝑎𝑖 × 10𝑖−1 (1)where 0 ≤ 𝑎𝑖 < 9.∑denotes Spiral Addition and × indi-cates Spiral Multiplication [7]. For example, the point at SIPaddress 867 can be located by accumulating the shifts corre-sponding to the locations of the addresses of 800, 60 and 7.To locate a SIP address corresponding to Cartesian coordi-nates (x,y) in a standard square image, we define the centreof the image as 𝐿(0) = (0, 0). Based on the SIP address-ing scheme, we have: L(1)=(-1,0), L(2)=(-1,1), L(3)=(0,1),L(4)=(1,1), L(5)=(1,0), L(6)=(1,-1), L(7)=(0,-1) and L(8)=(-1,-1). To locate the points in a higher SIP layer, we calculateof the shift required from the centre point to the target pointby𝐿(𝑎𝑖 × 10𝑖−1) = 3𝑖−1 × 𝐿(𝑎𝑖) (2)For example, the point L(87) can be located by 𝐿(87) =𝐿(80)+𝐿(7) = 3×𝐿(8)+𝐿(7) = 3×(−1,−1)+(0,−1) =(−3,−4). Similarly, a point at L(4536) can be located by𝐿(4536) = 𝐿(4000)+𝐿(500)+𝐿(30)+𝐿(6) = 33×𝐿(4)+32 × 𝐿(5) + 3 × 𝐿(3) + 𝐿(6) = (37, 29). Hence the pointL(4536) can be found by shifting the start point from (0,0) to(37,29). After conversion from a square image, the SIP imageis stored as a vector for further processing.3. SIP CONVOLUTION VIA EYE TREMOR3.1. Simulation of Eye TremorWhen using a standard 2D addressing scheme on an im-age, the addresses of a pixel’s neighbours can be determinedeasily. However, determining a pixel’s neighbours in a one-dimensional addressing scheme is not straightforward andrequires significant computation. Inspired by [9], we allevi-ate this computational burden by developing an eye tremorbased framework for SIP. The pixel offsets in a layer-1 eyetremor are illustrated in Fig.3 which includes nine offset im-ages. Consider 𝐼0 as a “base” SIP image; eight additionalimages 𝐼𝑗 , 𝑗 = 1, 2, ..., 8 is obtained by shifting 𝐼0 by onepixel in the image plane along the spiral addressing scheme.The “centre” of each image 𝐼𝑗 is located at a pixel within thelayer-1 neighbourhood centred at image 𝐼0. Each image isstored as a vector after being converted from the 2D imagestructure.Fig. 3. Pixel offsets for a layer-1 eye tremor using 9 images.3.2. SIP-based ConvolutionFeature detection operators are often based on first derivativeapproximations. Unlike HIP-based feature extraction, whichrequires x- and y-components of a hexagonal operator to bedesigned accordingly [2], the SIP-based approach supportsdirect convolution of standard image processing operators af-ter converting them to a SIP vector. For this study, we use So-bel (a layer-1 operator) as the example for edge detection. Fora given image 𝐼0, convolution of a Sobel operator (denoted as𝐻1) across the entire image plane is achieved by applying theoperator sparsely to each of the nine images 𝐼𝑗 , 𝑗 = 0, ..., 8and then combining the resultant outputs. Based on the eyetremor framework, for each image 𝐼𝑗 , we apply the opera-tor 𝐻1 only when centred at those pixels with spiral address0(mod 9), hence achieving non-overlapping convolution. Theconvolution of the image 𝐼𝑗 with an operator 𝐻𝜆 can be de-fined as,𝐺𝑗𝜆(𝑠0) =∑𝑠∈𝑁𝜆(𝑠0)𝐻𝜆(𝑠)× 𝐼𝑗(𝑠), (3)where ∀𝑠0 ∈ {𝑠∣𝑠 = 0(𝑚𝑜𝑑9))} and 𝑁𝜆(𝑠0) denotes the 𝜆-neighbourhood centred on the pixel with spiral address 𝑠0 inimage 𝐼𝑗 . For layer-1 operator 𝐻1, the matrix implementationof convolution with 𝐼0 in Eq. (3) can be written as:⎛⎜⎜⎜⎝𝐺01(0)𝐺01(10)...𝐺01(𝑘)⎞⎟⎟⎟⎠ =⎛⎜⎜⎜⎝𝐼0(0) 𝐼0(1) ... 𝐼0(8)𝐼0(10) 𝐼0(11) ... 𝐼0(18)............𝐼0(𝑘) 𝐼0(𝑘 + 1) ... 𝐼0(𝑘 + 8)⎞⎟⎟⎟⎠⎛⎜⎜⎜⎝𝐻1(0)𝐻1(1)...𝐻1(8)⎞⎟⎟⎟⎠(4)where 𝑘 = 0, 10, 20, 30, .... We can apply the same process tothe remaining eight images (𝐼1, ..., 𝐼8); each is an image cre-ated by shifting the origin by one pixel from 𝐼0. The overalloutcome can be obtained by assembling the values of 𝐺𝑗1 intoa vector with the following arrangement:[𝐺01(0)𝐺11(0)...𝐺81(0)𝐺01(10)𝐺11(10)...𝐺81(10)...𝐺01(𝑘)...𝐺81(𝑘)](5)This assembly within the one-dimensional vector achieves thesame outcome as standard 2D convolution. However, since allprocessing is executed in vector form, computation is signifi-cantly faster than standard convolution.4. APPLICATION TO VIDEO PROCESSINGMotivated by the real-time processing capabilities of the hu-man visual system, we consider how characteristics of thatsystem can simulated to reduce computational effort whenimplementing low-level feature extraction. As a consequenceof eye tremor, through rhythmic oscillations of the eye thehuman visual system processes series of temporal images thatare slightly offset [8]. Therefore, we use a set of similarly off-set images (video frames) that are each partially processed bynon-overlapping filters as described in Section 3. We thus ex-tend the proposed SIP approach to video processing, in whichSIP is applied to each video frame. For layer-1 eye tremor, weconsider nine consecutive video frames as nine eye tremor im-ages as shown in Fig.3, with each frame having a different off-set sequentially. For performance evaluation we compare theresults from combined nine consecutive frames by SIP tech-nique with the those obtained from the ninth frame by full(overlapping) convolution. This comparator represents theoutput from traditional convolution of the edge detection op-erator at every pixel, where full convolution would also havebeen applied to all previous frames. The difference is that intraditional convolution the operator is applied at every pixelin every frame, whereas in the SIP approach the operator isapplied at only one-ninth of the pixels in each frame.5. EXPERIMENT RESULTS5.1. Single Image Edge DetectionThe result of SIP-based edge detection for the lena image isshown in Fig. 4(b); Fig 4(a) shows the output from standard2D convolution. In Fig 4(a) the SIP region is outlined, whichcorresponds to the whole image shown in Fig 4(b). As SIPand standard convolution differ only in the approaches usedfor pixel indexing and image storage, the results of convolu-tion are the same for both methods. This can be seen in the(a) Standard 2D convolution (b) SIP convolutionFig. 4. Feature mapsfeature maps in Fig 4. The size of the SIP image is slightlysmaller because SIP is based on a square image that corre-sponds in size to the layer-4, and hence a border region be-yond that may not be processed. This is not a significant re-striction, as in most video applications the elements of inter-est are unlikely to be at the periphery of the image. Becausethe SIP image is stored as a vector, all of the computationalprocesses can be executed more efficiently than in the stan-dard 2D convolution approach as discussed in [6]. (We do notinclude the cost of SIP conversion and extra memory).5.2. Video Frame ImagesFor application to video data we deploy the eye tremor con-cept to video frame images by considering each nine consec-utive frames as eye tremor images, and we compare the edgefeature output with that obtained by application of the stan-dard full convolution to the ninth frame, as discussed in Sec-tion 4. To evaluate the performance under different degreesof motion within the video, we selected two short video clipsfrom the Weizmann [3], including two actions: “jump” and“walk”. To ensure that only small movements occur betweenconsecutive frames, we repeated each frame to increase theframe rate from 25 frames/sec to 50 frames/sec (though, ofcourse, this does not provide an accurate representation of theaction). Each video frame has a dimension of 250× 250, andwhich was then converted to a sequence of layer-5 SIP image(corresponded to a 243 × 243 image.) For each action wefurther selected two sets of nine consecutive frames as shownin Fig. 6: (a) the first set contained little visually perceptiblemovement, whilst (b) the second set contained clearly percep-tible changes between frames.The output from the first set for “jump” and “walk” areshown in Fig. 7 and Fig. 9, respectively. In each case theoutput from standard full convolution on the ninth frame areshown in (a) and those from application of SIP to frames 1-9 are shown in (b). For both “jump” and “walk” the edgefeature in (a) and (b) are visually similar. For the second setof frames, results for “jump” and “walk” are given in Fig.8and Fig.10. There are more obvious differences between theoutputs in (a) and (b), with (b) retaining a “history” of themovement across the nine frames to which SIP has been ap-plied.The results shown in Fig. 7 and Fig. 9 suggest that whenFig. 5. Selected video frame sets for action “jump: (a) the 1stset (frames 1-9); (b) the 2nd set (frames 26-34).Fig. 6. Selected video frame sets for action “walk: (a) the 1stset (frames 1-9); (b) the 2nd set (frames 26-34).movement between frames is small, the continuous, but par-tial processing of all nine video frames using SIP can provideedge feature output of a quality that is visually similar to thatprovided by full convolution on the 9th frame. When thereare greater movement, as across frames 26-34, the edge fea-ture output differs between full convolution on every frameand application of SIP (as in Fig. 8 and Fig. 10), with SIPyielding a more blurred output that represents motion acrossa sequence of nine frames. The current approach has beenbased on layer-1 eye tremor only, involving offset of just onepixel between successive video frames, and this is appropriatefor application of small operators such as Sobel. For futureextension of SIP we will consider extending the eye tremormodel to higher levels that are appropriate to operators atlarger-scale. With SIP then available over a range of layers,it is possible to implement adaptive multi-scale feature ex-traction on video data. The results presented here based onlayer-1 SIP case are promising and suggest the potential ofFig. 7. Edge detection obtained from: (a)frame 9; (b) frame1-9.Fig. 8. Edge detection obtained from: (a) frame 34; (b) frame26-34.Fig. 9. Edge detection obtained from: (a)frame 9; (b)frame1-9.Fig. 10. Edge detection obtained from: (a) frame 34; (b)frame 26-34.the proposed method for fast video processing.6. CONCLUSIONWe have introduced a novel spiral image processing frame-work for standard square pixel-based images. By incorpo-rating a spiral architecture in conjunction with eye tremor,a non-overlapping convolution is developed to facilitate fastprocessing. We have deployed the concept of eye tremor tovideo data analysis by considering a sequence of video framesas a set of eye tremor images. The preliminary results demon-strate the potential of the proposed method for fast video pro-cessing. Further extension will use higher order SIP layers forimplementation of adaptive multi-scale operators for featureextraction in real-time image and video content retrieval.7. REFERENCES[1] F. Asharindavida, N. Hundewale, S. Aljahdali,“Study onHexagonal Grid in Image Processing”, In Proc. ICIKM2012.[2] SA. Coleman, BW. Scotney and B. Gardiner,“A Biolog-ically Inspired Approach for Fast Image Processing”, InIAPR Proc. Machine Vision Applications, pp.129-132,2013.[3] L. Gorelick, M. Blank, E. Shechtman, M. Irani andRonen Basri, “Actions as Space-Time Shapes”, IEEETrans. on Pattern Analysis and Machine Intelligence,vol.29, no.12, pp. 2247-2253, 2007.[4] X. He, et al.,“An Approach to Edge Detection on a Vir-tual Hexagonal Structure”, Digital Image ComputingTechniques and Applications, pp. 340-345, 2007.[5] I. Her, “Geometric transformations on the hexagonalgrid”, IEEE Transactions on Image Processing, 4(9), pp.1213- 1222, 1995.[6] M. Jing, BW. Scotney, SA. Coleman and TM. McGin-nity, “Biologically Inspired Spiral Image Processing forSquare Images”, In Proc. IAPR MVA 2015.[7] L. Middleton and J. Sivaswamy,“Hexagonal Image Pro-cessing; A Practical Approach”, Springer 2005.[8] A. Roka, et al. “Edge Detection Model Based on Invol-untary Eye Movements of the Eye-Retina System”, ActaPolytechnica Hungarica, 4(1), pp31-46, 2007.[9] BW. Scotney, SA. Coleman and B. Gardiner, “Biologi-cally Motivated Feature Extraction Using the Spiral Ar-chitecture”, In Proc. IEEE ICIP, pp.221-224, 2011.[10] P. Sheridan, T. Hintz and D. Alexander,“Pseudo-invariant Image Transformations on a Hexagonal Lat-tice”, Image and Vision Computing, vol. 18, pp. 907-917, 2000.

BIOLOGICALLY MOTIVATED SPIRAL ARCHITECTURE FOR FAST VIDEO PROCESSING

https://pure.ulster.ac.uk/ws/files/11493747/BIOLOGICALLY-MOTIVATED-Jing.pdf

BIOLOGICALLY MOTIVATED SPIRAL ARCHITECTURE FOR FAST VIDEO PROCESSING

Abstract

Similar works

Full text

Available Versions

Ulster University's Research Portal