Argyriou, Vasileios

Doulgerakis, Matthaios

Pierscionek, Barbara K.

Tzimiropoulos, George

Kingston University Research Repository

DOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENT 1Non-Intrusive Action SymmetryMeasurement for Dementia andRehabilitationMatthaios Doulgerakis1M.Doulgerakis@kingston.ac.ukVasileios Argyriou1Vasileios.Argyriou@kingston.ac.ukGeorge Tzimiropoulos2gtzimiropoulos@lincoln.ac.ukBarbara Pierscionek1B.Pierscionek@kingston.ac.uk1Kingston UniversityLondon, UK2Lincoln UniversityLondon, UKAbstractThe purpose of this work is to quantify the symmetry in human actions, which is acommon objective in medical research. To this end, a novel non-intrusive method forquantifying action symmetry in the frequency domain is proposed. Reflectional sym-metry is estimated over time and a new metric is proposed based on dynamic symmetrywarping distance. The proposed method can be utilised to provide quantitative estimatesof symmetry in actions occurring during the examination of a patient. In order to evaluatethe proposed approach and metric, experiments were performed both with phantom datawhere ground truth is available and with real data offering quantitative and qualitativeanalysis.1 IntroductionThe purpose of this work is to quantify the symmetry in human actions, which is a commonobjective in medical research. The symmetry between the sides of the human body can be avaluable metric for many medical related procedures and applications. For example, moni-toring patients with cerebral palsy [9], in rehabilitation after a stroke [2] or amputation [10],diagnosis of Parkinson’s disease in early stages [8] and even diagnosis of numerous medicalconditions in infants [4].There are four main categories of symmetry as described in the literature [5], (see figure1). In research on human body motion analysis, the mirror-symmetry properties are exam-ined, assuming that the axis of symmetry is vertical and the differences between the left andright side motions are considered. In existing literature on human body motion analysis,symmetry has been estimated using a variety of sensors all of which are intrusive to the sub-ject. In particular, to acquire the measurements, a variety of sensors attached to the subjectshave been employed (gyroscopes [3], accelerometers [6], motion capture sensors [14]). Ad-ditionally, the placement of the sensors and the acquisition of the respective measurementsare not universal and they change according to the action or application.c© 2014. The copyright of this document resides with its authors.It may be distributed unchanged freely in print or electronic forms.2 DOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENTa) b) c) d) Figure 1: Examples of symmetry (a) rotational, (b) mirror, (c) radial and (d) circular.To overcome the aforementioned limitations a computer vision approach is proposed tomeasure symmetry in humans actions. Finding symmetry in images is a well-studied sub-ject. The symmetry detection task is performed either in the pixel [7] or in the frequencydomain [12], with the Fourier techniques, that are based on Phase Correlation and log-polartransform to offer measurements with high degree of accuracy and robustness. This ap-proach has the advantage of invariability to changes in illumination, is very fast, scale invari-ant and can estimate multiple symmetries simultaneously without additional computationalcost, [12]. Furthermore, symmetry is an important feature in computer vision and is used foraction classification [11].In this work the concept of symmetry estimation over time with applications in the mea-surement of the progress of a patient during the rehabilitation or the diagnosis of Parkinson’sand Alzheimer’s disease is introduced.A novel, non-invasive, method is proposed to quantifysymmetry in human actions using RGB or depth sequences. The technique of symmetry de-tection is extended to pairs of images. The contributions in this research are a novel methodto estimate symmetry between a pair of images in the frequency domain and extending thisapproach to the time dimension by introducing a innovative Dynamic Symmetry Warpingtechnique. The method introduced in this paper can quantify human action symmetry in avariety of cases where symmetry is an important feature for the monitoring of patients andthe diagnosis of diseases.2 MethodologyInitially, the subject is asked to perform some activities (e.g. flexion and extension of a limb)both from the left and the right side (see figure 2). The same activity is performed multipletimes and then the captured data is pre-processed to extract the action silhouettes. The aimof the proposed algorithm is to estimate the level of symmetry between two actions. Initiallysymmetry in a single image is defined using properties in the Fourier domain. Next,theapproach is extended to estimate the symmetry between a pair of images and finally temporalinformation is incorporated. As this research is aiming to identify symmetry between the leftand the right side of the body, the main symmetry of the body is assumed to lie in the verticalaxis of the image plane. The overall process is shown in figure 3.2.1 Estimation of Reflectional Symmetry in the Frequency DomainWorking in the frequency domain allows the robust estimation of translation between imagesthrough phase correlation. Furthermore, it has been shown that after converting an imageto polar coordinates the rotation can be measured as translation [13]. Exploiting this fact,the method proposed in [12] estimates the symmetry properties of images in the frequencydomain, suggesting that the magnitude of the Fourier transform of the polar representationof an image can be used to obtain the symmetry properties of the image. The best approachto achieve a high quality polar representation is to use the fast and accurate polar Fouriertransform proposed in [1]. This algorithm’s computational complexity is the same order asthe standard Cartesian Fast Fourier Transform (FFT), whilst the experimental results showDOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENT 3(a) (b) (c) Right Action Left Action Right Action Left Action (d) (e) (f) Figure 2: Examples of six symmetry actions performed by the patients in both sides.high accuracy [1].The aim is to identify reflectional symmetry in images. To this end symmetry detectionis formulated in the frequency domain which enables the use of robust correlation-basedtechniques. The key advantage of a Fourier-based formulation is that for any type of sym-metric pattern about a symmetry centre (x0,y0), the magnitude of its Fourier Transform isalso symmetric about the origin (0,0). This enables the computation of the axis of symme-try by a simple correlation in the polar domain. In particular, let denote by I a reflectionallysymmetric image and let us also denote by a0 the symmetry axis. Based on the Fourierproperties, the magnitude M of Fourier Transform of I resampled on polar coordinates (r,θ)contains the information about the degree and the axis of symmetry that can be obtained byperforming a 2D autocorrelation C, as shown below:C = F−1{M ·M∗} (1)where M is the Fourier transform of the polar magnitude, ∗ denotes the complex conjugateoperator and F−1 denotes the inverse Fourier transform.The higher the values in the output signal the higher the symmetry in the correspondingrotation, extrapolating this information allows the detection of the main axes of symmetry ofthe image as suggested in [12] and as shown can be used to quantify symmetry.2.2 Symmetry Distance MetricIn order to evaluate and quantify the level of symmetry between two given images I1 and I2, anovel metric is proposed. According to the proposed approach, the two images are combinedside by side into a larger image I, (see figure 3). Concatenating the images in this way ensuresthat the images found to be symmetrical also have a similar shape. Two dissimilar shapesmay have the same symmetry properties but with the undertaken approach the dissimilarityin shape will result in asymmetry. Additionally, the main symmetry is forced to be nearto the vertical axis so differences between the sides can be examined, which is the desiredresult.After combining the images a 2D autocorrelation is performed on the polar Fourier trans-formed magnitude of the combined image I (equation (1)). Considering the highest peak inthe output signal as the centre, dissimilarities either side suggest asymmetries between theimages. The difference between the left (CL) and the right side (CR) of the signal is obtainedby calculating their distance, as shown in equation (2):SD = ∆(CL,CR) (2)where ∆ is the Earth mover’s distance (EMD) and is calculated by the following equation:∆(CL,CR) =∑mi=1∑nj=1 di j fi j∑mi=1∑nj=1 fi j(3)4 DOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENT1 . . . . 2  . . . . m F Sequence 1 . . . . . . . . . n/2 . . . . . . . . n . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Combined Images ( ) Flip over the signal peak Flipped Signal Diﬀerence  Symmetry Distance Surface 1 . . . . . 2  . . . . m 1 . . . . . . . . . . . . . . n/2 . . . . . . . . . . . . . n For each combined image Symmetry Distance  i  Symmetry Correlation Surface in the Fourier log-polar domain R Sequence Dynamic Symmetry Warping ΣFigure 3: Each of the m×n combined images is represented in polar coordinates in the fre-quency domain and the magnitude is used for the 2D autocorrelation. The symmetry distanceis calculated as the EMD of the sides of the correlation surface. The symmetry distance sur-face is composed of the symmetry distance of each combination of images while the indexpreserves the temporal structure of the original sequences. The yellow line represents theoptimal path (wo) that minimises the global symmetry distance between the two sequences.where d denotes the ground distance between samples i and j along CL and CR and f theflow between them. According to the proposed metric, the smaller the symmetry distanceSD the higher is the symmetry between the images.2.3 Dynamic Symmetry WarpingA symmetry detection and evaluation approach over time is introduced in this section allow-ing the quantitative assessment of action symmetry. In order to find the symmetry betweentwo time series a novel approach based on the proposed symmetry distance metric is in-troduced. Dynamic Symmetry Warping (DSW) is a dynamic algorithm that computes theoptimal symmetry between two sequences, by stretching the time dimension and summingthe local symmetries of individual matched elements. The outputs of the DSW algorithm arethe optimal warping of the sequences and the minimum global symmetry distance betweenthe given series.Specifically, each image of the sequence R, of length m, is combined with each imageof the sequence F , of length n, and their symmetry distance is calculated as described inequation (2). The SD values are arranged in an m× n symmetry distance surface matrixwhere the indices of m and n describe the temporal structure of R and F respectively. Thewarping path w that minimises the global symmetry distance GW between the sequences,indicates the optimal global symmetry distance GS(R,F) = minwGw(R,F) where Gw are thepossible available paths of the accumulated symmetry distance surface:Gw =T∑t=1SD(wR(t)wF(t)/Nw) (4)where Nw is a normalisation factor andwR(t) ∈ {1...m}and wF(t) ∈ {1...n} (5)where t = 1...T , and T is the length of each path.The normalisation factor Nw can be chosen according to the needs of the application,and determines the flexibility of the path selection. The output GS is the optimal globalsymmetry difference between the image sequences after they are stretched in order to haveDOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENT 5the best possible symmetry. This metric can be used to assess symmetry in a variety ofhuman actions and to assist the monitoring of patients and the diagnosis of diseases.3 ResultsIn this section, the experiments performed and the results obtained are presented. In orderto evaluate the proposed method, a new Action Symmetry Database (ASDB) was createdthat consists of both phantom and real data. The phantom dataset is a moving dummy wherethe main attributes of the motion for each case are known, providing ground truth for eachcase. The differentiated attributes are the speed and the angle of movement; and the shapeof the limbs. Each one of the attributes is subjected to three variations: original state, 10%increment, and 20% increment, resulting in 27 different cases. The second part was capturedwith the Kinect sensor consisting of 5 subjects performing different actions, including: bodyweight shifting to each side, arm swings, leg swings, throwing a ball, kicking (see figure 2).Each person performs 6 actions and repeats each action 4 times on the right and 4 times onthe left side.The optimal global symmetry distance GS metric is used to quantitatively evaluate ouralgorithm. For the experiments, the actions are considered pre-segmented, with each subjectstarting in a specific pose, performing an action and finishing in a specific pose. The normal-isation factor that was used is N =m+n, providing a symmetric continuity constrain forcingeach image to be matched at least once, even if that means that some images are matchedmore than once. In figure 4, the results for the phantom data are displayed, indicating thatthe proposed metrics are close to ground truth and follow the same trend since the values ofour metric are increasing while we differentiate the motion attributes to induce asymmetry.The plots show the response of the system when we keep two parameters constant and in-crease the third. Also the variance is plotted to provide a more accurate representation of theperformance of the proposed metrics. All the results are summarised in table 1.Table 1: Phantom Data Estimated Mean Error and the Variance.Angle Changes Length Changes Speed Changes0% 10% 20% 0% 10% 20% 0% 10% 20%Length 0% Angle 0% Angle 0%0.336(0.014)1.462(0.001)3.03(0.008)0.336(0.014)2.227(0.005)4.686(0.001)2.345(1.839)2.433(1.763)2.471(1.740)Length 10% Angle 10% Angle 10%2.227(0.005)3.491(0.002)5.244(0.012)1.462(0.001)3.491(0.002)5.931(0.005)3.652(1.851)3.619(1.816)3.612(1.814)Length 20% Angle 20% Angle 20%4.686(0.001)5.930(0.005)7.738(0.014)3.03(0.008)5.244(0.012)7.738(0.014)5.448(1.949)5.331(1.912)5.234(1.908)Table 2: Real Data Estimated Mean Error and the Variance.Actor 1 2 3 4 5 6Symmetry6.237(1.167)5.876(1.563)7.308(1.101)8.186(2.549)5.452(1.562)9.399(3.004)Regarding the real data the obtained results are shown in table 2. The more complexaction of kicking (action 6) is scoring the highest value indicating that is performed theleast symmetrical, as expected. The results show that the overall methodology provides anaccurate non-intrusive action symmetry evaluation framework.6 DOULGERAKIS et al.: ACTION SYMMETRY MEASUREMENT0% 10% 20%012345678Angle Changes, averaged over Speed, standard deviation errorbar displayedAngle ChangesGlobal Symmetry Distance  Length:  0%, σ = 0.0612Length: 10%, σ = 0.0490Length: 20%, σ = 0.05160% 10% 20%012345678Length Changes, averaged over Speed, standard deviation errorbar displayedLength ChangesGlobal Symmetry Distance  Angle:  0%, σ = 0.0534Angle: 10%, σ = 0.0205Angle: 20%, σ = 0.08790% 10% 20%012345678Speed Changes, averaged over Length, standard deviation errorbar displayedSpeed ChangesGlobal Symmetry Distance  Angle:  0%, σ = 1.7811Angle: 10%, σ = 1.8272Angle: 20%, σ = 1.9232Figure 4: Obtained Action Symmetry Values for the phantom data using the proposedSymmetry Metric.4 DiscussionIn this paper a novel framework is proposed to detect and quantify symmetry between actionsin a non-intrusive way. The suggested method operates in the frequency domain based onthe properties of the Fourier transform. The proposed method has been quantitatively andqualitatively evaluated on the new Action Symmetry Database (ASDB) using both phantomand real samples.References[1] A. Averbuch, P. R. Coifman, D. L. Donoho, M. Elad, and M. Israeli. Fast and accurate Polar Fourier transform.Appl. Comput. Harmon. Anal., 21(2):145–167, 2006.[2] M. E. Brandstater, H. de Bruin, C. Gowland, and B. M. Clark. Hemiplegic gait: analysis of temporal variables.Archives of physical medicine and rehabilitation, 64:583–587, 1983.[3] S. M. N. A. Senanayake D. Gouwanda. Identifying gait asymmetry using gyroscopes–a cross-correlation andNormalized Symmetry Index approach. J. Biomech., 44(5):972–978, 2011.[4] P. Venuti G. Esposito. Symmetry in Infancy: Analysis of Motor Development in Autism Spectrum Disorders.Symmetry (Basel)., 1(2):215–225, 2009.[5] Y. Gong, Q. Wang, C. Yang, Y. Gao, and C. Li. Symmetry Detection for Multi-object Using Local PolarCoordinate. Computer Analysis of Images and Patterns, 5702:277–284, 2009.[6] X. Huang, J. M. Mahoney, M. M. Lewis, G. Du, S. J. Piazza, and J. P. Cusumano. Both coordination andsymmetry of arm swing are reduced in Parkinson’s disease. Gait Posture, 35(3):373–377, 2012.[7] S. Lee and Y. Liu. Curved Glide-Reflection Symmetry Detection. Pattern Analysis and Machine Intelligence,IEEE Transactions on, 34(2):266–278, 2012.[8] M. D. Lewek, R. Poole, J. Johnson, O. Halawa, and X. Huang. Arm swing magnitude and asymmetry duringgait in the early stages of Parkinson’s disease. Gait and Posture, 31:256–260, 2010.[9] W. L. Minear. Special article: A classification of cerebral palsy. Pediatrics, 18(5):841–852, 1956.[10] R. O. Andres and S. K. Stimmel. Prosthetic alignment effects on gait symmetry: A case study. Clin. Biomech.,5:88–96, 1990.[11] K. Schindler and L. van Gool. Action snippets: How many frames does human action recognition require?IEEE Conf. Comput. Vis. Pattern Recognit., pages 1–8, 2008.[12] G. Tzimiropoulos, V. Argyriou, and T. Stathaki. Symmetry detection using frequency domain motion estima-tion techniques. Acoust. Speech Signal Process ICASSP, pages 861–864, 2008.[13] G. Tzimiropoulos, V. Argyriou, and T. Stathaki. Subpixel Registration With Gradient Correlation. IEEETransactions on Image Processing, 20(6):1761–7, 2011.[14] R. A. Zifchock, I. Davis, J. Higginson, and T. Royer. The symmetry angle: a novel, robust method of quanti-fying asymmetry. Gait Posture, 27(4):622–627, 2008.

Non-Intrusive Action Symmetry Measurement for Dementia and Rehabilitation

Abstract

Similar works

Full text

Available Versions

Kingston University Research Repository