Skeleton tracking is a useful and popular application

of Kinect. However, it cannot provide accurate reconstructions

for complex motions, especially in the presence of occlusion. This

paper proposes a new 3-D motion recovery method based on lowrank

matrix analysis to correct invalid or corrupted motions.

We address this problem by representing a motion sequence as

a matrix, and introducing a convex low-rank matrix recovery

model, which fixes erroneous entries and finds the correct

low-rank matrix by minimizing nuclear norm and `1-norm

of constituent clean motion and error matrices. Experimental

results show that our method recovers the corrupted skeleton

joints, achieving accurate and smooth reconstructions even for

complicated motions

Jingyu, Yang

Lai, Yukun

Li, Kun

Wang, Meiyuan

Wu, Feng

Online Research @ Cardiff

3-D Motion Recovery via Low Rank MatrixAnalysisMeiyuan Wang 1, Kun Li 1∗, Feng Wu 2, Yu-Kun Lai 3, Jingyu Yang 11 Tianjin University, Tianjin, China2University of Science and Technology of China, Hefei, China3Cardiff University, Wales, UK1∗ Corresponding author: lik@tju.edu.cnAbstract—Skeleton tracking is a useful and popular applicationof Kinect. However, it cannot provide accurate reconstructionsfor complex motions, especially in the presence of occlusion. Thispaper proposes a new 3-D motion recovery method based on low-rank matrix analysis to correct invalid or corrupted motions.We address this problem by representing a motion sequence asa matrix, and introducing a convex low-rank matrix recoverymodel, which fixes erroneous entries and finds the correctlow-rank matrix by minimizing nuclear norm and `1-normof constituent clean motion and error matrices. Experimentalresults show that our method recovers the corrupted skeletonjoints, achieving accurate and smooth reconstructions even forcomplicated motions.Index Terms—Skeleton, Kinect, occlusion, motion recovery, lowrankI. INTRODUCTION3-D analysis of human motions has always been an ac-tive research topic in computer graphics and computer vi-sion, involving a variety of research problems such as 3-D reconstruction [1], pose estimation [2], [3], [4], motioncapture [5], [6], skeleton tracking [7], 3D model deformation[8], etc. Previous motion capture systems are challenging toimplement, because they are expensive, difficult to maintain,and in need of abundant manual operations. Microsoft Kinecthas shed a light on low-cost human motion capture. With itsadvantages of being convenient and inexpensive [9], Kinectgains its popularity in numerous works [10], and therefore hasbeen widely used in the motion capture field [11]. However,skeletons captured by Kinect suffer from severe joint driftingand motion jitter, especially in the case of self-occlusion orobject occlusion [12]. The accuracy of joint estimation is moresatisfactory in controlled scenarios with simple non-occludedmotions, such as standing upright, walking forward, whichlimits its wide applicability.It is of great interest to researchers to reconstruct humanmotion via different methods. Many work focus on humanmotion estimation from RGB images. Menier et al. [13]estimate skeletal poses from foreground silhouettes. Li et al.IEEE VCIP’16, Nov. 27 - Nov. 30, 2016, Chengdu, China[14] introduce sparse representation to estimate 3-D posesand camera motions. With the rapid development of neuralnetworks and deep learning, some works bring deep learningmodels into motion estimation. Toshev et al. [3] proposea Deep Neural Networks (DNNs) to estimate human posesfrom RGB images by formulating it as a joint regressionproblem. Ouyang et al. [4] present a deep model to solvethe pose estimation problem by utilizing information sourcesof mixture type, appearance score and deformation. Sincethe advent of depth cameras such as Kinect, pose estimationcan be better addressed through depth images. Shotton et al.[15] introduce body part classification (BPC) and offset jointregression (OJR) algorithms to estimate human poses withrobustness and efficiency from a single depth image. Wei et al.[5] develop an automatic motion capture system by integratingdepth data, full-body geometry, etc. together using a singledepth camera. However, these methods have limitations inrecovering skeletons with occlusions. Saito et al. [7] solvethis problem by finding subspace of valid motion, projectingcorrupted skeletons into the motion manifold and finallyrebuilding valid motion through inverse projection. However,this method needs a time-consuming training procedure.3-D motion recovery from corrupted skeletons is challeng-ing. The key is to impose proper priors to make the problemwell-posed. As skeleton sequence has high temporal correla-tion, the motion should lie in a low-dimensional subspace. Inthis paper, we propose a new 3-D motion recovery methodbased on low rank matrix analysis. The skeleton sequenceis rearranged into a (corrupted) matrix that contains errorsand noise. Then, the clean motion matrix is recovered fromthe corrupted matrix by minimizing the nuclear norm of theclean matrix and the `1-norm of the error matrix, exploitingthe characteristics of both matrices. Through this method, thecorrupted matrix that has high percentages of noise and errorscan be corrected. Experimental results show that our methodsuccessfully corrects the corrupted motion captured by Kinectv2.0, especially for complex motions.The remainder of this paper is organized as follows. SectionII gives the proposed 3-D motion recovery method. Validationexperiments are presented in Section III, and the paper isconcluded in Section IV.II. THE PROPOSED METHODA. Matrix Recovery ModelThe human skeleton is represented by a collection ofjoints, which are easily influenced by noises and have driftingproblems. Given the skeleton sequence captured by Kinectv2.0, an observation matrix D ∈ Rm×n is formed by stackingthe 3-D positions of all the joints together, where m is 3×the number of frames of the input skeleton sequence and nis the number of joints (21 in our case, ignoring the fingerjoints). Let A ∈ Rm×n be the recovered clean matrix, and letE ∈ Rm×n be the error matrix. We haveD = A+ E. (1)Classical works in the literature suggest that human motionslie in a subspace therefore can be effectively represented inlower dimensions [16]. This inspires us to focus on the timecoherence of skeleton data in a motion sequence–the rank ofthe clean matrix A should be low. To verify this, as shownin Fig.1, the rank of our input skeleton matrix for Dancingsequence is 5 using singular value decomposition, validatingour low-rank hypothesis, which forms the basis of our matrixrecovery model. 0 5 10 15 20 25050100150200250300350400diagonal matrix SFig. 1. Singular value decomposition of input skeleton matrix for sequencedancing.Given that the latent matrix A is low-rank and the errormatrix E is sparse, the problem can be formulated asmin rank(A) + γ ‖E‖0s.t. D = A+ E,(2)where rank(A) is the rank of matrix A, ‖E‖0 is the `0 norm ofmatrix E, i.e. the number of non-zero entries in the matrix andγ > 0 is a weighting parameter that balance the importanceof low-rank matrix A and sparsity of matrix E.Since the problem in Eq.(2) is NP-hard, we use the nuclearnorm of A as an approximate substitute for rank(A), and the`1 norm as an approximate substitute for `0 norm of matrixE [17]. Thus we obtain a semidefinite programing problem:min ‖A‖∗ + λ ‖E‖1s.t. D = A+ E,(3)where λ > 0 is a weighting parameter, ‖E‖1 =∑ij |Eij |,‖A‖∗ = tr((AAT)1/2) =∑i σi, where σi’s are singularvalues of matrix A. Theoretical considerations in [18] suggestthat λ must be of the form C√max(m,n), where C is a constant,typically set to unity. Additionally, we evaluate the influence ofthe weighting parameter λ in our experiments, which achievesthe balance between accuracy and smoothness of the recoveredskeletons. The value of λ is chosen small enough to thresholdaway the noise (by keeping the variance low to obtain highstability), and large enough not to overshrink the originalmatrix (by keeping the bias low to ensure flexible motion).B. Augmented Lagrangian AlgorithmAlgorithm 1 :ALM Algorithm1: Input: observed skeleton matrix D ∈ Rm×n2: Initialize: E0 = 0, Y0 = 0, µ > 0 maxIter = 10003: while not converged do4: Ak + 1 =M1/µ(D − Ek + 1µYk);5: Ek + 1 = Sλ/µ(D −Ak + 1 + 1µYk);6: Yk + 1 = Yk + µ(D −Ak + 1 − Ek + 1);7: µk+1 = ρµk, ρ > 1;8: end while9: Output: A, EThere are many algorithms to solve this minimization prob-lem, e.g.,singular value thresholding (SVT) [19], augmentedlagrangian method (ALM) [20], and accelerated proximategradient algorithm (APG) [21]. In this paper, we chooseaugmented lagrangian method (ALM) in an iterative frame-work, due to its high efficiency and accuracy. The augmentedLagrangian of Eq.(3) isL(A,E, Y, µ) = ‖A‖∗ + λ ‖E‖1 + 〈Y,D −A− E〉+µ2‖D −A− E‖2F ,(4)where ‖M‖F represents the Frobenious norm of a matrixM , Y is the Lagrangian multiplier, and 〈·, ·〉 denotes theinner product of two matrices considered as long vectors.Then, we solve two optimizations: minA L(A,E, Y ) andminE L(A,E, Y ) respectively.arg minAL(A,E, Y ) = Sλ/µ(D −A+1µY ) (5)arg minEL(A,E, Y ) = M1/µ(D − E +1µY ), (6)where Sδ(x) = sgn(x) max(|x| − δ, 0) denotes a shrinkageoperator. Similarly, Mδ(X) = USδ(Λ)V denotes a singularvalue thresholding operator.To address these minimization problems, we use a classicalapproach: First we minimize function L(A,E, Y ) with Afixed, then we similarly minimize L(A,E, Y ) with E fixed,and finally we update the Lagrange multiplier matrix Y . TheALM algorithm is summarized in Algorithm 1.III. EXPERIMENTAL RESULTSWe use real captured Kinect skeletons of 21-joints as ourinput D ∈ Rm×n, which suffers from severe joint driftingand motion jitter. Four motion sequences are tested: markingtime (463 frames), crossing and bending (1245 frames), facingaside (421 frames) and dancing (1432 frames). Detailed demovideo is presented in the supplementary material.First, we test the proposed method on a simple non-occludedmotion sequence: marking time (the first sequence in ourdemo video). As shown in Fig.2 and the video, the inputskeleton has joint drifting in some frames where the personlifts a leg or waves arms. The recovered skeleton has detectedthese disturbances and rectified the noisy skeleton as shownin Fig.2 (c). In the sequence of crossing and bending (thesecond sequence in our demo video), human motions are morecomplicated and have higher percentages of self-occlusion. Asshown in Fig.3 , the Kinect skeleton is obviously corrupted,while our method reconstructs a valid skeleton structure.(a)                        (b)                   (c)                                                                          Fig. 2. Estimated skeleton of frame 171 in marking-time sequence. (a)captured color image, (b) Kinect skeleton, (c) recovered skeleton by ourmethod.(a)                      (b)                     (c)                                                   Fig. 3. Estimated skeleton of frame 1178 in crossing and bending sequence.(a) captured color image, (b) Kinect skeleton, (c) recovered skeleton by ourmethod.Then, we evaluate our method on more complex motionsequences containing various human movements and high per-centages of self-occlusion: facing aside and dancing (the thirdand fourth sequence in the demo video). Note that in Fig.4, inorder to present skeletons more legibly, we additionally renderthe skeleton in a side view, since the outstretched arms tendto be elusive in a single frame in the front view. From theside view, we can more obviously observe that the occluded                                                            (a)             (b)            (c)            (d)          (e) (f)               (g)          (h)              (i)         (j) (k)                (l)           (m)            (n)          (o) Fig. 4. Estimated skeleton of frame 91, 104 and 148 in facing aside sequence.Frame 104: (a)(f)(k) captured color image, (b)(g)(l) Kinect skeleton in thefront view, (c)(h)(m) Kinect skeleton in the side view, (d)(i)(n) recoveredskeleton by our method in the front view, (e)(j)(o) recovered skeleton by ourmethod in the side view.arm is corrected estimated. The captured Kinect skeleton hasbeen corrupted largely as shown in Fig.4(b)(c)(g)(h)(l)(m)and Fig.5(d)-(f). This shows that our method gets reasonableskeletons for these complex motions.We test the running time of every input sequence on adesktop with an Intel Core i5-4690K CPU and 8GB RAM.The results are reported in Table I. Our method is fast and hasthe potential to achieve real-time online skeleton recovery.TABLE IRUNNING TIMESkeleton Sequence Number of Frames Running Time(s)Facing Aside 241 0.6190Marking Time 463 0.8001Crossing and Bending 1245 1.1470Dancing 1423 1.3740(a)                    (b)                   (c)  (d)                     (e)                   (f)  (g)                     (h)                   (i)                                                                     Fig. 5. Estimated skeleton of frame 141, 166 and 244 in dancnig sequence.(a) - (c) captured color image, (d) - (f) Kinect skeleton, (g) - (i) recoveredskeleton by our method.IV. CONCLUSIONThis paper proposes a novel approach to motion reconstruc-tion and skeleton recovery via low-rank matrix analysis. Weexploit the time coherence in a skeleton sequence which is for-mulated as a low-rank configuration in a mathematical model.Matrix recovery method shows its efficiency in addressing theskeleton recovery problem. An ALM algorithm is devised tosolve the optimization problem. We evaluate our method onreal skeletons captured by Kinect v2.0, which contain severeerrors. Experimental results show that our method accuratelyrecovers high quality skeletons from the invalid corruptedmotion data in high efficiency.Similar to the algorithm of Kinect, our method does notguarantee the invariance of bone lengths. One can obtain themotion capture result with changeless bone length using anIK (inverse kinematics) algorithm based on our recoveredskeletons [22]. In the future work, we will perform the low-rank matrix recovery on a graph to address this problem.V. ACKNOWLEDGEMENTSThis work was supported in part by the National NaturalScience Foundation of China (Grant 61571322, 61302059 and61372084).REFERENCES[1] D. Alexiadis, D. Zarpalas, and P. Daras, “Fast and smooth 3d recon-struction using multiple rgb-depth sensors,” in Visual Communicationsand Image Processing(VCIP),2014 IEEE, 2014, pp. 173–176.[2] C. P. Chen, Y. T. Chen, P. H. Lee, Y. P. Tsai, and S. Lei, “Real-timehand tracking on depth images,” in Visual Communications and ImageProcessing (VCIP), 2011 IEEE, 2011, pp. 1–4.[3] A. Toshev and C. Szegedy, “Deeppose: Human pose estimation via deepneural networks,” in Proc. IEEE Conference on Computer Vision andPattern Recognition (CVPR), 2013, pp. 1653–1660.[4] W. Ouyang, X. Chu, and X. Wang, “Multi-source deep learning forhuman pose estimation,” in IEEE Conference on Computer Vision andPattern Recognition, 2014, pp. 2337–2344.[5] X. Wei, P. Zhang, and J. Chai, “Accurate realtime full-body motioncapture using a single depth camera,” ACM Transactions on Graphics,vol. 31, no. 6, pp. 439–445, 2012.[6] C. Wang, Z. Liu, and S. C. Chan, “Superpixel-based hand gesture recog-nition with kinect depth camera,” IEEE Transactions on Multimedia,vol. 17, no. 1, pp. 29–39, 2015.[7] J. Saito, D. Holden, and T. Komura, “Learning motion manifolds withconvolutional autoencoders,” in SIGGRAPH Asia Technical Briefs, 2015.[8] M. Dou, J. Taylor, H. Fuchs, A. Fitzgibbon, and S. Izadi, “3d scanningdeformable objects with a single rgbd sensor,” in Computer Vision andPattern Recognition, 2015, pp. 493–501.[9] M. A. Livingston, J. Sebastian, Z. Ai, and J. W. Decker, “Performancemeasurements for the microsoft kinect skeleton,” in IEEE Virtual Reality,2012, pp. 119–120.[10] B. Wang, Z. Chen, and J. Chen, “Gesture recognition by using kinectskeleton tracking system,” in International Conference on IntelligentHuman-Machine Systems and Cybernetics, 2013, pp. 1119–1119.[11] D. S. Alexiadis and P. Daras, “Quaternionic signal processing techniquesfor automatic evaluation of dance performances from mocap data,” IEEETransactions on Multimedia, vol. 16, no. 5, pp. 1391–1406, 2014.[12] S. Obdrzalek, G. Kurillo, F. Ofli, R. Bajcsy, E. Seto, H. Jimison,and M. Pavel, “Accuracy and robustness of kinect pose estimation inthe context of coaching of elderly population.” in Proc. InternationalConference of the IEEE Engineering in Medicine and Biology Society,2012, pp. 1188–93.[13] C. Menier, E. Boyer, and B. Raffin, “3d skeleton-based body poserecovery,” in Proc. International Symposium on 3D Data ProcessingVisualization and Transmission, 2006, pp. 389–396.[14] K. Li, J. Yang, and J. Jiang, “Nonrigid structure from motion via sparserepresentation.” IEEE Transactions on Cybernetics, vol. 45, no. 8, pp.1401–1413, 2015.[15] J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio,R. Moore, P. Kohli, A. Criminisi, and A. Kipman, “Efficient human poseestimation from single depth images,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 35, no. 12, pp. 2821–2840, 2013.[16] J. Chai and J. K. Hodgins, “Performance animation from low-dimensional control signals,” ACM Transactions on Graphics, vol. 24,no. 3, pp. 686–696, 2005.[17] E. J. Cands, X. Li, Y. Ma, and J. Wright, “Robust principal componentanalysis,” Journal of the ACM, vol. 1, no. 1, pp. 1–73, 2000.[18] J. Wright, A. Ganesh, S. Rao, and Y. Ma, “Robust principal componentanalysis: Exact recovery of corrupted low-rank matrices via convexoptimization. submitted to the,” Journal of the ACM, 2009.[19] J. F. Cai, Cand, E. J. S, and Z. Shen, “A singular value thresholdingalgorithm for matrix completion,” Siam Journal on Optimization, vol. 20,no. 4, pp. 1956–1982, 2010.[20] Z. Lin, M. Chen, and Y. Ma, “The augmented lagrange multiplier methodfor exact recovery of corrupted low-rank matrices,” Eprint Arxiv, vol. 9,2010.[21] K. C. Toh and S. Yun, “An accelerated proximal gradient algorithm fornuclear norm regularized linear least squares problems,” Pacific Journalof Optimization, vol. 6, no. 615-640, p. 15, 2010.[22] P. Baerlocher and R. Boulic, “An inverse kinematics architecture en-forcing an arbitrary number of strict priority levels,” Visual Computer,vol. 20, no. 6, pp. 402–417, 2004.

3-D motion recovery via low rank matrix analysis

https://orca.cardiff.ac.uk/id/eprint/94061/1/VCIPMY_v5.pdf

3-D motion recovery via low rank matrix analysis

Abstract

Similar works

Full text

Available Versions

Online Research @ Cardiff