In this paper, we first briefly introduce the multidimensional Principal Component Analysis (PCA) techniques, and then amend our previous N-dimensional PCA (ND-PCA) scheme by introducing multidirectional decomposition into ND-PCA implementation. For the case of high dimensionality, PCA technique is usually extended to an arbitrary n-dimensional space by the Higher-Order Singular Value Decomposition (HO-SVD) technique. Due to the size of tensor, HO-SVD implementation usually leads to a huge matrix along some direction of tensor, which is always beyond the capacity of an ordinary PC. The novelty of this paper is to amend our previous ND-PCA scheme to deal with this challenge and further prove that the revised ND-PCA scheme can provide a near optimal linear solution under the given error bound. To evaluate the numerical property of the revised ND-PCA scheme, experiments are performed on a set of 3D volume datasets

Yu, Hongchuan

English

Bournemouth University Research Online

  1 N-Dimensional Principal Component Analysis Hongchuan YU and Jian J. ZHANG NCCA, Bournemouth University  Abstract  In this paper, we first briefly introduce the multidimensional Principal Component Analysis (PCA) techniques, and then amend our previous N-dimensional PCA (ND-PCA) scheme by introducing multidirectional decomposition into ND-PCA implementation. For the case of high dimensionality, PCA technique is usually extended to an arbitrary n-dimensional space by the Higher-Order Singular Value Decomposition (HO-SVD) technique. Due to the size of tensor, HO-SVD implementation usually leads to a huge matrix along some direction of tensor, which is always beyond the capacity of an ordinary PC. The novelty of this paper is to amend our previous ND-PCA scheme to deal with this challenge and further prove that the revised ND-PCA scheme can provide a near optimal linear solution under the given error bound. To evaluate the numerical property of the revised ND-PCA scheme, experiments are performed on a set of 3D volume datasets. 1 Introduction Principal component analysis (PCA) is a classical statistic technique that has been applied to many fields. Nevertheless, it can be noted that in the classical PCA the 2D data sample (e.g. image) must be initially converted to a 1D vector form. The resulting sample vector will lead to a high dimensional vector space. It is consequently difficult to evaluate the covariance matrix accurately when the sample vector is very long and the number of training samples is small. Furthermore, it can also be noted that the projection of a sample on each principal orthogonal vector is a scalar. This causes the sample data to be over-compressed. In order to overcome this kind of dimensionality problems, [5] and [6] separately proposed their individual PCA schemes for 2D case. For the case of high dimensionality, the higher order SVD (HO-SVD) has been applied to face recognition [2,8]. They both employed a higher order tensor form associated with people, view, illumination, and expression dimensions and applied the HO-SVD to it for face recognition. We formulated them into the N-Dimensional PCA scheme in [1]. However, the presented ND-PCA scheme still adopted the classical single directional decomposition. Besides, due to the size of tensor, HO-SVD implementation usually leads to a huge matrix along some dimension of tensor, which is always beyond the capacity of an ordinary PC. In [2,8], they all employed small sized intensity images or feature vectors and a limited number of viewpoints, facial expressions and illumination changes in their “tensorface”, so as to avoid this numerical challenge in HO-SVD computation. Motivated by the above-mentioned works, in this paper we will reformulate our ND-PCA scheme by introducing the multidirectional decomposition for a near optimal solution of the low rank approximation and overcome the above-mentioned numerical problems. 2 Overview of Multidimensional PCA Techniques [5] firstly presented a 2D-PCA scheme by using the single dimensional decomposition technique for 2D case. It has been noted that 2D-PCA only considers between column (or row) correlations [4]. In order to improve the accuracy of the low rank approximation,     2 [6] presented a 2D-SVD scheme, that is, SVD is applied respectively to two covariance matrices as follows, ( )( )( ) ( )T Ti i FiT Ti i GiF X X X X U UG X X X X V V         , (1) where 2iX R  denotes a sample, X  denotes the mean of a set of samples, and ,F G   denote the diagonal matrices of the eigenvalues respectively. Let kU  contains the first k principal eigenvectors of F and sV  contains the first s principal eigenvectors of G. The low-rank approximation of X can be expressed as, ˆ( )Tk sTk sX U MV XM U X X V   . It is clear that the 2D-SVD employs the 2-directional decomposition, i.e. both kU  and sV  appear in Xˆ , while the 2D-PCA only employs the classical single directional decomposition. It is proven that the 2D-SVD can obtain a near-optimal solution compared to the 2D-PCA in [6]. For the case of high dimensionality, we presented the ND-PCA scheme in [1], in which a difference tensor was used instead of the covariance tensor as follows,  1( ),...,( )MD X X X X   ,  (2) where 1 ... ...i NI I IiX R    and 1 ... ...i NI MI ID R    , i.e. N-order tensors ( ), 1,...,mX X m M   are stacked along the ith dimension in the tensor D. Furthermore, applying HO-SVD to D generate n-mode singular vectors contained in ( ) , 1,...,nU n N . (For HO-SVD computation, refer to [7] please.) Accordingly, our ND-PCA scheme in [1] is expressed as, ( )( )ˆ( )nn n kn Tn n kX Y U XY X X U     ,  (3) where ( )nkU  denotes the matrix of n-mode k principal vectors. It can be noted that the proposed ND-PCA scheme still adopted the classical single directional decomposition, i.e. only ( )nkU  is used in Xˆ . However, unfolding a tensor along some dimensions in the HO-SVD implementation usually leads to a huge matrix, which is always beyond the capacity of an ordinary PC, such as, unfolding D of Eq.(2) along the 1st dimension will generate a matrix of size 1 2 1( ... ... )i i NI I MI I I       in the HO-SVD computation. The size of the unfolded matrix depends upon the number of samples M and the sample size 1( ... )NI I  . 3 Reformulating ND-PCA Scheme Introducing the multidirectional decomposition to Eq.(3) yield, 11(1) ( )1 2(1) ( )1 2ˆ ...( ) ...NNNk N kT N Tk N kX Y U U XY X X U U         , (4) where ( )iikU  denotes the matrix of i-mode ik  principal vectors, i = 1,…N. The main challenge is that unfolding the tensor D of Eq.(2) in HO-SVD usually generates an overly large matrix. First, we consider the case of unfolding D of Eq.(2) along the ith dimension, which generates a matrix of size 1 1 1( ... ... )i i N iMI I I I I       . We prefer a unitary matrix ( )iU  of size i iI I  to that of the size i iMI MI . This can be achieved by reshaping the unfolded matrix as follows. Let jA  be a 1 1 1( ... ... )i i N iI I I I I        matrix and j=1,…M. The unfolded matrix is expressed as  1 ,...,TT TMA A A . Reshaping A into a 1 1 1( ... ... )i i N iI M I I I I        matrix  1ˆ ,..., MA A A , one can obtain an unitary matrix ( )iU  of size i iI I  by SVD. Then, consider the generic case. Since the size of each dimension 1,...,NI I  may be very large, this still leads to an overly large matrix along some dimension of sample X. Without loss of generality, we assume that the sizes of dimensions of sample X are independent of each other. Now, this numerical problem can be rephrased as follows, for a large sized matrix, how to carry out SVD decomposition. It is straightforward to apply matrix     3 partitioning approach to the large matrix. As a start point, we first provide the following lemma. Lemma: For any matrix n mM R  , if each column iM  of M, 1( ,..., )mM M M , maintains its own singular value i , i.e. 2( ,0,...,0)T Ti i i i iM M U diag U , while the singular values of M are 1 min( , ),...,m ns s , i.e. 1 min( , )( ,..., ) Tm nM Vdiag s s U , then min( , ) min( , )2 21 1m n m ni ii is   . Proof: Let n > m. Because,    21 12 21 1 1,..., ( ,..., ) ,...,m mT T Ti i i i ii iTm m mMM M M u uu u diag u u    , where iu  is the first column of each iU , while the SVD of TMM  is, 2 2 211( ,..., ,0,...,0)mT T Tm i i iiMM Vdiag s s V v s v  , where iv  is the ith column of V, we thus have, 2 2( )m mTi ii itr MM s   ,            End of proof. The lemma implies that each column of M corresponds to its own singular value. Moreover, let Mi be a submatrix instead of column vector, n riM R  . We have 2 21( ,... ,...,0)T Ti i i i ri iM M U diag s s U . It can be noted that there are more than one non-zero singular values 1... 0i ris s   . If we let ( ) 1Ti irank M M  , the approximation of Ti iM M  can be written as 21( ,0,...,0)T Ti i i i iM M U diag s U . In terms of the lemma, we can also approximate it as 21 1 1 1 1T T Ti i i i i i iM M M M u u  , where 1iM  is a column of Mi corresponding to the biggest singular value 1i  of column vector. On this basis, 1iM  is regarded as the principal column vector of the submatrix Mi. We can rearrange the matrix M by sorting these singular values { }i  and partition it into 2 block submatrices 1 2ˆ ( , )M M M  (assume m ≥ n below), so that 1M  contains the columns corresponding to the first k biggest singular values while 2M  contains others. Note that Mˆ  is different from the original M because of a column permutation (denoted as Permute). Applying SVD to each iM  respectively yields,   1 11 22 2ˆ ,TTVM U UV        . (5) Thus, matrix Mˆ  can be approximated as follows,   1 11 22ˆ ˆ ,0TTVM M U UV         . (6) In order to obtain the approximation of M, the inverse permutation of Permute needs to be carried out on the row-wise orthogonal matrix of 12TTVV    given in Eq.(6). The resulting matrix is the approximation of the original matrix M. The desired principal eigenvectors are therefore included in the matrix of 1U . Now, we can re-write our ND-PCA scheme as, 11(1) ( ) ( )1(1) ( )1( )ˆ ... ...( ) ... is from Eq.(6)i NNii Nk i k N kT N Tk N kikX Y U U U XY X X U UU        . (7) For comparison, the similarity metric can adopt the Frobenius-norms between the reconstructions of two samples X and X   as follows, ˆ ˆFFX X Y Y      .  (8) Furthermore, we can give out the following proposition. Proposition: Xˆ  of Eq.(7) is a near optimal approximation to sample X in a least-square sense. (For proof, please refer to appendix.) 4 Experiments     4  Fig.3. Comparison of the reconstruction through 1-mode, 3-mode and 1-mode+2-mode+3-mode principal subspace respectively. The scheme of Eq.(7) converges quicker than that of Eq.(3).  a. single direction.     b. multi-direction. Fig.5  Comparison of the reconstructions by using single directional decomposition and multidirectional composition in terms of the normalized residual errors. The proposed ND-PCA approach was performed on a 3D range database of human faces used for the Face Recognition Grand Challenge [3]. In order to establish an analogy with a 3D volume dataset or higher dimensional solid dataset, we embedded each 3D range dataset into a 3D array and mapped the pixels of the corresponding 2D face image to the voxels of the 3D array. For the sake of memory size, the reconstructed volume dataset was then re-sampled to the size of 180×180×90. The experiment is to test the quality of the reconstructed sample. Within our 3D volume dataset, we got the 1-mode, 2-mode and 3-mode singular vectors, which can span three independent orthogonal spaces respectively. Our objective is to test which manner leads to the best reconstruction quality based on these three spaces. To this end, we first compare the residual errors of reconstructions by performing Eq.(7) on the 1-mode, 3-mode and 1-mode+2-mode+3-mode principal subspaces respectively. (Note that when Eq.(7) is performed on 1-mode or 3-mode principal subspaces, it will degenerate into Eq.(3).) The residual errors and conclusion are shown in Fig.3. To (1)U  and (3)U , as their dimensions are different, the ranges of principal component numbers k are different too. If the curve of 3-mode (solid curve) is quantified to the same length of row coordinate as the curve of 1-mode (dashed line) in Fig.3, there isn't substantial difference compared to the 1-mode test, i.e. the curve of 3-mode is similar to that of 1-mode. The reconstructed results based on Eq.(3) are not affected by the difference between the different n-mode principal component subspaces. Furthermore, in the test of 1-mode+2-mode+3-mode principal component subspace, the numbers of principal components k are increased each time by 2 for both (1)U  and (2)U  while increased by 1 for (3)U , and the maximum k are set to 180 for (1)U  and (2)U  while 90 for (3)U . To compare the multidirectional decomposition with the single dimensional ones, we show the reconstructed results of the single directional decomposition (i.e. 2D-PCA and ND-PCA scheme of Eq.(3)) in Fig.5a and the multidirectional decomposition (i.e. 2D-SVD and ND-PCA scheme of Eq.(6-7)) in Fig.5b. The residual errors of reconstruction are normalized to the range of [0,1]. One can note that the multidirectional decomposition performs better than the single directional decomposition in the case of a small number of principal components. But then Fig.5a (or Fig.5b) also seems to show that 2D-PCA (or 2D-SVD) performs a     5 little better than ND-PCA scheme of Eq.(3) (or Eq.(6-7)) when only a small number of principal components are used. In our opinion, there is no visible difference in the reconstruction quality between 2D-PCA (or 2D-SVD) and ND-PCA schemes. This is because the reconstructed 3D volume dataset is a sparse 3D array (i.e. only the voxel values on the face surface are not equal to zero but all the others are equal to zero), it is therefore more sensitive to computational errors compared to a 2D still image. If the 3D volume datasets were solid, e.g. CT or MRI volume datasets, this difference between the two curves in Fig.5a (or Fig.5b) would not noticeably appear. 5 Conclusions In this paper, we amended our previous ND-PCA approach in [1] by introducing the multidimensional decomposition technique. The novelties of this paper include, 1) introducing the multidirectional decomposition and overcoming the numerical difficulty of large matrix SVD decomposition; 2) giving out the estimation of error bound. The experimental results indicate that the revised ND-PCA scheme could effectively improve the accuracy of reconstruction. In future work, we will apply the ND-PCA scheme to the multimodal face data fusion and recognition and develop a practical prototypical system.  Appendix Proof. According to the property 10 of HO-SVD in [10], we assume that the n-mode rank of ( )X X  be equal to (1 )nR n N   and ˆ( )X X  be defined by discarding the smallest n-mode singular values ( ) ( )1,...,n nn nI R    for given nI  . Then, the approximation Xˆ  is a near optimal approximation of sample X. The error is bounded by Frobenius-norm as follows, 111 12(1)2 ( )21 1ˆ ...NNN NRRNi iF i I i IX X           . (A1) This means that the tensor ˆ( )X X  is in general not the best possible approximation under the given n-mode rank constraints. But under the error upper-bound of Eq.(A1), Xˆ  is a near optimal approximation of sample X. Unfolding ( )X X  along ith dimension yields a large matrix which can be partitioned into two submatrices as shown in Eq.(5), i.e.     1 11 2 1 22 2ˆ , ,TTVM M M U UV         . Let   1 11 22ˆ ,0TTVM U UV          as shown in Eq.(6). Consider the difference of Mˆ  and ˆ n mM R   as follows,   11 22 20ˆ ˆ ,TTVM M U UV        , where , , , 1,2i i im m n mn ni i iU R V R R i      . It can be noted that the 2-norm of 12TTVV    is 1, and that of 20    is 2max{ : }   . As    1 2 11 2, ,n nn n n n TIU U U I IU U     , we can note that the 2-norm of both the orthogonal matrix 1U  and 1 2n nTIU U    are 1, and that of  ,n n n nI I   is 2  because of identity matrix n nI  . Therefore, we have, 2222ˆ ˆ 2max { : }M M     ,  (A2) in a 2-norm sense. Substituting Eq.(A2) into Eq.(A1) yields the error upper-bound of Xˆ  as follows,     22 (1) (1) (1) 2 ( ) ( ) ( )2 2ˆ2 max : ... max :FN N NX X       .     6 This implies that the approximation Xˆ  of Eq.(7) is a near optimal approximation of sample X under this error upper bound.               End of proof.  References [1]. H. Yu and M. Bennamoun: 1D-PCA 2D-PCA to nD-PCA, Proc. of IEEE 18th Int’l Conf. on Pattern Recognition (2006), pp.181-184. [2]. H. Wang and N. Ahuja: Facial Expression Decomposition, Proc. of IEEE 9th Int’l Conf. on Computer Vision (2003), Vol.2. [3]. P. Phillips and P. Flynn et al.: Overview of the Face Recognition Grand Challenge, Proc. of CVPR2005, Vol.1. [4]. L. Wang and X. Wang et al.: The equivalence of two-dimensional PCA to line-based PCA, Pattern Recognition Letters, Vol.26, No.1, pp.57-60. [5]. J. Yang and D. Zhang et al.: Two-Dimensional PCA: A New Approach to Appearance-Based Face Representation and Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.26, No.1, pp.131-137. [6]. C. Ding and J. Ye: Two-dimensional Singular Value Decomposition (2DSVD) for 2D Maps and Images, Proc. of SIAM Int'l Conf. Data Mining (2005), pp:32-43. [7]. L.D. Lathauwer, B.D. Moor and J. Vandewalle: A Multilinear Singular Value Decomposition, SIAM J. on Matrix Analysis and Applications, Vol.21, No.4, pp.1253-1278. [8]. M. Vasilescu, and D. Terzopoulos: Multilinear subspace analysis of image ensembles, Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (2003), Vol.2.  

1D-PCA 2D-PCA to nD-PCA,

A Multilinear Singular Value Decomposition,

Facial Expression Decomposition,

Multilinear subspace analysis of image ensembles,

Overview of the Face Recognition Grand Challenge,

The equivalence of two-dimensional PCA to line-based PCA,

Two-Dimensional PCA: A New Approach to Appearance-Based Face Representation and Recognition.

Two-dimensional Singular Value Decomposition (2DSVD) for 2D Maps and Images,

N-Dimensional Principal Component Analysis

N-Dimensional Principal Component Analysis

Abstract

Similar works

Full text

Available Versions

Bournemouth University Research Online