Claussen, H.

Damper, R.

Rosca, J.

English

Southampton (e-Prints Soton)

Mutual Interdependence Analysis (MIA)Heiko Claussen1,2, Justinian Rosca1,a n dR o b e r tD a m p e r 21 Siemens Corporate Research Inc.,755 College Road East, Princeton, New Jersey 08540{Heiko.Claussen,Justinian.Rosca}@siemens.com2 University of Southampton,School of Electronics and Computer Science,Southampton SO17 1BJ, UK{hc05r,rid}@ecs.soton.ac.ukAbstract. Functional Data Analysis (FDA) is used for datasets that aremore meaningfully represented in the functional form. Functional prin-cipal component analysis, for instance, is used to extract a set of func-tions of maximum variance that can represent the data. In this paper, amethod of Mutual Interdependence Analysis (MIA) is proposed that canextract an equally correlated function with a set of inputs. Formally, theMIA criterion deﬁnes the function whose mean variance of correlationswith all inputs is minimized. The meaningfulness of the MIA extractionis proven on real data. In a simple text independent speaker veriﬁcationexample, MIA is used to extract a signature function per each speaker,and results in an equal error rate of 2.9 % in the set of 168 speakers.1 IntroductionPrincipal Component Analysis (PCA) , discussed in [1] for nonfunctional and[2] for functional data, is an essential feature extraction method. Principal com-ponent functions are such that they can approximate new data with minimummean square error even if only a subset of all components is used. The prin-cipal component functions are given by the directions of maximum variance inthe projections of the data. The directions of minimum variance, or minor com-ponents, have received much less attention in the literature. Minor componentscorrespond to directions that are either orthogonal to the inputs or minimize thevariance of their projections. Hence, they represent an invariant or ”common”direction of the inputs. Further information about PCA and MCA, includingtheir probabilistic models, can be found in [3], [4], [5] and [6].[7] acknowledged that minor components are important in some signal pro-cessing applications. Examples are spectral estimation, curve and hyper-surfaceﬁtting, cognitive perception and computer vision. This work coined the nameMinor Component Analysis (MCA) to refer to neural network ﬁtting methodsthat compute minor components. The authors also suggested that MCA solvesthe Total Least Square (TLS) problem [8].Recently, a method called Extreme Component Analysis (XCA) was intro-duced that combines minor and principal components in order to represent aM.E. Davies et al. (Eds.): ICA 2007, LNCS 4666, pp. 446–453, 2007.c   Springer-Verlag Berlin Heidelberg 2007Mutual Interdependence Analysis (MIA) 447dataset ”optimally”. As discussed in [9], the ”optimal” representation, if by mi-nor or principal components, is dependent on the dataset. By freely choosingbetween minor, principal components and their combinations, datasets can berepresented with possibly a lower number of components.As previously mentioned, MCA can be used to extract minor componentsthat represent invariants of the data. However, when it is desired to extractthe mutual interdependencies of a set of input functions, MCA suﬀers becauseof the necessary preprocessing step of input data centering. In the usual case,where the input functions are linearly independent, centering reduces the spanof the data. Therefore, these centered functions can no longer fully representthe inputs. Furthermore, as discussed in [10] the TLS and therefore the MCAsolution is non trivial and can usually not be found in closed form.In this paper, we propose Mutual Interdependence Analysis (MIA) to solvea TLS-like optimization problem in the span of the original inputs in order toextract a mutually interdependent function from the input function set. We provethat in the case of linearly independent input functions, a closed form solutioncan be found that minimizes the MIA criterion.In section 2 we deﬁne the MIA problem, derive its solution and illustrate itsproperties. In the experimental section 3, MIA is used for simple text indepen-dent speaker recognition. We end the paper with conclusions and directions forfurther work.2 Mutual Interdependence Analysis (MIA)Throughout the paper, we use xT to denote the transpose of column vector x.Also, we denote X to be the matrix whose columns are xi with i =1 ,...,D.W euse 1 to represent a vector of ones, 1 to represent a matrix of ones and I to bethe identity matrix. The dimension will be clear from the context.Consider D real inputs, xi(tj)w i t hi =1 ,...,Dand j =1 ,...,N,w h e r ee a c hinput xi =[ xi(t1),...,x i(tN)]T is viewed as a single entity (i.e. has the intrinsicstructure of a function) rather than a series of individual observations. FunctionalData Analysis (FDA) normally treats data this way, therefore we refer to each xias an input. In our case, N is typically larger or much larger than D.T L Ss o l v e salinear equation XT ·s = b by ﬁnding a direction s that minimizes the squared,orthogonal distances to the data points xi:m i nsDi=1|xTi ·s−bi|2sT·s+1 . On the otherhand, the ordinary Least Square (LS) problem ﬁnds a direction s that minimizesminsDi=1 xTi · s − bi 2. While LS assumes that only vector b to contain errors,the TLS approach models uncertainties in both b a n di nt h ed a t aX. Our goalis diﬀerent: Extract a new vector (function) s that ”optimally” represents the Dinput functions. An optimal solution is a function that is maximally correlatedwith all inputs with the constraint that it is a linear combination of the inputs.We call this problem Mutual Interdependence Analysis (MIA). Formally, theoptimality criterion J(X|s), given a functional data series s,i sa sf o l l o w s :448 H. Claussen, J. Rosca, and R. DamperJ(X|s)=D i=1sT · xi −1DD k=1sT · xk2(1)Our problem is to ﬁnd the maximum likelihood vector ˆ s of norm one, in the spanof the inputs, that minimizes J(X|s):ˆ s =a r g m i ns, s =1,s=D k=1ck·xkJ(X|s)( 2 )2.1 Solution to MIALet us ﬁnd an equivalent formulation for the MIA problem in (2). Consider themean function x(m) = 1DDi=1 xi and the centered functions x(c)i = xi − x(m).I tc a nb ee a s i l ys h o w nt h a tsT · xi − 1DDk=1 sT · xk = sT · x(c)i . Hence,(2) becomes:ˆ s =a r g m i ns, s =1,s=Dk=1 ck·xk sT · X(c) 2, (3)where X(c) has as columns x(c)i with i =1 ,...,D. Then, x(m) = 1D X · 1 andX(c) =[ x1 − x(m)|...|xD − x(m)]=X − x(m) · 1T . It follows thatX(c) = X · P with P = I −1D1 . (4)Obviously,Di=1 x(c)i = 0 . Hence, the nullspace NULL(x(c)1 ,x(c)2 ,...,x(c)D )i snon trivial. All vectors s ∈NUL L (x(c)1 ,x(c)2 ,...,x(c)D ) will minimize J(X|s). Inthe next theorem, we show that the problem given by (3) has at least onesolution.Theorem 1. Assume x1,x2,...,xD are linearly independent. Then, there existss  = 0 in NULL(x(c)1 ,x(c)2 ,...,x(c)D ) such that s is in the span of the inputs xi,i =1 ,...,D.Proof. As o l u t i o ns  = 0 and c =[ c1,c 2,...,c D]T ∈ RD of the system ofequations:sT · X(c) = 0 (5)s = X · c (6)will also satisfy the theorem and solve the optimization criterion of problem(3). Indeed, (5) is equivalent to the existence of s such that s ∈N U L L (x(c)1 ,x(c)2 ,...,x(c)D ) and (6) speciﬁes that s is in the span of the inputs xi and s  = 0.Let us substitute s from (6) and X(c) from (4) in (5):⇒ cT · XT · X · P = 0 (7)Given that G =( XT · X) is a Gram matrix formed by linearly independentvectors x1,x2,...,xD, G is invertible (see theorem 7.2.10 in [11]). LetMutual Interdependence Analysis (MIA) 449cT = dT · (XT · X)−1. (8)Therefore, (7) becomes: dT · P = 0 withd = ζ 1 (9)and ζ ∈ R. When substituting (9) into (8): c = ζ (XT · X)−1 · 1. Hence:ˆ s = ζ X · (XT · X)−1 · 1. (10)Then, ˆ s ˆ s  is a solution to (6) and (5) for all ζ ∈ R . Therefore, any MIAproblem with linearly independent inputs has a solution given by (10). An al-ternative interpretation of the MIA solution ˆ s ˆ s  is that of a direction in theN-dimensional space which minimizes the variance of the projections of all pointsxi,i =1 ,...,D.    Should the inputs be translated by a constant λ ∈ R, i.e.xi ← xi − λ1 ∀i (11)then the solution of J(X|s) changes. Indeed, it can be easily proven that thecriterion (1) itself is invariant to a translation (11), however the constraintof (2) will require that the solution ˆ s is in the span of xi.2.2 Example with Synthetic DataA synthetic example is given below to compare MIA, MCA, PCA and Indepen-dent Component Analysis (ICA) (see [12]). Assume three inputs given by:x1 = f1 + f2x2 = f1 − 0.5f2x3 =2f1 +2 f3 +101where f1 =s i n ( 2ΠiN ) i =1 ,...,N ,f2 is Gaussian noise N(0,1) and f3 is Lapla-cian noise. The inputs and results of the methods are illustrated in Fig. 1. TheMIA solution closely approximates f1, in contrast to the other methods.3 Application: Text Independent Speaker VeriﬁcationIn this section, we apply MIA to the problem of extracting signatures from speechdata for the purpose of text independent speaker recognition. This problem ischallenging when we need to verify the identity of a person but can not controlthe way data is acquired (i.e. recording equipment, environment, etc.). For thisstudy, we have used the TIMIT database [13]. Data from 168 speakers waspartitioned 50-50 for training and testing. The data was preprocessed by silenceremoval and normalization of each recording. Data for a given speaker was usedas input to MIA in order to generate a speaker signature as described below.We compare the equal error rate (EER) results for speaker veriﬁcation obtainedwith MIA versus the PCA and ICA-based methods described in [14].450 H. Claussen, J. Rosca, and R. Damper0 200 400 600 800 1000−505101520Sample NumberAmplitudex1x2x3(a) Inputs xi0 500 1000−0.1−0.0500.050.10.15Sample NumberNormalized Amplitude0 500 1000−0.1−0.0500.050.10.15Sample NumberNormalized Amplitude0 500 1000−0.1−0.0500.050.10.15Sample NumberNormalized Amplitude0 500 1000−0.1−0.0500.050.10.15Sample NumberNormalized Amplitude1st IC (ICA) 1st MC (MCA)1st PC (PCA) s (MIA)(b) Results of Diﬀerent MethodsFig.1. (a) Inputs xi are linear combinations of three basis functions. (b) Signalsextracted using ICA, MCA, PCA and MIA. The MIA result, with λ = 0, is meaningful.3.1 Data ModelA speech signal can be modeled as an excitation that is convolved with a lineardynamic ﬁlter which represents the vocal tract. The excitation signal can bemodeled for voiced speech as a periodic signal and for unvoiced speech as randomnoise. It is common to analyze the voiced and unvoiced speech separately [15].In this example, only the voiced speech is used for speaker recognition. LetE(p), H(p) and V(p) be the spectral representations of the excitation, vocal tractﬁlter and the voiced signal parts of a person p respectively. Moreover, let Mrepresent speaker independent signal parts in the spectral domain (i.e. recordingequipment, environment, etc.). Therefore, the data can be modeled as:V(p) = E(p) · H(p) · M. (12)By cepstral deconvolution, the model can be represented as a linear combinationof its basis functions:x(p)i =l o gV(p) =l o gE(p) +l o gH(p) +l o gM. (13)This model suggests that we could use MIA to extract a function that repre-sents the speaker’s signature. In practice, we take speech segments of about onesecond as MIA inputs x(p)i in order to achieve spectral accuracy. An exampleof inputs x(p)i is shown in Fig. 2(a). Therefore, MIA will extract signatures thatcapture typical speaker dependent correlations in the logarithmic spectral do-main. Speaker independent signal parts M will be minimized if they are notequally present in all MIA inputs.3.2 MIA-Based Text Independent Speaker VeriﬁcationWe partition the training and testing data for each person p into D =8s e g m e n t s{x(p)i }i=1,...,D of one second. For each person, we extract a voice signature s(p)Mutual Interdependence Analysis (MIA) 451using MIA. The cosine distances between the testing data signatures and trainingdata signatures is used as a measure of similarity. A matrix that represents thecosine distances between all signatures in the database is illustrated in Fig. 2(b).Let NFA and NCA represent the number of false and correct acceptances re-spectively. NU is the number of registered users. The speaker recognition resultsare evaluated by a comparison of the false acceptance rate FA with the falserejection rate FR, calculated as follows:FA =NFAN2U − NUand FR =NU − NCANU. (14)NFA and NCA result from the entire database by testing each learned speakersignature against all test speaker signatures. This means that a number of N2Utests is performed, including NU correct combinations and N2U −NU impostors.By changing a threshold value, more people can be accepted which results ina trade oﬀ between FA and FR. The FA versus FR plot of this example isillustrated in Fig. 3(a). The equal error rate (EER), where FA equals FR, isused to compare to previous results in [14].0 2 4 6 8−20−15−10−50Frequency [kHz]Logarithmic Amplitude(a) Inputs xi in the Fourier DomainLearned Signature of Speaker Nr.Test Signature of Speaker Nr.0 50 100 150050100150(b) Similarity ScoresFig.2. MIA applied to text independent speaker recognition. (a) Representation ofinput data {x(p)i }i=1,...,8, given by speech segments of a single speaker p in the loga-rithmic Fourier domain. (b) Matrix of similarity scores between diﬀerent signatures.Bright gray stands for high and dark gray for low similarity between signatures.3.3 ResultsWe used a set of 168 speakers from the TIMIT database [13]. For each signatureextraction from the training and testing data, we used 8 seconds of voiced speech.The data was partitioned into 8 windows with non overlapping, nearly rectangu-lar windowing functions of one second lengths and Gaussian tails of 120 second.The input functions had their mean subtracted. Thereafter, each extracted sig-nature was down-sampled to 256 points. The mean signature was subtracted452 H. Claussen, J. Rosca, and R. Damper0 5 10 15 20 2505101520False Acceptance RateFalse Rejection RateSpeaker Dependent Variance ThresholdTrace Dependent, Equal Threshold(a) FA vs. FR percentage0 2 4 6 8−0.1−0.0500.050.10.15Frequency [kHz]Normalized Amplitude0 2 4 6 8−0.1−0.0500.050.10.15Frequency [kHz]Normalized Amplitude0 1 2 3 4 5 6 7 8−0.1−0.0500.050.10.15Frequency [kHz]Normalized Amplitudes  (MIA)1st MC (MCA) 1st PC (PCA)(b) Results of Diﬀerent MethodsFig.3. Results of MIA-based text independent speaker veriﬁcation. (a) False rejection(FR) versus false acceptance (FA) rate. (b) Speaker signature extracted by MIA from{x(p)i }i=1,...,8. Also plotted are the ﬁrst minor component (MCA) and the ﬁrst principalcomponent (PCA) of the data.from all signatures to focus on the evaluation of diﬀerences. The comparison ofthe MIA-based signature with the 1st minor component and the 1st principalcomponent is illustrated in Fig. 3(b). Note visually that only MIA extracts signalamplitudes in accordance with the well known result that low frequencies containmost information about a speaker. In order to alter the proportion between FAand FR, two diﬀerent thresholds were used. First, a trace dependent thresholdtreats every user the same. Secondly, the variance dependent threshold uses in-formation about the similarity between test cases to learn a speaker dependentweighting. The EER of this MIA-based text independent speaker recognitionsystem was 2.9 % for the variance dependent threshold and 5.4 % for a tracedependent threshold. For a similar experiment, [14] reports EER’s between 4.3% and 6.1 % using ICA and PCA features. Here it has to be noted that thistest was done with 462 speakers. However, one person was represented by 16 or32 features of 128 or 256 samples length. On the other hand, MIA only uses asingle signature of 256 sampes length per speaker.4C o n c l u s i o nWe proposed a novel feature extraction method, Mutual Interdependence Anal-ysis (MIA), which ﬁnds an invariant function of the input function set, repre-senting the direction of minimum variance of input projections. Intuitively, thisfunction is mutually interdependent with all inputs. The proof of the minimiza-tion problem exploits the unconstrained span of the original input data to infer aclosed form solution. Furthermore, we showed the eﬀect of input data translationby a constant value λ. In this way, one can control the degree of correlatednesswith outlier functions in the input set. Indeed, one can choose a value of λ toMutual Interdependence Analysis (MIA) 453discriminate between inputs and bias towards a result which correlates only witha subset of them. Further work will analyze the robustness of MIA to noise. More-over, the eﬀect of changes in the span constraint, or the choice of a basis functionset, will be explored.AcknowledgmentsWe would like to thank Radu Balan for his invaluable input to this paper.References1. Jolliﬀe, I.T.: Principal Component Analysis, 2nd edn. Springer, Heidelberg (2002)2. Ramsay, J.O., Silverman, B.W.: Functional Data Analysis, 2nd edn. Springer, Hei-delberg (2006)3. Oja, E.: Principal components, minor components, and linear neural networks.Neural Networks 5, 927–935 (1992)4. Tipping, M.E., Bishop, C.M.: Probabilistic principal component analysis. Journalof the Royal Statistical Society 61(3), 611–622 (1999) (Series B)5. Williams, C., Agakov, F.: Products of gaussians and probabilistic minor compo-nents analysis. Neural Computation 14(5), 1169–1182 (2002)6. Thompson, P.A.: An adaptive spectral analysis technique for unbiased frequencyestimation in the presence of white noise. In: Systems and Computers, pp. 529–533(1980)7. Xu, L., Oja, E., Suen, C.Y.: Modiﬁed hebbian learning for curve and surface ﬁtting.Neural Networks 5, 441–457 (1992)8. Moon, T.K., Stirling, W.C.: Mathematical methods and algorithms for signal pro-cessing. Prentice-Hall, Upper Saddle River, NJ (2000)9. Welling, M., Agakov, F., Williams, C.K.I.: Extreme components analysis. In:Thrun, S., Saul, L., Sch¨ olkopf, B. (eds.) Advances in Neural Information ProcessingSystems 16, MIT Press, Cambridge, MA (2004)10. Rao, Y.N., Principe, J.: Eﬃcient total least squares method for system modelingusing minor component analysis. In: Proceedings of international workshop onneural networks for signal processing, pp. 259–268 (2002)11. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, Cam-bridge (1999)12. Hyv¨ arinen, A., Karhunen, J., Oja, E.: Independent component analysis. John Wileyand Sons, Chichester (2001)13. Garofolo, J.S., Lamel, L.F., Fisher, W.M., Fiscus, J.G., Pallett, D.S., Dahlgren,N.L., Zue, V.: Timit acoustic-phonetic continuous speech corpus. CDROM (1993)14. Rosca, J., Kofmehl, A.: Cepstrum-like ica representations for text independentspeaker recognition. In: ICA, pp. 999–1004 (2003)15. Deng, L., O’Shaughnessy, D.: Speech Processing: A Dynamic and Optimization-Oriented Approach. Signal Processing and Communications. Marcel Dekker, Inc.(2003)

A.: Cepstrum-like ica representations for text independent speaker recognition.

An adaptive spectral analysis technique for unbiased frequency estimation in the presence of white noise. In:

B.W.: Functional Data Analysis, 2nd edn.

C.K.I.: Extreme components analysis.

C.M.: Probabilistic principal component analysis.

C.R.: Matrix Analysis.

E.: Independent component analysis.

Eﬃcient total least squares method for system modeling using minor component analysis. In:

Modiﬁed hebbian learning for curve and surface ﬁtting.

Principal Component Analysis, 2nd edn.

Principal components, minor components, and linear neural networks.

Products of gaussians and probabilistic minor components analysis.

Speech Processing: A Dynamic and OptimizationOriented Approach. Signal Processing and Communications.

Timit acoustic-phonetic continuous speech corpus. CDROM

W.C.: Mathematical methods and algorithms for signal processing. Prentice-Hall, Upper Saddle River,

Mutual interdependence analysis (MIA)

Mutual interdependence analysis (MIA)

Abstract

Similar works

Full text

Available Versions

Southampton (e-Prints Soton)