Biehl, M.

Lange, M.

Villmann, T.

English

In the present contribution we tackle the problem of nonlinear independent component analysis by non-Euclidean Hebbian-like learning. Independent component analysis (ICA) and blind source separation originally were introduced as tools for the linear unmixing of the signals to detect the underlying sources. Hebbian methods became very popular and succesfully in this context. Many nonlinear ICA extensions are known. A promising strategy is the application of kernel mapping. Kernel mapping realizes an usually nonlinear but implicite data mapping of the data into a reproducing kernel Hilbert space. After that a linear demixing can be carried out there. However, explicit handling in this non-Euclidean kernel mapping space is impossible. We show in this paper an alternative using an isomorphic mapping space. In particular, we show that the idea of Hebbian-like learning of kernel ICA can be transferred to this non-Euclidean space realizing an non-Euclidean ICA

Dissertations of the University of Groningen

   University of GroningenNon-Euclidean Independent Component Analysis and Oja's LearningLange, M.; Biehl, M.; Villmann, T.Published in:Proc. 21st Europ. Symp. Artificial Neural Networks (ESANN)IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2013Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):Lange, M., Biehl, M., & Villmann, T. (2013). Non-Euclidean Independent Component Analysis and Oja'sLearning. In M. Verleysen (Ed.), Proc. 21st Europ. Symp. Artificial Neural Networks (ESANN) (pp. 125-130). d-side publishing.CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Download date: 04-06-2022Non-Euclidean Independent Component Analysisand Oja's LearningM. Lange1, M. Biehl2, and T. Villmann11- University of Appl. Sciences Mittweida - Dept. of MathematicsMittweida, Saxonia - Germany2- University Groningen - J.-Bernoulli-Inst. of Mathematics and Computer SciencesGroningen, The NetherlandsAbstract. In the present contribution we tackle the problem of nonlinearindependent component analysis by non-Euclidean Hebbian-like learning.Independent component analysis (ICA) and blind source separation orig-inally were introduced as tools for the linear unmixing of the signals todetect the underlying sources. Hebbian methods became very popular andsuccesfully in this context. Many nonlinear ICA extensions are known. Apromising strategy is the application of kernel mapping. Kernel mappingrealizes an usually nonlinear but implicite data mapping of the data intoa reproducing kernel Hilbert space. After that a linear demixing can becarried out there. However, explicit handling in this non-Euclidean ker-nel mapping space is impossible. We show in this paper an alternativeusing an isomorphic mapping space. In particular, we show that the ideaof Hebbian-like learning of kernel ICA can be transferred to this non-Euclidean space realizing an non-Euclidean ICA.1 IntroductionIndependent component analysis (ICA) and blind source separation (BSS) con-stitute a paradigm to extract independent sources from a sequence of mixtures[5, 10, 15]. It can be seen as a generalization of principal component analysis(PCA). The generic problem consists in separating useful independent signalsfrom noise and interferences [6]. Originally, respective methods were consideredas linear models. There exists a whole bunch of methods to tackle this problem.They dier mainly on the a priori assumptions about the underlying mixingmodel. One of the key principles of ICA is 'non-Gaussianity' as an equivalent ofindependence together with the central limit theorem (CLT,[13]) [5, 10]. Dier-ent approaches were developed to contrast this property: approximations of thenegentropy [19] were developed in [10, 14], mutual information and the infomaxprinciple were considered in [4, 22, 25, 26, 9]. ICA and projection pursuit isaddressed in [14]. A large family of Hebbian-like learning algorithms for ICAutilize implicitly the kurtosis as a contrast function [12, 11, 17, 18].Nonlinear approaches of ICA and BSS are natural generalizations of the linearapproaches and were addressed in [12, 16]. Several investigations consider kernelmethods to deal with nonlinear separation [2, 7, 20]. However, these models arenot based on Hebbian-like learning ICA. Recently, kernel PCA has been studiedin terms of Hebbian learning by a kernelized variant of Oja-learning realizing anon-Euclidean PCA [3]. We transfer this idea to Hebbian-like ICA in the presentcontribution and provide the theoretical basis. We demonstrate the abilities ofthis new approach for exemplary source separation problems.125ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.The paper is organized as follows: First, we briey review ICA based onkurtosis contrast. Then, we reconsider the Hebbian-like algorithms for learn-ing independent components. Finally we introduce our kernel based approachillustrate it in terms of an application example..2 Linear Independent Component Analysis by Hebbian-like Learning Using the KurtosisAs mentioned in the introduction, one key principle of ICA is to estimate in-dependence by non-Gaussianity. In this context we consider sequences of n-dimensional mixture vectors v (t) ∈ VdE ⊆ Rn with the Euclidean distance dE .We assume that a pre-whitening x (t) = Pv (t) takes place such that the expec-tation becomes E[xx>]= I. Further, we suppose a linear mixing modelv (t) = As (t) (1)such that x (t) = Ms (t) is valid with M = PA. It turns out that M has to beorthonormal to ensure E[xx>]= I. Letsi =〈m>i ,x〉=n∑j=1mi,jxj (2)be the ith source where mi is a column vector of M. Here xj are stochasticquantities such that the central limit theorem (CLT) is valid, i.e. the quantitysi is more Gaussian than the single summands. Thus, ICA can be performed bymaximization of the absolute value of the kurtosis kurt (si) as a measure of Non-Gaussianity. The kurtosis is dened by kurt (y) = E[y4]−3(E[y2])2using thefourth and the second moments E[y4]and E[y2], respectively. If we considerw = mi, Hebbian-like learning interprets the vector w as a weight vector of alinear perceptron, which is trained by a sequence x (t) of input vectors [8, 23].A it was shown in [11, 12, 17, 24], Hebbian-like ICA-learning can be achievedapplying the weight vector dynamicw (t+ 1) = w (t) + εt[−σ · x (t) · g (〈w (t) ,x (t)〉)− f(〈w,x〉2)w (t)](3)with a nonlinear modulation function g (y) = ay− by3. The function f is a non-vanishing scalar and σ = ±1 is a sign that determines whether we are minimizing(σ = −1) or maximizing (σ = +1) the kurtosis. The constants are chosen to bea ≥ 0 and b > 0. The maximization of the positive kurtosis is obtained for a = 0,the optimization of the negative kurtosis requires a > 0 [11]. The value 0 < εt 1 is the learning rate. The term x (t) · g (〈w (t) ,x (t)〉) reects the Hebb-signal-enhancing idea with the perceptron output learning function g (〈w (t) ,x (t)〉)whereas f(E[〈w,x〉2])w (t) denes a constraint term to prevent w (t) frominnite growing. The learning rule (3) performs a stochastic gradient descent onthe cost functionJ (w) = σ(−a2· E[〈w,x〉2]− b4· E[〈w,x〉4])+ F(E[〈w,x〉2])(4)with F ′ (y) = f (y).126ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.3 Nonlinear Kernel Independent Component Analysis byHebbian-like LearningKernel ICA and BSS make use of the nonlinear kernel mapping to perform anonlinear ICA v (t) = h (s (t)) for an unknown non-linear function h. Severalapproaches are discussed [2, 7, 20]. We concentrate on the approach presentedby Harmeling et al. in [7]. We consider a generally nonlinear kernel data mapΦ : V 3 v 7−→ Φ(v) ∈ H (5)with a positive denite kernelκΦ(v,w) = 〈Φ(v),Φ(w)〉H (6)for all v,w ∈ V and 〈·, ·〉H such that the space H is a reproducing kernel Hilbertspace (RKHS) H uniquely corresponding the kernel to a reproducing kernel κΦa canonical manner [1, 21]. The norm ‖Φ(v)‖H =√κΦ(Φ(v),Φ(v)) of thisRKHS induces a metric dH (Φ(v),Φ(w)) =√κΦ(v,v)− 2κΦ(v,w) + κΦ(w,w)based on the kernel κΦ [27]. Steinwart has shown that continuous, universalkernels induce the continuity1 and separability of the corresponding feature mapΦ and the image IκΦ = span (Φ (V )) is a subspace of H [28]. Let bi form anorthonormal basis in IκΦ . ThenΦ(v) =∑i〈Φ(v),bi〉H · bi (7)is the representation of an image vector Φ(v) in IκΦ . In analogy to the linearICA/BSS we consider now the linear mixing problem in the Hilbert space H:Φ(v) = MH [Φ(s)] (8)with MH being a linear operator in H. We denote by MkH the kth componentof MH. Because H is a RKHS, each linear operator can be expressed in termsof the inner product, i.e.MkH [Φ(s)] = 〈MkH,Φ(s)〉H. (9)Using the basis representation form(7) we obtainΦk(v) = 〈MkH,∑i〈Φ(s),bi〉H · bi〉H =∑i〈Φ(s),bi〉H · 〈MkH,bi〉H . (10)We remark at this point that 〈Φ(s),bi〉H is a random variable because of thestochastic character of s and, hence, Φ(s) is random too. Therefore, we can statethat the sum in (10) is more Gaussian than the single components according tothe CLT. In consequence we can take Φk(v) as a quantity the absolute kurtosisof which has to be maximized for separating independent components in IκΦ bythe same arguments as for linear ICA.1Note that continuity ensures the existence of the inverse mapping Φ−1.127ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.Figure 1: Visualization of the original signals (top) and the mixed sources (bottom).For a nite number N of samples in V the number of basis elements bi isD ≤ min (n,N). In that case we can simplify (10) toΦk(v) = D · κΦ(Φ−1(MkH), s) (11)using the reproducing property of the inner product of a RKHS and keeping inmind denition (6) of the kernel.In the last step of our argument we replace the kernel map Φ byΨ : VdE −→ VdκΦ (12)with dκΦ(v,w) = dH (Φ(v),Φ(w)). For universal continuous kernels VdκΦ is acompact vector space with the kernel induced metric dκΦ . Note that Ψ is for-mally the identity map but changing the metric, and, hence, generally nonlinear.Moreover, it turns out that the kernel space VdκΦ is isometric and isomorphicto IκΦ [29]. Under this assumption, the operator MH is equivalent to a conven-tional matrix M but M [Ψ(s)] is dened byMk [Ψ(s)] = D · κΦ(mk, s) (13)where mk is the kth row vector of M. Hence, we can replace the Euclidean innerproduct in (3) by κΦ(mk, s) to obtain an ICA in VdκΦ as a Hebbian-like kernelICA (kICA) based of the original data, which realizes a non-linear de-mixingbecause of the non-linear kernel mapping Φ or its analagon Ψ.4 Exemplary SimulationsIn an illustrative simulation we consider a non-linear mixture of two signals ob-tained by linear mixing in the kernel space and subsequent back transformationinto the data space. The sources and the mixed signals are depicted in Fig.1. Then we applied the original linear Hebbian-like ICA according to (3). Thekurtosis was estimated using the online learningµ4 (t+ 1) = (1− ξ)µ4 (t) + ξ 〈w (t) ,x (t)〉4 (14)128ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.Figure 2: Estimated source signals by original Hebbian-like ICA based on the Eu-clidean inner product (top) and by the kernelized variant (bottom).suggested in [11] with an averaging parameter of ξ = 0.05. In comparison weestimated the source signals by the kernelized variant replacing the inner prod-uct 〈w (t) ,x (t)〉 by the Gaussian kernel G (w (t) ,x (t)). The results of bothapproaches are visualized in Fig. 2. We observe that the kernelized variant isbetter able to de-mix the signals and to reconstruct the original signals thanthe original Euclidean approach. The obtained integrated squared errors oft thelinear model are e1ICA = 22.3 and e2ICA = 55.0 for the two signals, respectively,whereas the kICA yields e1kICA = 23.1 and e2kICA = 34.8. The respective corre-lation coecients are ρ1ICA = 0.86 and ρ2ICA = 0.81 for the linear model. ThekICA yields ρ1kICA = 0.88 and ρ2kICA = 0.84. The improved performance ofkICA is due to its non-linear character implicitly realized by the kernel trick.However, the kICA is very sensitive.5 ConclusionIn this paper we introduced a kernelized variant of the original Hebbian-like ICAproposed by Hyvärinen&Oja. We showed that the Euclidean inner product inthe original approach can be replaced by an universal and continuous kernel withan appropriate interpretation in the resulting generally non Euclidean kernelizeddata space. Thus, a non-linear demixing can be realized. We demonstrated inan exemplary application that this non-Euclidean variant of Hebbian-like ICAis able to extract non-linearly mixed signals, however, it is dicult to stabilizethe model.References[1] N. Aronszajn. Theory of reproducing kernels. Transactions of the American MathematicalSociety, 68:337404, 1950.[2] F. Bach and M. Jordan. Kernel independent component analysis. Journal of MachineLearning Research, 3:148, 2002.[3] M. Biehl, M. Kästner, M. Lange, and T. Villmann. Non-euclidean principal compo-nent analysis and Oja's learning rule  theoretical aspects. In P. Estevez, J. Principe,and P. Zegers, editors, Advances in Self-Organizing Maps: 9th International Workshop129ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.WSOM 2012 Santiage de Chile, volume 198 of Advances in Intelligent Systems and Com-puting, pages 2334, Berlin, 2012. Springer.[4] J.-F. Cardoso. Infomax and maximum likelihood for source separation. IEEE Letters onSignal Processing, 4:112114, 1997.[5] A. Cichocki and S. Amari. Adaptive Blind Signal and Image Processing. John Wiley,2002.[6] P. Comon and C. Jutten. Handbook of Blind Source Separation. Academic Press, 2010.[7] S. Harmeling, A. Ziehe, and M. K. and. K.-R. Müller. Kernel-based nonlinear blind sourceseparation. Neural Computation, 15(5):10891124, 2003.[8] D. Hebb. The Organization of Behavior. A Neuropsychological Theory. John Wiley, NewYork, 1949.[9] A. Hyvärinen. The xed-point algorithm and maximum likelihood estimation for inde-pendent component analysis. IEEE Transactions on Neural Networks, 10(3):626634,1999.[10] A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis. J. Wiley &Sons, 2001.[11] A. Hyvärinen and E. Oja. Simple neuron models for independent component analysis.International Journal of Neural Systems, 7(6):671687, 1996.[12] A. Hyvärinen and E. Oja. Independent component analysis by general nonlinear hebbian-like learning rules. Signal Processing, 64:301313, 1998.[13] I. Jolie. Principal Component Analysis. Springer, 2nd edition, 2002.[14] M. Jones and R. Sibson. What is projection pursuit? Journal of the Royal StatisticalSociety, Series A, 150:136, 1987.[15] C. Jutten and J. Hérault. Blind separation of sources, Part I: An adaptive algorithmbased on neuromimetic architecture. Signal Processing, 24:110, 1991.[16] C. Jutten and J. Karhunen. Advances in blind source separation (BSS) and indepen-dent component analysis (ICA) for nonlinear mixtures. International Journal of NeuralSystems, 14(5):267292, 2004.[17] J. Karhunen, E. Oja, L. Wang, R. Vigário, and J. Joutsensalo. A class of neural networksfor independnet component analysis. IEEE Transactions on Neural Networks, 8(3):486504, 1997.[18] J. Karhunen, P. Pajunen, and E. Oja. The nonlinear PCA criterion in blind sourceseparation: Relations with other approaches. Neurocomputing, 22:520, 1998.[19] D. Mackay. Information Theory, Inference and Learning Algorithms. Cambridge Univer-sity Press, 2003.[20] D. Martinez and A. Bray. Nonlinear blind source separation using kernels. IEEE Trans-actions on Neural Networks, 14(1):228235, 2003.[21] J. Mercer. Functions of positive and negative type and their connection with the theory ofintegral equations. Philosophical Transactions of the Royal Society, London, A, 209:415446, 1909.[22] J.-P. Nadal and N. Parga. Non-linear neurons in the low noise limit: a factorial codemaximizes information transfer. Netw, 5:565581, 1994.[23] E. Oja. Neural networks, principle components and subspaces. International Journal ofNeural Systems, 1:6168, 1989.[24] E. Oja. The nonlinear PCA learning rule in independent component analysis. Neurocom-puting, 17:2545, 1997.[25] D. Pham. Mutual information approach to blind separation of stationary sources. IEEETransactions on Information Theory, 48:19351946, 2002.[26] D.-T. Pham, P. Garat, and C. Jutten. Separation of a mixture of independent sourcesthrough a maximum likelihood approach. In J. Vandewalle, R. Boite, M. Moonen, andA. Oosterlinck, editors, Signal Processing VI: Theories and Applications (EUSIPCO),pages 771774, 1997.[27] B. Schölkopf and A. Smola. Learning with Kernels. MIT Press, 2002.[28] I. Steinwart. On the inuence of the kernel on the consistency of support vector machines.Journal of Machine Learning Research, 2:6793, 2001.[29] T. Villmann and S. Haase. A note on gradient based learning in vector quantiza-tion using dierentiable kernels for Hilbert and Banach spaces. Machine Learning Re-ports, 6(MLR-02-2012):129, 2012. ISSN:1865-3960, http://www.techfak.uni-bielefeld.de/˜fschleif/mlr/mlr_02_2012.pdf.130ESANN 2013 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Bruges (Belgium), 24-26 April 2013, i6doc.com publ., ISBN 978-2-87419-081-0. Available from http://www.i6doc.com/en/livre/?GCOI=28001100131010.

Non-Euclidean Independent Component Analysis and Oja's Learning

https://pure.rug.nl/ws/files/41826186/es2013_42.pdf

Non-Euclidean Independent Component Analysis and Oja's Learning

Abstract

Similar works

Full text

Available Versions

Dissertations of the University of Groningen