International audienceIn a number of real-life applications, the user is interested in analyzing several sources of information together: a graph combined with the additional information known on its nodes, numerical variables measured on individuals and factors describing these individuals... The combination of all sources of information can help him to understand the dataset in its whole better. The present article focuses on such an issue, by using self-organizing maps. The use a kernel version of the algorithm allows us to combine various types of information and automatically tune the data combination. This approach is illustrated on a simulated example

Cierco-Ayrolles, Christine

Olteanu, Madalina

Villa-Vialaneix, Nathalie

Villa-Vialaneix, Nathalie, N.

HAL-Paris1

Multiple kernel self-organizing maps

Villa-Vialaneix, Nathalie,

HAL Descartes

Abstract. In a number of real-life applications, the user is interested in analyzing several sources of information together: a graph combined with the additional information known on its nodes, numerical variables measured on individuals and factors describing these individuals... The combination of all sources of information can help him to understand the dataset in its whole better. The present article focuses on such an issue, by using self-organizing maps. The use a kernel version of the algorithm allows us to combine various types of information and automatically tune the data combination. This approach is illustrated on a simulated example. 

Madalina Olteanu

Nathalie Villa-vialaneix

Christine Cierco-ayrolles

CiteSeerX

Hal-Diderot

Multiple kernel self-organizing mapsMadalina Olteanu, Nathalie Villa-Vialaneix, Christine Cierco-AyrollesTo cite this version:Madalina Olteanu, Nathalie Villa-Vialaneix, Christine Cierco-Ayrolles. Multiple kernel self-organizing maps. Michel Verleysen. European Symposium on Artificial Neural Networks,Computational Intelligence and Machine Learning, Apr 2013, Bruges, Belgium. pp.83, 2013.<hal-00817920>HAL Id: hal-00817920https://hal.archives-ouvertes.fr/hal-00817920Submitted on 25 Apr 2013HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.Multiple Kernel Self-Organizing MapsMadalina Olteanu1, Nathalie Villa-Vialaneix2 and Christine Cierco-Ayrolles21- SAMM, Universite´ Paris 1, Paris, France2- Unite´ BIA, INRA de Toulouse, Auzeville, FranceAbstract. In a number of real-life applications, the user is interestedin analyzing several sources of information together: a graph combinedwith the additional information known on its nodes, numerical variablesmeasured on individuals and factors describing these individuals... Thecombination of all sources of information can help him to understand thedataset in its whole better. The present article focuses on such an issue,by using self-organizing maps. The use a kernel version of the algorithmallows us to combine various types of information and automatically tunethe data combination. This approach is illustrated on a simulated example.1 IntroductionIn a number of real-life applications, the user is interested in analyzing severalsources of information together: a graph combined with the additional infor-mation known on its nodes, numerical variables measured on individuals andfactors describing these individuals... The combination of all sources of infor-mation can help him to understand the dataset in its whole better. The presentarticle focuses on such an issue, by using self-organizing maps (SOM).SOM has already been extended to the framework of non numerical data,using various approaches. Historically, median SOM, which consists in replac-ing the standard computation of the prototypes by an approximation in theoriginal dataset, was introduced first. This principle was used to extend SOMto dissimilarity data in [1]. Several variants of the original algorithm have beenproposed since: [2, 3, 4, 5] extended the framework of SOM for dissimilarity databy proposing a more flexible representation for the prototypes, either in batchor in online versions. A closely related approach is to rely on kernels to map theoriginal data into an (implicit) Euclidean space where the standard SOM can beused [6, 7, 8]. Several kernels have been designed to handle complex data suchas strings, nodes in a graph or several graphs [9].In the present paper, we propose to extend kernel SOM by using a combi-nation of kernels, each of them dedicated to a particular aspect of the entiredataset. The methodology is described in Section 2 and illustrated in Section 3.2 MethodThe purpose of the present paper is to design a self-organizing map algorithmin which data coming from various sources can be used as inputs in an op-timal combination. Let us suppose that we are given D sets of input data,(xdi )i=1,...,n, d=1,...,D, all measured on the same individuals (i = 1, . . . , n), andtaking values in arbitrary spaces (Gd)d. For example, (xdi )i may represent p-dimensional numerical vectors or factor (non-numerical) variables describing then individuals. More specifically, (xdi )i may describe the n nodes in a graph andthe kind of relation between them or some text variables naming or describingthe data.Multiple kernel SOMSuppose also that for each of these D datasets, a kernel is known, i.e., afunction Kd : Gd × Gd → R such that Kd is symmetric (∀ z, z′ ∈ Gd, Kd(z, z′) =Kd(z′, z)) and positive (∀N ∈ N, ∀ (zj)j=1,...,N ⊂ Gd and ∀ (αj)j=1,...,N ⊂ R,∑j,j′ αjαj′Kd(zj , zj′) ≥ 0). For example, the linear kernel, the Gaussian kernelor the polynomial kernel, among others, may be used for numerical variables.For nodes of a graph, the heat kernel [10], the commute time kernel [11] or anyother regularized version of the Laplacian can be used as kernels [12], while fortextual variables, string kernels, based on the number of occurences of commonsubstrings [13] may be considered.1From these D kernels, a new kernel can be defined by computing a convexcombination: K(xi, xi′) =∑Dd=1 αdKd(xdi , xdi′), where αd ∈ [0, 1],∑d αd = 1and xi = (x1i , . . . , xdi ) ∈ G = G1 × ... × Gd. Obviously, such an application isalso symmetric and positive and thus [14] proves that there is a Hilbert space,(H, 〈., .〉) (called the feature space) and an application φ : G → H (called thefeature map) such that K(xi, xi′) = 〈φ(xi), φ(xi′)〉.In the SOM algorithm, the individuals i = 1, . . . , n have to be clustered into alow dimensional grid made of M neurons, {1, . . . ,M}. These neurons are relatedto each other by a neighborhood relationship, h. Each neuron is also representedby prototypes, (pj)j (j = 1, . . . ,M) taking values in the previously definedfeature space H. Following the general framework of kernel SOM described in[6, 15, 8], the prototypes are written as a convex combination of the input datain the feature space H: pj =∑i γjiφ(xi). The (squared-)distance between anindividual xi and a prototype pj is then computed by‖pj − φ(xi)‖2 = K(xi, xi)− 2n∑l=1γjlK(xi, xl) +n∑l,l′=1γjlγjl′K(xl, xl′). (1)Online multiple kernel SOMIf the (αd)d are given, the standard kernel SOM aims at optimizing (over(γji)ji) the following energy function :E((γji)ji, (αd)d) =n∑i=1M∑j=1h (f(xi), j)∥∥φα(xi)− pαj (γj)∥∥2α ,where f(xi) ∈ {1, . . . ,M} is the index of the cluster to which xi is assigned. Thenotations φα, pαj and ‖.‖α are used to emphasize that these quantities depend1Note that this methodology can also been applied to combine several kernels on the samedata (e.g., different kinds of string kernels, encapsulating different features of the texts).on α. The novelty of this paper is to include the optimization of the convexcombination of kernels into the on-line algorithm which trains the map. Moreprecisely, a stochastic gradient descent step is added to the original on-line SOMalgorithm to optimize the energy E((γji)ji, (αd)d) over both (γji)ji and (αd)d.To perform the stochastic gradient descent step on the (αd), the computation ofthe derivative of E|xt =∑Mj=1 h (f(xt), j)∥∥φα(xt)− pαj (γj)∥∥2α (the contributionof the randomly chosen observation xt to the energy) with respect to α is needed.As ∂∂αd [K(xi, xi′)] = Kd(xdi , xdi′), we have thatDd := ∂E|xt∂αd=M∑j=1h (f(xt), j)(Kd(xdt , xdt )− 2n∑l=1γjlKd(xdt , xdl )+n∑l,l′=1γjlγjl′Kd(xdl , xdl′)Following an idea similar to that of [16], the SOM is trained by performing,alternatively, the standard steps of the SOM algorithm (i.e., affectation andrepresentation steps) and a gradient descent step for the (αd)d. The full processis given in Algorithm 1.Algorithm 1 Multiple online kernel SOM1: ∀ j = 1, . . . ,M and ∀ i = 1, . . . , n, initialize γ0ji randomly in R such thatγ0ji ≥ 0 and∑ni=1 γ0ji = 1 and ∀ d = 1, . . . , D, initialize α0d in [0, 1] such that∑d α0d = 1.2: for t = 1→ T do3: Randomly choose an input xl ∈ {xi}i4: Assignment step. Find the unit of the closest prototypef t(xl)← arg minj=1,...,M∥∥∥φαt−1(xl)− pαt−1j (γt−1j )∥∥∥αt−15: Representation step. Update all the prototypes: ∀ j = 1, . . . ,M ,γtji ← γt−1ji + µ(t)h(f t(xl), j)(δli − γt−1ji)with δli = 1 if l = i and 0 otherwise.6: Gradient descent step. Update the kernel:∀ d = 1, . . . , D, αtd ← αt−1d + ν(t)Dtd and Kt ←∑dαtdKd.7: end forTo ensure that the gradient step respects the constraints on α (αd ≥ 0and∑d αd = 1), the following strategy can be applied: first, the gradientDtd =(∂Et|xt∂αd)dis reduced and it is projected so that the non-negativity of α isensured (the reduced gradient projection procedure is described in [17, 18, 16]).This leads to the descent direction Dtd. The descent step is decreased with thestandard rate of ν0/t.In practice, to ensure similar scales for all kernels, a normalization step isperformed before computing the first convex combination: all kernels are scaledto have a unit norm, according to the Frobenius norm. The tuning procedurethen automatically finds out the (αd) that give the best map, i.e., that bestorganizes the observations according to the global kernel K. As illustrated in[19], this can be particularly useful when the data can be described by severalkernels, all briging up a different insight on the similarities between individuals,as it is often the case in social sciences (see [20, 21] for a discussion on this topic).Finally, the additional cost of tuning the (αd) is moderate because it is per-formed online: in practice, we simply multiplied by two the number of stepsusually needed to train the map and this choice led to a converging algorithm.3 ApplicationIn this section, a simple example is used to test the algorithm and illustrateits behavior in the presence of complementary information. 200 observations,divided into 8 groups (numbered from 1 to 8 in the following), were generatedusing three different kinds of data: 1/ an unweighted graph, simulated similarlyas described in [22]. The nodes of the groups 1 to 4 and the nodes of thegroups 5 to 8 could not be distinguished in the graph structure: the edgeswithin these two sets of nodes were randomly generated with a probability equalto 0.3. The edges between these two sets of nodes were randomly generatedwith a probability equal to 0.01; 2/ numerical data coming from two dimensionalGaussian distributions. The variables were simulated by Gaussian vectors withindependent components having a variance equal to 0.3 and a mean equal to(0, 0) for the odd groups and to (1, 1) for the even groups; 3/ a factor with 2levels. Observations of groups 1, 2, 5, and 7 were affected to the first level andobservations of the other groups to the second level.Hence, only the combined knowledge of the three datasets gave access to theeight original groups. The online multiple kernel SOM algorithm was applied tothis problem with the commute time kernel for the graph and a Gaussian kernelfor the numerical data and for the factor variable (re-coded in disjunctive form).The algorithm was compared with a standard kernel SOM approach using oneof the three datasets only. It was also compared to kernel SOM using numericaland factor data or all the three datasets but used as if they were issued from thesame dataset with a single Gaussian kernel (when the graph was added to thenumerical and factor data, it was under the form of its adjacency matrix). Theperformance of the different approaches were compared by using the normalizedmutual information (NMI) [23] compared to the original classes. This measurehas values between 0 and 1 and is equal to 1 when the two partitions are identical.The results are given in Figure 1.The multiple kernel SOM clearly outperformed all the other approaches, in-Fig. 1: NMI (left) of online kernel SOM when using all datasets in the multiplekernel framework or only one dataset or all datasets in a single kernel framework.Example of a map resulting from the use of the algorithm (right): each noderepresents a neuron of the map and has an area proportional to the numberof observations classified inside. The colors display the proportion of the eightoriginal classes in the clusters.cluding the one where all datasets are simply aggregated together. The exampleof the resulting map (given on the right hand side of the figure) shows a goodclassification and a good organization according to the three types of informa-tion: the eight groups are well distinguished by the algorithm.4 ConclusionThis article proposes a way to combine multiple sources of heterogenous informa-tion in self-organizing maps. Our approach can help to select the most relevantsources of information or the most relevant ways to describe similarities betweenobservations. The computational cost of such an approach is moderate but canhowever be prohibitive when run with very large datasets or a large number ofkernels. A sparse approach is currently under study for improving this aspect ofthe algorithm.References[1] T. Kohohen and P.J. Somervuo. Self-organizing maps of symbol strings. Neurocomputing,21:19–30, 1998.[2] B. Conan-Guez, F. Rossi, and A. El Golli. Fast algorithm and implementation of dissim-ilarity self-organizing maps. Neural Networks, 19(6-7):855–863, 2006.[3] B. Hammer, A. Hasenfuss, F. Rossi, and M. Strickert. Topographic processing of relationaldata. In Bielefeld University Neuroinformatics Group, editor, Proceedings of the 6thWorkshop on Self-Organizing Maps (WSOM 07), Bielefeld, Germany, September 2007.[4] B. Hammer, A. Gisbrecht, A. Hasenfuss, B. Mokbel, F.M. Schleif, and X. Zhu. Topo-graphic mapping of dissimilarity data. In J. Laaksonen and T. Honkela, editors, Advancesin Self-Organizing Maps (Proceedings of the 8th Workshop on Self-Organizing Maps,WSOM 2011), volume 6731 of Lecture Notes in Computer Science, pages 1–15, Espoo,Finland, 2011. Springer.[5] M. Olteanu, N. Villa-Vialaneix, and M. Cottrell. On-line relational som for dissimilaritydata. In P.A. Estevez, J. Principe, P. Zegers, and G. Barreto, editors, Advances inSelf-Organizing Maps (Proceedings of WSOM 2012), volume 198 of AISC (Advances inIntelligent Systems and Computing), pages 13–22, Santiago, Chile, 2012. Springer Verlag,Berlin, Heidelberg.[6] D. Mac Donald and C. Fyfe. The kernel self organising map. In Proceedings of 4th In-ternational Conference on knowledge-based intelligence engineering systems and appliedtechnologies, pages 317–320, 2000.[7] P. Andras. Kernel-Kohonen networks. International Journal of Neural Systems, 12:117–135, 2002.[8] R. Boulet, B. Jouve, F. Rossi, and N. Villa. Batch kernel SOM and related laplacianmethods for social network analysis. Neurocomputing, 71(7-9):1257–1273, 2008.[9] T. Ga¨rtner. Kernel for Structured Data. World Scientific, 2008.[10] R.I. Kondor and J. Lafferty. Diffusion kernels on graphs and other discrete structures. InProceedings of the 19th International Conference on Machine Learning, pages 315–322,2002.[11] F. Fouss, A. Pirotte, J.M. Renders, and M. Saerens. Random-walk computation of sim-ilarities between nodes of a graph, with application to collaborative recommendation.IEEE Transactions on Knowledge and Data Engineering, 19(3):355–369, 2007.[12] A.J. Smola and R. Kondor. Kernels and regularization on graphs. In M. Warmuth andB. Scho¨lkopf, editors, Proceedings of the Conference on Learning Theory (COLT) andKernel Workshop, Lecture Notes in Computer Science, pages 144–158, 2003.[13] C. Watkins. Dynamic alignment kernels. In A. Smola, P. Bartlett, B. Scho¨lkopf, andD. Schuurmans, editors, Advances in Large Margin Classifiers, pages 39–50, Cambridge,MA, USA, 2000. MIT P.[14] N. Aronszajn. Theory of reproducing kernels. Transactions of the American MathematicalSociety, 68(3):337–404, 1950.[15] N. Villa and F. Rossi. A comparison between dissimilarity SOM and kernel SOM forclustering the vertices of a graph. In 6th International Workshop on Self-Organizing Maps(WSOM), Bielefield, Germany, 2007. Neuroinformatics Group, Bielefield University.[16] A. Rakotomamonjy, F.R. Bach, S. Canu, and Y. Grandvalet. SimpleMKL. Journal ofMachine Learning Research, 9:2491–2521, 2008.[17] D. Luenberger. Linear and Nonlinear Programming. Addison-Wesley, 1984.[18] F. Bonnans. Optimisation Continue. Dunod, 2006.[19] M. Olteanu, S. Massoni, and N. Villa-Vialaneix. Which distance use when extractingtypologies in sequence analysis? An application to school to work transitions. Submittedfor publication.[20] A. Abbott and Tsay. Sequence analysis and optimal matching methods in sociology.Review and Prospect. Sociological Methods and Research, 29(1):3–33”, 2000.[21] L. Wu. Some comments on “Sequence analysis and optimal matching methods in sociology,review and prospect”. Sociological Methods and Research, 29(1):41–64, 2000.[22] A. Condon and R.M. Karp. Algorithms for graph partitioning on the planted partitionmodel. Random Structures and Algorithms, 18(2):116–140, 2001.[23] L. Danon, A. Diaz-Guilera, J. Duch, and A. Arenas. Comparing community structureidentification. Journal of Statistical Mechanics, page P09008, 2005.

In a number of real-life applications, the user is interested in analyzing several sources of information together: a graph combined with the additional information known on its nodes, numerical variables measured on individuals and factors describing these individuals... The combination of all sources of information can help him to understand the dataset in its whole better. The present article focuses on such an issue, by using self-organizing maps. The use a kernel version of the algorithm allows us to combine various types of information and automatically tune the data combination. This approach is illustrated on a simulated example

ProdInra

https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2013-99.pdf

Multiple kernel self-organizing maps

Abstract

Similar works

Full text

Available Versions

HAL-Paris1

HAL Descartes

CiteSeerX

HAL Descartes

Hal-Diderot

HAL-Paris1

Hal-Diderot

ProdInra