We study the extraction of nonlinear data models in high dimensional spaces with modified self-organizing maps. Our algorithm maps lower dimensional lattice into a high dimensional space without topology violations by tuning the neighborhood widths locally. The approach is based on a new principle exploiting the specific dynamical properties of the first order phase transition induced by the noise of the
data. The performance of the algorithm is demonstrated for one- and two-dimensional principal manifolds and for sparse data sets

Balzuweit, Gerd

Der, Ralf

Herrmann, Michael

English

Qucosa - Publikationsserver der Universität Leipzig

Building Nonlinear Data Models withSelf{Organizing MapsRalf Der1, Gerd Balzuweit1, Michael Herrmann21 University of Leipzig, Institute of InformaticsP. O. Box 920, 04109 Leipzig, Germany2 RIKEN, Lab. of Information Representation2-1 Hirosawa, Wako-shi, 351-01 Saitama, JapanAbstract. We study the extraction of nonlinear data models inhigh dimensional spaces with modied self-organizing maps. Our algo-rithm maps lower dimensional lattice into a high dimensional space with-out topology violations by tuning the neighborhood widths locally. Theapproach is based on a new principle exploiting the specic dynamicalproperties of the rst order phase transition induced by the noise of thedata. The performance of the algorithm is demonstrated for one- andtwo-dimensional principal manifolds and for sparse data sets.1 IntroductionArticial neural networks provide convenient tools for reconstructing non-parametric data models from noisy data. Consider data representing a func-tional relation y = f (x) corrupted by noise, i.e. the observations are given byy = f (x)+ , where  is the noise which may or may not depend on x. Buildinga data model means extracting the systematic term f(x) from the noisy data.If the noise also aects the x values the problem transforms into the construc-tion of principal manifolds (PMs) that model data distributions by providinga curvilinear coordinate system for a dimension reduced representation of thedata. PMs, principal curves e.g., may be dened self-consistently by the require-ment that each point on the PM is the average of the data points projectingto it, cf. [1]. Thus, the mean square deviations (MSD) of the data from the PMare minimized locally at each base point. Averaged over the input distribution,however, the PM is only a stationary point with respect to the MSD [1], i.e. forsome directions in the function space the PM belongs to, it is stable, for othersunstable. So far there exist no explicit criteria for the existence, uniqueness andstability of PMs. It is known from the simulation of many data sets [1] that sta-bility is guaranteed by local averaging the span of average being guided globallyby cross validation [1].Kohonen's algorithm provides a nonlinear mapping of some lattice (repre-senting the physical positions of neurons) into the data manifold in input space.Thus, a piecewise linear approximation of the desired curvilinear coordinate sys-tem is established (cf. [3, 4]). Whether or not these coordinates reect the mainfeatures of the data or, more precisely, whether they are principal curves of thedata becomes in the context of self-organizing maps (SOMs) a stability prob-lem. For this problem { and in accordance with the empirical observations forprincipal manifolds { it could be shown [3, 5] that stability is ensured by a su-cient range of local collaboration, i.e. by local averaging over a suciently widespan. This indicates the principal manifold are saddle points with respect to theMSD which are stable with respect to long ranged variations and unstable forshort ranged variations. Averaging or neighborhood interaction stabilizes PMsbecause the short ranged deviations are suppressed. In [3, 4] stability problemshave not been addressed, i.e. the neighborhood parameter has been assumed tovary only in the stable range. Since on the other hand the averaging disruptsthe representation of specic data features, and thus tends to increase the MSD,the neighborhood coherence should not be to long-ranged.The present paper focuses on a choice of this parameter that compromisesbetween the requirements of smoothness and small MSD error. We start fromKohonen's unsupervised learning rule cf. [2] to develop a simple and robust algo-rithm for learning a good approximation of the principal manifold. As comparedto the HS algorithm, it has the advantages of being on-line learning, not dimen-sion specic and uses a locally dened width for the smoothing operation. Thisis important for data distributions with locally varying variance of the noise(multiplicative noise). Most importantly these widths are self-regulating. Ourapproach rests on a new principle which exploits the specic dynamical proper-ties of the rst-order phase transition [3, 5] induced by the noise. The approachis shown to work also for sparse data sets and should therefore be favorable alsoin the case of high-dimensional inputs.2 Maps with locally adaptive smoothnessLet us assume that the data embedded in some d-dimensional space X ac-tually are scattering about some manifold of the lower dimension D < d. ThenKohonen's algorithm may be used to map the data topographically onto a D-dimensional lattice A, where the lattice points r 2 [1;N ]D may be considered asthe physical positions of ND neurons. Upon presentation of a data vector x, thelearning step for the synaptic vectors wr iswr = " hr;r0 (x wr) ; (1)where the position r0 of the winner (best matching) neuron is deter-mined by r0 = argmaxr kx wrk . The neighborhood function hr;r0 = 22 D2 exp  (r  r0)2=(22)denes the range of cooperativity betweenneurons which controls the smoothness of the map: Its radius of curvature obeys  > 2d where d is about the Euclidean distance between neurons in inputspace which are a distance  apart on the lattice. (The normalization factor 22 D2 was introduced in order that the average force exerted on wr is inde-pendent on ). For varying width of the data scattering we need the smoothnessof the map to be dened locally. This may be achieved by an individual neigh-borhood r for each neuron, so thathr;r0 =1p2~rDexp  (r  r0)22~2r!(2)where ~r = min fr; r0g with the additional constraints 1  ~r  max.The crucial point now is the determination of the local values r. The prin-cipal manifold can be mapped topographically onto the lattice because of thedimensions match by denition. However, with the data points scattering aboutthe PM we have to compromise between two options. On the one hand the lat-tice should be mapped tightly to the PM which requires a small  (curvature)if the PM is nonlinear. On the other hand the stiness and hence  should besuciently large in order to avoid the folding due to the dimensional mismatch.For the denition of the optimal r we exploit the dynamics of the phasetransitions induced by the scattered data points. Namely, if  is lowered below avalue crit the representation of the main data feature by the map gets distortedby other `noisy' features. The transition has has been shown earlier [5] to proceedthrough a either of two phases, one preserving and one violating topology. Sincethe latter is more pronounced, emerging folding into secondary features willbe signaled immediately by topology violations. Concentrating on the rst andsecond winner the criterion for the occurrence of a fold is  > 1 where  =kr0   r00k is the distance between the rst and second winner in the lattice.In contrast to [6], where the neighborhood width has been updated followingan energy function resulting in the occurrence of local minima and a slow con-vergence, our approach here consists in keeping r uctuating around its criticalvalue critr . The result of the algorithm is given in terms of sliding averages overthe uctuations rather than a convergent network state. For this purpose wedecrement r at each step asr =  1NTr 8r (3)and reset whenever  > 1 the 's in the vicinity of the topology distortion asr := max r;  exp  2 (r  R)22!!; where R =12(r0 + r00) : (4)As a result, the map uctuates around the PM due to the phase transition takingplace each time the critical value crit is crossed. In order to average over theuctuations each neuron keeps a second pointer wr obtained by the movingaveragewr =1KNT(wr  wr) (5)over the uctuations, where K is of the order of 10. The wr provide in mostcases a very good rst order data model. Further improvements depend on thetask. In the case of modelling a functional relationship (see introduction) one-10-8-6-4-202468-12 -10 -8 -6 -4 -2 0 2 4 600.511.522.533.540 5 10 15 20 25 30 35 40 45 50Fig. 1. The map of a two-dimensional data distribution of varying scattering widthonto a one-dimensional chain of 50 neurons (left). Final values of r along the neuralchain produced by enforcing topology preservation (right).051015202530 0510152025300510051015 05101555.566.5Fig. 2. Embedding a two-dimensional neural lattice in a three-dimensional data set(left). Values of the neighborhood widths r as a function of the net position r (right).may use the wr to investigate the properties of the noise  in order to improvethe model. For the PM case, an essential improvement consists in using the wras starting positions for a nal step in the sense of the iterative HS algorithm.This can be implemented more easily by monitoring directly the averages overthe data in each domain. Hence, instead of wr each neuron gets a second pointervr measuring the local average of the input data which may deviate from wrif the principal manifold is curved. vr is updated if neuron r is the winneras vs =1KT(v   vs) The set fvr j r = 1; :::; Ng are the nal result of thealgorithm, i. e. they represent the PM in input space.3 Sparse data setsThe above algorithm hinges on the abundance of data points which signalthe folding via the topology violations. This may fail if the number of datapoints is small. For this case, a very sensitive criterion for the emergence of thecritical uctuations was found to be a wavelet transform [7] of the map. For aone-dimensional SOM we use the Gabor transformgr0 =1p2ur0NXk=1wr0 exp  (k   r0)22u2r0!exp ( i k!r0 ) (6)where both the frequency !r0 of the kernel and the width are functions of thecurrent values of r0 so that the kernel is always in resonance with potential-0.5-0.4-0.3-0.2-0.100.10.20.30.40.5-3 -2 -1 0 1 2 3Fig. 3. Mapping a noisy sin-wave onto a chain of neurons. The strong noise in thecenter of the data distribution is clipped by the map.-0.200.20.40.60.81-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 116.1516.216.2516.316.3516.40 10 20 30 40 50 60 70 80 90 100Fig. 4. Using wavelet transform to control the neighborhood widths in a sparse dataset. Display of the emerged map (left) and r-values (right).foldings. At the critical point  = crit the wavelength of the emerging foldsis  = 4:04l, where l is the average distance between the neurons in thatregion, cf. [3]. Choosing !r0 = ur0 = 4r0 causes gr0 to jump by an order ofmagnitude when r0 drops below critr0 . Hence, gr0 is the desired sensitive criterionfor detecting the onset of the phase transition. In the algorithm we use (3) asbefore. If for the winner gr0 exceeds a small threshold we use  = r0 in (4),observing 1    max, where  = 1:2 is an empirical factor. In the simulationsa control of  is obtained from monitoring the uctuations of  which optimallyshould stay in the region of a few percent.4 Numerical simulationsWe have applied the above algorithms to map both one- and two-dimensionallattices into higher-dimensional input spaces with inhomogeneous data distribu-tions of eective dimension D = 1 or D = 2, respectively. The local scatteringof the data points around the central manifold varied by up to an order of mag-nitude. In all simulations the algorithms were stable and produced the desiredresults. In particular the cooling time T could be varied by more than an orderof magnitude without instability problem. Parameters used in the simulationswere T = 1500, max = N=3, and " = 0:15 (kept constant during the simu-lations. It should be noted that such relatively high values of " allow for fastconvergence and improve the eciency of the average (5)), which in turn com-pensates the uctuation in the nal map wr. The rst and the second example(Figs. 1 and 2, respectively) are carried out by the basic algorithm explained in00.050.10.150.20.250.30.350.41000 2000 3000 4000 5000 6000 7000Fig. 5. Time course of the wavelet transform of one of the neurons during 7000 steps.the second section. In the third example shown in g. 4 we use wavelet trans-form to adjust the r. A data set of 80 points and a chain of 100 neurons arechosen. In practical computation it is easy to use FFT for computing the Gabortransform function.5 ConclusionThe present contribution is dedicated to the problem of building nonlineardata models with a modied Kohonen learning rule which learns its own parame-ters. In particular, the neighborhood range of the output units, which determinesthe smoothness of the produced maps, is an essential prerequisite for the forma-tion of principal manifolds in the input data set. Our algorithm solves the task ofthe determination of the neighborhood parameter stably and in a local manner.This is of importance for processing sparse data sets with spatial heterogeneities.Therefore, the algorithm appears to be well suited for more complex tasks. Inparticular, the algorithm was found to be also stable with higher-dimensional(up to d = 8) inputs. The application to real world data is underway.Acknowledgement: One of the authors (RD) gratefully acknowledges the hospitalityof RIKEN (Tokyo) he received during a visit in February 1996.References1. T. Hastie, W. Stuetzle, Principal curves. Journal of the American Statistical Asso-ciation 84, 502{516 (1989)2. T. Kohonen, The Self-Organizing Map, Springer-Verlag, 1995.3. H. Ritter, T. Martinetz, K. Schulten, Neural Computation and Self-OrganizingMaps, Addison-Wesley, 1992.4. F. Mulier, V. Cherkassky, Self-Organization as an Iterative Kernel Smoothing Pro-cess. Neural Computation 7, 1165-1177, 1995.5. R. Der, M. Herrmann, Critical Phenomena in Self-Organized Feature Maps: AGinzburg-Landau Approach, Phys. Rev. E 49:6, 5840{5848, 1994.6. M. Herrmann, \Self-Organizing Feature Maps with Self-Organizing NeighborhoodWidths", Proc. 1995 IEEE Intern. Conf. on Neur. Networks, 2998{3003, 1995.7. C. K. Chui, An Introduction to Wavelets, Academic Press, 1992.This article was processed using the LaTEX macro package with LLNCS style

Building nonlinear data models with self-organizing maps

https://ul.qucosa.de/api/qucosa%3A32410/attachment/ATT-0/

Building nonlinear data models with self-organizing maps

Abstract

Similar works

Full text

Available Versions

Qucosa - Publikationsserver der Universität Leipzig