We investigate, by a systematic numerical study, the parameter dependence of
the stability of the Kohonen Self-Organizing Map and the Zheng and Greenleaf
concave and convex learning with respect to different input distributions,
input and output dimensions

E. Erwin

H. Ritter

J.C. Claussen

R. Linsker

S. Zrehen

T. Heskes

T. Kohonen

T. Villmann

T.M. Martinetz

Y. Zheng

English

arXiv

Crossref

Investigation of Topographical Stability of the Concave and Convex Self-Organizing Map Variant

We investigate, by a systematic numerical study, the parameter dependence of the stability of the Kohonen Self-Organizing Map and the Zheng and Greenleaf concave and convex learning with respect to different input distributions, input and output dimensions

Molle, Fabien

Claussen, Jens Christian

Aston Publications Explorer

arXiv:cond-mat/0609510v1  [cond-mat.dis-nn]  20 Sep 2006 Published in: Lecture Notes in Computer Science 4131, 208-215 (2006).Investigation of topographical stability of theconcave and convex Self-Organizing Map variantFabien Molle1,2 and Jens Christian Claussen2∗1Theoretical Physics, Chalmers Tekniska Ho¨gskola, Go¨teborg2Institut fu¨r Theoretische Physik und AstrophysikLeibnizstr. 15, 24098 Christian-Albrechts-Universita¨t zu Kiel, Germanyhttp://www.theo-physik.uni-kiel.de/~claussen/AbstractWe investigate, by a systematic numerical study, the parameter depen-dence of the stability of the Kohonen Self-Organizing Map and the Zhengand Greenleaf concave and convex learning with respect to different inputdistributions, input and output dimensions.Topical groups: Advances in Neural Network Learning Methods, Neural andhybrid architectures and learning algorithms, Self-organization.Neural vector quantizers have become a widespreadly used tool to explore high-dimensional data sets by self-organized learning schemes. Compared to the vastliterature on variants and applications that appeared the last two decades, the the-oretical description proceeded more slowly. Even for the coining Self-OrganizingMap (SOM) [1], still open questions remain, as a proper description of the dy-namics for the case of dimension reduction and varying data dimensionality, orthe question for what parameters stability of the algorithm can be guaranteed.This paper is devoted to the latter question. The stability criteria are especiallyinteresting for modifications and variants, as the concave and convex learning [2],whose magnification behaviour has been discussed recently [3]. Especially for thevariants, analytical progress becomes quite difficult, and in any case one will ex-pect that the stability will depend on the input distribution to some —apart fromspecial cases— unknown extent. As the invariant density in general is analyti-cally unaccessible for input dimensions larger than one (see [4] for recent tractablecases), we expect a general theory not to be available immediately, and insteadproceed with a systematic numerical exploration.The Kohonen SOM, and the nonlinear variant of Zheng and Greenleaf. – Theclass of algorithms investigated here is defined by the learning rule, that for eachstimulus v ∈ V each weight vector wr is updated according townewr= woldr+ ε · grs · (v −woldr)K (1)(grs being a gaussian function (width σ) of euclidian distance |r− s| in the neurallayer, thus describing the neural topology). Herein|ws − v| = minr∈R |wr − v| (2)determines for each stimulus v the best-matching unit or winner neuron.∗Corresponding author.208F. Molle and J. C. Claussen Investigation of topographical stability 209The case K = 1 is the original SOM [1], corresponding to a linear or Hebbianlearning rule. The generalization to K or 1/K taking integer values has been pro-posed by Zheng and Greenleaf [2], but arbitrary nonzero real values of K can beused [3], and the choice of K → 0 has been shown (for the onedimensional case)to have an invariant density with the information-theoretically optimal value ofthe magnification exponent one [3], i.e., the neural density is proportional to theinput density and hence can be used as a density estimator.Convergence and stability. – It is well known that for the learning rate ε, onehas to fulfill the Robbins-Munro conditions (see, e.g. [4]) to ensure convergence,with all other parameters fixed. However, practically it is necessary to use a largeneighborhood width at the beginning, to have the network of weight vectors or-dered in input space, and decrease this width in the course of time downto a smallvalue that ensures topology preservation during further on-line learning. Thus thesituation becomes more involved when additionally also σ is made time-dependent.Here we consider the strategy where the stability border in the (ε, σ) plane alwaysis approached from small ε with σ fixed during this final phase. An ordered statehas to be generated by preceding learning phases.Measures for Topographical Stability. – To quantify the ordered state and thetopology preservation, a variety of measures is used, e.g. the topographic product[5], the Zrehen measure [6], and the average quadratic reconstruction error. Todetect instable behaviour, all measures should be suitable and give similar results.For an unstable and disordered map, also the total sum over all (squared) distancesbetween adjacent weight vectors will increase significantly; so a threshholded in-crease will indicate instability as well. This indicator is used below; however, forthe case of a large neighborhood (of network size), the weight vectors shrink to asmall volume, thus influencing the results; however, this applies to a neighborhoodwidths larger than that commonly used for the pre-ordering.In addition we use here an even more simple approach than the Zrehen mea-sure (which counts the number of neurons that lie within a circle between eachpair of neurons that are adjacent in the neural layer). For a mapping from d to ddimensions, we consider the determinant of the i vectors spanned by ~w~r+~ei − ~w~r,with ~ei being the ith unit vector. The sign of this determinant, where 1 ≤ i ≤ d,thus gives the orientation of the set of d vectors. Note that the 1-dimensional casejust reads sgn(wr+1−wr), which has been widely considered to detect the orderedstate in the 1 to 1 dimensional case. Hence, we can defineχ({~w~r}) := (3)1−1N∣∣∣∣∣∑~rsgn(det((~w~r+~e1 − ~w~r), . . . (~w~r+~ei − ~w~r), . . . (~w~r+~ed − ~w~r)))∣∣∣∣∣ .This evaluates the number of neurons N+ (resp. N−), where this sign is pos-itive (resp. negative), hence the relative fraction of minority signs is given by(1− |N+−N−|/N). A typical single defect is shown in Fig. 1. Due to its simplic-ity, this measure χ will be used in the remainder.F. Molle and J. C. Claussen Investigation of topographical stability 2100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91Figure 1: Situation where one defect is detected by the crossproduct measure (3).Modification of learning rules and data representation. – A classical result[7, 8, 9] states that the neural density for 1-D SOM (in the continuum limes)approaches not the input density itself, but a power of it, with exponent 2/3, theso-called magnification exponent. As pointed out by Linsker [10], the case of anexponent 1 would correspond to the case of maximal mutual information betweeninput and output space. Different modifications of the winner mechanism or thelearning rule, by additive or multiplicative terms, have been suggested and influ-ence the magnification exponent [11, 12, 13, 14]. Here we investigate the case ofconcave and convex learning [2, 3], which defines a nonlinear generalization of theSOM.Topographical Stability for the Self-Organizing Map. – Before investigating thecase of concave and convex learning, the stability measures should be tested forthe well-established SOM algorithm. Using the parameter path of Fig. 2, we firstanalyze the 2D → 2D case, for three input distributions: the homogeneous inputdensity (equidistribution), an inhomogeneous input distribution ∼ sin(πxi) [14],and a varying-dimension dataset (Figs. 3, 4). The results are shown in Fig. 5.σεFigure 2: Schematic diagram of the parameterpath in (ε, σ) space. Starting with high val-ues, σ is slowly decreased to the desired value,while the learning rate still is kept safely low.From there, at constant σ the learning rate isincreased until instability is observed; givingan upper border to the stability area. – Thesame scheme is applied for the concave and con-vex learning, where the nonlinearity exponent isconsidered as a fixed parameter.Different input dimensions and varying intrinsic dimension. – As the input di-mensionality is of pronounced influence on the maximal stable learning rate (Fig.6), we also investigate an artificial dataset combining different dimensions: theF. Molle and J. C. Claussen Investigation of topographical stability 211box-plane-stick-loop dataset [15] (Fig. 3), or its 2D counterpart, the plane-stick-loop (Fig. 4). Here the crossproduct detection will become problematic where theinput space is intrinsically 1D (stick and loop), thus the average distance criterionis used, and we restrict to the case σ ≤ 1.Figure 3: Schematic view of the classical box-plane-stick-loop dataset. Its moti-vation is to combine locally different input data dimensions within one data set.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.050.10.150.20.25PlaneStick−loop data inputFigure 4: Part of the input data for the 2D plane-stick-loop data set (Fig. 3).F. Molle and J. C. Claussen Investigation of topographical stability 21210−2 10−1 100 10110−210−1100sigmaepsilonmaxAverage distance based stability measuresSin input spaceEquiprobable input spacePlaneStick−loop input space10−2 10−1 100 10110−310−210−1100sigmaepsilonmaxCrossproduct based stability measuresSin input spaceEquiprobable input spaceFigure 5: Critical εmax(σ) where (coming from small ε values, see Fig. 2) the SOMlearning loses stability. Here a 2D array of 10×10 neurons was used with decayexp(−t/k) exponentially in time t, with k between 30000 and 60000 dependingon ε0 (for σ between 0.1 and 0.001, k = 300000). Top: Unstable ε detected fromgrowth of the averaged distance of neurons; here a threshold of 15% was chosen.For large σ, this measure becomes less reliable due to shrinking of the network,i.e. ∀~r ~w~r → 〈~v〉. Bottom: Unstable ε detected from the crossproduct measure, eq.(3), with threshold of 1 defect per 100 iterations. The ε value depends on the datadistribution, but the qualitative behaviour remains similar. In all cases, below acertain σcrit of about 0.3, ε has to be decreased significantly. This independentlyreproduces [16], here we investigate also different ε values.F. Molle and J. C. Claussen Investigation of topographical stability 21310−2 10−1 100 10110−210−1100101sigmaepsilonmaxAverage distance based stability measures1D−1D2D−2D3D−3D10−2 10−1 100 10110−210−1100101sigmaepsilonmaxCrossproduct based stability measures1D−1D2D−2D3D−3DFigure 6: Stability border dependence on input dimension (1D, 2D, 3D). Theknown 1D case is included for comparison. Top: Using the average length criterion(for σ > 1, the result can be misleading due to total shrinking of the network, seetext). Bottom: Using the crossproduct detection for defects, similar results areobtained; for large σ instabilities are detected earlier.F. Molle and J. C. Claussen Investigation of topographical stability 214Concave and convex learning: Stability of nonlinear learning. – The simulationresults are given in Fig. 7: Clearly, a strong influence of the nonlinearity parameterK is observed. Especially one has to take care when decreasing σ, because for toolarge ǫ the network becomes instable. For K < 1, much smaller values of ǫ arepossible, thus considerably longer learning phases have to be taken into accountcompared to original SOM. For K > 1 the stability range becomes larger.10−2 10−1 100 10110−310−210−1100101sigmaepsilonmaxCrossproduct based stability measuresK=2.0K=1.5K=1.0K=0.5K=0.2K=0.1K=010−2 10−1 100 10110−310−210−1100101sigmaepsilonmaxAverage distance based stability measuresK=2.0K=1.5K=1.0K=0.5K=0.2K=0.1K=0Figure 7: Critical εmax(σ) for different values of the nonlinearity parameter fromK = 2.0 (top) to K = 0 (bottom). K = 1 corresponds to the SOM case.F. Molle and J. C. Claussen Investigation of topographical stability 215Discussion. – We have defined a standardized testbed for the stability analysis ofSOM vector quantizers with serial pattern presentation, and compared the SOMwith the recently introduced variants of concave and convex learning. The stabilityregions for different input distribution and dimension are of the same shape, thusqualitatively similar, but not coinciding exactly. The neighborhood width, butunfortunately also the input distribution affect the maximal stable learning rate.For the concave and convex learning, the exponent steering the nonlinear learningalso crucially influences the learning rate. In all cases, a plateau for σ ≪ 1 isfound where the learning rate must be quite low compared to the intermediaterange 0.3 ≤ σ ≤ 1. As a too safe choice of the learning rate simply increasescomputational cost, an accurate knowledge of the stability range of neural vectorquantizers is of direct relevance in many applications.References[1] T. Kohonen. Self-Organized Formation of Toplogically Correct FeatureMaps. Biological Cybernetics 43 (1982) 59-69[2] Y. Zheng and J. F. Greenleaf. The effect of concave and convex weightadjustments on self-organizing maps. IEEE Transactions on Neural Networks,7 (1996) 87-96[3] T. Villmann and J. C. Claussen, Investigation of Magnification Controlin Self-Organizing Maps and Neural Gas. Neural Computation 18 (2006)449-469[4] T. Kohonen. Comparison of SOM point densities based on different criteria.Neural Computation 11 (1999) 2081-2095[5] T. Villmann, R. Der, M. Herrmann, and T. Martinetz. Topology Preservationin Self-Organizing Feature Maps: Exact Definition and Measurement. IEEETransactions on Neural Networks, 8 (1997) 256-266[6] S. Zrehen. Analyzing Kohonen maps with geometry. In Gielen, S. andKappen, B., editors, Proc. ICANN’93, p. 609-612, Springer, London (1993)[7] H. Ritter and K. Schulten. On the Stationary State of Kohonen’s Self-Organizing Sensory Mapping. Biological Cybernetics 54 (1986) 99-106[8] H. Ritter and K. Schulten. Convergence Properties of Kohonen’s TopologyConserving Maps: Fluctuations, Stability and Dimension Selection. Biologi-cal Cybernetics 60 (1988) 59-71[9] H. Ritter, T. Martinetz, and K. Schulten. Neural Computation and Self-Organizing Maps: An Introduction. Addison-Wesley (1992)[10] R. Linsker. How to generate maps by maximizing the mutual informationbetween input and output signals. Neural Computation 1 (1989) 402-411[11] T. Heskes. Energy functions for self-organizing maps. In E. Oja and S. Kaski,editors, Kohonen Maps, p. 303-316. Elsevier, Amsterdam (1999)[12] E. Erwin, K. Obermayer, and K. Schulten. Self-organizing maps: Ordering,convergence properties and energy functions. Biol. Cyb. 67 (1992) 47–55[13] J. C. Claussen, Generalized Winner Relaxing Kohonen Feature Maps, NeuralComputation 17 (2005) 996-1009[14] J. C. Claussen and T. Villmann, Magnification Control in Winner RelaxingNeural Gas, Neurocomputing 63 (2005) 125-137 2005.[15] T. M. Martinetz, S. G. Berkovich, and K. J. Schulten. ’Neural-gas’ networkfor vector quantization and its application to time-series prediction. IEEETrans. on Neural Networks, 4 (1993) 558–569[16] T. Villmann. PhD thesis, Leipzig, Verlag Harri Deutsch, Frankfurt (1998)

Investigation of topographical stability of the concave and convex self-organizing map variant

Chalmers Research

Investigation of topographical stability of the concave and convex
  Self-Organizing Map variant

Investigation of topographical stability of the concave and convex Self-Organizing Map variant

Abstract

Similar works

Full text

Available Versions

Crossref

Aston Publications Explorer

Chalmers Research