A large number of practical problems involves elements that are described as a mixture of qualitative and quantitative infomation, and whose description is probably incomplete. The self-organizing map is an effective tool for visualization of high-dimensional continuous data. In this work, we extend the network and training algorithm to cope with heterogeneous information, as well as missing values. The classification performance on a collection of benchmarking data sets is compared in different configurations. Various visualization methods are suggested to aid users interpret post-training results.Peer ReviewedPostprint (author's final draft

Belanche Muñoz, Luis Antonio

Negri, Sergio

English

A large number of practical problems involves elements that are described as a mixture of qualitative and quantitative infomation, and whose description is probably incomplete. The self-organizing map is an effective tool for visualization of high-dimensional continuous data. In this work, we extend the network and training algorithm to cope with heterogeneous information, as well as missing values. The classification performance on a collection of benchmarking data sets is compared in different configurations. Various visualization methods are suggested to aid users interpret post-training results.Peer Reviewe

UPCommons

Heterogeneous Kohonen networks

UPCommons. Portal del coneixement obert de la UPC

Heterogeneous Kohonen networksSergio Negri and Llus A Belanche Politecnico di TorinoCorso Duca degli Abruzzi Torino ItalysernegritinitDept de Llenguatges i Sistemes InformaticsUniversitat Politecnica de CatalunyacJordi Girona Salgado 		 Barcelona SpainbelanchelsiupcesAbstract A large number of practical problems involves elements thatare described as a mixture of qualitative and quantitative informationand whose description is probably incomplete The selforganizing mapis an eective tool for visualization of highdimensional continuous dataIn this work we extend the network and training algorithm to cope withheterogeneous information as well as missing values The classicationperformance on a collection of benchmarking data sets is compared indierent congurations Various visualization methods are suggested toaid users interpret posttraining results  IntroductionKohonen networks also known as selforganizing maps 	  were bornto emulate the human brain characteristic of topological and geometricalorganization of information The training algorithm aims at nding analogiesbetween similar incoming data in a nonsupervised processThe algorithm places the weight vectors such that geometrically close vectorsin weight space are also topologically close in the grid represented by thenetwork In other words for similar incoming vectors in input space theneurons responding more vigorously should also be similar in terms of theirweight vectors and located in nearby positions in the network gridThe interest in using these networks is not limited to the discovery ofregularities or to tracking the input data density Once the training process hasended the result can be visualized Furthermore it the class labels are availablethey can be superimposed to the discovered regularities Much information canbe extracted from such graphical plots although it has to be identied with care   Data heterogeneityRealworld data come from many dierent sources described by mixturesof numeric and qualitative variables These variables include continuous orNegri, S.; Belanche, L. Heterogeneous Kohonen networks. A: International Work-Conference on Artificial and Natural Neural Network. "Connectionist models of neurons, learning processes, and artificial intelligence: 6th International Work-Conference on Artificial and Natural Neural Networks, IWANN 2001: Granada, Spain, June 13-15, 2001: proceedings, part 1". Berlín: Springer, 2001, p. 243-252. The final authenticated version is available online at https://doi.org/10.1007/3-540-45720-8_28discrete numerical processes symbolic information etc In particular qualitativevariables might have a dierent nature Some are ordinal in the usual statisticalsense ie with a discrete domain composed by k categories but totally orderedwrt a given relation or nominal discrete but without an ordering relationThe data also come with their own peculiarities vagueness uncertaintyincompleteness and thus may require completely dierent treatmentsIn the neural network paradigm this heterogeneity is traditionally handledif at all by preparing the data using a number of coding methods so that allvariables are treated as real quantities However this preprocessing is not partof the original task and may have deep consequences in the structure of theproblem These consequences range from a change in input distribution to anincrease in dimension which results in a growth in the number of weights thenetwork is forced to learn an added diculty in their interpretation an increasein training time and so on The choice of representation if any should beas faithful as possible in the sense that the relations between the representedentities should correspond to meaningful relations on the original data items  Euclidean geometryThe common assumption of articial neural models about the Euclidean natureof the input space leads naturally to the use of a scalar product or a distancemetric as the standard neuron models usually followed by a nonlinear activationfunction In particular this space is taken to be Rn with the customarydenition of scalar product This assumption not only means that the features ofthe problem at hand can be expressed in terms of vectors of real quantities butalso that scalar product and Euclidean distance are adequate ways of measuringthe similarity between elements in the spaceIn consequence in order to determine which neuron has a weight vector thatis more similar to the input there are two possibilities choose the neuron whichmaximizes the scalar product or choose that which minimizes the distance Thesemethods may or may not be meaningful for a particular problem and in principleare only appropriate for realvalued vectors What to do in cases where inputpatterns contain heterogeneous information In this case a similarity index canbe used as a more exible way to measure likeness  Aims and structureThe aim of this work is to extend the main ideas of Kohonen networks in sucha way that they can work in generic heterogeneous spaces making the neuronscompute a similarity measure among the elements of the space This is to bedone in stages which we call network congurations in order to appreciatethe eect of each decision The resulting networks are studied regarding twoaspects classication accuracy and ability to express meaningful information ina visual way From the point of view of classication accuracy it is shown howthe consideration of heterogeneous andor incomplete information without theneed of a coding scheme results in signicantly better classiers On the otherhand it is illustrated how a solution can be visually analyzedThe paper is organized as follows In section  we introduce the datacharacteristics considered in this work Sections  and 	 briey review theKohonen algorithm and the basics of a similarity measure Section  describesthe proposed extension of the algorithm The last two sections present practicalmatters about classication ability in benchmarking data sets section  andvisualization of the results section  Data heterogeneityWe consider in this work the following types of variables for which correspondingsimilarity measures are to be denedNominal categorical  nonnumerical variable on which no order relationhas been dened It thus can be seen as having a set of values nite or notOrdinal  variable numerical or not for which a linear order relation has beendened on a nite number of values where each value has a precise senseContinuous  numerical and crisp variable for which a linear order relation hasbeen dened on a continuum of valuesLinguistic  variable whose values are expressing uncertainty in the form ofvagueness eg cool fast youngThe values of the last type can be obtained where appropriate byconverting an existing set of values ordinal or continuous into fuzzy quantitiesIn all cases we assume some values may be missing in a particular data set The Kohonen network and algorithmThe classical Kohonen network 	 assumes a set of laterally interactingadaptive neurons usually arranged as a twodimensional sheet a rectangulargrid of neurons Each neuron r is represented by an ndimensional prototypevector wr where n is the dimension of the input space On each training step ta data sample xt is presented to the network and the unit wsmost similar toxt is identied the Best Matching Unit or BMU The adaptation step shiftswsand those wrcorresponding to neighbouring units r to s towards xtwrt  wrtthrst xt wrt for all units r in the grid In this formula hrst establishes the scope and amount of the changescentered on the BMU s at time t In other words it represents the varyingneighbourhood The factor t acts as a learning ratio controlling the size of theadaptive steps towards the input vector at time t Both are decreasing functionsof time In this work we use the following commonly found formulashrst  exp  dr stwith t  infininttmaxt  tt t    if t  t t t t tmax t t if t  t  for all t  tmax In formula  dr s is the topological distance between units in thenetwork structure In case this structure is a rectangular grid the standardEuclidean distance dr s  kr   sk can be used Note also that formula ensures that the BMU would be the same in the hypothetical case xt  xtThe initial vectors wr are set to small random valuesAs stated in  in order to determine the BMU there are two basicpossibilities pick the neuron with the greatest scalar product to the input vectoror pick that with the smallest Euclidean distance The rst choice usuallyinvolves vector normalization both input and weight The Euclidean distance isa more general criterion which reduces to scalar product for normalized vectorsand is usually adopted to compute similarity closeness in this case in inputspace In this work all realvalued variables are standardized to zeromeanunit standard deviation so that all of them have a uniform inuence in thecomputation of distance~~ Similarity measures  DenitionLet us represent input patterns belonging to a space X   about which the onlyassumption is the existence of an equality relation as vectors xiof n componentswhere each component xijrepresents the value of a particular feature descriptivevariable ajfor object i from a predened set of features A  fa  a     angjudged by the investigator as relevant to the problem A similarity measure isa unique number expressing how like two given objects are given only thefeatures in A  Let us denote by sijthe similarity between xiand xj that iss  X X  R fg and sij sxixjDenition    A similarity measure in X fullls the following properties Nonnegativity sij  	xixj X Symmetry sij sji	xixj X Boundedness There is a maximum attained similarity 	that of an object withitself smax R sij smax	xixj X Minimality sij smax xi xj	xixj X Semantics The meaning of sij sikis that object i is more similar to objectj than is to object k Heterogeneous measuresIn this section we dene similarity measures with smax  for the types ofvariables mentioned in x Let xy be two heterogeneous vectors	 For nominal variablessxk yk   if xk yk if xk yk		 For ordinal variablessxk yk  jxk ykjrankkwhere rankk is the number of values the variable k can take	 For continuous variablessxk yk   jxk  ykjwhere xk ykare standardized continuous variables	 For linguistic variables where their values are fuzzy intervals of the LRtype   the problem is a bit more complex We started using trapezoidsbut we found that better results were in general achieved with the followingbellshaped functionAx   ax mwhere m is the mean and a controls the fuzziness Let FX be the crisp setof all such fuzzy intervals in X Given!A!B  F with  a real intervaland respective support sets AB  we dene I!A!B ZABABudu" U !A!B ZABABudu #and thensxk yk Ixk ykU xk yk  Gowers similarity indexA basic but very useful similaritybased neuron can be devised using a Gowerlikesimilarity index wellknown in the literature on multivariate data analysis For any two objects xi xjof cardinality n this index is given by the expressionsGxixj Pnk skxik xjk 	ijkPnk 	ijkwhere skis the partial similarity index according to variable k and 	ijkis abinary function expressing whether the objects are comparable or not accordingto variable k Let X represent a missing value then	ijk  if xikX  xjkX otherwise Heterogeneous algorithm for the Kohonen networkThe weight propagation in equation  can only be applied to real values Forother variable types the following propagation formulas are used Let writrepresent the ith component of weight vector wrt	 For ordinal variableswrit   bwrit  thrst xit  writ  c 	 For linguistic variables formula  is applied to both m and a in 	 For nominal variables a deeper explanation is needed In all of the previouscases there is a linear order continuous or discrete Hence the notion ofgetting closer to the input vector makes sense and the basic formula can be applied In absence of an order there is no shift but a change in valuewhich can be regarded as a more general concept In addition though theintuition behind the product thrst must be kept its practical role hasto be dierent We take it as the probability of such a changeIn essence a random number     with uniform probability is generatedThen the updating rule is given bywrit   xit if thrst  writ otherwiseThis scheme is intuitively pleasing but has a serious drawback it ignores thepast We hence propose an updating rule in which the probability of changingthe weight vector component increases proportionally to the number of timesthis change was already attempted relative to the number of attempts of thecurrent value of the weight and the rest of possible values and inuencedby the distance dr s where s is the BMU for xt see the Appendix fordetails An experimental comparisonA number of experiments are carried out to illustrate the validity of the approachusing several benchmarking problems These are selected as representativesbecause of the diversity in the kind of problem and richness in data heterogeneity  Problem descriptionA total of ten learning tasks are worked out taken from  and  and altogetherrepresentative of the kinds of variables typically found in real problems whiledisplaying dierent degrees of missing information from $ to $ Their maincharacteristics are displayed in Table Table  Basic features of the data sets Missing refers to the percentage of missingvalues R real N nominal I ordinal and L linguisticName Cases Heterogeneity Classes Class distribution MissingCredit Card  RNIL   Heart Disease 		 RNIL   Horse Colic 	 RNIL  	 Solar Flares  RNIL   Gene Sequences 	 RNIL 	  Mushroom  RNIL   Cylinder bands  RNIL   Meta data  RNIL 	 			 Servo data  RNIL  	 Annealing data  RN	IL    Dierent network congurationsThe extended algorithm has been tested in seven dierent congurations In allcases formula  is used as follows Kohonen network treating all attributes as realvalued with the usual outofk coding for nominal ones with k the number of values and an extraattribute signaling a missing value for those attributes that can be missing As  but coding nominals as ~~ As  without the coding for missing values thus treating them directly byusing formula 	 As  using formula  for ordinal attributes considering all linguisticattributes as ordinals and  for nominals and  as 	 but using two dierent ways of enhancing  for nominals seethe Appendix As  considering linguistic attributes as indicatedIn all cases the interest is not in building a classier per se but inassessing the relative dierences between network congurations Hence all theinformation is used to train the network without the class labels After thisprocess has ended the network is colored using the weighted global method see section  and its classication accuracy computed The results are showngraphically in Fig Fig  Graphical impression of the results The horizontal axis stands for the networkconguration The vertical axis shows classication accuracy Each plotted value is theaverage of forty runs Visualization methodsIn order to help users understand the posttraining result of the network we usetwo dierent visualization methods  Weighted global BMUAt the end of the training process we assign a class to each neuron of the gridthat is we color the grid For each neuron r we compute its receptive eld F rcomposed of all those input vectors for which its BMU is r A basic criterion isthen to assign to r the majority class in F r This criterion does not take intoaccount the relevance of each vector in F r A simple way to do this is to weighteach vector in F r by its similarity to wr Afterwards the class with greaterweighted similarity to wris chosen to label r If a neuron is never a BMU it isnot assigned to any class and is painted neutral white in Fig  left UmatrixAs shown in # the Umatrix is useful to visualize similarity among contiguousmap units The minimum similarity between each grid unit and its four adjacentneighbours is computed and displayed using grey shade Fig  center Wehave also found very useful displaying the contour plot of the same UmatrixFig  right and compare it to the weighted global BMU ConclusionsAn extension of Kohonen%s selforganizing network and the correspondingtraining algorithm has been introduced that works in heterogeneous spaces forFig  Visualization of the results in a twoclass example Left Global BMU CenterUmatrix Right Umatrix contour plotboth input and weight vectors The inner workings of a neuron are grounded ona similarity measure which allows for more exibility and a builtin treatmentof heterogeneous or incomplete data It is also shown how to visualize the resultof the training process Some results that have been obtained for illustrativepurposes in several benchmarking data sets indicate a superior performance ofthe extended algorithmAcknowledgementsThis work is supported by the Spanish CICYT grant TAP  	References Belanche Ll Similaritybased Heterogeneous Neuron Models In Procs ofECAI European Conf on Articial Intelligence IOS Press  Chandon J Pinson S Analyse Typologique Theorie et ApplicationsMasson 	 Gower JC A General Coecient of Similarity and some of its PropertiesBiometrics    Kohonen T SelfOrganization and Associative Memory SpringerVerlag  Murphy PM Aha D UCI Repository of machine learning databases UCI Deptof Information and Computer Science  Prechelt L Proben A set of Neural Network Benchmark Problems andBenchmarking Rules Fac fur Informatik Univ Karlsruhe Tech Report   Ritter H Kohonen T SelfOrganizing Semantic Maps Biological Cybernetics   Vesanto J SOMbased data visualization methods Intel data analysis 	  Zimmermann HJ Fuzzy set theory and its applicationsKluwer Acad Publ A AppendixDuring weight propagation for nominal variables a random number    with uniform probability is generated Then writ   is updated to xit ifpt   with pt  thrst otherwise it is left unchangedLet nbe the number of times writ has been proposed to be the value ofwriup to time t nthe number of times xit has been proposed and let ngenerally denote the number of times any other possible value of nominal variablei has been proposed to be wri up to time t If there are no more possible valuesthat is if ranki   then nis undened Dene thenft nnsranki rankiP     nif ranki  nnif ranki  	In these conditions the new probability of change is dened aspt  thrst lne ftWhereas a simpler possibility isp~~t  thrst ftNetwork conguration 	 uses pt conguration  uses pt andconguration  uses p~~t

https://upcommons.upc.edu/bitstream/2117/182259/3/iwann01.pdf

Heterogeneous Kohonen networks

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC