Adams, R.G.

Davey, N.

Pensuwon, W.

English

University of Hertfordshire Research Archive

OPTIMISING A STOCHASTIC DYNAMIC NEURAL TREEW.Pensuwon1,2, R.G.Adams2, and N.Davey21 Department of Computer Science,University of Hertfordshire, Hatfield, Herts,AL10 9AB,United Kingdom., Tel: +44 01707 2843212 Department of Electrical and Electronics Engineering,Faculty of Engineering, Ubonratchathani University,34190, Thailand., Tel: +66 045 288376-8ABSTRACTThis paper describes experiments performed using aGenetic Algorithm (GA) to optimise the parameters of anovel model of a stochastic hierarchical neural clusterer.Two issues of enhancing and optimising the model arediscussed. Two fitness functions were created from twoselected clustering measures, and a population ofgenotypes, specifying parameters of the model wereevolved. Using the idea of optimising the model by a GAhas been proven to be useful. This process mirrorsgenomic evolution and ontogeny.1. INTRODUCTIONOrganising the output nodes of a competitive networkinto a tree structure may produce potential improvementsin both convergence and recall rates.  It may also make thenetwork more stable in responding to changes in the databeing classified.  However a limitation of most clusteringalgorithms is that they require the number of classes to bepredefined.  This is a problem if no a-priori knowledgeabout the data is available [1,2].Dynamic neural tree networks (DNTN) may avoid theabove limitation by dynamically creating nodes andautomatically creating a tree structure.  However, all theoriginal models require parameters to be externally set,which will significantly influence the hierarchical structureand the classification that is produced [10].A more recent DNTN is the Stochastic CompetitiveEvolutionary Neural Tree (SCENT), [8,9]. This model iscomparatively robust, with respect to its parameter settingswhen compared with other DNTNs [8]. Nonetheless, it isunclear precisely how the internal parameters affect theperformance, which lead to the investigation reported here.In order to explore the parameter space a good default setof parameters was needed, and a Genetic Algorithm (GA)was used as the search tool. A given genome (a set ofparameters), together with a data set allowed SCENT toproduce a classificatory tree. Twelve data sets were used.The fitness of these trees was calculated using twoclustering measures.2.  STOCHASTIC COMPETITIVEEVOLUTIONARY NEURAL TREEIn SCENT, the tree structure is created dynamically inresponse to structure in the data set. The neural tree startswith a root node with its tolerance (the radius of itsclassificatory hypersphere) set to the standard deviation ofinput vectors and its position set to the mean of inputvectors.  It has 2 randomly positioned children.  Each nodehas two counters, called inner and outer, which count thenumber of occasions that a classified input vector is withinor outside tolerance, respectively.  These counters are usedto determine whether the tree should grow children orsiblings once it has been determined that growth is to beallowed.2.1. AlgorithmAt each input presentation, a recursive search throughthe tree is made for a winning branch of the tree.  Eachnode on this branch is moved towards the input using thestandard competitive neural network update rule.Any winning node is allowed to grow if it satisfies 2conditions. It should be mature (have existed for anepoch), and the number of times it has won compared tothe number of times its parent has won needs to exceed athreshold. A finite limit is put on the number of times anode attempts growth.C hild  no d e s a re  c rea ted S ib l ing  no d e  is c re a te dInp ut  d a taW innerFigure 1.  Process of growing a treeWhen a node is allowed to grow, if it represents adense cluster, then its inner counter will be greater than itsouter counter and it creates two children. Otherwise, itproduces a sibling node. The process of growing isillustrated in figure 1. To improve the tree two pruningalgorithms, short and long term, are applied to delete theinsufficiently useful nodes.  The short-term pruningprocedure deletes nodes early in their life if their existenceis not improving the previous classificatory error. Thelong-term pruning procedure removes a leaf when itsactivity is not greater than an epsilon parameter. See figure2 for a pruning process. (a) Node to be pruned.(b) Singleton is removed, the tree is reconstructed.(c) Final tree after pruning process.Figure 2. Pruning process of an inactive node from thetree. The final tree is restructured so that a singleton isremoved.The addition of randomness to a hierarchical neuraltree clusterer have some benefit in helping the modelavoid local minima in its implicit cost function. Thisapproach is well known in the field of optimisation [9].There are two different ways in which stochasticity canbe added to the model. Firstly the deterministic decisionsrelating to growth and pruning can be made probabilistic,we call this Decision Based Stochasticity.Secondly the attributes inherited by nodes when theyare created can be calculated with a stochastic element, wecall this Generative Stochasticity. To both of theseapproaches a simulated annealing process can be added tomediate the amount of non-determinism in a controlledway, so that a decreasing temperature allows for lessrandomness later in the life of the network.3. PARAMETER SETTINGSThe parameters that are the object of this experimentsare briefly described in this section. The Growthparameteris the maximum times a node is allowed to grow.  The Kparameter influences how easy it is for a node to grow - itsactivity must satisfy.  As K is increased, it makes nodegrowth easier, and thereby bigger trees will result.KNumOfChildParent's1ActivitysParentActivitysNode+>''The α and β parameters determine the new tolerance ofany children produced. ×+×=innerouterParentTollNewChildTo βαAn increase in α, and/or an increase in β tend to makethe trees deeper.  When a node produces a sibling thesetwo nodes have larger tolerance values, mediated bySiblingToleranceSetting. Increasing this parameter limitssibling growth.)1( TolSibceOldToleranceNewToleran +×=ErrorAcceptance is a parameter used in short-termpruning. The condition to determine when a node shouldbe deleted is as below:ncetaErroracceprorPreviousErsParenting)(No.OfSibl)SumOfErroring(TotalSibl>×'In a long-term pruning procedure, a leaf node isdeleted when its activity is not greater than Epsilon.4. CLUSTER MEASURESThe general goal in many clustering applications is toarrive at clusters of objects that show small within-clustervariation relative to the between-cluster variation [3].Clustering is difficult as many reasonableclassifications may exist for a given data set, moreover itis easy for a clusterer to identify too few or too manyclusters.  Suitable cluster criterion measures are thereforeneeded [6].There are two types of clustering measures, ones thatgrade the flat clustering performance of the leaf nodes andones that grade the hierarchical structure.An initial investigation, concentrated on 10 clusteringcriterion selected from a comparative evaluation ofMilligan and Cooper [6] and another 2 hierarchicalmethods [4].  From this study, 2 measures were chosen:the gamma measure, which measures the flat partitioningperformance and the hierarchical correlation that assessesa hierarchical clustering against the data.5. OPTIMISING PARAMETERS BY GENETICALGORITHMSSearching a complex space of problem solutions ofteninvolves a balance between two apparently conflictingobjectives. The first is exploiting the best solutionscurrently available and the second is exploring the space.GAs have been identified as a general purpose searchstrategy that strikes a reasonable balance betweenexploration and exploitation [7].The seven parameters, described earlier, were encodedin a binary genome of 40 bits.  A population of 50individuals with random initialisation, were evolved usingthe GENESIS package from John Grefenstette [5], for 100generations. The mutation rate was 0.001 and thecrossover rate was 0.6, using dual point crossover.Selection was rank based using an elitist strategy [7].1 0 1 1G e neEncode d parame te r1 0 1 1M in: - 0 .95 M a x: 5 .4 0K : 7  b its1 0 1 1M in: 0 .0 05 M ax: 0 .9 50α: 6  b its1 0 1 1M in: 1 M a x: 32B : 5  b its1 0 1 1M in: 1 M a x: 32G ro w t h p ara : 5  b its1 0 1 1M in: 0 .0 0 M a x: 0 .6 3S ib T o l: 6  b its1 0 1 1M in: 0 .0 5 M a x: 1 .6 0E rra ccp t : 5  b its 1 0 1 1M in: 1 .0 0 M a x: 4 .1 5E psilo n : 6  b itsFigure 3.  A Genome picture of 7 encoded parameters5.1. Fitness EvaluationThe fitness function in the GA was constructed interms of the two best criteria described in the previoussection. The gamma measure and the hierarchical measureboth need to be maximised. Two fitness functions wereused, one just used the 2 cluster measures and wasotherwise unconstrained and the other constrained theresulting tree to have fewer than 80 nodes. So the fitnessfunctions are:Fitness Function1 = Hierarchical Correlation + GammaFitness Function2 = Hierarchical Correlation + Gamma- (Total nodes if > 80)The genome for each individual was decoded intoparameter values for SCENT.  The corresponding modelwas applied to all twelve data sets three times; theresulting trees were evaluated using the two criteriondescribed above, and the overall fitness of an individualcalculated from the average fitness of the three runs acrossthe twelve data sets.6. RESULTSResults in this section are divided into 2 sets of results.The first set of results illustrates the convergence of thefitness functions to a final value and gives the sets ofparameters found by the best individual over a 100generation run. These results are shown in Figures 4 and 5.The second set of results is to show the effects on eachparameter during the run and are depicted in Figures 6 and7. 1.201.251.301.351.401.451.501.551.601.651.700 10 20 30 40 50 60 70 80 90 100GenerationsFitnessBestAverageFigure 4. The average and best fitness of thepopulation when using the first fitness function.Figure 4 depicts clearly that for the first 20 generationsthere is a big difference in value of the population averagefitness and the best individual. Nonetheless, the differencewas reduced as the number of generations increased.Toward the end of the run, the average fitness wasincreased almost to the level of the best. An analysis of theresults for the first fitness function showed that the bestindividual settled to a set of parameters that alwaysproduced network trees with less than 80 nodes.This is what motivated the creation of the secondfitness function. It was designed to see if a reduced searchspace would produce better results. Since the final resultalways had less than 80 nodes could a better performancebe produced by restricting the whole search space.-2-1.5-1-0.500.511.520 20 40 60 80 100GenerationsFitnessBestAverageFigure 5. The average and best fitness of thepopulation when using the second fitness function.Interestingly results show in Figure 5 that the averagefitness started with a negative value as it penalised largetrees which were more than 80 total nodes in size.However, the average fitness was positive after 10generations. The average fitness value in Figure 5 wasmore erratic than that in Figure 4, due to the large penaltyassociated with large trees.Nevertheless, this average fitness value alsoapproached the value of the best individual over 100generations as had been the case with the first fitnessfunction. It can be seen that in both cases the overallfitness level of the population and the best individualfitness increase over time.During the 100 generations, the best individual in thepopulation changed 19 times. Information regarding eachbest of population using the first fitness function wasrecorded and these 19 points represent the x-axis on agraph in Figure 6. In the experimental run using thesecond fitness function, the best individual only changed12 times. Again information from each of these individualis plotted in Figure 7 as the change occurred.Figure 6 and 7 depict the number of total nodes andleaf nodes and the depth of tree when the best ofpopulation changed. The 22 clusters data sets was selectedto illustrate the changes that occurred to the parameters aseach best of population was found. Similar results wereobtained for most of the data sets.From Figure 6, it can be seen that the number of totalnodes and leaf nodes and depth of the tree had settled byabout the 7th and 8th best of population. In Figure 7, thebasic tree shape in terms of number of nodes, number ofleaf nodes and tree depth had basically settled by the 5thbest of population.22 Clusters0204060801001200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18TimeNo.of nodes0123456789Depth of treeTotal nodesLeaf nodesDepth of TreeFigure 6. Results in term of number of total nodes andleaf nodes and depth of tree, using the first fitness functionand a representative data set. 22clusters010203040506070800 1 2 3 4 5 6 7 8 9 10 11TimeNo.of nodes012345678Depth of treeTotal nodesLeaf nodesDepth of TreeFigure 7. Results in term of  number of total nodes andleaf nodes and depth of tree,using the second fitnessfunction and a representative data set.7. DISCUSSION AND CONCLUSIONOne of the interesting features of this study is that thefitness function is not applied directly to the genotypes onthe population that the GA is evolving but to neuralclassifiers that have grown in response to environmentalstimuli.This provides a direct analogue with the normalgenotype to phenotype mapping that occurs in ontogeny.The results presented here demonstrate that a GA canoptimise the network performance by using an indirectcoding between the genotype (a set of parameters) andphenotype (the grown tree).Moreover different fitness functions clearly produceddifferent parameter sets.  Previous experimentation hadshown that certain parameters had a more critical impacton the tree structures produced, and the GAs producedparameter settings that corresponded with intuition,varying with the aim of the specific fitness function.                             8. REFERENCES[1] R.G. Adams, K. Butchart, and N. Davey,   HierarchicalClassification with a Competitive Evolutionary Neural Tree.Neural Networks, Vol. 12, pp 541-551,1999.[2] N. Davey, R.G. Adams, and S.G. George, The Architectureand Performance of a Stochastic Competitive EvolutionaryNeural Tree Network, Applied Intelligence, Vol. 12, No. 1/2,pp.75-93, 1999.[3] B.S. Everitt, Cluster Analysis, Edward Arnold, London,1993.[4] A.D. Gordon, A Review of Hierarchical Classification,Journal of the Royal Statistical Society, Vol. 150, pp 119-137,1987.[5] J.J.Grefenstette, Genesis5.0,  ftp://www.aic.nrl.navy.mil/pub/galist/source-code/ga-source, 1995.[6] G.W. Milligan, and M.C. Cooper, M.C., An Examination ofProcedures for Determing the Number of Clusters in a Data Set.Psychometrika, Vol. 50, No. 2, pp 159-179, 1985.[7]  M. Mitchell, An Introduction to Genetic Algorithms, MITPress, USA, 1998.[8] W. Pensuwon, R.G. Adams, and N.Davey, HierarchicalDynamic Neural Networks, Ph.D. Thesis. University ofHertfordshire, 2001..[9]  W. Pensuwon, R.G. Adams, and N.Davey, The Analysis ofthe Addition of Stochasticity to a Neural Tree Classifier, Journalof Applied Soft Computing, Vol 1 (3), pp 189-200, 2001.[10] H. Song, and S. Lee, A Self-Organising Neural Tree forLarge-Set Pattern Classification. IEEE Transaction on NeuralNetworks, Vol. 9, No. 3, pp 369-380, 1998.

A Review of Hierarchical Classification,

A Self-Organising Neural Tree for Large-Set Pattern Classification.

An Examination of Procedures for Determing the Number of Clusters in a Data Set.

An Introduction to Genetic Algorithms,

and N.Davey, The Analysis of the Addition of Stochasticity to a Neural Tree Classifier,

Cluster Analysis,

Hierarchical Classification with a Competitive Evolutionary Neural Tree.

The Architecture and Performance of a Stochastic

Optimising a stochastic dynamic neural tree

Optimising a stochastic dynamic neural tree

Abstract

Similar works

Full text

Available Versions

University of Hertfordshire Research Archive