This paper documents experiments performed using a GA to optimise the parameters of a dynamic neural tree model. Two fitness functions were created from two selected clustering measures, and a population of genotypes, specifying parameters of the model were evolved. This process mirrors genomic evolution and ontogeny. It is shown that the evolved parameter values improved performanceFinal Accepted Versio

Adams, R.G.

Davey, N.

Pensuwon, W.

English

This paper documents experiments performed using a GA to optimise the parameters of a dynamic neural tree model. Two fitness functions were created from two selected clustering measures, and a population of genotypes, specifying parameters of the model were evolved. This process mirrors genomic evolution and ontogeny. It is shown that the evolved parameter values improved performanc

University of Hertfordshire Research Archive

Optimising a Neural Tree Classifier Using a Genetic AlgorithmWanida PensuwonW.Pensuwon@herts.ac.ukRod AdamsR.G.Adams@herts.ac.ukNeil DaveyN.Davey@herts.ac.ukComputer Science Dept. University of Hertfordshire, Hatfield, Herts, AL10 9AB,United Kingdom.,Tel: +44 01707 284321Abstract. This paper documents experimentsperformed using a GA to optimise the parameters of adynamic neural tree model.  Two fitness functionswere created from two selected clustering measures,and a population of genotypes, specifying parametersof the model were evolved.  This process mirrorsgenomic evolution and ontogeny.  It is shown that theevolved parameter values improved performance.1 INTRODUCTIONOrganising the output nodes of a competitive networkinto a tree structure may produce potentialimprovements in both convergence and recall rates.  Itmay also make the network more stable in respondingto changes in the data being classified.  However alimitation of most clustering algorithms is that theyrequire the number of classes to be predefined.  This isa problem if no a-priori knowledge about the data isavailable [3].Dynamic neural tree networks (DNTN) may avoid theabove limitation by dynamically creating nodes andautomatically creating a tree structure.  However, allthe original models require parameters to be externallyset, which will significantly influence the hierarchicalstructure and the classification that is produced [8, 11,12].A more recent DNTN is the Competitive EvolutionaryNeural Tree (CENT), [1,2,3]. This model iscomparatively robust, with respect to its parametersettings when compared with other DNTNs [2].Nonetheless, it is unclear precisely how the internalparameters affect the performance, which lead to theinvestigation reported here. In order to explore theparameter space a good default set of parameters wasneeded, and a genetic algorithm (GA) was used as thesearch vehicle. A given genome (a set of parameters),together with a data set allowed CENT to produce aclassificatory tree.  The fitness of these trees wascalculated using two clustering measures (described insection 4).  Nine data sets were used.2 ALGORITHMIn CENT, the tree structure is created dynamically inresponse to structure in the data set. The neural treestarts with a root node with its tolerance (the radius ofits classificatory hypersphere) set to the standarddeviation of input vectors and its position set to themean of input vectors.  It has 2 randomly positionedchildren.  Each node has two counters, called innerand outer, which count the number of occasions that aclassified input vector is within or outside tolerance,respectively.  These counters are used to determinewhether the tree should grow children or siblings onceit has been determined that growth is to be allowed.2.1 TOP-LEVEL ALGORITHMAt each input presentation, a recursive search throughthe tree is made for a winning branch of the tree.Each node on this branch is moved towards the inputusing the standard competitive neural network updaterule.Any winning node is allowed to grow if it satisfies 2conditions. It should be mature (have existed for anepoch), and the number of times it has won comparedto the number of times its parent has won needs toexceed a threshold. A finite limit is put on the numberof times a node attempts growth.W inner nodeC hild  nodes  a re  c rea tedW inne r nodeS ib ling  node  is  crea tedInput d a taW inne r nodeFigure 1.  Process of growing a treeWhen a node is allowed to grow, if it represents adense cluster, then its inner counter will be greaterthan its outer counter and it creates two children.Otherwise, it produces a sibling node. The process ofgrowing is illustrated in figure 1. To improve the treetwo pruning algorithms, short and long term, areapplied to delete the insufficiently useful nodes.  Theshort-term pruning procedure deletes nodes early intheir life if their existence is not improving theprevious classificatory error according to anErrorAcceptance (EA) parameter value. The long-termpruning procedure removes a leaf when its activity isnot greater than an epsilon parameter. See figure 2 fora pruning process.(b) Singleton is removed, the tree is  reconstructed.(a)  Node to be pruned.(c) Final  t ree af ter pruning processFigure 2. Pruning process3 PARAMETER SETTINGSThe parameters that are the object of this investigationare briefly described in this section.The growthparameter is the maximum times a node isallowed to grow.  The K parameter influences howeasy it is for a node to grow - its activity must satisfy:KchildofnumbersParentactivitysParentactivitysNode+>'1''As K is increased, it makes node growth easier, andthereby bigger trees will result.The A and B parameters determine the new tolerance(tol) of any children produced.⎟⎠⎞⎜⎝⎛+==InnerBOuterACwhereCtolsParenttolschildrenNew**''An increase in  A, and/or a decrease in B tend to makethe trees deeper.When a node produces a sibling these two nodes havelarger tolerance values, mediated by Sbtol.)1(* tolSbTolOldTolNew +=Increasing this parameter limits sibling growth.EA is a parameter used in short-term pruning. Thecondition to determine when a node should be deletedis as below:EAerrorprevioussParentsiblingofNumbererrorofsumssiblingTotal>')(*)'(In a long-term Pruning procedure, a leaf node isdeleted when its activity is not greater than epsilon.4 CLUSTER MEASURESThe general goal in many clustering applications is toarrive at clusters of objects that show small within-cluster variation relative to the between-clustervariation [7]. Clustering is difficult as manyreasonable classifications may exist for a given dataset, moreover it is easy for a clusterer to identify toofew or too many clusters.  Suitable cluster criterionmeasures are therefore needed [4].An initial investigation, concentrated on 10 clusteringcriterion selected from a comparative evaluation ofMilligan and Cooper [9] and another 2 hierarchicalmethods [5].  From this study, 2 measures werechosen: the gamma measure, which measures the flatpartitioning performance and the hierarchicaldistortion that assesses a hierarchical clusteringagainst the data.5 GENETIC ALGORITHMOPTIMISATIONSearching a complex space of problem solutions ofteninvolves a balance between two apparently conflictingobjectives. The first is exploiting the best solutionscurrently available and the second is exploring thespace. GAs have been identified as a general purposesearch strategy that strikes a reasonable balancebetween exploration and exploitation [10].The seven parameters, described earlier, were encodedin a binary genome of 40 bits.  A population of 50individuals with random initialisation, were evolvedusing the GENESIS package from John Grefenstette[6], for 50 generations. The mutation rate was 0.001and the crossover rate was 0.6, using dual pointcrossover.  Selection was rank based using an elitiststrategy [10].1 0 1 1GeneEncoded parameter1 0 1 1Min: -0.95 Max:  5.40K :  7 bits1 0 1 1Min:  0.005 Max:  0.950A :  6 bits1 0 1 1Min: 1 Max:  32B :  5 bits1 0 1 1Min: 1 Max:  32Growthpara :  5 bits1 0 1 1Min:  0.00 Max:  0.63Sb tol:  6 bits1 0 1 1Min:  0.05 Max:  1.60E A :  5 bits 1 0 1 1Min:  1.00 Max:  4.15Epsi lon :  6 bitsFigure 3.  A Genome picture of 7 encoded parameters5.1 FITNESS EVALUATIONThe genome for each individual was decoded intoparameter values for CENT.  The correspondingmodel was applied to all nine data sets three times; theresulting trees were evaluated using the two criteriondescribed above, and the overall fitness of anindividual calculated from the average fitness of thethree runs across the nine data sets.We investigated two different fitness functions, one inwhich the two measures were simply combined:Fitness function1 = Hierarchical distortion-GammaAnd a second which contained an additional factorpenalising large trees, namely:Fitness function2 = Hierarchical distortion–Gamma +(Totalnode if > 40)In both cases minimisation of the fitness function wasrequired.6 RESULTSFigures 4 and 5 show the average and best fitness inthe population for the two fitness functions:0.000.501.001.502.002.503.003.504.000 10 20 30 40 50GenerationsFitnessBestAverageFigure 4. The average and best fitness of thepopulation when using the first fitness function.It can be seen that in both cases the overall fitnesslevel of the population and the best individual fitnessreduced over time.  An improvement in performancewas obtained by using the parameters found by theGA compared to the set originally thought to beappropriate.  For the second fitness function, the finalfitness level was significantly higher, due to thepropensity of the hierarchical distortion measure toprefer large trees, something heavily penalised by thisfitness function.  This difference in tree sizes isillustrated in Figures 6 and 7, where, for arepresentative data set, the total number of nodes andthe number of leaf nodes is plotted whenever the bestindividual changes.The CENT model is fairly tolerant to its parametervalues.  Ranges of values can produce reasonableresults, hence the GA quickly finds an adequatesolution.  However a really good solution was harderto find, and the GA needed to run for longer.  Therelative change in the clustering measures can often besmall even though the classification may be visiblybetter, so that the observed small changes inpopulation fitness can be significant.0.002.004.006.008.0010.0012.000 10 20 30 40 50GenerationsFitnessBestAverageFigure 5. The average and best fitness of thepopulation when using the second fitness function.0204060801001201400 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18Best of populationNo. of NodesTotal NodeLeaf NodeFigure 6. Results in term of number of total nodes,and number of leaf nodes, using the first fitnessfunction and a representative data set.0246810121416180 5 10 15 20Best of populationNo. of NodesTotal NodeLeaf NodeFigure 7. Results in term of number of total nodes,and number of leaf nodes, using the second fitnessfunction and a representative data set.7 DISCUSSION ANDCONCLUSIONSOne of the interesting features of this study is that thefitness function is not applied directly to the genotypeson the population that the GA is evolving but to neuralclassifiers that have grown in response toenvironmental stimuli.  This provides a directanalogue with the normal genotype to phenotypemapping that occurs in ontogeny.The results presented here demonstrate that a GA canhelp in optimizing the internal parameter settings of aneural network.  Moreover different fitness functionsclearly produced different parameter sets.  Previousexperimentation had shown that certain parametershad a more critical impact on the tree structuresproduced, and the GAs produced parameter settingsthat corresponded with intuition, varying with the aimof the specific fitness function.AcknowledgmentsPart of this research work is financially supported bythe Royal Thai Government.References[1] Adams, R.G., Butchart, K. and Davey, N. (1999)Hierarchical Classification with a CompetitiveEvolutionary Neural Tree.  Neural Networks,Vol. 12, pp 541-551.[2] Butchart, K. (1996) Hierarchical Clustering UsingDynamic Self Organising Neural Network.Ph.D. Thesis. University of Hertfordshire.[3] Davey,N., Adams, R.G. and George, S.G. (1999)The Architecture and Performance of aStochastic Competitive Evolutionary NeuralTree Network, Applied Intelligence, Vol. 12,No. 1/2, pp.75-93.[4] Everitt, B.S. (1993) Cluster Analysis, EdwardArnold, London.[5] Gordon, A.D. (1987) A Review of HierarchicalClassification.  Journal of the Royal StatisticalSociety  ,Vol. 150, pp 119-137.[6] Grefenstette J.J. (1995) Genesis 5.0,ftp://www.aic.nrl.navy.mil/pub/galist/source-code/ga-source[7] Hartigan, J.A. (1975) Clustering Algorithms, JohnWiley & Sons, USA.[8] Li, T., Tang, Y.Y. and Fang, L.Y. (1995) AStructure-Parameter-Adaptive (SPA) NeuralTree for the Recognition of Large CharacterSet. Pattern Recognition, Vol.28, No. 4, pp315-329.[9] Milligan, G.W. and Cooper, M.C. (1985) AnExamination of Procedures for Determing theNumber of Clusters in a Data Set.Psychometrika, Vol. 50, No. 2, pp 159-179.[10]  Mitchell, M.  (1998) An Introduction to GeneticAlgorithms, MIT Press, USA.[11] Racz, J. and Klotz, T. (1991) KnowledgeRepresentation by Dynamic CompetitiveLearning Algotithm. Proceedings  SPIEApplications of Artificial Neural Network II ,pp778-783.[12] Song, H. and Lee, S. (1998) A Self-OrganisingNeural Tree for Large-Set PatternClassification. IEEE Transaction on NeuralNetworks, Vol. 9, No. 3, pp 369-380.

Optimising a neural tree classifier using a genetic algorithm

Abstract

Similar works

Full text

Available Versions

University of Hertfordshire Research Archive