Comunicação aprovada à ICANGA March 2005, Coimbra.The Multilayer Perceptrons (MLPs) are the most popular class of Neural Networks. When applying MLPs, the search for the ideal architecture is a crucial task, since it should should be complex enough to learn the input/output mapping, without overfitting the training data. Under this context, the use of Evolutionary Computation makes a promising global search approach for model selection. On the other hand, ensembles (combinations of models) have been boosting the performance of several Machine Learning (ML) algorithms. In this work, a novel evolutionary technique for MLP design is presented, being also used an ensemble based approach. A set of real world classification and regression tasks was used to test this strategy, comparing it with a heuristic model selection, as well as with other ML algorithms. The results 
favour the evolutionary MLP ensemble method.Fundação para a Ciência e Tecnologia - Project
POSI/ROBO/43904/2002; FEDER

Cortez, Paulo

Neves, José

Rocha, Miguel

Universidade do Minho: RepositoriUM

Evolutionary design of neural networks for classification and regression

Evolutionary Design of Neural Networksfor Classification and Regression1Miguel Rocha?, Paulo Cortez†, Jos·e Neves??Dep. Inform·atica, University of Minho, Portugal†Dep. Sistemas de Informac‚ ao, University of Minho, PortugalE-mail: mrocha@di.uminho.pt pcortez@dsi.uminho.pt jneves@di.uminho.ptAbstractThe Multilayer Perceptrons (MLPs) are the mostpopular class of Neural Networks. When applyingMLPs, the search for the ideal architecture is a crucialtask, since it should should be complex enough tolearn the input/output mapping, without overfittingthe training data. Under this context, the use ofEvolutionary Computation makes a promising globalsearch approach for model selection. On the otherhand, ensembles (combinations of models) have beenboosting the performance of several Machine Learning(ML) algorithms. In this work, a novel evolutionarytechnique for MLP design is presented, being alsoused an ensemble based approach. A set of real worldclassification and regression tasks was used to test thisstrategy, comparing it with a heuristic model selection,as well as with other ML algorithms. The results favourthe evolutionary MLP ensemble method.Keywords: Supervised Machine Learning, MultilayerPerceptrons, Evolutionary Algorithms, Ensembles.1 IntroductionNeural Networks (NNs) are important Machine Learn-ing (ML) techniques, denoting a set of connectionistmodels inspired in the behavior of the human brain. Inparticular, the Multilayer Perceptron (MLP) is a populararchitecture, where neurons are grouped in layers andonly forward connections exist. This provides a pow-erful base-learner, capable of nonlinear mappings [1].When compared to other ML methods, MLPs are knownto behave well in terms of predictive knowledge [2],and there has also been research in terms of explanatoryknowledge (e.g. extracting rules from MLPs) [3].However, one of the major issues when applyingMLPs is the topology (i.e. connectivity) design. This isa complex and crucial task, with a strong impact in per-formance (a small network may provide poor learning1This work was supported by the FCT projectPOSI/ROBO/43904/2002, which is partially funded by FEDER.capabilities, while a large one will overfit the data). Itis common to address this task by trial-and-error proce-dures (e.g. exploring different number of hidden nodes),in a blind search strategy, which only goes through asmall set of possible configurations. More elaboratedmethods have been proposed, such as pruning [4] andconstructive [5] algorithms, although these perform hill-climbing, being prone to local minima.An alternative is offered by Evolutionary Computa-tion (EC), which performs a global multi-point (or beam)search, quickly locating areas of high quality, even whenthe search space is very large and complex. The combi-nation of EC and NN, often called Evolutionary NeuralNetworks (ENNs), is a better candidate for the topologydesign, due to the characteristics of the error surface [6]:• the number of nodes/connections is unbounded;• the mapping from the structure to its performanceis indirect;• changes are discrete and can provide discontinu-ous effects in the NN behaviour; and• similar topologies may present different perfor-mances.In addiction, this approach is biologically more plau-sible; i.e., living creatures have succesfully adapted totheir environments as a result of the interaction of evolu-tion and learning.Another emergent ML research area is related to theuse of ensembles, where a set of models are combined toproduce an answer, being often more accurate than indi-vidual learners [7]. One interesting way to build NN en-sembles is based on heterogeneous topologies [8]. Thisapproach can be easily adapted to ENNs with no com-putacional effort increase, since ENNs already use a pop-ulation of NNs with different connectivities.The present work presents a novel ENN to the designof MLP topologies, where a direct MLP representation(closer to the phenotype) is used. This approach will betested in classification and regression tasks, using bothsingle and ensemble based models. Finally, results willbe compared with a heuristic NN selection procedure, aswell with other ML methods.2 Materials and Methods2.1 Data SetsEight classification and eight regression data sets wereselected from the UCI ML repository [9], and its mainfeatures are listed in Table 1, namely the number of nu-meric (Nu), binary (Bi) and nominal (No, i.e. discretewith 3 or more labels) input attributes, as well as thenumber of examples (Ex) and classes (Cl). The regres-sion tasks are identified by the symbol < in the Cl col-umn (last eight rows).Table 1. A summary of the data sets used.Task Inputs Ex ClNu Bi NoBalance 4 0 0 625 3Bupa 6 0 0 345 2Car 0 0 6 1728 4Cmc 5 3 1 1473 3Dermatology 34 0 0 366 6Ionosphere 34 0 0 351 2Sonar 60 0 0 104 2Yeast 7 1 0 1484 10Abalone 7 0 1 4177 <Auto-mpg 5 0 2 398 <Autos 17 3 5 205 <Breast-cancer 1 4 4 286 <Heart-disease 6 3 4 303 <Housing 12 1 0 506 <Servo 2 0 2 167 <WPBC 32 0 0 194 <2.2 Neural NetworksBefore feeding the MLPs, the data was preprocessed:a 1-of-C encoding (one binary variable per class) wasapplied to the nominal attributes and all input valueswere rescaled within the range [−1, 1]. For example, thesafety attribute from the task car was encoded accordingto: low→ 1 -1 -1, med→ -1 1 -1 and high→ -1 -1 1.Regarding the outputs, the discrete variables were nor-malized within the range [0, 1] (using also a 1-of-C en-coding for the nominal attributes). Therefore, the pre-dicted class is given by the nearest class value to thenode’s output, if one single node is used (binary vari-able), otherwise the node with the highest output valueis considered. On the other hand, regression problemswill be modeled by one real-valued output, which di-rectly represents the dependent target variable.The MLPs used make use of biases and sigmoid ac-tivation functions, with one hidden layer, containing avariable number of nodes. A different approach was fol-lowed for the regression tasks, since outputs may lie outof the logistic output range ([0, 1]). In this case, the lo-gistic function was applied on hidden nodes, while theoutput ones used shortcut connections and linear func-tions, to scale the range of the outputs (Figure 1). Thissolution avoids the need of filtering procedures, whichmay give rise to information loss and has been success-fully adopted in other regression applications, such asTime Series Forecasting [10].i,0wij wi,jHidden Layer Output LayerInput Layerxx12Fig. 1. A 2− 2 − 1 MLP topology with bias and shortcuts.The initial weights will be randomly set within therange [−1, 1]. Then, the RPROP algorithm [11] is se-lected for training, due to its faster convergence and sta-bility, being stopped after a maximum of 200 epochs orwhen the error slope is approaching zero.Two distinct accuracy measures were adopted: thePercentage of Correctly Classified Examples (PCCE),used in classification tasks; and the Normalized RootMean Squared Error (NRMSE), applied in the regressionones. These metrics are given by the equations:PCCE =PNi=1 1 , if(Ti=Pi)N× 100 (%)NRMSE =r PNi=1(Ti−Pi)2NPNi=1TiN× 100 (%)(1)where N denotes the number of examples; Pi, Ti thepredicted and target values for the i-th example.An Heuristic approach (HNN) to model selection isdefined by a simple trial-and-error procedure, wherefully connected MLPs, with a number of hidden nodesranging from 0 to 20, are trained. Then, the topol-ogy with the lowest validation error (computed over nontraining data) is selected. The trained MLPs will also beused to build an Ensemble (HNNE), where the output isgiven by the average over all 21 MLPs.2.3 Evolutionary Neural NetworkIn this work, an Evolutionary Algorithm with a directrepresentation is embraced, where the genotype is thewhole MLP. Each individual of the initial population isset by choosing a random number of hidden nodes (be-tween 0 and 10). Then, each possible connection is setwith probability of 50%. New individuals are bred bystructural mutation, which works by adding or deletinga random number (from 1 to 5) of nodes or connections.The population size was set to 20, being the selectiondone by converting the fitness value (the error computedover a validation set) into its ranking, and then applyinga roulette wheel scheme, being used a substitution rate of50%. Finally, the ENN is stopped after 20 generations.This scalable ENN is able to search through any kindof MLP connectivity, ranging from linear models to com-plex nonlinear MLPs. The ENN Ensemble (ENNE) willbe built using the best 20 individuals (with lower vali-dation error) obtained during the evolutionary process,being the output computed as the average of the MLPs.3 ResultsThe NN/EC experiments were conducted using a soft-ware package developed in Java by the authors. Theother techniques were computed using WEKA ML soft-ware (with its default parameters) [12]:• J48 – a classification decision tree based on theC4.5 algorithm;• M5P – a regression decision tree (M5 algorithm);• IB5 – a 5-Nearest Neigboor;• KStar – an instance based algorithm; and• SVM – a Support Vector Machine.For each model, 10 runs of a 5-fold cross-validation pro-cess [13] (stratified in the classification tasks) were exe-cuted. This means that in each of these 50 experiments,80% of the data is used for learning and 20% for testing.Regarding the MLP based approaches, the learning datawas divided into training (50% of the original dataset)and validation sets (30%). Tables 2 and 3 show the av-erage errors of the 10 runs for each learning model andclassification/regression task. In both tables, the last rowaverages the global behaviour of each technique.First, the classification results will be analyzed. TheNN based approaches (last four columns) are competivewhen compared with the other ML algorithms. The fewexceptions are the dermatology and sonar tasks, wherethe SVM and KStar get the best results. As expected, theENN outperforms the HNN, with a 1% increase in perfor-mance. Furthermore, the ensemble approaches (HNNEand ENNE) obtain better results when compared with thesingle based methods (1.4% improvement in both cases).Indeed, the ENNE reveals the best overall behaviour, out-performing all other algorithms in 5 of the 8 tasks.A similar scenario occurs in the regression tasks. Ingeneral, the NN methods are better than the other ML al-gorithms, although the M5P outperforms the HNN andHNNE approaches. When compared with the ENN, theHNN is outperformed by a wider difference (2.5%). Asbefore, the ensembles behave better, although the impactis higher with the ENNE (1.5% improvement, being thebest method in 5 tasks) than with the HNNE (0.7% im-provement).4 ConclusionsThe surge of bio-inspired techniques, such as Multi-layer Perceptrons (MLPs) and Evolutionary Computa-tion (EC), has created new exciting possibilities for thefield of Machine Learning (ML). Considering ensemblesof learning models to improve its accuracy has also beena focus of attention by the research community. In thiswork, an evolutionary approach to MLP topology designwas presented, considering single and ensemble combi-nations, being tested in supervised learning tasks (e.g.,classification and regression).The results obtained confirm than Evolutionary Neu-ral Network (ENN) approach outperforms a heuristictrial-and-error MLP design procedure (HNN), as well asother ML algorithms (e.g. K-Nearest Neigboor). How-ever, this improvement in performance has the handicapof increasing the computational complexity (the HNN re-quires only a tenth of the ENN computational effort).On the other hand, the use of the EC population struc-ture to construct ensembles is a recent research field. In-deed, the proposed ENN Ensemble (ENNE), based on theaverage of the outputs from the best EC individuals (orMLPs), is competitive. The ENNE has the advantage ofpresenting the best overall performance while requiringthe same computational effort, when compared with thesingle based ENN.In future work, it is intended to explore similar ap-proaches with different neural architectures (e.g., Recur-rent Neural Networks). Moreover, more elaborated en-sembles should be considered, by designing fitness func-tions which reward specialization [14].References[1] Haykin, S. (1999) Neural Networks - A Compreen-sive Foundation. 2nd ed. Prentice-Hall[2] Quinlan, J.R. (1994) Comparing Connectionist andSymbolic Learning Methods. In: Hanson, S. et al.(eds.) Computation Learning: Theory and NaturalLearning Systems, MIT Press, pp. 445-456Table 2. The classification results (PCCE values, in %).Task J48 IB5 KStar SVM HNN ENN HNNE ENNEBalance 78.1 87.6 88.3 87.7 94.8 95.8 95.7 96.5Bupa 64.8 60.7 65.9 58.0 68.4 69.0 68.5 69.7Car 91.3 92.3 87.1 93.5 97.4 98.2 98.3 98.7Cmc 51.2 47.2 49.6 48.4 50.6 54.2 52.1 55.9Dermatology 95.7 96.6 94.5 97.4 94.9 95.5 95.9 96.8Ionosphere 89.4 84.6 84.0 87.9 88.9 89.6 92.3 92.6Sonar 72.5 80.5 85.2 76.7 79.9 79.0 80.9 81.2Yeast 56.0 57.1 53.1 56.6 58.2 59.3 59.9 60.2Mean 74.9 75.8 76.0 75.8 79.1 80.1 80.5 81.5Table 3. The regression results (NRMSE values, in %).Task M5P IB5 KStar SVM HNN ENN HNNE ENNEAbalone 24.3 25.3 24.8 25.0 23.2 23.2 23.2 23.2Auto-mpg 11.8 15.1 14.6 13.5 14.2 13.0 12.2 12.2Autos 13.3 21.4 21.6 13.8 14.6 16.1 14.2 14.9Breast-cancer 40.8 40.8 43.6 44.0 42.6 40.6 47.4 40.2Heart-disease 21.2 21.0 25.5 21.5 21.9 21.7 22.2 21.5Housing 18.4 22.2 18.0 22.6 18.4 17.3 16.6 16.0Servo 50.4 60.4 67.6 70.5 60.8 45.0 49.6 39.4Wpbc 73.2 73.6 98.2 71.7 75.4 74.4 80.3 71.7Mean 33.0 35.0 39.2 35.3 33.9 31.4 33.2 29.9[3] Setiono, R. (2003) Techniques for ExtractingClassification and Regression Rules from Artifi-cial Neural Networks. In D. Fogel and C. Robin-son, (eds.), Computational Intelligence: The Ex-perts Speak, IEEE Press/Wiley, pp 99–114[4] Thimm, G. and Fiesler, E. (1995) Evaluating prun-ing methods. In: Proc. of the Int. Symp. on Artifi-cial Neural Networks, pp 20–25[5] Kwok, T. and Yeung, D. (1997) Constructive al-gorithms for structure learning in feedforward neu-ral networks for regression problems problems: Asurvey. IEEE Transactions on Neural Networks,8(3):630–645[6] Yao, X. (1999) Evolving Artificial Neural Net-works. In: Proc. of the IEEE, 87(9): 1423-1447[7] Dietterich, T. (1997) Machine Learning Research:Four Current Directions. AI Magazine, 18(4):97–136[8] Rocha, M., Cortez, P. and Neves, J. Ensemblesof Artificial Neural Networks with HeterogeneousTopologies. In: Proc. of the 4th Symposium on En-gineering of Intelligent Systems (EIS2004). ICSCAcademic Press[9] Blake, C. and Merz, C. (1998) UCI Repository ofMachine Learning Databases, University of Cali-fornia[10] Cortez, P., Rocha, M. and Neves, J. (2001) Evolv-ing Time Series Forecasting Neural Network Mod-els. In: Proc. of the 3rd Int. Symposium on Adap-tive Systems: Evolutionary Computation and Prob-abilistic Graphical Models (ISAS 2001), pp. 84–91[11] Riedmiller, M. (1994) Supervised Learning inMultilayer Perceptrons - from Backpropagation toAdaptive Learning Techniques. Computer Stan-dards and Interfaces 16[12] Witten, I. and Frank, E. (2000) Data Mining: Prac-tical Machine Learning Tools and Techniques withJava Implementations. Morgan Kaufmann[13] Kohavi, R. (1995) A Study of Cross-Validation andBootstrap for Accuracy Estimation and Model Se-lection. In: Proc. of the Int. Joint Conference onArtificial Intelligence (IJCAI)[14] Liu, Y., Yao, X. and Higuchi, T. (2000) Evolution-ary Ensembles with Negative Correlation Learning.IEEE Transactions on Evolutionary Computation,4(4):380–387

http://repositorium.sdum.uminho.pt/bitstream/1822/892/1/ednncr.pdf

Evolutionary design of neural networks for classification and regression

Abstract

Similar works

Full text

Available Versions

Universidade do Minho: RepositoriUM

Universidade do Minho: RepositoriUM