Evolving connectionist systems: Characterisation, simplification, formalisation, explanation and optimisation

There are several well-known problems with conventional artificial neural networks (ANN), such as difficulties with selecting the structure of the network, and problems with forgetting previously-learned knowledge after further training. Constructive neural network algorithms attempt to solve these problems, but in turn have problems of their own. The Evolving Connectionist System (ECoS) is a class of open architecture artificial neural networks that are similar in the way in which neurons are added to their structures, and in the way in which their connection weights are modified. The ECoS algorithm is intended to address the problems with constructive neural networks. Several problems with ECoS are identified and discussed in this thesis. These problems are: the lack of comparison of ECoS with constructive neural networks; the excessive complexity of the Evolving Fuzzy Neural Network (EFuNN), which is the seminal ECoS network: the lack of a testable formalisation of ECoS; the dependence on fuzzy logic elements embedded within the network for fuzzy rule extraction; and the lack of methods for optimising ECoS networks. The research in this thesis addresses these problems. The overall theme of the research can be summarised as the characterisation, simplification, formalisation, explanation and optimisation of ECoS. Characterisation in this thesis means the comparison of ECoS with existing constructive ANN. Simplification means reducing the network to a minimalist implementation. Formalisation means the creation of a testable predictive model of ECoS training. Explanation means explaining ECoS networks via the extraction of fuzzy rules. Finally, optimisation means creating ECoS networks that have a minimum number of neurons with maximum accuracy. Each of these themes is approached in ways that build upon, and are complementary to, the basic ECoS network and ECoS training algorithm. The basic ECoS structure and algorithm is left unchanged, and the problems are addressed by extending that structure, rather than altering it as has been done in other work on EcoS. The principal contributions of this thesis are: a qualitative comparison of ECoS to constructive neural network algorithms; a proposed simplified version of EFuNN called SECoS; an experimentally tested formalisation of ECoS: novel algorithms for explicating SECoS via the extraction of fuzzy rules; and several novel algorithms for the optimisation of ECoS networks. The formalisation of ECoS and the proposed algorithms are evaluated on data from a set of standard benchmarking problems. Further experiments are performed with a data set with real-world applications, namely the recognition of isolated New Zealand English phonemes. The analyses of the experimental results show that the proposed algorithms are effective across both the benchmark data sets and the case study data set.UnpublishedAbraham, A. (2002). Optimization of evolutionary neural networks using hybrid learning algorithms. In Proceedings of IJCNN 2002, pages 2797-2802. Abreu, A. and Pinto-Ferreira, L. C. (1996). Fuzzy modeling: a rule based approach. In Proceedings of the fifth IEEE International Conference on Fuzzy Systems, pages 162-168. Aguiler, J. and Colmenares, A. (1997). Recognition algorithm using evolutionary learning on the random neural network. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 2, pages 1023-1028. IEEE Press. Alander, J. T. (1993). On robot navigation using a genetic algorithm. In Albrecht, R., Reeves, C., and Steele, N., editors, Artificial Neural Nets and Genetic Algorithms, pages 471-478. Alpaydin, E. (1994). GAL: Networks that grow when they learn and shrink when they forget. International Journal of Pattern Recognition and Artificial Intelligence, 8(1):391-414. Anderson, E. (1935). The irises of the gaspe peninsula. Bulletin of the American Iris Society, 59. Anderson, S., Merrill, J., and Port, R. (1988). Dynamic speech categorization with recurrent networks. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School, pages 398-396. Morgan Kaufmann. Andreassen, H., Bohr, H., Bohr, J., Brunak, S., Bugge, T., Cotterill, R., Jacobsen, C., Kusk, P., Lautrup, B., Petersen, S., Saermark, T., and Ulrich, K. (1990). Analysis of the secondary structure of the human immunodeficiency virus (HIV) proteins p17, gp120, and gp41 by computer modeling based on neural networks methods. Journal of Acquired Immune Deficiency Syndromes, 3:615-622. Andrews, R., Diederich, J., and Tickle, A. B. (1995). Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8(6):373-389. Andrews, R. and Geva, S. (1997). Refining expert knowledge with an artificial neural network. In Kasabov, N., Kozma, R., Ko, K., S'Shea, R., Coghill, G., and Gedeon, T., editors, Progress in Connectionist-Based Information Systems, volume 2, pages 847-850. Springer. Angeline, P. K., Saunders, G. M., and Pollack, J. B. (1994). An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1):54-65. Antonisse, J. (1989). A new interpretation of schema notation that overturns the binary encoding constraint. In Schaffer, J., editor, Proceedings of the Third International Conference on Genetic Algorithms, pages 86-91. Arena, P., Caponetto, R., Fortuna, L., and Xibilia, M. G. (1993). M.L.P. optimal topology via genetic algorithms. In Artificial Neural Nets and Genetic Algorithms, pages 670-674. Springer-Verlag Wien New York. Ash, T. (1989). Dynamic node creation in backpropagation networks. Connection Science, 1(4):365-375. Baba, N., Marume, M., and Itoh, K. (1992). Utilization of stochastic automaton and genetic algorithm for neural network design. In Proceedings of the 2nd International Conference on Fuzzy Logic and Neural Networks, volume 2, pages 837-840, Iizuka, Japan. Back, T., Hoffmeister, F., and Schwefel, H.-P. (1991). A survey of evolution strategies. In Belew, R. K. and Booker, L. B., editors, Proceedings of the Fourth International Conference on Genetic Algorithms, pages 2-9. Balakrishnan, K. and Honavar, V. (1996). Analysis of neurocontrollers designed by simulated evolution. In International Conference on Neural Networks 1996: Plenary, Panel and Special Sessions, pages 130-135. Baldi, P. and Brunak, S. (1998). Bioinformatics: The Machine Learning Approach. MIT Press. Bebis, G., Georgiopoulos, and Kasparis, T. (1996). Coupling weight elimination and genetic algorithms. In Proceedings of the 1996 IEEE International Conference on Neural Networks, pages 1115-1120. Belew, R. K., McInerney, J., and Schraudolph, N. N. (1990). Evolving networks: Using the genetic algorithm with connectionist learning. In Langton, C. G., Taylor, C., Farmer, J. D., and Rasmussen, S., editors, Artificial Life II, pages 511-547, Santa Fe, New Mexico. Addison-Wesley Publishing Company. Bengio, Y. and De Mori, R. (1988). Speaker normalization and automatic speech recognition using spectral lines and neural networks. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School, pages 388-397. Morgan Kaufmann. Billings, S. A. and Zheng, G. L. (1995). Radial basis function network configuration using genetic algorithms. Neural Networks, 8(6):877-890. Bisant, D. and Maizel, J. (1995). Identification of ribosome binding sites in Escherichia coli using neural network models. Nucleic Acids Research, 23(9):1632-1639. Bohr, H., Bohr, J., Brunak, S., Cotterill, R. M., Lautrup, B., Norskov, L., Olsen, 0. H., and Petersen, S. B. (1988). Protein secondary structure and homology by neural networks. The a-helices in rhodopsin. FEBS Letters, 241(1,2):223-228. Bornholdt, S. and Graudenz, D. (1992). General asymmetric neural networks and structure design by genetic algorithms. Neural Networks, 5:327-334. Bose, N. and Garga, A. K. (1993). Neural network design using voronoi diagrams. IEEE Transactions on Neural Networks, 4(5):778-787. Bourland, H. and Wellekens, C. (1987). Multiplayer perceptrons and automatic speech recognition. In IEEE First Annual Conference on Neural Networks, volume IV, pages 407-416, San Diego. Box, G. E. and Jenkins, G. M. (1970). Time Series Analysis forecasting and control. Holden-Day. Brasil, L. M., de Azevedo, F. M., and Barreto, J. M. (2000). A hybrid expert system for the diagnosis of epileptic crisis. Artificial Intelligence in Medicine, 585:1-7. Brown, A. and Card, H. (1997). Evolutionary artificial neural networks for competitive learning. In Proceedings of /CNN, pages 1558-1562. Brunak, S., Engelbrecht, J., and Knudsen, S. (1991). Prediction of human mRNA donor and acceptor sites from the DNA sequence. Journal of Molecular Biology, 220:49-65. Bruske, J. and Sommer, G. (1995a). Dynamic cell structure learns perfectly topology preserving map. Neural Computation, 7 :845-865. Bruske, J. and Sommer, G. (1995b). Dynamic cell structures. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 497-504. The MIT Press. Bud, A. and Nocholson, A. (1997). Scheduling trains with genetic algorithms. In Kasabov, N., Kozma, R., Ko, K., O'Shea, R., Coghill, G., and Gedeon, T., editors, Progress in Connectionist-Based Information Systems, volume 2, pages 1017-1020. Carpenter, G., Grossberg, S., Markuzon, M., Reynolds, J., and Rosen, D. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3:698-713. Casdagli, M. (1989). Nonlinear prediction of chaotic time-series. Physica D, 35:335-356. Castellano, G. and Fanelli, A. (2000). Fuzzy inference and rule extraction using a neural network. Neural Network World Journal, 3:361-371. Cechin, A. L., Epperlin, U., Rosentiel, W., and Koppenhoefer, B. (1996). The extraction of sugeno fuzzy rules from neural networks. In Andrews, R. and Diederich, J., editors, Rules and Networks, pages 16-24. Queensland University of Technology, Neurocomputing Research Centre. Chandonia, J.-M. and Karplus, M. (1995). Neural networks for secondary structure and structural class predictions. Protein Science, 4:275-285. Chauvin, Y. (1990). A back-propagation algorithm with optimal use of hidden units. In Touretzky, D., editor, Advances in Neural Information Processing Systems (Denver 1988), pages 519-526. Morgan Kaufmann, San Mateo. Chellapilla, K. and Fogel, D. B. (1999). Evolving neural networks to play checkers without relying on expert knowledge. IEEE Transactions on Neural Networks, 10(6):1382-1391. Chiu, S. (1994). Fuzzy model identification based on cluster estimation. Journal of Intelligent and Fuzzy Systems, 2. Cho, S.-B. and Shimohara, K. (1998). Cooperative behavior in evolved modular neural networks. In Methodologies for the Conception, Design and Application of Soft Computing: Proceedings of IIZUKA '98, pages 606-609. Choi, B. and Bluff, K. (1995). Genetic optimisation of control parameters of a neural network. In Kasabov, N. K. and Coghill, G., editors, Artificial Neural Networks and Expert Systems, pages 174-177. IEEE Computer Society Press. Cortes, C. and Vapnik, V. (1995). Support vector networks. Machine Learning, 20:273-297. Crick, F. (1989). The recent excitement about neural networks. Nature, 337:129-132. Crowder, R. (1990). Predicting the Mackey-Glass timeseries with cascade-correlation learning. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1990 Connectionist Models Summer School, pages 117-123, Carnegie Mellon Univ. Cybenko, G. (1989). Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals, and Systems, 2:303-314. Darwin, C. (1859). The Origin of Species by means of natural selection. John Murray, London. Davis, L., editor (1996). Handbook of Genetic Algorithms. International Thomson Computer Press. Davis, S. B. and Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions of Acoustics, Speech, and Signal Processing, 28(4):357-366. de Castro, L. N., Iyoda, E. M., Von Zuben, F. J., and Gudwin, R. (1998). Feedforward neural network initialization: an evolutionary approach. In Proceedings of Vth Brazilian Conference on Neural Networks, December 9-11, 1998, pages 43-48. Deng, D. and Kasabov, N. (1999). Evolving self-organizing map and its application in generating a world macroeconomic map. In Kasabov, N. and Ko, K., editors, Emerging Knowledge Engineering and Connectionist-based Systems (Proceedings of the ICONIP/ANZIIS/ANNES'99 Workshop "Future directions for intelligent systems and information sciences", Dunedin, 22-23 November 1999), pages 7-12. University of Otago Press. Diederichs, K., Freigang, J., Umhau, S., Zeth, K., and Breed, J. (1998). Prediction by a neural network of outer membrane /3-strand protein topology. Protein Science, 7:2413-2420. Dorado, J., Rabunal, J., Rivero, D., Santos, A., and Pazos, A. (2002). Automatic recurrent ANN rule extraction with genetic programming. In Proceedings of IJCNN 2002, pages 1552-1557. East, I. R. and Rowe, J. (1997). Abstract genetic representation of dynamical neural networks using kauffman networks. Artificial Life, 3:67-80. Eldracher, M. (1992). Classification of non-linear-separable real-world-problems usin 8-rule, perceptrons and topologically distributed encoding. In Proceedings of the 1992 ACM/SIGAPP Symposium on Applied Computing, volume 2, pages 1098-1104. ACM Press. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14:179-211. Esat, I., Kothari, B., and Wrathall, P. (1999). Encoding neural networks for GA based structural construction. In ICONIP '99 6th International Conference on Neural Information Processing, pages 359-365. Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks. Technical Report CMU-CS-88-162, Department of Computer Science, Carnegie-Mellon University. Fahlman, S. E. and Lebiere, C. (1990). The cascade-correlation learning architecture. In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 2, pages 524-532. Morgan Kaufman Publishers. Faraq, W. and Tawfik, A. (2000). On fuzzy model identification and the gas furnace data. In Proceedings of the LASTED International Conference. Faraq, W. A., Quintana, V. H., and Lambert-Torres, G. (1997). Neuro-fuzzy modeling of complex systems using genetic algorithms. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 1, pages 444-449. IEEE Press. Fisher, R. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7:179-188. Fogel, D. B., Wasson, E. C., Boughton, E. M., and Porto, V. W. (1997). A step toward computer-assisted mammography using evolutionary programming and neural networks. Cancer Letters, 119:93-97. Fogel, L. J., Owens, A. J., and Walsh, M. J. (1965). Artificial intelligence through a simulation of evolution. In Maxfield, M., Callahan, A., and Fogel, L., editors, Biophysics and Cybernetic Systems: Proceedings of the 2nd Cybernetic Sciences Symposium, pages 131-155. Fontanari, J. and Meir, R. (1991). Evolving a learning algorithm for the binary perceptron. Network, 2:353-359. Franzini, M. A. (1988). Learning to recognize spoken words: A study in connectionist speech recognition. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer, pages 407-416. Morgan Kaufmann. Frean, M. (1990). The upstart algorithm: A method for constructing and training feedforward neural networks. Neural Computation, 2(2):198-209. Fritzke, B. (1991). Unsupervised clustering with growing cell structures. In Proceedings of the IJCNN-91 Seattle. IEEE Press. Fritzke, B. (1993a). Growing cell structures - a self organizing network for unsupervised and supervised learning. Technical Report TR-93-026, International Computer Science Institute. Fritzke, B. (1993b). Kohonen feature maps and growing cell structures - a performance comparison. In Giles, C., Hanson, S., and Cowan, J., editors, Advances in Neural Information Processing Systems 5. Morgan Kaufmann. Fritzke, B. (1994). Supervised learning with growing cell structures. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 255-262. Morgan Kaufmann. Fritzke, B. (1995). A growing neural gas network learns topologies. In Tesauro, G., Tourezky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 625-632. The MIT Press. Fu, L. (1999). An expert network for DNA sequence analysis. IEEE Intelligent Systems, 14(January / February):65-71. Fukuda, T., Komata, Y., and Arakawa, T. (1997a). Recurrent neural networks with self-adaptive GAs for biped locomotion robot. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 3, pages 1710-1715. IEEE Press. Fukuda, T., Komata, Y., and Arakawa, T. (1997b). Recurrent neural networks with self-adaptive GAs for biped locomotion robot. In 1997 International Conference on Neural Networks (ICNN '97), volume 3, pages 1710- 1715, Westin Galleria Hotel, Houston, Texas, USA. IEEE Press. Fukumi, M. and Akamatsu, N. (1996). A genetic approach to feature selection for pattern recognition systems. In Methodologies for the Conception, Design and Application on Intelligent Systems: Proceedings of IIZUKA '96, pages 907-910. Furuhashi, T., Hasegawa, T., Horikawa, S.-i., and Uchikawa, Y. (1993). An adaptive fuzzy controller using fuzzy neural networks. In Proceedings of Fifth IFSA World Congress, pages 769-772. Furuhashi, T., Matushita, S., Tsutsui, H., and Uchikawa, Y. (1997). Knowledge extraction from hierarchical fuzzy model obtained by fuzzy neural networks and genetic algorithm. In Proceedings of the 1997 International Conference on Neural Networks (ICNN'97), volume 4, pages 2374-2379. IEEE Press. Gallant, S. I. (1993). Neural Network Learning and Expert Systems. MIT Press. Gan, M., Lan, H., and Zhang, L. (1995). A genetic-based method of generating fuzzy rules and membership functions by learning from example. In Proceedings of International Conference on Neural Information Processing (ICONIP'95), volume 1, pages 335-338. Gates, G. (1972). The reduced nearest neighbor rule. IEEE Transactions on Information Theory, pages 431-433. Gaweda, A. E., Zurada, J. M., and Aronhime, P. B. (2002). Efficient data-driven modeling with fuzzy relational rule network. In Proceedings of FUZZ-IEEE 2002, pages 174-178. Ghobakhlou, A., Watts, M., and Kasabov, N. (2000). On-line expansion of output space in evolving fuzzy neural networks. In Proceedings ICONIP 2000, Taejon, Korea, November, 2000, volume 2, pages 891-896. Ghobakhlou, A. A. and Seesink, R. (2001). An interactive multi modal system for mobile robotic control. In Proceedings of the Fifth Biannual Conference on Artificial Neural Networks and Expert Systems (ANNES2001 ), pages 93-99. Glaeser, A. (1998). Modular neural networks for low-complex phoneme recognition. In Proceedings of ICSLP'98, pages 1303-1306. Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimisation and Machine Learning. Addison-Wesley. Grefenstette, J. J. (1986). Optimization of control parameters for genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 16(1):122-128. Gueriot, D. and Maillard, E. (1996). A local approach for a fuzzy error function used in multilayer perceptron training through a genetic algorithm. In Proceedings of the 1996 IEEE international conference on neural networks, pages 1050-1055. Gupta, M. M. and Ding, H. (1994). Fuzzy neuronal networks and genetic algorithms. In Proceedings of the 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing (Iizuka, Japan), pages 187-188. Hakim, B. A. (2001). Extraction and optimization of fuzzy rules. In Zhang, L. and Gu, F., editors, Proceedings of ICONIP 2001, November 14-18, 2001, Shanghai, China, volume 1, pages 361-365. Fudan University Press. Hamker, F. H. (2001). Life-long learning cell structures-continuously learning without catastrophic interference. Neural Networks, 14:551-573. Hanebeck, D. and Schmidt, G. K. (1994). Optimization of fuzzy networks via genetic algorithms. In Proceedings of International Conference on Neural Information Processing, volume 3, pages 1583-1588. Hansen, L., Rasmussen, C., Svarer, C., and Larsen, J. (1994). Adaptive regularization. In Proceedings of the IEEE Workshop on Neural Networks for Signal Processing IV, pages 78-87, Piscataway, New Jersey. IEEE Press. Harp, S. A., Samad, T., and Guha, A. (1990). Designing application-specific neural networks using the genetic algorithm. In Toretzky, D. S., editor, Advances in Neural Information Processing Systems 5, pages 447-454. Morgan Kauffman Publishers. Hasegawa, T., Horikawa, Furuhashi, T., and Uchikawa, Y. (1992). A study'on fuzzy modeling of BOF using a fuzzy neural network. In Proceedings of the 2nd International Conference on Fuzzy Logic and Neural Networks (lizuka, Japan, July 17-22, 1992), pages 1061-1064. Hasegawa, T., Horikawa, Furuhashi, T., and Uchikawa, Y. (1993). An application of fuzzy neural networks to design of adaptive fuzzy controllers. In Proceedings of 1993 International Joint Conference on Neural Networks, pages 1761-1764. Hashiyama, T., Furuhashi, T., and Uchikawa, Y. (1993a). A fuzzy neural network for identifying changes of degrees of attent

Evolving connectionist systems: Characterisation, simplification, formalisation, explanation and optimisation

There are several well-known problems with conventional artificial neural networks (ANN), such as difficulties with selecting the structure of the network, and problems with forgetting previously-learned knowledge after further training. Constructive neural network algorithms attempt to solve these problems, but in turn have problems of their own. The Evolving Connectionist System (ECoS) is a class of open architecture artificial neural networks that are similar in the way in which neurons are added to their structures, and in the way in which their connection weights are modified. The ECoS algorithm is intended to address the problems with constructive neural networks. Several problems with ECoS are identified and discussed in this thesis. These problems are: the lack of comparison of ECoS with constructive neural networks; the excessive complexity of the Evolving Fuzzy Neural Network (EFuNN), which is the seminal ECoS network: the lack of a testable formalisation of ECoS; the dependence on fuzzy logic elements embedded within the network for fuzzy rule extraction; and the lack of methods for optimising ECoS networks. The research in this thesis addresses these problems. The overall theme of the research can be summarised as the characterisation, simplification, formalisation, explanation and optimisation of ECoS. Characterisation in this thesis means the comparison of ECoS with existing constructive ANN. Simplification means reducing the network to a minimalist implementation. Formalisation means the creation of a testable predictive model of ECoS training. Explanation means explaining ECoS networks via the extraction of fuzzy rules. Finally, optimisation means creating ECoS networks that have a minimum number of neurons with maximum accuracy. Each of these themes is approached in ways that build upon, and are complementary to, the basic ECoS network and ECoS training algorithm. The basic ECoS structure and algorithm is left unchanged, and the problems are addressed by extending that structure, rather than altering it as has been done in other work on EcoS. The principal contributions of this thesis are: a qualitative comparison of ECoS to constructive neural network algorithms; a proposed simplified version of EFuNN called SECoS; an experimentally tested formalisation of ECoS: novel algorithms for explicating SECoS via the extraction of fuzzy rules; and several novel algorithms for the optimisation of ECoS networks. The formalisation of ECoS and the proposed algorithms are evaluated on data from a set of standard benchmarking problems. Further experiments are performed with a data set with real-world applications, namely the recognition of isolated New Zealand English phonemes. The analyses of the experimental results show that the proposed algorithms are effective across both the benchmark data sets and the case study data set.UnpublishedAbraham, A. (2002). Optimization of evolutionary neural networks using hybrid learning algorithms. In Proceedings of IJCNN 2002, pages 2797-2802. Abreu, A. and Pinto-Ferreira, L. C. (1996). Fuzzy modeling: a rule based approach. In Proceedings of the fifth IEEE International Conference on Fuzzy Systems, pages 162-168. Aguiler, J. and Colmenares, A. (1997). Recognition algorithm using evolutionary learning on the random neural network. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 2, pages 1023-1028. IEEE Press. Alander, J. T. (1993). On robot navigation using a genetic algorithm. In Albrecht, R., Reeves, C., and Steele, N., editors, Artificial Neural Nets and Genetic Algorithms, pages 471-478. Alpaydin, E. (1994). GAL: Networks that grow when they learn and shrink when they forget. International Journal of Pattern Recognition and Artificial Intelligence, 8(1):391-414. Anderson, E. (1935). The irises of the gaspe peninsula. Bulletin of the American Iris Society, 59. Anderson, S., Merrill, J., and Port, R. (1988). Dynamic speech categorization with recurrent networks. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School, pages 398-396. Morgan Kaufmann. Andreassen, H., Bohr, H., Bohr, J., Brunak, S., Bugge, T., Cotterill, R., Jacobsen, C., Kusk, P., Lautrup, B., Petersen, S., Saermark, T., and Ulrich, K. (1990). Analysis of the secondary structure of the human immunodeficiency virus (HIV) proteins p17, gp120, and gp41 by computer modeling based on neural networks methods. Journal of Acquired Immune Deficiency Syndromes, 3:615-622. Andrews, R., Diederich, J., and Tickle, A. B. (1995). Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowledge-Based Systems, 8(6):373-389. Andrews, R. and Geva, S. (1997). Refining expert knowledge with an artificial neural network. In Kasabov, N., Kozma, R., Ko, K., S'Shea, R., Coghill, G., and Gedeon, T., editors, Progress in Connectionist-Based Information Systems, volume 2, pages 847-850. Springer. Angeline, P. K., Saunders, G. M., and Pollack, J. B. (1994). An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks, 5(1):54-65. Antonisse, J. (1989). A new interpretation of schema notation that overturns the binary encoding constraint. In Schaffer, J., editor, Proceedings of the Third International Conference on Genetic Algorithms, pages 86-91. Arena, P., Caponetto, R., Fortuna, L., and Xibilia, M. G. (1993). M.L.P. optimal topology via genetic algorithms. In Artificial Neural Nets and Genetic Algorithms, pages 670-674. Springer-Verlag Wien New York. Ash, T. (1989). Dynamic node creation in backpropagation networks. Connection Science, 1(4):365-375. Baba, N., Marume, M., and Itoh, K. (1992). Utilization of stochastic automaton and genetic algorithm for neural network design. In Proceedings of the 2nd International Conference on Fuzzy Logic and Neural Networks, volume 2, pages 837-840, Iizuka, Japan. Back, T., Hoffmeister, F., and Schwefel, H.-P. (1991). A survey of evolution strategies. In Belew, R. K. and Booker, L. B., editors, Proceedings of the Fourth International Conference on Genetic Algorithms, pages 2-9. Balakrishnan, K. and Honavar, V. (1996). Analysis of neurocontrollers designed by simulated evolution. In International Conference on Neural Networks 1996: Plenary, Panel and Special Sessions, pages 130-135. Baldi, P. and Brunak, S. (1998). Bioinformatics: The Machine Learning Approach. MIT Press. Bebis, G., Georgiopoulos, and Kasparis, T. (1996). Coupling weight elimination and genetic algorithms. In Proceedings of the 1996 IEEE International Conference on Neural Networks, pages 1115-1120. Belew, R. K., McInerney, J., and Schraudolph, N. N. (1990). Evolving networks: Using the genetic algorithm with connectionist learning. In Langton, C. G., Taylor, C., Farmer, J. D., and Rasmussen, S., editors, Artificial Life II, pages 511-547, Santa Fe, New Mexico. Addison-Wesley Publishing Company. Bengio, Y. and De Mori, R. (1988). Speaker normalization and automatic speech recognition using spectral lines and neural networks. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer School, pages 388-397. Morgan Kaufmann. Billings, S. A. and Zheng, G. L. (1995). Radial basis function network configuration using genetic algorithms. Neural Networks, 8(6):877-890. Bisant, D. and Maizel, J. (1995). Identification of ribosome binding sites in Escherichia coli using neural network models. Nucleic Acids Research, 23(9):1632-1639. Bohr, H., Bohr, J., Brunak, S., Cotterill, R. M., Lautrup, B., Norskov, L., Olsen, 0. H., and Petersen, S. B. (1988). Protein secondary structure and homology by neural networks. The a-helices in rhodopsin. FEBS Letters, 241(1,2):223-228. Bornholdt, S. and Graudenz, D. (1992). General asymmetric neural networks and structure design by genetic algorithms. Neural Networks, 5:327-334. Bose, N. and Garga, A. K. (1993). Neural network design using voronoi diagrams. IEEE Transactions on Neural Networks, 4(5):778-787. Bourland, H. and Wellekens, C. (1987). Multiplayer perceptrons and automatic speech recognition. In IEEE First Annual Conference on Neural Networks, volume IV, pages 407-416, San Diego. Box, G. E. and Jenkins, G. M. (1970). Time Series Analysis forecasting and control. Holden-Day. Brasil, L. M., de Azevedo, F. M., and Barreto, J. M. (2000). A hybrid expert system for the diagnosis of epileptic crisis. Artificial Intelligence in Medicine, 585:1-7. Brown, A. and Card, H. (1997). Evolutionary artificial neural networks for competitive learning. In Proceedings of /CNN, pages 1558-1562. Brunak, S., Engelbrecht, J., and Knudsen, S. (1991). Prediction of human mRNA donor and acceptor sites from the DNA sequence. Journal of Molecular Biology, 220:49-65. Bruske, J. and Sommer, G. (1995a). Dynamic cell structure learns perfectly topology preserving map. Neural Computation, 7 :845-865. Bruske, J. and Sommer, G. (1995b). Dynamic cell structures. In Tesauro, G., Touretzky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 497-504. The MIT Press. Bud, A. and Nocholson, A. (1997). Scheduling trains with genetic algorithms. In Kasabov, N., Kozma, R., Ko, K., O'Shea, R., Coghill, G., and Gedeon, T., editors, Progress in Connectionist-Based Information Systems, volume 2, pages 1017-1020. Carpenter, G., Grossberg, S., Markuzon, M., Reynolds, J., and Rosen, D. (1992). Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps. IEEE Transactions on Neural Networks, 3:698-713. Casdagli, M. (1989). Nonlinear prediction of chaotic time-series. Physica D, 35:335-356. Castellano, G. and Fanelli, A. (2000). Fuzzy inference and rule extraction using a neural network. Neural Network World Journal, 3:361-371. Cechin, A. L., Epperlin, U., Rosentiel, W., and Koppenhoefer, B. (1996). The extraction of sugeno fuzzy rules from neural networks. In Andrews, R. and Diederich, J., editors, Rules and Networks, pages 16-24. Queensland University of Technology, Neurocomputing Research Centre. Chandonia, J.-M. and Karplus, M. (1995). Neural networks for secondary structure and structural class predictions. Protein Science, 4:275-285. Chauvin, Y. (1990). A back-propagation algorithm with optimal use of hidden units. In Touretzky, D., editor, Advances in Neural Information Processing Systems (Denver 1988), pages 519-526. Morgan Kaufmann, San Mateo. Chellapilla, K. and Fogel, D. B. (1999). Evolving neural networks to play checkers without relying on expert knowledge. IEEE Transactions on Neural Networks, 10(6):1382-1391. Chiu, S. (1994). Fuzzy model identification based on cluster estimation. Journal of Intelligent and Fuzzy Systems, 2. Cho, S.-B. and Shimohara, K. (1998). Cooperative behavior in evolved modular neural networks. In Methodologies for the Conception, Design and Application of Soft Computing: Proceedings of IIZUKA '98, pages 606-609. Choi, B. and Bluff, K. (1995). Genetic optimisation of control parameters of a neural network. In Kasabov, N. K. and Coghill, G., editors, Artificial Neural Networks and Expert Systems, pages 174-177. IEEE Computer Society Press. Cortes, C. and Vapnik, V. (1995). Support vector networks. Machine Learning, 20:273-297. Crick, F. (1989). The recent excitement about neural networks. Nature, 337:129-132. Crowder, R. (1990). Predicting the Mackey-Glass timeseries with cascade-correlation learning. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1990 Connectionist Models Summer School, pages 117-123, Carnegie Mellon Univ. Cybenko, G. (1989). Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals, and Systems, 2:303-314. Darwin, C. (1859). The Origin of Species by means of natural selection. John Murray, London. Davis, L., editor (1996). Handbook of Genetic Algorithms. International Thomson Computer Press. Davis, S. B. and Mermelstein, P. (1980). Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions of Acoustics, Speech, and Signal Processing, 28(4):357-366. de Castro, L. N., Iyoda, E. M., Von Zuben, F. J., and Gudwin, R. (1998). Feedforward neural network initialization: an evolutionary approach. In Proceedings of Vth Brazilian Conference on Neural Networks, December 9-11, 1998, pages 43-48. Deng, D. and Kasabov, N. (1999). Evolving self-organizing map and its application in generating a world macroeconomic map. In Kasabov, N. and Ko, K., editors, Emerging Knowledge Engineering and Connectionist-based Systems (Proceedings of the ICONIP/ANZIIS/ANNES'99 Workshop "Future directions for intelligent systems and information sciences", Dunedin, 22-23 November 1999), pages 7-12. University of Otago Press. Diederichs, K., Freigang, J., Umhau, S., Zeth, K., and Breed, J. (1998). Prediction by a neural network of outer membrane /3-strand protein topology. Protein Science, 7:2413-2420. Dorado, J., Rabunal, J., Rivero, D., Santos, A., and Pazos, A. (2002). Automatic recurrent ANN rule extraction with genetic programming. In Proceedings of IJCNN 2002, pages 1552-1557. East, I. R. and Rowe, J. (1997). Abstract genetic representation of dynamical neural networks using kauffman networks. Artificial Life, 3:67-80. Eldracher, M. (1992). Classification of non-linear-separable real-world-problems usin 8-rule, perceptrons and topologically distributed encoding. In Proceedings of the 1992 ACM/SIGAPP Symposium on Applied Computing, volume 2, pages 1098-1104. ACM Press. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14:179-211. Esat, I., Kothari, B., and Wrathall, P. (1999). Encoding neural networks for GA based structural construction. In ICONIP '99 6th International Conference on Neural Information Processing, pages 359-365. Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks. Technical Report CMU-CS-88-162, Department of Computer Science, Carnegie-Mellon University. Fahlman, S. E. and Lebiere, C. (1990). The cascade-correlation learning architecture. In Touretzky, D. S., editor, Advances in Neural Information Processing Systems 2, pages 524-532. Morgan Kaufman Publishers. Faraq, W. and Tawfik, A. (2000). On fuzzy model identification and the gas furnace data. In Proceedings of the LASTED International Conference. Faraq, W. A., Quintana, V. H., and Lambert-Torres, G. (1997). Neuro-fuzzy modeling of complex systems using genetic algorithms. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 1, pages 444-449. IEEE Press. Fisher, R. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7:179-188. Fogel, D. B., Wasson, E. C., Boughton, E. M., and Porto, V. W. (1997). A step toward computer-assisted mammography using evolutionary programming and neural networks. Cancer Letters, 119:93-97. Fogel, L. J., Owens, A. J., and Walsh, M. J. (1965). Artificial intelligence through a simulation of evolution. In Maxfield, M., Callahan, A., and Fogel, L., editors, Biophysics and Cybernetic Systems: Proceedings of the 2nd Cybernetic Sciences Symposium, pages 131-155. Fontanari, J. and Meir, R. (1991). Evolving a learning algorithm for the binary perceptron. Network, 2:353-359. Franzini, M. A. (1988). Learning to recognize spoken words: A study in connectionist speech recognition. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionist Models Summer, pages 407-416. Morgan Kaufmann. Frean, M. (1990). The upstart algorithm: A method for constructing and training feedforward neural networks. Neural Computation, 2(2):198-209. Fritzke, B. (1991). Unsupervised clustering with growing cell structures. In Proceedings of the IJCNN-91 Seattle. IEEE Press. Fritzke, B. (1993a). Growing cell structures - a self organizing network for unsupervised and supervised learning. Technical Report TR-93-026, International Computer Science Institute. Fritzke, B. (1993b). Kohonen feature maps and growing cell structures - a performance comparison. In Giles, C., Hanson, S., and Cowan, J., editors, Advances in Neural Information Processing Systems 5. Morgan Kaufmann. Fritzke, B. (1994). Supervised learning with growing cell structures. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Advances in Neural Information Processing Systems 6, pages 255-262. Morgan Kaufmann. Fritzke, B. (1995). A growing neural gas network learns topologies. In Tesauro, G., Tourezky, D., and Leen, T., editors, Advances in Neural Information Processing Systems 7, pages 625-632. The MIT Press. Fu, L. (1999). An expert network for DNA sequence analysis. IEEE Intelligent Systems, 14(January / February):65-71. Fukuda, T., Komata, Y., and Arakawa, T. (1997a). Recurrent neural networks with self-adaptive GAs for biped locomotion robot. In Proceedings of the 1997 IEEE International Conference on Neural Networks, volume 3, pages 1710-1715. IEEE Press. Fukuda, T., Komata, Y., and Arakawa, T. (1997b). Recurrent neural networks with self-adaptive GAs for biped locomotion robot. In 1997 International Conference on Neural Networks (ICNN '97), volume 3, pages 1710- 1715, Westin Galleria Hotel, Houston, Texas, USA. IEEE Press. Fukumi, M. and Akamatsu, N. (1996). A genetic approach to feature selection for pattern recognition systems. In Methodologies for the Conception, Design and Application on Intelligent Systems: Proceedings of IIZUKA '96, pages 907-910. Furuhashi, T., Hasegawa, T., Horikawa, S.-i., and Uchikawa, Y. (1993). An adaptive fuzzy controller using fuzzy neural networks. In Proceedings of Fifth IFSA World Congress, pages 769-772. Furuhashi, T., Matushita, S., Tsutsui, H., and Uchikawa, Y. (1997). Knowledge extraction from hierarchical fuzzy model obtained by fuzzy neural networks and genetic algorithm. In Proceedings of the 1997 International Conference on Neural Networks (ICNN'97), volume 4, pages 2374-2379. IEEE Press. Gallant, S. I. (1993). Neural Network Learning and Expert Systems. MIT Press. Gan, M., Lan, H., and Zhang, L. (1995). A genetic-based method of generating fuzzy rules and membership functions by learning from example. In Proceedings of International Conference on Neural Information Processing (ICONIP'95), volume 1, pages 335-338. Gates, G. (1972). The reduced nearest neighbor rule. IEEE Transactions on Information Theory, pages 431-433. Gaweda, A. E., Zurada, J. M., and Aronhime, P. B. (2002). Efficient data-driven modeling with fuzzy relational rule network. In Proceedings of FUZZ-IEEE 2002, pages 174-178. Ghobakhlou, A., Watts, M., and Kasabov, N. (2000). On-line expansion of output space in evolving fuzzy neural networks. In Proceedings ICONIP 2000, Taejon, Korea, November, 2000, volume 2, pages 891-896. Ghobakhlou, A. A. and Seesink, R. (2001). An interactive multi modal system for mobile robotic control. In Proceedings of the Fifth Biannual Conference on Artificial Neural Networks and Expert Systems (ANNES2001 ), pages 93-99. Glaeser, A. (1998). Modular neural networks for low-complex phoneme recognition. In Proceedings of ICSLP'98, pages 1303-1306. Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimisation and Machine Learning. Addison-Wesley. Grefenstette, J. J. (1986). Optimization of control parameters for genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 16(1):122-128. Gueriot, D. and Maillard, E. (1996). A local approach for a fuzzy error function used in multilayer perceptron training through a genetic algorithm. In Proceedings of the 1996 IEEE international conference on neural networks, pages 1050-1055. Gupta, M. M. and Ding, H. (1994). Fuzzy neuronal networks and genetic algorithms. In Proceedings of the 3rd International Conference on Fuzzy Logic, Neural Nets and Soft Computing (Iizuka, Japan), pages 187-188. Hakim, B. A. (2001). Extraction and optimization of fuzzy rules. In Zhang, L. and Gu, F., editors, Proceedings of ICONIP 2001, November 14-18, 2001, Shanghai, China, volume 1, pages 361-365. Fudan University Press. Hamker, F. H. (2001). Life-long learning cell structures-continuously learning without catastrophic interference. Neural Networks, 14:551-573. Hanebeck, D. and Schmidt, G. K. (1994). Optimization of fuzzy networks via genetic algorithms. In Proceedings of International Conference on Neural Information Processing, volume 3, pages 1583-1588. Hansen, L., Rasmussen, C., Svarer, C., and Larsen, J. (1994). Adaptive regularization. In Proceedings of the IEEE Workshop on Neural Networks for Signal Processing IV, pages 78-87, Piscataway, New Jersey. IEEE Press. Harp, S. A., Samad, T., and Guha, A. (1990). Designing application-specific neural networks using the genetic algorithm. In Toretzky, D. S., editor, Advances in Neural Information Processing Systems 5, pages 447-454. Morgan Kauffman Publishers. Hasegawa, T., Horikawa, Furuhashi, T., and Uchikawa, Y. (1992). A study'on fuzzy modeling of BOF using a fuzzy neural network. In Proceedings of the 2nd International Conference on Fuzzy Logic and Neural Networks (lizuka, Japan, July 17-22, 1992), pages 1061-1064. Hasegawa, T., Horikawa, Furuhashi, T., and Uchikawa, Y. (1993). An application of fuzzy neural networks to design of adaptive fuzzy controllers. In Proceedings of 1993 International Joint Conference on Neural Networks, pages 1761-1764. Hashiyama, T., Furuhashi, T., and Uchikawa, Y. (1993a). A fuzzy neural network for identifying changes of degrees of attent