Modelling the emergence of speech sound categories in evolving connectionist systems

We report on the clustering of nodes in internally represented acoustic space. Learners of different languages partition perceptual space distinctly. Here, an Evolving Connectionist-Based System (ECOS) is used to model the perceptual space of New Zealand English. Currently, the system evolves in an unsupervised, self-organising manner. The perceptual space can be visualised, and the important features of the input patterns analysed. Additionally, the path of the internal representations can be seen. The results here will be used to develop a supervised system that can be used for speech recognition based on the evolved, internal sub-word units.Unpublished[1] P. Jusczyk, The Discovery of Spoken Language, Cambridge, MA: MIT Press, 1997. [2] P. K. Kuhl, "Speech Perception," in Introduction to Communication Sciences and Disorders, F. Minifie, Ed., San Diego, CA: Singular Pub Group, 1994, pp. 77-142. [3] P. Lieberman, Uniquely Human: The Evolution of Speech, Thought, and Selfless Behavior, Cambridge, MA: Harvard University Press, 1991 [4] Liberman, Speech: A Special Code, Cambridge, MA: MIT Press, 1996. [5] N. Chomsky, The Minimalist Program, Cambridge, MA: MIT Press, 1995. [6] M. S. Seidenberg, "Language acquisition and use: Learning and applying probabilistic constraints," Science, vol. 275, pp. 1599-1603, 1997. [7] E. Bates and J. Elman, "Learning rediscovered," Science, vol. 274, pp 1849-1850, 1996. [8] K. Plunkett, "Connectionist approaches to language acquisition," in The Handbook of Child Language, P. Fletcher and B. MacWhinney, Eds., Oxford: Blackwell, 1995, pp. 36-72. [9] N. Kasabov, "The ECOS framework and the 'eco' training method for evolving connectionist systems," Journal of Advanced Computational Intelligence, vol. 2, no. 6, pp. 195-202, 1998. [10] N. Kasabov, "Evolving fuzzy neural networks: Theory and applications for on-line adaptive prediction, decision making and control," Australian Journal of Intelligent Information Processing Systems, vol. 5 (3), pp. 154-160, 1998. [11] N. Kasabov, "Evolving connectionist and fuzzy connectionist systems – theory and applications for adaptive, on-line intelligent systems," in Neuro-Fuzzy Techniques for Intelligent Information Systems, N. Kasabov and R. Kozma, Eds., Heidelberg: Physica Verlag, 1999, pp. 111-146. [12] S. Sinclair, and C. Watson, "The Development of the Otago Speech Database," in Proceedings of ANNES ’95, 1995, pp. 298-301. [13] N. Kasabov, R. Kilgour and S. Sinclair, "From hybrid adjustable neuro-fuzzy systems to adaptive connectionist-based systems for phoneme and word recognition," Fuzzy Sets and Systems, 130 (2), 1999. [14] N. Kasabov, "A framework for intelligent conscious machines and its application to multilingual speech recognition systems," Brain-like computing and intelligent information systems, S. Amari and N. Kasabov, Eds., Singapore: Springer Verlag, 1998

Modelling the emergence of speech sound categories in evolving connectionist systems

We report on the clustering of nodes in internally represented acoustic space. Learners of different languages partition perceptual space distinctly. Here, an Evolving Connectionist-Based System (ECOS) is used to model the perceptual space of New Zealand English. Currently, the system evolves in an unsupervised, self-organising manner. The perceptual space can be visualised, and the important features of the input patterns analysed. Additionally, the path of the internal representations can be seen. The results here will be used to develop a supervised system that can be used for speech recognition based on the evolved, internal sub-word units.Unpublished[1] P. Jusczyk, The Discovery of Spoken Language, Cambridge, MA: MIT Press, 1997. [2] P. K. Kuhl, "Speech Perception," in Introduction to Communication Sciences and Disorders, F. Minifie, Ed., San Diego, CA: Singular Pub Group, 1994, pp. 77-142. [3] P. Lieberman, Uniquely Human: The Evolution of Speech, Thought, and Selfless Behavior, Cambridge, MA: Harvard University Press, 1991 [4] Liberman, Speech: A Special Code, Cambridge, MA: MIT Press, 1996. [5] N. Chomsky, The Minimalist Program, Cambridge, MA: MIT Press, 1995. [6] M. S. Seidenberg, "Language acquisition and use: Learning and applying probabilistic constraints," Science, vol. 275, pp. 1599-1603, 1997. [7] E. Bates and J. Elman, "Learning rediscovered," Science, vol. 274, pp 1849-1850, 1996. [8] K. Plunkett, "Connectionist approaches to language acquisition," in The Handbook of Child Language, P. Fletcher and B. MacWhinney, Eds., Oxford: Blackwell, 1995, pp. 36-72. [9] N. Kasabov, "The ECOS framework and the 'eco' training method for evolving connectionist systems," Journal of Advanced Computational Intelligence, vol. 2, no. 6, pp. 195-202, 1998. [10] N. Kasabov, "Evolving fuzzy neural networks: Theory and applications for on-line adaptive prediction, decision making and control," Australian Journal of Intelligent Information Processing Systems, vol. 5 (3), pp. 154-160, 1998. [11] N. Kasabov, "Evolving connectionist and fuzzy connectionist systems – theory and applications for adaptive, on-line intelligent systems," in Neuro-Fuzzy Techniques for Intelligent Information Systems, N. Kasabov and R. Kozma, Eds., Heidelberg: Physica Verlag, 1999, pp. 111-146. [12] S. Sinclair, and C. Watson, "The Development of the Otago Speech Database," in Proceedings of ANNES ’95, 1995, pp. 298-301. [13] N. Kasabov, R. Kilgour and S. Sinclair, "From hybrid adjustable neuro-fuzzy systems to adaptive connectionist-based systems for phoneme and word recognition," Fuzzy Sets and Systems, 130 (2), 1999. [14] N. Kasabov, "A framework for intelligent conscious machines and its application to multilingual speech recognition systems," Brain-like computing and intelligent information systems, S. Amari and N. Kasabov, Eds., Singapore: Springer Verlag, 1998

From hybrid adjustable neuro-fuzzy systems to adaptive connectionist-based systems for phoneme and word recognition

This paper discusses the problem of adaptation in automatic speech recognition systems (ASRS) and suggests several strategies for adaptation in a modular architecture for speech recognition. The architecture allows for adaptation at different levels of the recognition process, where modules can be adapted individually based on their performance and the performance of the whole system. Two realisations of this architecture are presented along with experimental results from small-scale experiments. The first realisation is a hybrid system for speaker-independent phoneme-based spoken word recognition, consisting of neural networks for recognising English phonemes and fuzzy systems for modelling acoustic and linguistic knowledge. This system is adjustable by additional training of individual neural network modules and tuning the fuzzy systems. The increased accuracy of the recognition through appropriate adjustment is also discussed. The second realisation of the architecture is a connectionist system that uses fuzzy neural networks FuNNs to accommodate both a prior linguistic knowledge and data from a speech corpus. A method for on-line adaptation of FuNNs is also presented.Unpublished[1] S. Amari, N.K. Kasabov (Eds.), Brain-like Computing and Intelligent Information Systems, Springer, Berlin, 1997. [2] Clark, C. Yallop, An Introduction to Phonetics and Phonology, Blackwell, Cambridge MA, 1990. [3] Cole et al., The challenge of spoken language systems: research directions for the Nineties, IEEE Trans. Speech Audio Process. 3 (1) (1995) 1-21. [4] Li-Min Fu, Building expert systems on neural architectures, Proc. Ist IEEE Internat. Conf. on Artificial Neural Networks, 1989, pp. 221-225. [5] Goldberg, Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley, New York, 1989. [6] Q. Huo, C.-H. Lee, A study of on-line Quasi-Bayes adaptation for CDHMM-based speech recognition, Proc. IEEE Internat. Conf. on Acoustic, Speech, and Signal Processing, 1996, pp. 705-708. [7] J.S.R. Jang, ANFIS: adaptive-network-based fuzzy inference system, IEEETrans. Systems Man Cybernet. 23 (3) (1993) 665-684. [8] N.K. Kasabov, Building comprehensive AI and the task of speech recognition, in: J. Alspector, R. Goodman, T. Brown (Eds.), Applications of Neural Networks to Telecommunications 2, Lawrence Erlbaum, Hillsdale, NJ, 1995, pp. 178-185. [9] N.K. Kasabov, Hybrid connectionist fuzzy production systems - towards building comprehensive AI, Intell. Automat. Soft Comput. 1 (1995) 351-360. [10] N.K. Kasabov, Hybrid Connectionist Fuzzy Rule-based Systems for Speech Recognition, Lecture Notes in Computer Science/Artificial Intelligence, vol. 1011, Springer, Berlin, 1995, pp. 20-33. [11] N.K. Kasabov, Adaptable counectionist production systems, Neurocomputing 13 (1996) 95-117. [12] N.K. Kasabov, Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, MIT Press, Cambridge, MA, 1996. [13] N.K. Kasabov, Learning and approximate reasoning in fuzzy neural networks and hybrid systems, Fuzzy Sets and Systems 82 (1996) 135-149. [14] N.K. Kasabov, Learning strategies for modular neuro-fuzzy systems: a case study on phoneme-based speech recognition, J. Intell. Fuzzy Systems 5 (1997) 1-10. [15] N.K. Kasabov, A framework for intelligent conscious machines and applications for adaptive speech recognition, in: Amari, N.K. Kasabov (Eds.), Brain-like Computing and Intelligent Systems, Springer, Berlin, 1997. [16] N.K. Kasabov, ECOS: Evolving connectionist systems - methods, algorithms, applications, in: Proc. ICONIP'98 Conf. (International Conference on Neuro-Information Processing), Kitakyushu, Japan, 21-23 October 1998, pp. 793-796. [17] N.K. Kasabov, J.S. Kim, M. Watts, A. Gray, FuNN/2 - a fuzzy neural network architecture for adaptive learning and knowledge acquisition, Inform. Sci. 101 (3-4) (1997) 155-175. [18] N.K. Kasabov, R. Kozma, M. Watts, Optimization and adaption of fuzzy neural networks through genetic algorithms and learning-with-forgetting methods and applications for phoneme based speech recognition, Inform. Sci. 110 (1998) 61-79. [19] N.K. Kasabov, E. Postma, J. van en Herik, AVIS: a connectionist framework for integrated audio and visual information processing systems, in: Proc. Iizuka'98 Conf., 16-20 October, Iizuka, Japan, 1998. [20] N.K. Kasabov, S.J. Sinclair, R. Kilgour, C. Watson, M. Laws, D. Kassabova, Intelligent human computer interfaces and the case study of building English-to-Māori talking dictionary, in: N.K. Kasabov, G. Coghill (Eds.), Proc. ANNES'95, Dunedin, IEEE Computer Society Press, Los Alamitos, 1995, pp. 294-297. [21] R.I, Kilgour, Hybrid fuzzy systems and neural networks for speech recognition, Unpublished Masters Thesis, University of Otago, 1996. [22] K. Kim, N. Relkin, K.-M. Lee, J. Hirsch, Distinct cortical areas associated with native and second languages, Nature 388 (1997) 171-174. [23] D. Massaro, Perceiving Talking Faces, MIT Press, Cambridge, MA, 1997. [24] D. Massaro, M. Cohen, Integration of visual and auditory information in speech perception, J. Experimental Psychol.: Human Perception Performance 9 (1983) 753-771. [25] Mitra, S. Pal, Fuzzy multi-layer perceptron, inferencing and rule generation, IEEE Trans. Neural Networks 6 (1995) 51-63. [26] Morgan, C. Scofield, Neural Networks and Speech Processing, Kluwer Academic Publishers, Amsterdam, 1991. [27] N. Pal, E. Kumar, Neural networks for dimensionality reduction, in: Kasabov et al. (Eds.), Connectionist Based Information Systems, Proc. ICONIP'97 Conf., Dunedin, Springer, Singapore, 1997, pp. 221-224. [28] R. Rabiner, Applications of voice processing to telecommunications, Proc. IEEE 82 (1994) 199-228. [29] D. Robinson, Artificial Intelligence and Expert Systems, McGraw Hill, New York, 1988. [30] G. Rummery, M. Niranjan, On-line Q-learning using connectionist systems, CUED/F-INFENG/TR 166, Cambridge University Engineering Department, 1994. [31] A. Sankar, L. Neumeyer, M. Weintraub, An experimental study of acoustic adaptation algorithms, Proc. IEEE Internat. Conf. on Acoustic, Speech, and Signal Processing, 1996, pp. 713-716. [32] M.A-S. Seyed, Bayesian and predictive techniques for speaker adaptation, Unpublished PhD Thesis, University of Cambridge, 1996. [33] S.J. Sinclair, Development of an isolated speech digit recognition system based on backpropagation neural networks, Unpublished Masters Thesis, University of Otago, 1996. [34] S.J. Sinclair, C. Watson, The development of the Otago speech database, in: N. Kasabov, G. Coghill (Eds.), Proc. ANNES '95, Dunedin, IEEE Computer Society Press, Los Alamitos, 1995, pp. 294-297. [35] T. Yamakawa, H. Kusanagi, E. Uchino, T. Miki, A new effective algorithm for neo fuzzy neuron model, in: Proc. 5th IFSA World Congress, 1993, pp. 1017-1020. [36] Yamazaki, Research activities on spontaneous speech, in: N. Kasabov, G. Coghill (Eds.), Proc. ANNES '95, Dunedin, IEEE Computer Society Press, Los Alamitos, 1995, pp. 280-283. [37] S. Young, Large vocabulary continuous speech recognition: a review, Internal Report, Cambridge University Engineering Department, 1996

Evolving connectionist systems for on-line, knowledge-based learning: Principles and applications

The paper introduces evolving connectionist systems (ECOS) as an effective approach to building on-line, adaptive intelligent systems. ECOS evolve through incremental, hybrid (supervised/unsupervised), on-line learning. They can accommodate new input data, including new features, new classes, etc. through local element tuning. New connections and new neurons are created during the operation of the system. The ECOS framework is presented and illustrated on a particular type of evolving neural networks---evolving fuzzy neural networks (EFuNNs). EFuNNs can learn spatial-temporal sequences in an adaptive way, through one pass learning. Rules can be inserted and extracted at any time of the system operation. The characteristics of ECOS and EFuNNs are illustrated on several case studies that include: adaptive pattern classification; adaptive, phoneme-based spoken language recognition; adaptive dynamic time-series prediction; intelligent agents.Unpublished1. Albus, J.S., A new approach to manipulator control: The cerebellar model articulation controller (CMAC), Tarns. of the ASME: Journal of Dynamic Systems, Measurement, and Control, pp.220:227, Sept. (1975) 2. Amari, S. and Kasabov, N. eds, “Brain-like Computing and Intelligent Information Systems”, Springer Verlag,1998. 3. Amari, S., Mathematical foundations of neuro-computing, Proc. of IEEE, 78 (9), Sept. (1990) 4. Arbib, M. (ed) The Handbook of Brain Theory and Neural Networks,The MIT Press, 1995. 5. Bollacker, K., S.Lawrence and L.Giles, CiteSeer: An autonomous Web agent for automatic retrieval and identification of interesting publications, 2nd International ACM conference on autonomous agents, ACM Press, 1998, 116-123 6. Bottu and Vapnik, “Local learning computation”, Neural Computation, 4, 888-900 (1992) 7. Carpenter, G. and Grossberg S., Pattern recognition by self-organizing neural networks , The MIT Press, Cambridge, Massachusetts (1991) 8. Carpenter, G. and S. Grossberg, “ART3: Hierarchical search using chemical transmitters in self-organising pattern-recognition architectures”, Neural Networks, 3(2) 129-152(1990). 9. Carpenter, G. S. Grossberg, N. Markuzon, J.H. Reynolds, D.B. Rosen, “FuzzyARTMAP: A neural network architecture for incremental supervised learning of analog multi-dimensional maps,” IEEE Transactions of Neural Networks , vol.3, No.5, 698-713 (1991). 10. Cybenko, G., Approximation by super-positions of sigmoidal function, Mathematics of Control, Signals and Systems, 2, 303-314 (1989) 11. DeGaris, H., “Circuits of Production Rule - GenNets – The genetic programming of nervous systems”, in: Albrecht, R., Reeves, C. and Steele, N. (eds) Artificial Neural Networks and Genetic Algorithms, Springer Verlag (1993) 12. Duda and Hart, “Pattern classification and scene analysis”, New York: Willey (1973) 13. Edelman, G., Neuronal Darwinism: The theory of neuronal group selection, Basic Books (1992). 14. Elman, J., E.Bates, M.Johnson, A.Karmiloff-Smith, D.Parisi and K.Plunkett, Rethinking Innateness (A Connectionist Perspective of Development), The MIT Press, 1997 15. Fahlman, C., and C. Lebiere, "The Cascade-Correlation Learning Architecture", in: Turetzky, D (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 524-532 (1990). 16. Farmer, J.D., and Sidorowitch, Predicting chaotic time series, Physical Review Letters, 59, 845 (1987) 17. Freeman, J., D. Saad, “On-line learning in radial basis function networks”, Neural Computation vol. 9, No.7 (1997). 18. French, “Semi-destructive representations and catastrophic forgetting in connectionist networks, Connection Science, 1, 365-377 (1992) 19. Fritzke, B. “A growing neural gas network learns topologies”, Advances in Neural Information Processing Systems, vol.7 (1995). 20. Fukuda, T., Y. Komata, and T. Arakawa, "Recurrent Neural Networks with Self-Adaptive GAs for Biped Locomotion Robot", In: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press (1997) 21. Funuhashi, K., On the approximate realization of continuous mappings by neural networks, Neural Networks, 2, 183-192 (1989) 22. Gaussier, T., and S. Zrehen, “A topological neural map for on-line learning: Emergence of obstacle avoidance in a mobile robot”, In: From Animals to Animats No.3, 282-290, (1994). 23. Goldberg, D.E., Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley (1989) 24. Goodman, R., C.M. Higgins, J.W. Miller, P.Smyth, "Rule-based neural networks for classification and probability estimation", Neural Computation, 14, 781-804 (1992). 25. Hashiyama, T., T. Furuhashi, Y Uchikawa,. “A Decision Making Model Using a Fuzzy Neural Network”, in: Proceedings of the 2nd International Conference on Fuzzy Logic & Neural Networks, Iizuka, Japan, 1057-1060, (1992). 26. Hassibi and Stork, “Second order derivatives for network pruning: Optimal Brain Surgeon,” in: Advances in Neural Information Processing Systems, 4, 164-171, (1992). 27. Hech-Nielsen, R. “Counter-propagation networks”, IEEE First int. conference on neural networks, San Diego, vol.2, pp.19-31 (1987) 28. Heskes, T.M., B. Kappen, “On-line learning processes in artificial neural networks”, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993). 29. Ishikawa, M., "Structural Learning with Forgetting", Neural Networks 9, 501-521, (1996). 30. Kasabov, N. "Adaptable connectionist production systems”. Neurocomputing, 13 (2-4) 95-117, (1996). 31. Kasabov, N. The ECOS Framework and the ECO Learning Method for Evolving Connectionist Systems, Journal of Advanced Computational Intelligence, 2 (6) 1998, 1-8 32. Kasabov, N., "Investigating the adaptation and forgetting in fuzzy neural networks by using the method of training and zeroing", Proceedings of the International Conference on Neural Networks ICNN'96, Plenary, Panel and Special Sessions volume, 118-123 (1996). 33. Kasabov, N., "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 34. Kasabov, N., “A framework for intelligent conscious machines utilising fuzzy neural networks and spatial temporal maps and a case study of multilingual speech recognition", in: Amari, S. and Kasabov, N. (eds) Brain-like computing and intelligent information systems, Springer Verlag, 106-126 (1998) 35. Kasabov, N., “ECOS: A framework for evolving connectionist systems and the ECO learning paradigm”, Proc. of ICONIP'98, Kitakyushu, Japan, Oct. 1998, IOS Press, 1222-1235 36. Kasabov, N., “Evolving Fuzzy Neural Networks - Algorithms, Applications and Biological Motivation”, in: in: Yamakawa and Matsumoto (eds), Methodologies for the Conception, design and Application of Soft Computing, World Scientific, 1998, 271-274 37. Kasabov, N., E. Postma, and J. Van den Herik, “AVIS: A Connectionist-based Framework for Integrated Audio and Visual Information Processing”, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998. 38. Kasabov, N., Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA, (1996). 39. Kasabov, N., J. S Kim, M. Watts, A. Gray, “FuNN/2- A Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition”, Information Sciences - Applications, 101(3-4): 155-175 (1997) 40. Kasabov, N., M. Watts, “Genetic algorithms for structural optimisation, dynamic adaptation and automated design of fuzzy neural networks”, in: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press, Houston (1997). 41. Kasabov, N., R. Kozma, R. Kilgour, M. Laws, J. Taylor, M. Watts, and A. Gray, “A Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks and Self Organising Maps”, in: Kasabov and Kozma (eds) Neuro-fuzzy techniques for intelligent information systems, Physica Verlag (Springer Verlag) 1999 42. Kasabov, N., Song, Q. “Dynamic, evolving fuzzy neural networks with ‘m-out-of-n’ activation nodes for on-line adaptive systems” , TR 99/04, Department of Information Science, University of Otago (1999) 43. Kasabov, N., Watts, M. Spatial-temporal evolving fuzzy neural networks STEFuNNs and applications for adaptive phoneme recognition, TR 99/03 Department of Information Science, University of Otago (1999) 44. Kasabov, N., Woodford, B. Rule Insertion and Rule Extraction from Evolving Fuzzy Neural Networks: Algorithms and Applications for Building Adaptive, Intelligent Expert Systems, in Proc. of Int. Conf. FUZZ-IEEE, Seoul, August 1999 (1999) 45. Kasabov,N. "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 46. Kawahara, S., Saito, T. “On a novel adaptive self-organising network”, Cellular Neural Networks and Their Applications, 41-46 (1996) 47. Kohonen, T., “The Self-Organizing Map”, Proceedings of the IEEE, vol.78, N-9, pp.1464-1497, (1990). 48. Kohonen, T., Self-Organizing Maps, second edition, Springer Verlag, 1997 49. Krogh, A., and J.A. Hertz, “A simple weight decay can improve generalisation”, Advances in Neural Information Processing Systems, 4 951-957, (1992). 50. Le Cun, Y., J.S. Denker and S.A. Solla, “Optimal Brain Damage”, in: Touretzky, D.S., ed., Advances in Neural Information Processing Systems, Morgan Kaufmann, 2, 598-605 (1990). 51. Lin, C.T. and C.S. G. Lee, Neuro Fuzzy Systems, Prentice Hall (1996). 52. Maeda, M., Miyajima, H. and Murashima, S., “A self organizing neural network with creatinga nd deleting methods, Nonlinear theory and its applications, 1, 397-400 (1996) 53. Mandziuk, J., Shastri, L. “Incremental class learning approach and its application to hand-written digit recognition, Proc. of the fifth int. conf. on neuro-information processing, Kitakyushu, Japan, Oct. 21-23, 1998 54. Massaro, D., and M.Cohen, "Integration of visual and auditory information in speech perception", Journal of Experimental Psychology: Human Perception and Performance, Vol 9, pp.753-771, (1983). 55. McClelland, J., B.L. McNaughton, and R.C. Reilly "Why there are Complementary Learning Systems in the Hippocampus and Neo-cortex: Insights from the Successes and Failures of Connectionist Models of Learning and Memory", CMU TR PDP.CNS.94.1, March, (1994). 56. Miller, D.J., Zurada and J.H. Lilly, "Pruning via Dynamic Adaptation of the Forgetting Rate in Structural Learning," Proc. IEEE ICNN'96, Vol.1, p.448 (1996). 57. Mitchell, M.T., "Machine Learning", MacGraw-Hill (1997) 58. Moody, J., Darken, C., Fast learning in networks of locally-tuned processing units, Neural Computation, 1, 281-294 (1989) 59. Mozer, M., and P. Smolensky, “A technique for trimming the fat from a network via relevance assessment”, in: D. Touretzky (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 598-605 (1989). 60. Murphy, P. and Aha, D. “UCI Repository of machine learning databases, Irvin, CA: University of California, Department of Information and Computer Science (1994), (http://www.ics.uci.edu/~mlearn/MLRepository.html) 61. Port, R., and T.van Gelder (eds) Mind as motion (Explorations in the Dynamics of Cognition) , The MIT Press, 1995 62. Quartz, S.R., and T.J. Sejnowski, “The neural basis of cognitive development: a constructivist manifesto”, Behavioral and Brain Science, to appear. 63. R. Jang, “ANFIS: adaptive network-based fuzzy inference system”, IEEE Trans. on Syst., Man, Cybernetics, 23(3), May-June, 665-685, (1993). 64. Reed, R., “Pruning algorithms - a survey”, IEEE Trans. Neural Networks, 4 (5) 740-747, (1993). 65. Robins, A. and Frean, M. “Local learning algorithms for sequential learning tasks in neural networks, Journal of Advanced Computational Intelligence, vol.2, 6 (1998) 66. Robins, A., “Consolidation in neural networks and the sleeping brain, Connection Science”, 8, 2, 259-275, (1996). 67. Rummery, G.A., and M. Niranjan, “On-line Q-learning using connectionist systems”, Cambridge University Engineering Department, CUED/F-INENG/TR 166 (1994) 68. S.R.H. Joseph, “Theories of adaptive neural growth”, PhD Thesis, University of Edinburgh, 1998 69. Saad, D. (ed) On-line learning in neural networks, Cambridge University Press, 1999 70. Sankar, A., and R.J. Mammone, “Growing and Pruning Neural Tree Networks”, IEEE Trans. Comput. 42(3) 291-299 (1993). 71. Schiffman, W., M. Joost, and R. Werner, “Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons” In: Albrecht, R.F., Reeves, Segalowitz, S.J. Language functions and brain organization, Academic Press, 1983 72. Segev, R. and E.Ben-Jacob, From neurons to brain: Adaptive self-wiring of neurons, TR /98 Faculty of Exact Sciences, Tel-Aviv University (1998) 73. Selverston, A. (ed) Model neural networks and behaviour, Plenum Press, 1985 74. Sinclair, S., and C. Watson, “The Development of the Otago Speech Database”, In Kasabov, N. and Coghill, G. (Eds.), Proceedings of ANNES ’95, Los Alamitos, CA, IEEE Computer Society Press (1995). 75. Towel, G., J. Shavlik, and M. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks", Proc. of the 8th National Conf. on Artificial Intelligence AAAI'90, Morgan Kaufmann, 861-866 (1990). 76. Van Ooyen, and J. Van Pelt, “Activity-dependent outgrowth of neurons and overshoot phenomena in developing neural networks”, Journal Theoretical Biology, 167, 27-43 (1994). 77. Waibel, A., M.Vo, P.Duchnovski, S.Manke, "Multimodal Interfaces", Artificial Intelligence Review, 1997. 78. Watts, M., and N. Kasabov, “Genetic algorithms for the design of fuzzy neural networks”, in Proc. of ICONIP'98, Kitakyushu, Oct. 1998. 79. Whitley, D., and C. Bogart, The evolution of connectivity: Pruning neural networks using genetic algorithms. Proc. Int. Joint Conf. Neural Networks, No. 1, 17-22. (1990). 80. Woldrige, M., and N. Jennings, “Intelligent agents: Theory and practice”, The Knowledge Engineering review (10) 1995. 81. Wong, R.O. “Use, disuse, and growth of the brain”, Proc. Nat. Acad. Sci. USA, 92 (6) 1797-99, (1995). 82. Yamakawa, T., H. Kusanagi, E. Uchino and T. Miki, "A new Effective Algorithm for Neo Fuzzy Neuron Model", in: Proceedings of Fifth IFSA World Congress, 1017-1020, (1993)

Evolving connectionist systems for on-line, knowledge-based learning: Principles and applications

The paper introduces evolving connectionist systems (ECOS) as an effective approach to building on-line, adaptive intelligent systems. ECOS evolve through incremental, hybrid (supervised/unsupervised), on-line learning. They can accommodate new input data, including new features, new classes, etc. through local element tuning. New connections and new neurons are created during the operation of the system. The ECOS framework is presented and illustrated on a particular type of evolving neural networks---evolving fuzzy neural networks (EFuNNs). EFuNNs can learn spatial-temporal sequences in an adaptive way, through one pass learning. Rules can be inserted and extracted at any time of the system operation. The characteristics of ECOS and EFuNNs are illustrated on several case studies that include: adaptive pattern classification; adaptive, phoneme-based spoken language recognition; adaptive dynamic time-series prediction; intelligent agents.Unpublished1. Albus, J.S., A new approach to manipulator control: The cerebellar model articulation controller (CMAC), Tarns. of the ASME: Journal of Dynamic Systems, Measurement, and Control, pp.220:227, Sept. (1975) 2. Amari, S. and Kasabov, N. eds, “Brain-like Computing and Intelligent Information Systems”, Springer Verlag,1998. 3. Amari, S., Mathematical foundations of neuro-computing, Proc. of IEEE, 78 (9), Sept. (1990) 4. Arbib, M. (ed) The Handbook of Brain Theory and Neural Networks,The MIT Press, 1995. 5. Bollacker, K., S.Lawrence and L.Giles, CiteSeer: An autonomous Web agent for automatic retrieval and identification of interesting publications, 2nd International ACM conference on autonomous agents, ACM Press, 1998, 116-123 6. Bottu and Vapnik, “Local learning computation”, Neural Computation, 4, 888-900 (1992) 7. Carpenter, G. and Grossberg S., Pattern recognition by self-organizing neural networks , The MIT Press, Cambridge, Massachusetts (1991) 8. Carpenter, G. and S. Grossberg, “ART3: Hierarchical search using chemical transmitters in self-organising pattern-recognition architectures”, Neural Networks, 3(2) 129-152(1990). 9. Carpenter, G. S. Grossberg, N. Markuzon, J.H. Reynolds, D.B. Rosen, “FuzzyARTMAP: A neural network architecture for incremental supervised learning of analog multi-dimensional maps,” IEEE Transactions of Neural Networks , vol.3, No.5, 698-713 (1991). 10. Cybenko, G., Approximation by super-positions of sigmoidal function, Mathematics of Control, Signals and Systems, 2, 303-314 (1989) 11. DeGaris, H., “Circuits of Production Rule - GenNets – The genetic programming of nervous systems”, in: Albrecht, R., Reeves, C. and Steele, N. (eds) Artificial Neural Networks and Genetic Algorithms, Springer Verlag (1993) 12. Duda and Hart, “Pattern classification and scene analysis”, New York: Willey (1973) 13. Edelman, G., Neuronal Darwinism: The theory of neuronal group selection, Basic Books (1992). 14. Elman, J., E.Bates, M.Johnson, A.Karmiloff-Smith, D.Parisi and K.Plunkett, Rethinking Innateness (A Connectionist Perspective of Development), The MIT Press, 1997 15. Fahlman, C., and C. Lebiere, "The Cascade-Correlation Learning Architecture", in: Turetzky, D (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 524-532 (1990). 16. Farmer, J.D., and Sidorowitch, Predicting chaotic time series, Physical Review Letters, 59, 845 (1987) 17. Freeman, J., D. Saad, “On-line learning in radial basis function networks”, Neural Computation vol. 9, No.7 (1997). 18. French, “Semi-destructive representations and catastrophic forgetting in connectionist networks, Connection Science, 1, 365-377 (1992) 19. Fritzke, B. “A growing neural gas network learns topologies”, Advances in Neural Information Processing Systems, vol.7 (1995). 20. Fukuda, T., Y. Komata, and T. Arakawa, "Recurrent Neural Networks with Self-Adaptive GAs for Biped Locomotion Robot", In: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press (1997) 21. Funuhashi, K., On the approximate realization of continuous mappings by neural networks, Neural Networks, 2, 183-192 (1989) 22. Gaussier, T., and S. Zrehen, “A topological neural map for on-line learning: Emergence of obstacle avoidance in a mobile robot”, In: From Animals to Animats No.3, 282-290, (1994). 23. Goldberg, D.E., Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley (1989) 24. Goodman, R., C.M. Higgins, J.W. Miller, P.Smyth, "Rule-based neural networks for classification and probability estimation", Neural Computation, 14, 781-804 (1992). 25. Hashiyama, T., T. Furuhashi, Y Uchikawa,. “A Decision Making Model Using a Fuzzy Neural Network”, in: Proceedings of the 2nd International Conference on Fuzzy Logic & Neural Networks, Iizuka, Japan, 1057-1060, (1992). 26. Hassibi and Stork, “Second order derivatives for network pruning: Optimal Brain Surgeon,” in: Advances in Neural Information Processing Systems, 4, 164-171, (1992). 27. Hech-Nielsen, R. “Counter-propagation networks”, IEEE First int. conference on neural networks, San Diego, vol.2, pp.19-31 (1987) 28. Heskes, T.M., B. Kappen, “On-line learning processes in artificial neural networks”, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233, (1993). 29. Ishikawa, M., "Structural Learning with Forgetting", Neural Networks 9, 501-521, (1996). 30. Kasabov, N. "Adaptable connectionist production systems”. Neurocomputing, 13 (2-4) 95-117, (1996). 31. Kasabov, N. The ECOS Framework and the ECO Learning Method for Evolving Connectionist Systems, Journal of Advanced Computational Intelligence, 2 (6) 1998, 1-8 32. Kasabov, N., "Investigating the adaptation and forgetting in fuzzy neural networks by using the method of training and zeroing", Proceedings of the International Conference on Neural Networks ICNN'96, Plenary, Panel and Special Sessions volume, 118-123 (1996). 33. Kasabov, N., "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 34. Kasabov, N., “A framework for intelligent conscious machines utilising fuzzy neural networks and spatial temporal maps and a case study of multilingual speech recognition", in: Amari, S. and Kasabov, N. (eds) Brain-like computing and intelligent information systems, Springer Verlag, 106-126 (1998) 35. Kasabov, N., “ECOS: A framework for evolving connectionist systems and the ECO learning paradigm”, Proc. of ICONIP'98, Kitakyushu, Japan, Oct. 1998, IOS Press, 1222-1235 36. Kasabov, N., “Evolving Fuzzy Neural Networks - Algorithms, Applications and Biological Motivation”, in: in: Yamakawa and Matsumoto (eds), Methodologies for the Conception, design and Application of Soft Computing, World Scientific, 1998, 271-274 37. Kasabov, N., E. Postma, and J. Van den Herik, “AVIS: A Connectionist-based Framework for Integrated Audio and Visual Information Processing”, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998. 38. Kasabov, N., Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA, (1996). 39. Kasabov, N., J. S Kim, M. Watts, A. Gray, “FuNN/2- A Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition”, Information Sciences - Applications, 101(3-4): 155-175 (1997) 40. Kasabov, N., M. Watts, “Genetic algorithms for structural optimisation, dynamic adaptation and automated design of fuzzy neural networks”, in: Proceedings of the International Conference on Neural Networks ICNN'97, IEEE Press, Houston (1997). 41. Kasabov, N., R. Kozma, R. Kilgour, M. Laws, J. Taylor, M. Watts, and A. Gray, “A Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks and Self Organising Maps”, in: Kasabov and Kozma (eds) Neuro-fuzzy techniques for intelligent information systems, Physica Verlag (Springer Verlag) 1999 42. Kasabov, N., Song, Q. “Dynamic, evolving fuzzy neural networks with ‘m-out-of-n’ activation nodes for on-line adaptive systems” , TR 99/04, Department of Information Science, University of Otago (1999) 43. Kasabov, N., Watts, M. Spatial-temporal evolving fuzzy neural networks STEFuNNs and applications for adaptive phoneme recognition, TR 99/03 Department of Information Science, University of Otago (1999) 44. Kasabov, N., Woodford, B. Rule Insertion and Rule Extraction from Evolving Fuzzy Neural Networks: Algorithms and Applications for Building Adaptive, Intelligent Expert Systems, in Proc. of Int. Conf. FUZZ-IEEE, Seoul, August 1999 (1999) 45. Kasabov,N. "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 46. Kawahara, S., Saito, T. “On a novel adaptive self-organising network”, Cellular Neural Networks and Their Applications, 41-46 (1996) 47. Kohonen, T., “The Self-Organizing Map”, Proceedings of the IEEE, vol.78, N-9, pp.1464-1497, (1990). 48. Kohonen, T., Self-Organizing Maps, second edition, Springer Verlag, 1997 49. Krogh, A., and J.A. Hertz, “A simple weight decay can improve generalisation”, Advances in Neural Information Processing Systems, 4 951-957, (1992). 50. Le Cun, Y., J.S. Denker and S.A. Solla, “Optimal Brain Damage”, in: Touretzky, D.S., ed., Advances in Neural Information Processing Systems, Morgan Kaufmann, 2, 598-605 (1990). 51. Lin, C.T. and C.S. G. Lee, Neuro Fuzzy Systems, Prentice Hall (1996). 52. Maeda, M., Miyajima, H. and Murashima, S., “A self organizing neural network with creatinga nd deleting methods, Nonlinear theory and its applications, 1, 397-400 (1996) 53. Mandziuk, J., Shastri, L. “Incremental class learning approach and its application to hand-written digit recognition, Proc. of the fifth int. conf. on neuro-information processing, Kitakyushu, Japan, Oct. 21-23, 1998 54. Massaro, D., and M.Cohen, "Integration of visual and auditory information in speech perception", Journal of Experimental Psychology: Human Perception and Performance, Vol 9, pp.753-771, (1983). 55. McClelland, J., B.L. McNaughton, and R.C. Reilly "Why there are Complementary Learning Systems in the Hippocampus and Neo-cortex: Insights from the Successes and Failures of Connectionist Models of Learning and Memory", CMU TR PDP.CNS.94.1, March, (1994). 56. Miller, D.J., Zurada and J.H. Lilly, "Pruning via Dynamic Adaptation of the Forgetting Rate in Structural Learning," Proc. IEEE ICNN'96, Vol.1, p.448 (1996). 57. Mitchell, M.T., "Machine Learning", MacGraw-Hill (1997) 58. Moody, J., Darken, C., Fast learning in networks of locally-tuned processing units, Neural Computation, 1, 281-294 (1989) 59. Mozer, M., and P. Smolensky, “A technique for trimming the fat from a network via relevance assessment”, in: D. Touretzky (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 598-605 (1989). 60. Murphy, P. and Aha, D. “UCI Repository of machine learning databases, Irvin, CA: University of California, Department of Information and Computer Science (1994), (http://www.ics.uci.edu/~mlearn/MLRepository.html) 61. Port, R., and T.van Gelder (eds) Mind as motion (Explorations in the Dynamics of Cognition) , The MIT Press, 1995 62. Quartz, S.R., and T.J. Sejnowski, “The neural basis of cognitive development: a constructivist manifesto”, Behavioral and Brain Science, to appear. 63. R. Jang, “ANFIS: adaptive network-based fuzzy inference system”, IEEE Trans. on Syst., Man, Cybernetics, 23(3), May-June, 665-685, (1993). 64. Reed, R., “Pruning algorithms - a survey”, IEEE Trans. Neural Networks, 4 (5) 740-747, (1993). 65. Robins, A. and Frean, M. “Local learning algorithms for sequential learning tasks in neural networks, Journal of Advanced Computational Intelligence, vol.2, 6 (1998) 66. Robins, A., “Consolidation in neural networks and the sleeping brain, Connection Science”, 8, 2, 259-275, (1996). 67. Rummery, G.A., and M. Niranjan, “On-line Q-learning using connectionist systems”, Cambridge University Engineering Department, CUED/F-INENG/TR 166 (1994) 68. S.R.H. Joseph, “Theories of adaptive neural growth”, PhD Thesis, University of Edinburgh, 1998 69. Saad, D. (ed) On-line learning in neural networks, Cambridge University Press, 1999 70. Sankar, A., and R.J. Mammone, “Growing and Pruning Neural Tree Networks”, IEEE Trans. Comput. 42(3) 291-299 (1993). 71. Schiffman, W., M. Joost, and R. Werner, “Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons” In: Albrecht, R.F., Reeves, Segalowitz, S.J. Language functions and brain organization, Academic Press, 1983 72. Segev, R. and E.Ben-Jacob, From neurons to brain: Adaptive self-wiring of neurons, TR /98 Faculty of Exact Sciences, Tel-Aviv University (1998) 73. Selverston, A. (ed) Model neural networks and behaviour, Plenum Press, 1985 74. Sinclair, S., and C. Watson, “The Development of the Otago Speech Database”, In Kasabov, N. and Coghill, G. (Eds.), Proceedings of ANNES ’95, Los Alamitos, CA, IEEE Computer Society Press (1995). 75. Towel, G., J. Shavlik, and M. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks", Proc. of the 8th National Conf. on Artificial Intelligence AAAI'90, Morgan Kaufmann, 861-866 (1990). 76. Van Ooyen, and J. Van Pelt, “Activity-dependent outgrowth of neurons and overshoot phenomena in developing neural networks”, Journal Theoretical Biology, 167, 27-43 (1994). 77. Waibel, A., M.Vo, P.Duchnovski, S.Manke, "Multimodal Interfaces", Artificial Intelligence Review, 1997. 78. Watts, M., and N. Kasabov, “Genetic algorithms for the design of fuzzy neural networks”, in Proc. of ICONIP'98, Kitakyushu, Oct. 1998. 79. Whitley, D., and C. Bogart, The evolution of connectivity: Pruning neural networks using genetic algorithms. Proc. Int. Joint Conf. Neural Networks, No. 1, 17-22. (1990). 80. Woldrige, M., and N. Jennings, “Intelligent agents: Theory and practice”, The Knowledge Engineering review (10) 1995. 81. Wong, R.O. “Use, disuse, and growth of the brain”, Proc. Nat. Acad. Sci. USA, 92 (6) 1797-99, (1995). 82. Yamakawa, T., H. Kusanagi, E. Uchino and T. Miki, "A new Effective Algorithm for Neo Fuzzy Neuron Model", in: Proceedings of Fifth IFSA World Congress, 1017-1020, (1993)

Looking for a new AI paradigm: Evolving connectionist and fuzzy connectionist systems—Theory and applications for adaptive, on-line intelligent systems

Please note that this is a searchable PDF derived via optical character recognition (OCR) from the original source document. As the OCR process is never 100% perfect, there may be some discrepancies between the document image and the underlying text.The paper introduces one paradigm of neuro-fuzzy techniques and an approach to building on-line, adaptive intelligent systems. This approach is called evolving connectionist systems (ECOS). ECOS evolve through incremental, on-line learning, both supervised and unsupervised. They can accommodate new input data, including new features, new classes, etc. The ECOS framework is presented and illustrated on a particular type of evolving neural networks—evolving fuzzy neural networks. ECOS are three to six orders of magnitude faster than the multilayer perceptrons, or the fuzzy neural networks, trained with the backpropagation algorithm, or with a genetic programming technique. ECOS belong to the new generation of adaptive intelligent systems. This is illustrated on several real world problems for adaptive, on-line classification, prediction, decision making and control: phoneme-based speech recognition; moving person identification; wastewater flow time-series prediction and control; intelligent agents; financial time series prediction and control. The principles of recurrent ECOS and reinforcement learning are outlined.Unpublished1. Almeida,L., T. Langlois, J. Amaral, J. On-line Step Size Adaptation, Technical Report, INESC RT07/97, 1997 2. Altman, G., Cognitive Models of Speech Processing, MIT Press, 1990 3. Amari, S. and Kasabov, N. eds (1997) Brain-like computing and intelligent information systems, Springer Verlag 4. Andrews, R., J. Diederich, A.B.Tickle, "A Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks", Knowledge-Based Systems, 8, 373-389 (1995). 5. Arbib, M. (ed) (1995) The Handbook of Brain Theory and Neural Networks. The MIT Press 6. Baestalus, Dik-Emma, van den Bergh, W.M., Wood, D. Neural network solutions for trading financial market, Pitman Publications, 1994 7. Beltraffi, A., Margarita, S., Terna, P. Neural networks for economics and financial modelling, Int. Thomson Computer Press, 1996 8. Carpenter, G. and Grossberg, S., Pattern recognition by self-organizing neural networks, The MIT Press, Cambridge, Massachusetts (1991) 9. Carpenter, G.A. and Grossberg, S., ART3: Hierarchical search using chemical transmitters in self-organising pattern-recognition architectures, Neural Networks, 3(2) (1990) 129-152. 10. Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, IH., Rosen, D.B., FuzzyARTMAP: A neural network architecture for incremental supervised learning of analog multi-dimensional maps, IEEE Transactions of Neural Networks, vol.3, No.5 (1991), 698-713 11. Chauvin, L, A backpropagation algorithm with optimal use of hidden units, Advances in Neuro Information Processing Syste, 1 (1989) 519-526. 12. Cole, R., et al. The Challenge of Spoken Language Systems: Research Directions for the Nineties, IEEE Transactions on Speech and Audio Processing, vol.3, No.1, 1-21, 1995 13. DeBoeck, L. Trading on the edge. Kluwer Academics, 1994 14. DeGaris, H. Circuits of Production Rule - GenNets - The genetic programming of nervous systems, in: Albrecht, R., Reeves, C. and N. Steele (eds) Artifical Neural Networks and Genetic Algorithms, Springer Verlag (1993) 15. Edelman, G., Neuronal Darwinism: The theory of neuronal group selection, Basic Books (1992) 16. Fahlman, C .,and C. Lebiere, "The Cascade- Correlation Architecture", in: Turetzky, D (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 524-532 (1990). 17. Freeman, I.A.S., Saad, D., On-line learning in radial basis function networks, Neural Computation vol. 9, No.7 (1997) 18. Fritzke, B., A growing neural gas network learns topologies, Advances in Neural Information ProcessingSystems, vol.7 (1995) 19. Fukuda, T., Komata, Y., and Arakawa, T. "Recurrent Neural Networks with Self-Adaptive GAs for Biped Locomotion Robot", In: Proceedings of the International Conference on Neural Networks [CNN ’97, IEEE Press (1997) 20. Gaussier, P. and Zrehen, S., A topological neural map for on-line learning: Emergence of obstacle avoidance in a mobile robot, In: From Animals to Animats No.3, (1994) 282--290 21. Goldberg, D.E., Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley (1989) 22. Goodman, R.M., C.M. Higgins, I.W. Miller, P.Smyth, "Rule-based neural networks for classification and probability estimation", Neural Computation, 14, 781-804 (1992) 23. Gray, M.S., J.R.Movellan, and T.J.Sejnowski, Dynammic features for visual speech reading: A systematic comparison. In M.C. Mozer, M.I. Jordan, and T. Petsche (Eds.), Advances in Neural Inform. Proc. Systems, Vol.9, pp.751- 757. Morgan-Kaufmann: San Fransisco, CA, 1997. 24. Hashiyama,T., Furuhashi, T., Uchikawa, Y. (1992) A Decision Making Model Using a Fuzzy Neural Network, in: Proceedings of the 2nd International Conference on Fuzzy Logic & Neural Networks, Iizuka, Japan, 1057-1060. 25. Hassibi and Stork, Second order derivatives for network pruning: Optimal Brain Surgeon, in: Advances in Neural Information Processing Systems, 4, (1992) 164-171 26. Heskes, T.M., Kappen, B. (1993) On-line learning processes in artificial neural networks, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233 27. Ishikawa, M. (1996) "Structural Learning with Forgetting", Neural Networks 9, 501-521. 28. Jang, R. (1993) ANFIS: adaptive network-based fuzzy inference system, IEEE Trans. on Syst.,Man, Cybernetics, 23(3), May-June 1993, 665-685 29. Joseph, S.R.H. Theories of adaptive neural growth, PhD Thesis, University of Edinburgh, 1998 30. Kasabov, N. A framework for intelligent conscious machines utilising fuzzy neural networks and spatial temporal maps and a case study of multilingual speech recognition", in: Amari, S. and Kasabov, N. (eds) Brain-like computing and intelligent information systems, Springer, 106-126 (1997) 31. Kasabov, N. and Fedrizzi, M. (1999) Fuzzy Neural Networks and Evolving Connectionist Systems for Intelligent Decision Making, Proc. of IFSA'99, Taiwan, 1999, submitted. 32. Kasabov, N. and Kozma, R. Multi-scale analysis of time series based on neuro-fuzzy-chaos methodology applied to financial data. In: Refenes, A., Burges, A. and Moody, B. eds. Computational Finance 1997, Kluwer Academic, 1998, accepted 33. Kasabov, N. ECOS: A framework for evolving connectionist systems and the eco learning paradigm, Proc. of ICONIP'98, Kitakyushu, Oct. 1998 34. Kasabov, N. Evolving conneetionist and fuzzy connectionist system for on-line decision making and control, Proc. of the 3rd On-line WWW World Congress on Soft Computing in Engineering Design, June 1998, Springer Verlag, to appear. 35. Kasabov, N. Evolving connectionist and fuzzy connectionist systems. IEEE Transactions on Man, Machine and Cybernetics, submitted 36. Kasabov, N. Evolving Fuzzy Neural Networks - Algorithms, Applications and Biological Motivation, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998 37. Kasabov, N.(1996) Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA. 38. Kasabov, N., "Adaptable conneetionist production systems". Neurocomputing, 13 (2-4) 95-117 (1996). 39. Kasabov, N., "Investigating the adaptation and forgetting in fuzzy neural networks by using the method of training and zeroing", Proceedings of the International Conference on Neural Networks ICNN'96, Panel and Plenary, Special Sessions volume, 118-123 (1996). 40. Kasabov, N., "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 41. Kasabov, N., Kim J S, Watts, M., Gray, A (1997) FuNN/2- A Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition, Information Sciences - Applications, 101(3-4): 155-175 (1997) 42. Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M. and Gray, A. "A Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks and Self Organising Maps, in the same volume. 43. Kasabov, N., Postma, E., and Van den Herik, J AVIS: A Connectionist-based Framework tor Integrated Audio and Visual Information Processing, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998 44. Kasabov, N., Watts, M. Genetic algorithms for structural optimisation, dynamic adaptation and automated design of fuzzy neural networks. in: Proceedings of the International Conference on Neural Networks ICNN’97, IEEE Press, Houston (1997) 45. Kater, S.B., Mattson, N.P., Cohan, C. and Connor, J. Calcium regulation of the neuronal cone growth, Trends in Neuroscience, 11 (1988) 315-321. 46. Kohonen, T. (1990) The Self-Organizing Map. Proceedings of the IEEE, vol.78, N-9, pp.146-4-1497. 47. Kohonen, T.,. Self-Organizing Maps, second edition, Springer Verlag, 1997 48. Kozma, R. and Kasabov, N Generic neuro-fuzzy-chaos methodologies and techniques for intelligent time-series analysis. In: Soft Computing in Financial Engineering. R. Ribeiro, R.Yager, H. J. Zimmermann and J. Kacprzyk eds. Heidelberg, Physica-Verlag (1998) 49. Kozma, R. and N.Kasabov, Rules of chaotic behaviour extracted from the fuzzy neural network FUNN, Proc. of the WCCI'98 FUZZ-IEEE International Conference on Fuzzy Systems, Anchorage, May (1998). 50. Kozma, R., M. Sakuma, Y. Yokoyama, M. Kitamura, On the Accuracy of Mapping by Neural Networks T rained by Backpropagation with Forgetting, Neurocomputing, 13 (2-4) (1996). 51. Krogh, A. and Hertz, J.A., A simple weight decay can improve generalisation. Advances in Neural Information Processing Systems, 4 (1992) 951-957 52. Lawrence, S., Fong, S., Giles, L. natural language grammatical inference: A comparison of recurrent neural networks and machine learning methods, in: S.Wermtner, E.Riloff and G.Scheler (eds) Symbolic, Connectionist and Statistical: Approaches to Learning for Natural language processing, Lecture Notes in AI, (1996) 33--47 53. Le Cun, Y., J.S. Denker and S.A. Solla, Optimal Brain Damage, in: D.S. Touretzky, ed., Advances in Neural Information Processing Systems, Morgan Kaufmann, 2, 598-605 (1990). 54. Lin, C.T. and C.S. G. Lee, Neuro Fuzzy Systems, Prentice Hall (1996). 55. Massaro, D., and M.Cohen, "Integration of visual and auditory information in speech perception", Journal of Experimental Psychology: Human Perception and Performance, Vol 9, pp.753-771, 1983. 56. McClelland, J., B.L. McNaughton, and R.C. Reilly (1994) "Why there are Complementary Learning Systems in the Hippocampus and Neocortx: Insights from the Successes and Failures of Connectionist Models of Learning and Memeory", CMU Technical Report PDP_CNS.94.1, March 57. Miller, D.,J.Zurada and JH. Lilly, "Pruning via Dynamic Adaptation of the Forgetting Rate in Structural Learning," Proc. IEEE ICNN'96, Vol.1, p.448 (1996). 58. Mitchell, M.T. "Machine Learning", MacGraw-Hill (1997) 59. Mitchell, Melanie, An Introduction to Genetic Algorithms, MIT Press, Cambridge, Massachusetts (1996). 60. Mozer. M, and P.Smolensky, A technique for trimming the fat from a network via relevance assessment, in: D.Touretzky (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 598-605 (1989). 61. Quartz, S.R., and Sejnowski, T.J. The neural basis of cognitive development: a constructivist manifesto, Behavioral and Brain Science, to appear 62. Reed, R. (1993) Pruning algorithms - a survey, IEEE Trans. Neural Networks, 4 (5) 740-747. 63. Robins, A. Consolidation in neural networks and the sleeping brain, Connection Science, 8, 2 (1996) 259-275 64. Rummery, G.A. and Niranjan, M., On-line Q-learning using connectionist systems, Cambridge University Engineering Department, CUED/F-INENG/TR 166 (1994) 65. Sanchez, E. , DNA Biosoft Computing, in: Methodology for the Conception, design, and Application of Intelligent Systems, Proc. Iizuka'96, 30-37 66. Sankar, A. and R.J. Mammone, Growing and Pruning Neural Tree Networks, IEEE Trans. Comput. 42(3) 291-299 (1993). 67. Schiffman, W., Joost, M. and Werner. R., Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons. In: Albrecht, R.F., Reeves, C. R., Steele, N. C. (Eds.), Artificial Neural Nets and Genetic Algorithms, Spring-Verlag Wien, New York (1993) 68. Segev, R. and E.Ben-Jacob, from neurons to brain: Adaptive self-wiring of neurons, TR, Faculty of Exact Sciences, Tel-Aviv University (1998) 69. Sinclair, S., and Watson, C., The Development of the Otago Speech Database. In Kasabov, N. and Coghill, G. (Eds.), Proceedings of ANNES ’95, Los Alamitos, CA, IEEE Computer Society Press (1995). 70. Towel, G., J. Shavlik, J. and M. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks", Proc. of the 8th National Conf. on Artificial Intelligence AAAI’9O, Morgan Kaufmann, 861-866 (1990). 71. Van Ooyen, A. Activity-dependent neural network development, Network: Computation in Neural Systems, 5 (1994) 401-423. Van Ooyen, A. and Van Pell, I., Activity-dependent outgrowth of neurons and overshoot in developing neural networks, Journal Theoretical Biology, 167 (1994) 27-43 72. von Helmholtz, H. , Handbuch der Physiologishe Optic, Hamburg and Leipzig: Voss, 1866, 1896 73. Waibel, A., M.Vo, P.Duchnovski, S.Manke, "Multimodal Interfaces", Artificial Intelligence Review, 1997 74. Watts, M., and Kasabov, N. Genetic algorithms tor the design of fuzzy neural networks, in Proc. of ICONIP’98, Kitakyushu, Oct. 1998 75. Whitley, D. and Bogart, C., The evolution of connectivity: Pruning neural networks using genetic algorithms. Proc. Int. Joint Conf. Neural Networks, No.1 (1990) 17-22. 76. Winson, J. The meaning of dreams, Scientific American, November (1990) 42-48 77. Woldrige, M. and Jennings, N. Intelligent agents: Theory and practice, The Knowledge Engineering review (10) 1995 78. Wong, R.O.L. Use, disuse, and growth of the brain, Proc. Nat. Acad. Sci. USA, 92 (6) (1995) 1797-99. 79. Yamakawa, T., H. Kusanagi,E. Uchino and T.Miki, (1993) "A new Effective Algorithm for Neo Fuzzy Neuron Model", in: Proceedings of Fifth IFSA World Congress, 1017-1020 80. Zadeh, L. 1965. Fuzzy Sets, Information, and Control, vol.8, 338-353

Looking for a new AI paradigm: Evolving connectionist and fuzzy connectionist systems—Theory and applications for adaptive, on-line intelligent systems

Please note that this is a searchable PDF derived via optical character recognition (OCR) from the original source document. As the OCR process is never 100% perfect, there may be some discrepancies between the document image and the underlying text.The paper introduces one paradigm of neuro-fuzzy techniques and an approach to building on-line, adaptive intelligent systems. This approach is called evolving connectionist systems (ECOS). ECOS evolve through incremental, on-line learning, both supervised and unsupervised. They can accommodate new input data, including new features, new classes, etc. The ECOS framework is presented and illustrated on a particular type of evolving neural networks—evolving fuzzy neural networks. ECOS are three to six orders of magnitude faster than the multilayer perceptrons, or the fuzzy neural networks, trained with the backpropagation algorithm, or with a genetic programming technique. ECOS belong to the new generation of adaptive intelligent systems. This is illustrated on several real world problems for adaptive, on-line classification, prediction, decision making and control: phoneme-based speech recognition; moving person identification; wastewater flow time-series prediction and control; intelligent agents; financial time series prediction and control. The principles of recurrent ECOS and reinforcement learning are outlined.Unpublished1. Almeida,L., T. Langlois, J. Amaral, J. On-line Step Size Adaptation, Technical Report, INESC RT07/97, 1997 2. Altman, G., Cognitive Models of Speech Processing, MIT Press, 1990 3. Amari, S. and Kasabov, N. eds (1997) Brain-like computing and intelligent information systems, Springer Verlag 4. Andrews, R., J. Diederich, A.B.Tickle, "A Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks", Knowledge-Based Systems, 8, 373-389 (1995). 5. Arbib, M. (ed) (1995) The Handbook of Brain Theory and Neural Networks. The MIT Press 6. Baestalus, Dik-Emma, van den Bergh, W.M., Wood, D. Neural network solutions for trading financial market, Pitman Publications, 1994 7. Beltraffi, A., Margarita, S., Terna, P. Neural networks for economics and financial modelling, Int. Thomson Computer Press, 1996 8. Carpenter, G. and Grossberg, S., Pattern recognition by self-organizing neural networks, The MIT Press, Cambridge, Massachusetts (1991) 9. Carpenter, G.A. and Grossberg, S., ART3: Hierarchical search using chemical transmitters in self-organising pattern-recognition architectures, Neural Networks, 3(2) (1990) 129-152. 10. Carpenter, G.A., Grossberg, S., Markuzon, N., Reynolds, IH., Rosen, D.B., FuzzyARTMAP: A neural network architecture for incremental supervised learning of analog multi-dimensional maps, IEEE Transactions of Neural Networks, vol.3, No.5 (1991), 698-713 11. Chauvin, L, A backpropagation algorithm with optimal use of hidden units, Advances in Neuro Information Processing Syste, 1 (1989) 519-526. 12. Cole, R., et al. The Challenge of Spoken Language Systems: Research Directions for the Nineties, IEEE Transactions on Speech and Audio Processing, vol.3, No.1, 1-21, 1995 13. DeBoeck, L. Trading on the edge. Kluwer Academics, 1994 14. DeGaris, H. Circuits of Production Rule - GenNets - The genetic programming of nervous systems, in: Albrecht, R., Reeves, C. and N. Steele (eds) Artifical Neural Networks and Genetic Algorithms, Springer Verlag (1993) 15. Edelman, G., Neuronal Darwinism: The theory of neuronal group selection, Basic Books (1992) 16. Fahlman, C .,and C. Lebiere, "The Cascade- Correlation Architecture", in: Turetzky, D (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 524-532 (1990). 17. Freeman, I.A.S., Saad, D., On-line learning in radial basis function networks, Neural Computation vol. 9, No.7 (1997) 18. Fritzke, B., A growing neural gas network learns topologies, Advances in Neural Information ProcessingSystems, vol.7 (1995) 19. Fukuda, T., Komata, Y., and Arakawa, T. "Recurrent Neural Networks with Self-Adaptive GAs for Biped Locomotion Robot", In: Proceedings of the International Conference on Neural Networks [CNN ’97, IEEE Press (1997) 20. Gaussier, P. and Zrehen, S., A topological neural map for on-line learning: Emergence of obstacle avoidance in a mobile robot, In: From Animals to Animats No.3, (1994) 282--290 21. Goldberg, D.E., Genetic Algorithms in Search, Optimisation and Machine Learning, Addison-Wesley (1989) 22. Goodman, R.M., C.M. Higgins, I.W. Miller, P.Smyth, "Rule-based neural networks for classification and probability estimation", Neural Computation, 14, 781-804 (1992) 23. Gray, M.S., J.R.Movellan, and T.J.Sejnowski, Dynammic features for visual speech reading: A systematic comparison. In M.C. Mozer, M.I. Jordan, and T. Petsche (Eds.), Advances in Neural Inform. Proc. Systems, Vol.9, pp.751- 757. Morgan-Kaufmann: San Fransisco, CA, 1997. 24. Hashiyama,T., Furuhashi, T., Uchikawa, Y. (1992) A Decision Making Model Using a Fuzzy Neural Network, in: Proceedings of the 2nd International Conference on Fuzzy Logic & Neural Networks, Iizuka, Japan, 1057-1060. 25. Hassibi and Stork, Second order derivatives for network pruning: Optimal Brain Surgeon, in: Advances in Neural Information Processing Systems, 4, (1992) 164-171 26. Heskes, T.M., Kappen, B. (1993) On-line learning processes in artificial neural networks, in: Math. foundations of neural networks, Elsevier, Amsterdam, 199-233 27. Ishikawa, M. (1996) "Structural Learning with Forgetting", Neural Networks 9, 501-521. 28. Jang, R. (1993) ANFIS: adaptive network-based fuzzy inference system, IEEE Trans. on Syst.,Man, Cybernetics, 23(3), May-June 1993, 665-685 29. Joseph, S.R.H. Theories of adaptive neural growth, PhD Thesis, University of Edinburgh, 1998 30. Kasabov, N. A framework for intelligent conscious machines utilising fuzzy neural networks and spatial temporal maps and a case study of multilingual speech recognition", in: Amari, S. and Kasabov, N. (eds) Brain-like computing and intelligent information systems, Springer, 106-126 (1997) 31. Kasabov, N. and Fedrizzi, M. (1999) Fuzzy Neural Networks and Evolving Connectionist Systems for Intelligent Decision Making, Proc. of IFSA'99, Taiwan, 1999, submitted. 32. Kasabov, N. and Kozma, R. Multi-scale analysis of time series based on neuro-fuzzy-chaos methodology applied to financial data. In: Refenes, A., Burges, A. and Moody, B. eds. Computational Finance 1997, Kluwer Academic, 1998, accepted 33. Kasabov, N. ECOS: A framework for evolving connectionist systems and the eco learning paradigm, Proc. of ICONIP'98, Kitakyushu, Oct. 1998 34. Kasabov, N. Evolving conneetionist and fuzzy connectionist system for on-line decision making and control, Proc. of the 3rd On-line WWW World Congress on Soft Computing in Engineering Design, June 1998, Springer Verlag, to appear. 35. Kasabov, N. Evolving connectionist and fuzzy connectionist systems. IEEE Transactions on Man, Machine and Cybernetics, submitted 36. Kasabov, N. Evolving Fuzzy Neural Networks - Algorithms, Applications and Biological Motivation, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998 37. Kasabov, N.(1996) Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA. 38. Kasabov, N., "Adaptable conneetionist production systems". Neurocomputing, 13 (2-4) 95-117 (1996). 39. Kasabov, N., "Investigating the adaptation and forgetting in fuzzy neural networks by using the method of training and zeroing", Proceedings of the International Conference on Neural Networks ICNN'96, Panel and Plenary, Special Sessions volume, 118-123 (1996). 40. Kasabov, N., "Learning fuzzy rules and approximate reasoning in fuzzy neural networks and hybrid systems", Fuzzy Sets and Systems 82 (2) 2-20 (1996). 41. Kasabov, N., Kim J S, Watts, M., Gray, A (1997) FuNN/2- A Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition, Information Sciences - Applications, 101(3-4): 155-175 (1997) 42. Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M. and Gray, A. "A Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks and Self Organising Maps, in the same volume. 43. Kasabov, N., Postma, E., and Van den Herik, J AVIS: A Connectionist-based Framework tor Integrated Audio and Visual Information Processing, in Proc. of Iizuka'98, Iizuka, Japan, Oct.1998 44. Kasabov, N., Watts, M. Genetic algorithms for structural optimisation, dynamic adaptation and automated design of fuzzy neural networks. in: Proceedings of the International Conference on Neural Networks ICNN’97, IEEE Press, Houston (1997) 45. Kater, S.B., Mattson, N.P., Cohan, C. and Connor, J. Calcium regulation of the neuronal cone growth, Trends in Neuroscience, 11 (1988) 315-321. 46. Kohonen, T. (1990) The Self-Organizing Map. Proceedings of the IEEE, vol.78, N-9, pp.146-4-1497. 47. Kohonen, T.,. Self-Organizing Maps, second edition, Springer Verlag, 1997 48. Kozma, R. and Kasabov, N Generic neuro-fuzzy-chaos methodologies and techniques for intelligent time-series analysis. In: Soft Computing in Financial Engineering. R. Ribeiro, R.Yager, H. J. Zimmermann and J. Kacprzyk eds. Heidelberg, Physica-Verlag (1998) 49. Kozma, R. and N.Kasabov, Rules of chaotic behaviour extracted from the fuzzy neural network FUNN, Proc. of the WCCI'98 FUZZ-IEEE International Conference on Fuzzy Systems, Anchorage, May (1998). 50. Kozma, R., M. Sakuma, Y. Yokoyama, M. Kitamura, On the Accuracy of Mapping by Neural Networks T rained by Backpropagation with Forgetting, Neurocomputing, 13 (2-4) (1996). 51. Krogh, A. and Hertz, J.A., A simple weight decay can improve generalisation. Advances in Neural Information Processing Systems, 4 (1992) 951-957 52. Lawrence, S., Fong, S., Giles, L. natural language grammatical inference: A comparison of recurrent neural networks and machine learning methods, in: S.Wermtner, E.Riloff and G.Scheler (eds) Symbolic, Connectionist and Statistical: Approaches to Learning for Natural language processing, Lecture Notes in AI, (1996) 33--47 53. Le Cun, Y., J.S. Denker and S.A. Solla, Optimal Brain Damage, in: D.S. Touretzky, ed., Advances in Neural Information Processing Systems, Morgan Kaufmann, 2, 598-605 (1990). 54. Lin, C.T. and C.S. G. Lee, Neuro Fuzzy Systems, Prentice Hall (1996). 55. Massaro, D., and M.Cohen, "Integration of visual and auditory information in speech perception", Journal of Experimental Psychology: Human Perception and Performance, Vol 9, pp.753-771, 1983. 56. McClelland, J., B.L. McNaughton, and R.C. Reilly (1994) "Why there are Complementary Learning Systems in the Hippocampus and Neocortx: Insights from the Successes and Failures of Connectionist Models of Learning and Memeory", CMU Technical Report PDP_CNS.94.1, March 57. Miller, D.,J.Zurada and JH. Lilly, "Pruning via Dynamic Adaptation of the Forgetting Rate in Structural Learning," Proc. IEEE ICNN'96, Vol.1, p.448 (1996). 58. Mitchell, M.T. "Machine Learning", MacGraw-Hill (1997) 59. Mitchell, Melanie, An Introduction to Genetic Algorithms, MIT Press, Cambridge, Massachusetts (1996). 60. Mozer. M, and P.Smolensky, A technique for trimming the fat from a network via relevance assessment, in: D.Touretzky (ed) Advances in Neural Information Processing Systems, vol.2, Morgan Kaufmann, 598-605 (1989). 61. Quartz, S.R., and Sejnowski, T.J. The neural basis of cognitive development: a constructivist manifesto, Behavioral and Brain Science, to appear 62. Reed, R. (1993) Pruning algorithms - a survey, IEEE Trans. Neural Networks, 4 (5) 740-747. 63. Robins, A. Consolidation in neural networks and the sleeping brain, Connection Science, 8, 2 (1996) 259-275 64. Rummery, G.A. and Niranjan, M., On-line Q-learning using connectionist systems, Cambridge University Engineering Department, CUED/F-INENG/TR 166 (1994) 65. Sanchez, E. , DNA Biosoft Computing, in: Methodology for the Conception, design, and Application of Intelligent Systems, Proc. Iizuka'96, 30-37 66. Sankar, A. and R.J. Mammone, Growing and Pruning Neural Tree Networks, IEEE Trans. Comput. 42(3) 291-299 (1993). 67. Schiffman, W., Joost, M. and Werner. R., Application of Genetic Algorithms to the Construction of Topologies for Multilayer Perceptrons. In: Albrecht, R.F., Reeves, C. R., Steele, N. C. (Eds.), Artificial Neural Nets and Genetic Algorithms, Spring-Verlag Wien, New York (1993) 68. Segev, R. and E.Ben-Jacob, from neurons to brain: Adaptive self-wiring of neurons, TR, Faculty of Exact Sciences, Tel-Aviv University (1998) 69. Sinclair, S., and Watson, C., The Development of the Otago Speech Database. In Kasabov, N. and Coghill, G. (Eds.), Proceedings of ANNES ’95, Los Alamitos, CA, IEEE Computer Society Press (1995). 70. Towel, G., J. Shavlik, J. and M. Noordewier, "Refinement of approximate domain theories by knowledge-based neural networks", Proc. of the 8th National Conf. on Artificial Intelligence AAAI’9O, Morgan Kaufmann, 861-866 (1990). 71. Van Ooyen, A. Activity-dependent neural network development, Network: Computation in Neural Systems, 5 (1994) 401-423. Van Ooyen, A. and Van Pell, I., Activity-dependent outgrowth of neurons and overshoot in developing neural networks, Journal Theoretical Biology, 167 (1994) 27-43 72. von Helmholtz, H. , Handbuch der Physiologishe Optic, Hamburg and Leipzig: Voss, 1866, 1896 73. Waibel, A., M.Vo, P.Duchnovski, S.Manke, "Multimodal Interfaces", Artificial Intelligence Review, 1997 74. Watts, M., and Kasabov, N. Genetic algorithms tor the design of fuzzy neural networks, in Proc. of ICONIP’98, Kitakyushu, Oct. 1998 75. Whitley, D. and Bogart, C., The evolution of connectivity: Pruning neural networks using genetic algorithms. Proc. Int. Joint Conf. Neural Networks, No.1 (1990) 17-22. 76. Winson, J. The meaning of dreams, Scientific American, November (1990) 42-48 77. Woldrige, M. and Jennings, N. Intelligent agents: Theory and practice, The Knowledge Engineering review (10) 1995 78. Wong, R.O.L. Use, disuse, and growth of the brain, Proc. Nat. Acad. Sci. USA, 92 (6) (1995) 1797-99. 79. Yamakawa, T., H. Kusanagi,E. Uchino and T.Miki, (1993) "A new Effective Algorithm for Neo Fuzzy Neuron Model", in: Proceedings of Fifth IFSA World Congress, 1017-1020 80. Zadeh, L. 1965. Fuzzy Sets, Information, and Control, vol.8, 338-353

A bilingual speech interface for New Zealand English to Māori

The 'Bilingual Speech Interface for New Zealand English to Māori ' is part of the 'Intelligent Human Computer Interfaces' project under 'Objective 3' of the 'Connectionist-Based Information Systems' programme (FoRST UOO606) Department of Information Science, University of Otago. The project experiments with artificial intelligent knowledge-based engineering methodologies and techniques for designing tools that utilise a hybrid system approach that is both adaptable and flexible to different speakers and languages-namely New Zealand English and Māori. Artificial neural networks, fuzzy rule-based inferencing techniques, genetic algorithms and multimedia based applications form the multiple paradigm approaches for solving real applied problems for speech generation. All are fundamental in the developmental direction of an intelligent human computer interface. Multimedia interfaces to databases use conventional software engineering techniques for the management, access and retrieval of information, therefore, implementing an interface application to access a speech and language database means that a change in computer interaction from manual to automatic control is facilitated between the user and the database. The 'Hybrid Neuro-Fuzzy Speech Recognition System' called HySpeech is based on isolated word recognition for New Zealand English that utilises an English-Māori lexical database to provide an automatic language translator or 'Talking Dictionary'. The current development with version two of HySpeech incorporates the advanced 'Fuzzy Neural Network' models with new 'Learning with Forgetting' algorithms for better speaker adaptive capabilities, an aggregation function provides cleaner phoneme compression and 'Self-Organising Maps' for phoneme and word lists-all to facilitate new language modelling techniques. The language information is housed in a separate database management system designed for HySpeech, it contains; the speakers' characteristics, English and Māori words, phonetic transcriptions and pronunciations, segmental labels, phoneme activation and co-ordinate vectors, and a growing set of digitised Māori speech examples. The interface component of HySpeech thus comprises and text examples from the aforementioned database, interactive dialogues, and a graphical user interface generated environment. Also, experiments in artificial language generation for speech synthesis provide the system with some knowledge and information about the languages. The system could therefore be capable of recreating a form of machine generated speech. Many approaches are available to facilitate synthesised speech, so by utilising the present methodologies and techniques, there is a future solution that can run parallel to the existing direction of HySpeech, for a complete bilingual speech interface. Given that there is presently no operational system that can synthesise New Zealand English and Māori speech, this thesis will also provide some basis to facilitate the further research and development in that area.ndrewes, M. and Tamblyn, R. (1994) On Call™ Innovative Language Learning Systems, Language Learning Centre, University of Otago, Dunedin. Apple Computer Inc. (1991) Inside Macintosh, Volume VI. Addison Wesley. Apple Computer Inc. (1996) Inside Macintosh, On-Line Version. [http://speech.apple.com/dev/insidemac/] Apple Computer Inc. (1997) PlainTalk: An Apple Speech Technology White Paper. [http://speech.apple.com/] Bäckström, C. and Sandewall, E. (1994) Current Trends in AI Planning. EWSP’93 – 2nd European Workshop on Planning, IOS Press, Amsterdam. Barlow, C. (1990a) An Alternative Format for Maori Archival Materials, Archifacts, April 1990, pp. 22-28. Barlow, C. (1990b) Me Ako Taatou i Te Reo Maaori , By Bruce Biggs, Uniprint, Orakei. Barlow, C. (1994) Tikanga Whakaaro: Key concepts in Mäori culture. Oxford University Press, Auckland. Bauer, L. (1986) Notes on New Zealand English Phonetics and Phonology. English World- Wide, Volume 7, No. 2, pp. 255-258. Bauer, L. (1994) Introducing the Wellington Corpus of Written New Zealand English, Te Reo, Journal of the Linguistic Society of New Zealand, Volume 37, University of Auckland. Bauer, W., Parker, W., Evans, T. (1993) Maori Descriptive Grammars, Routledge, London and New York. Bell Laboratories, Lucent Technologies, Murray Hill, New Jersey. [http://www.bell-labs.com/project/tts/] Bender, B. W. (1971) Linguistic Factors in Maori Education, A Report. New Zealand Council for Educational Research, Wellington. Benton, R. A. (1979) Is Bilingualism a Handicap? New Zealand Council for Educational Research, Wellington. Benton, R. A. (1981) The Flight of the Amokura, Oceanic Languages and Formal Education In the South Pacific, New Zealand Council for Educational Research, Wellington. Benton, R. A., Tumoana, H., Robb, A. (1982) Ko Ngä Kupu Pü Noa o Te Reo Mäori: The First Basic Maori Word List, New Zealand Council for Educational Research, Wellington. Benton, R. A. (1982) The Anatomy of a Word List, New Zealand Council for Educational Research, Wellington. Benton, R. A. (1990) The History and Development of the Mäori Language, Dirty Silence Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Auckland. Benton, R. A. (1992) Maori English: A New Zealand Myth?, New Zealand English Newsletter, Number 6, Department of English, University of Canterbury, Christchurch, pp. 27-35. Bezdek, J. and Pal, N. (1993) An Index of Topological Preservation and its Application to Self-Organising Feature Maps. Proceedings of 1993 International Joint Conference on Neural Networks, pp. 2435-2440. Biggs, B. (1981) The Complete English-Maori Dictionary, Auckland University Press, Auckland. Biggs, B. (1996) Lets Learn Maaori. A Guide to the Study of the Maori Language, Reprinted by Uniprint, Auckland. Black, A. W. and Taylor, P. (1994) CHATR: a generic speech synthesis system. COLING-94, volume II, pp. 983-986, Kyoto. Japan. Borland (1992). Paradox for Windows ObjectPAL Reference. Borland International, Inc. Braspenning, P., Taylor, J., Gallinari, P., Kasabov, N. (1990) Connectionism & AI Lectures. ISAI’90-International School in Artificial Intelligence, Albena, Bulgaria. Bregman, A. S. (1994) Auditory Scene Analysis, The Perceptual Organization of Sound. A Bradford Book, The MIT Press, Cambridge. Campbell, N. (1996) CHATR: A High-Definition Speech Re-Sequencing System. Acoustic Society of America & Acoustic Society of Japan, 3rd Joint Meeting, Honolulu, pp. 1223- 1228. Campbell, N. (1997) Processing a Speech Corpus for CHATR Synthesis. ATR Interpreting Telecommunications Research Laboratories, Kyoto. Capron, H. (1990) Computers – Tools for an Information Age, 2nd Edition, Redwood City, Benjamin/Cummings Publishing. Chomsky, N. and Halle, M. (1968) The Sound Pattern of English. New York, Harper and Row. Clark, J. and Yallop, C. (1990) An Introduction to Phonetics and Phonology. Cambridge, Massachusetts, B. Blackwell. Cole, R., Noel, M., Burnett, D. C., Fanty, M., Lander, T., Oshika, B., Sutton, S. (1994) Corpus Development Activities at the Center for Spoken Language Understanding, In Proceedings of the ARPA Workshop on Human Language Technology. Creative Technology Ltd. (1997) Creative TextAssist™. [http://www-nt-ok.creaf.com/techsupt/devcnr/tassist.html] Cruttenden, A. (1994) Gimson’s Pronunciation of English, 5th Edition, Hodder Headline Group, London. Crystal, D. (1980) A First Dictionary of Linguistics and Phonetics. The Language Library, University Press, Cambridge. Crystal, D. (1992) The Cambridge Encyclopedia of Language, Cambridge University Press, Cambridge. Date, C. J. (1990) An Introduction to Database Systems. Volume 1, 5th Edition. Addison- Wesley Publishing Co. Demuth, H. and Beale, M. (1996) Neural Network ToolBox For Use with MatLab. The Math Works, Inc. Natick, Massachusetts. Dutoit, T., V, Pagel., N, Pierret., F, Bataille., O, Van Der Vrecken. (1996) “The MBROLA Project: Towards a Set of High-Quality Speech Synthesisers Free of Use for Non-Commercial Purposes”. Proc ICSLP’96, Philadelphia, Vol. 3, pp. 1393-1396. Fahlman, S. (1988) Faster-Learning Variations on Back-Propagation: An Empirical Study, in Proceedings Connectionist Models Summer School, Morgan Kaufmann. Foster, J. (1993) He Whakamärama A New Course in Mäori, Reed Books, Auckland. Fraser, S. (1995) DictionaryEdit Online User Manual. Centre for Population Biology, Imperial College at Silwood Park, Birkshire. Fromkin, V., Rodman, R., Collins, P., Blair, D. (1988). An Introduction to Language , Second Australian Edition, Holt, Rinehart and Winston, Australia. FuzzyCOPE V1.0, (1995) User Manual, Information Science Department, University of Otago, Dunedin. FuzzyCLIPS Version 6.02, (1994) User’s Guide, Knowledge Systems Laboratory, Institute for Information Technology, National Research Council, Canada. Giegerich, H. (1992) English Phonology: An Introduction. Cambridge University Press, Cambridge. Goodman, D. (1993) The Complete HyperCard 2.2 Handbook. 4th Edition, Random House, New York. Hanks, P. (1980) Meaning and Grammar. Editor, Collins Dictionary of the English Language. An extensive coverage of contemporary international and Australian English. William Collins Sons & Co. Ltd, Sydney. Harawira, K. T; revised by Käretu, T. (1994) Teach Yourself Mäori, Reeds Books, Auckland. Harlow, R. (1987) A Word-List of South Island Maori, Te Reo Monographs, Linguistic Society of New Zealand, University of Auckland. Harlow, R. (1990) A Name and Word Index to Ngä Mahi a Ngä Tüpuna, University of Otago Press, Dunedin. Holland, J. (1994) Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence. MIT Press, CA, MA. Holmes, J. (1990) The Role of the Sociolinguist in Society, Dirty Silence, Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Oxford. Hunt, A. J. and Black, A. W. (1996) Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database. Proceedings: ICASSP, Atlanta, GA. Karaali, O., Corrigan, G., Gerson, I. (1996) Speech Synthesis with Neural Networks. World Congress on Neural Networks, San Diego, pp. 45-50. Kärena-Holmes, D. (1993) Grammar Basics English – Mäori, 60 Royal Tce, Dunedin. Kärena-Holmes, D. (1995) Mäori Language Understanding the Grammar, The Printing Department, University of Otago, Dunedin. Kasabov, N., Watson, C., Sinclair, S., Kilgour, R. (1994) Integrating Neural Networks and Fuzzy Systems for Speech Recognition. 5th International Australian Speech Science and Technology Conference, Perth, pp. 462-467. Kasabov, N., Watson, C. (1994) Automatic Speech Recognition: Methods, Tools and Their Application for Communication and Intelligent Information Systems. A Report on a Research Project Funded by Telecom NZ Ltd. Information Science, University of Otago, Dunedin, New Zealand. Kasabov, N., Sinclair, S., Kilgour, R., Watson, C., Laws, M., Kassabova, D. (1995) Intelligent Human Computer Interfaces and the Case Study of Building English-to-Mäori Talking Dictionary, Kasabov, N., Coghil, G. (Eds.), Proceedings ANNES 95’ University of Otago, Los Alamitos, CA: IEEE Computer Society Press, pp. 294-297. Kasabov, N., Purvis, M., Sallis, P. (1996a) Connectionist-Based Information Systems: A Proposed Research Theme, The Information Science Discussion Paper Series, No 96/03, Dunedin, New Zealand. Kasabov, N., Kim, J., Gray, A., Watts, M. (1996b) Fuzzy Neural Network Architecture for Adaptive Learning and Knowledge Acquisition, International Journal of Information Sciences: Applications, North Holland. Kasabov, N. (1996) Foundations of Neural Networks, Fuzzy Systems and Knowledge Engineering, The MIT Press, CA, MA. Kasabov, N. and Gray, A. (1996) Round-trip System Engineering in Neuro-Fuzzy Hybrid Systems. Submitted to Journal of Intelligent & Fuzzy Systems. Kasabov, N. (1997) A Framework for Intelligent ‘Conscious’ Machines Utilising Fuzzy Neural Networks and Spatial temporal Maps and a Case Study of Multilingual Speech Recognition, In: S. Amari and N. Kasabov (eds) “Brain-like Computing and Intelligent Information Systems”, Springer Verlag, Singapore Kasabov, N. and Kozma, R. (1997) Fuzzy Neural Networks and Fuzzy Spatio Temporal Maps, In: N. Kasabov and R. Kozma (eds) Neuro-fuzzy Tools and Techniques, Physica Verlag, Heidelberg. Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M., Gray, A. (1997) Neuro-fuzzy Techniques for Speech Data Analysis and Adaptive Speech Recognition, In: N. Kasabov and R. Kozma (eds) Neuro-fuzzy Tools and Techniques, Physica Verlag, Heidelberg Kasabov, N., Kozma, R., Kilgour, R., Laws, M., Taylor, J., Watts, M., Gray, A. (1997a) Methodology for Speech Data Analysis and a Framework for Adaptive Speech Recognition Using Fuzzy Neural Networks. ICONIP/ANZIIS/ANNES`97. vol 2, pp. 1055-1060. Keegan, T. and Treweek, P. (1994) Ngä Tautono Rorohiko Course Manual. Department of Computer Science, University of Waikato, Hamilton. Keegan, P. (1996) Te Wahapu – Mäori/English Lexical Database Web Public Access, New Zealand Council for Educational Research, Wellington. [http://www.dia.govt.nz/dia/general/dictionary/maori/public/wordsearch.html] Keegan, P. (1997) Kimikupu Hou Mäori Lexical Database on the Web: Reflections and Possible Future Directions. In Proceedings NAMMSAT Conference, October 1997, Massey University, Palmerston North. Kilgour, R. “Hybrid Fuzzy Systems and Neural Networks for Speech Recognition.” Unpublished M.Sc. (Cognitive) thesis, University of Otago, 1996. Kilgour, R. (1997) Report on file formats for sound and speech files. Department of Information Science, University of Otago. Knowles, M. (1983) Andragogy: An Emerging Technology for Adult Learning, M. Tight (ed.) In, Adult Learning and Education. London: Open University. 53-70. Kohonen, T. (1990) The Self-Organizing Map, Proceedings of the IEEE, vol.78, N-9, pp. 1464-1479. Kohonen, T. (1997) Self-Organizing Maps, Second Edition, Springer-Verlag, New York. Kosko, B. (1992) Neural Networks and Fuzzy Systems: A Dynamical Approach to Machine Intelligence, Prentice Hall, New Jersey. Kozma, R., Sakuma, M., Yokoyama, Y., Kitamura, M. (1996) On the accuracy of mapping by neural networks trained with backpropagation with forgetting, Neurocomputing, 13, pp. 295- 311. Kuchera, H. and Francis, W. N. (1967) Computational Analysis of Modern-Day American English, Brown University Press, Providence. Kumar, S. R. and Ghoshal, J. (1997) Neuro Fuzzy Approach to Pattern Recognition. Neural Networks, Vol. 10, No. 1, pp. 161-182. Kurzwell, R. (1997) When Will HAL Understand What We Are Saying, In: Stork, D. G (Ed) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Ladefoged, P. (1993) A Course In Phonetics, Harcourt Brace Jovanovich, Inc. Laws, M. (1995) A Computer Assisted Mäori Language Laboratory. The Design, Development and Implementation of an integrated package of Mäori language course material in a computerised environment—is it possible and practical? Proceedings of the Inaugural NAMMSAT Conference, Auckland University. Laws, M. (1995) Building Speech Interfaces to Databases: A Case Study for an English-to- Mäori Talking Dictionary, Addendum to ANNES 95’, University of Otago, Dunedin, New Zealand. Laws, M. “New Zealand English to Mäori Lexical Database: A Talking Dictionary”. Unpublished Research Dissertation, University of Otago, 1995. Laws, M. (1996) The ongoing development of an Intelligent Human Computer Interface to a New Zealand English-Mäori Speech Database: A Talking Dictionary. In Proceedings, NAMMSAT Conference, Hamilton. Laws, M. (1997) Integrating Text and Speech into Databases, Information Systems and Knowledge Engineering for Human Computer Interaction. The progressive development of an Integrated Bilingual Interface. In Proceedings NAMMSAT Conference, October 1997, Massey University, Palmerston North. Lewis, G. (1996) The Origins of New Zealand English: A Report on Work in Progress. The New Zealand English Journal 10, pp. 25-30. Lewis, K. R., Pettey, M., Shneiderman, B. (1993) Speech versus mouse commands for word processing: an empirical evaluation. International Journal of Man-Machine Studies, 39, pp. 667-687. Linguistic Data Consortium. (1996) LDC Catalog, University of Pennsylvania, Philadelphia. Linggard, R. (1994) Speech Science and Technology - Review and Perspective. 5th International Australian Speech Science and Technology Conference, Perth, pp. 576-578. Luce, P., Pisoni, D., Goldinger, S. (1990) Similarity Neighborhoods of Spoken Words. Cognitive Models of Speech Processing: Psycholinguistic and Computational Perspectives. Ed G. Altmann, The MIT Press, Massachusetts. Maclagan, M (1987) An acoustic study of New Zealand Vowels. New Zealand Speech Therapists' Journal Vol. 37, No 1, pp. 20-26. Maclagan, M. and Gordon, E. (1995) The Changing Sound of New Zealand English. The New Zealand Speech-Language Therapists' Journal Vol. 50, pp. 32-40. McCrone, J. (1993) Computers that Listen. New Scientist, 4th December 1993, pp. 30-35. McCulloch, N., Bedworth, M., Brible, J. (1987) NETspeak—A re-implementation of NETtalk. Computer Speech and Language, Vol 2, pp. 289-301. McGregor, G. and Williams M. (1990) (Eds.) Dirty Silence, Aspects of Language and Literature in New Zealand, Essays arising from the University of Waikato Winter Lecture Series of 1990, Oxford University Press, Oxford. Malfrère, F. and Dutoit, T. (1996) High-Quality Speech Synthesis For Phonetic Speech Segmentation. Circuits Theory and Signal Processing Lab, Faculté Polytechnique de Mons, Belgium. Mäori Language Commission. (1992) Te Matatiki – Ngä Kupu Hou a Te Taura Whiri i te Reo Mäori, Mäori Language Commission, Wellington. Mäori Language Commission. (1996) Te Matatiki – Ngä Kupu Hou a Te Taura Whiri i te Reo Mäori . Oxford Press, Auckland. Meisel, W. S. (1993) Talk to Your Computer: Voice technology lets you verbally command your computer or convert speech to text. BYTE October 1993, pp. 113-120. Microsoft (1994) Access® Version 2.0. Relational Database Management System for Windows™, Building Applications, Microsoft Corporation. Microsoft (1995)Building Applications with Microsoft Access for Windows 95. Programming with Visual Basic for Applications, Microsoft Corporation, Australia. Mitra, S. and Pal, S. K. (1995) Fuzzy Multi-Layer Perceptron, Inferencing and Rule Generation. IEEE Transactions on Neural Networks, Vol. 6, No. 1, pp. 51-63. Möbius, B., Schroeter, J., Santen, J., Sproat, R., Olive, J. (1996) Recent Advances in Multilingual Text-To-Speech Synthesis. AT&T Bell Laboratories, Murray Hill, New Jersey. Mohri, M., Riley, M., Sproat, R. (1996) Algorithms for Speech Recognition and Language Processing. A Tutorial presented at COLING’96 (Computational Linguistics), Copenhagen. Molich, R. and Nielsen, J. (1990) Improving a Human-Computer Dialogue, Communications of the ACM, Vol 33, No 3, March, pp. 338-343. Moorfield, J. C. (1989) Te Käkano, Longman Paul Ltd, Auckland. Moorfield, J. C. (1988) Te Whanake 1 Te Käkano: Pukapuka ärahi i te kaiwhakaako, Te Whare Wänanga o Waikato. Morgan, D. and Scofield, C. (1991) Neural Networks and Speech Processing. Kluwer Academic Publisher, Massachusetts. Ngata, H. M. (1993) English-Mäori Dictionary, Learning Media, Wellington. Olive, J. P. (1997) “The Talking Computer”, In: Stork, D. G (Ed) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Owens, F. (1994) Signal Processing of Speech. The MacMillan, McGraw-Hill, Inc. Pagel, V. “New Database” Personal email communication. 10 June 1997. Faculté Polytechnique de Mons, Belgium. [[email protected]] Rabiner, L. (1994) Applications of Voice Processing to Telecommunications, Proceedings of the IEEE, Vol 82, No. 2, Feb, pp. 199-288. Rheingold, H. (1992) Virtual Reality, Mandarin Paperbacks, London. Ritter, H. and Kohonen, T. (1989) Self-Organising Semantic Maps. Biological Cybernetics vol 61, Springer-Verlag, pp. 241-254. Rob, P. and Williams, T. R. (1995) Database Design and Applications Development with Microsoft Access 2.0. McGraw-Hill, San Francisco. Ryan, P. M. (1985) The Revised Dictionary of Modern Mäori, Heinemann Publishers. Ryan, P. M. (1990) The Revised Dictionary of Modern Mäori, Heinemann Publishers. Sejnowski, T. J. and Rosenberg, C. (1986) NetTalk: A Parallel Network that Learns to Read Aloud, Johns Hopkins University Electrical and Computer Science Technical Report, Johns Hopkins University, Baltimore. Sejnowski, T. J. and Rosenberg, C. (1988) NetTalk Corpus, Johns Hopkins University, Cognitive Science Center, Baltimore. Sinclair, S. and Watson, C. (1995) The Development of the Otago Speech Database. In Proceedings ANNES 95’, University of Otago, Dunedin. Sinclair, S. “Development of an Isolated Speech Digit Recognition System Based on Back Propagation Neural Networks.” Unpublished M.Com. thesis, University of Otago, 1996. Sproat, R. (1996) Multilingual Text Analysis for Text-to-Speech Synthesis. 12th European Conference on Artificial Intelligence, Edited by W. Wahlster, Sproat, R. and Shih, C. “Maori Info.” Personal email communication. 27 May 1997. Bell Laboratories, Lucent Technologies, Murray Hill. [[email protected]] Stork, D. G (1997) HAL’S Legacy: 2001’s Computer As Dream and Reality. The MIT Press, Cambridge, Massachusetts. Sullivan, K. P. H. and Damper, R. I. (1995) Novel-word pronunciation: A cross-language study. Artificial Intelligence Laboratory, Computer & Information Science, University of Otago, Dunedin. Sullivan, K. P. H. (1995) Text-to-speech conversion fo