Generation, selection and combination of components in neural network ensembles applied to classification problems

Orientador: Fernando Jose Von ZubenDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Eletrica e ComputaçãoResumo: O uso da abordagem ensembles tem sido bastante explorado na última década, por se tratar de uma técnica simples e capaz de aumentar a capacidade de generalização de soluções baseadas em aprendizado de máquina. No entanto, para que um ensemble seja capaz de promover melhorias de desempenho, os seus componentes devem apresentar bons desempenhos individuais e, ao mesmo tempo, devem ter comportamentos diversos entre si. Neste trabalho, é proposta uma metodologia de criação de ensembles para problemas de classificação, onde os componentes são redes neurais artificiais do tipo perceptron multicamadas. Para que fossem gerados bons candidatos a comporem o ensemble, atendendo a critérios de desempenho e de diversidade, foi aplicada uma meta-heurística populacional imuno-inspirada, denominada opt-aiNet, a qual é caracterizada por definir automaticamente o número de indivíduos na população a cada iteração, promover diversidade e preservar ótimos locais ao longo da busca. Na etapa de seleção dos componentes que efetivamente irão compor o ensemble, foram utilizadas seis técnicas distintas e, para combinação dos componentes selecionados, foram adotadas cinco estratégias. A abordagem proposta foi aplicada a quatro problemas de classificação de padrões e os resultados obtidos indicam a validade da metodologia de criação de ensembles. Além disso, foi verificada uma dependência entre o melhor par de técnicas de seleção e combinação e a população de indivíduos candidatos a comporem o ensemble, assim como foi feita uma análise de confiabilidade dos resultados de classificaçãoAbstract: In the last decade, the ensemble approach has been widely explored, once it is a simple technique capable of increasing the generalization capability of machine learning based solutions. However, an ensemble can only promote performance enhancement if its components present good individual performance and, at the same time, diverse behavior among each other. This work proposes a methodology to synthesize ensembles for classification problems, where the components of the ensembles are multi-layer perceptrons. To generate good candidates to compose the ensemble, meeting the performance and diversity requirements, it was applied a populational and immune-inspired metaheuristic, named opt-aiNet, which is characterized as being capable of automatically determining the number of individuals in the population at each iteration, promoting diversity and preserving local optima through the search. In the component selection phase, six distinct techniques were applied and, to combine these selected components, five strategies were adopted. The proposed approach was applied to four pattern classification problems and the obtained results indicated the validity of the methodology to synthesize ensembles. It was also verified a dependence of the best pair of selection and combination techniques on the population of candidates to compose the ensemble, and it was made an analysis of the confidence of the classification resultsMestradoEngenharia de ComputaçãoMestre em Engenharia Elétric

Omni-ainet: An Immune-inspired Approach For Omni Optimization

This work presents omni-aiNet, an immune-inspired algorithm developed to solve single and multi-objective optimization problems, either with single and multi-global solutions. The search engine is capable of automatically adapting the exploration of the search space according to the intrinsic demand of the optimization problem. This proposal unites the concepts of omni-optimization, already proposed in the literature, with distinctive procedures associated with immune-inspired concepts. Due to the immune inspiration, the omni-aiNet presents a population capable of adjusting its size during the execution of the algorithm, according to a predefined suppression threshold, and a new grid mechanism to control the spread of solutions in the objective space. The omni-aiNet was applied to several optimization problems and the obtained results are presented and analyzed. © Springer-Verlag Berlin Heidelberg 2006.4163 LNCS294308Bazaraa, M.S., Sherali, H.D., Shetty, C.M., (1993) Nonlinear Programming: Theory and Algorithms, , John-Wiley & Sons, 2nd. edBurnet, F.M., Clonal selection and after (1978) Theoretical Immunology, pp. 63-85. , Bell, G. I., Perelson, A. S., Pimgley Jr, G. H. (Eds.), Marcel Dekker IncCoello Coello, C.A., Cortés, N.C., Solving multiobjective optimization problems using an artificial immune system (2005) Genetic Programming and Evolvable Machines, 6, pp. 163-190De Castro, L.N., Timmis, J., (2002) An Introduction to Artificial Immune Systems: A New Computational Intelligence Paradigm, , Springer-VerlagDe Castro, L.N., Timmis, J., An artificial immune network for multimodal funcion optimization (2002) Proc. IEEE CEC, pp. 669-674. , USADe Castro, L.N., Von Zuberi, F.J., aiNet: An artificial immune network for data analysis (2001) Data Mining: A Heuristic Approach, pp. 231-259. , Abbass, H. A., Sarker, R. A., Newton, C. S. (Eds.), Idea Group Publishing, USA, Chapter XIIDe França, F.O., Von Zuben, F.J., De Castro, L.N., An artificial immune network for multimodal funcion optimization on dynamic environments (2005) Proc. GECCO, pp. 289-296. , Washington, DC, USADeb, K., Tiwari, S., Omni-optimizer: A procedure for single and multi-objective optimization (2005) LNCS, 3410, pp. 47-61. , Coello Coello, C. A., Aguirre, A. H., Zitzler, E. (Eds.) Proc. 3rd. EMO, MexicoGomes, L.C.T., De Sousa, J.S., Bezerra, G.B., De Castro, L.N., Von Zuben, F.J., Copt-aiNet and the gene ordering problem (2003) Information Technology Magazine, 3 (2), pp. 27-33. , Catholic University of BrasiliaHolland, P.W.H., Garcia-Fernandez, J., Williams, N.A., Sidow, A., Gene duplications and origins of the vertebrate development (1994) Dev. Supp., pp. 125-133Jerne, N.K., Towards a network theory of the immune system (1974) Ann. Immunol., Inst. Pasteur, 125 C, pp. 373-389Ohno, S., (1970) Evolution by Gene Duplication, , Allen and Unwin, London, UKZitzler, E., Thiele, L., Deb, K., Comparison of multiobjective evolutionary algorithms: Empirical results (2000) Evol. Computation, 8 (2), pp. 173-19

Applying Biclustering To Perform Collaborative Filtering

Collaborative filtering (CF) is a method to perform automated suggestions for a user based on the opinion of other users with similar interest. Most of the CF algorithms do not take into account the existent duality between users and items, considering only the similarities between users or only the similarities between items. In this paper we propose a novel methodology for the CF capable of dealing with this situation. By proposing an immune-inspired biclustering technique to carry out clustering of rows and columns at the same time, our algorithm is able to group similarities between users and items. In order to evaluate the proposed methodology, we have applied it to MovieLens dataset which contains user's ratings to a large set of movies. The results indicate that our proposal is able to provide useful recommendations for the users, outperforming other methodologies for CF reported in the literature. © 2007 IEEE.421426Agrawal, R., Gehrke, J., Gunopulus, D., Raghavan, P., Automatic subspace clustering of high dimensional data for data mining applications (1998) Proc. of the ACM/SIGMOD Int. Conference on Management of Data, pp. 94-105Cheng, Y., Church, G.M., Biclustering of expression data (2000) Proc. of the 8th Int. Conf. on Int. Systems for Molecular Biology, pp. 93-103de Castro, L.N., Von Zuben, F.J., (2001) aiNet: An Artificial Immune Network for Data Analysis, pp. 231-259. , Data Mining: A Heuristic Approachde França, F.O., Bezerra, G., Von Zuben, F.J., New Perspectives for the Biclustering Problem (2006) IEEE Congress on Evolutionary Computation, pp. 2768-2775Dhillon, I.S., Co-clustering documents and words using bipartite spectral graph partitioning (2001) Proc. of the 7th Int. Conf. on Knowledge Discovery and Data Mining, pp. 269-274Goldberg, D., Nichols, D., Brian, M., Terry, D., Using collaborative filtering to weave an information tapestry (1992) ACM Communications, 35, pp. 61-70Gomes, L.C.T., de Sousa, J.S., Bezerra, G.B., de Castro, L.N., Von Zuben, F.J., (2003) Copt-aiNet and the Gene Ordering Problem, 3 (2), pp. 27-33. , Information Technology MagazineHaixun, W., Wei, W., Jiong, Y., Yu, P.S., Clustering by pattern similarity in large data sets (2002) Proc. of the 2002 ACM SIGMOD Int. Conference on Management of Data, pp. 394-405Hartigan, J. A, Direct clustering of a data matrix. Journal of the American Statistical Association (JASA), 1972, 67, no. 337, pp. 123-129Moscato, P., Berretta, R., Mendes, A., A New Memetic Algorithm for Ordering Datasets: Applications in Microarray Analysis (2005) Proc. of the 6th Metaheuristics Int. Conference, pp. 695-700. , Austria, AugustResnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J., Grouplens: An open architecture for collaborative filtering on netnews (1994) In Proc. of the Computer Supported Collaborative Work Conference, pp. 175-186Segal, E., Taskar, B., Gasch, A., Friedman, N., Koller, D., Rich probabilistic models for gene expression (2001) In Bioinformatics, 17 (SUPPL. 1), pp. S243-S252Sheng, Q., Moreau, Y., De Moor, B., Biclustering micrarray data by Gibbs sampling (2003) Bioinformatics, 19 (SUPPL. 2), pp. 196-205Symeonidis, P., Nanopoulos, A., Papadopoulos, A., Manolopoulos, Y., Nearest-Biclusters Collaborative Filtering (2006) Proc. of the WebKDD - Workshop held in conjuction with KDDTang, C., Zhang, L., Zhang, I., Ramanathan, M., Interrelated two-way clustering: An unsupervised approach for gene expression data analysis (2001) Proc. of the 2nd IEEE Int. Symposium on Bioinformatics and Bioengineering, pp. 41-48Yu, K., Schwaighofer, A., Tresp, V., Xu, X., Kriegel, H.-P., Probabilistic Memory-based Collaborative Filtering (2004) In IEEE Transactions on Knowledge and Data Engineering, pp. 56-5

Designing Ensembles Of Fuzzy Classification Systems: An Immune-inspired Approach

In this work we propose an immune-based approach for designing of fuzzy systems. From numerical data and with membership function previously defined, the immune algorithm evolves a population of fuzzy classification rules based on the clonal selection, hypermutation and immune network principles. Once AIS are able to find multiple good solutions of the problem, accurate and diverse fuzzy systems are built in a single run. Hence, we construct an ensemble of these classifier in order to achieve better results. An ensemble of classifiers consists of a set of individual classifiers whose outputs are combined when classifying novel patterns. The good performance of an ensemble is strongly dependent of individual accuracy and diversity of its components. We evaluate the proposed methodology through computational experiments on some datasets. The results demonstrate that the performance of the obtained fuzzy systems in isolation is very good. However when we combine these systems, a significant improvement is obtained in the correct classification rate, outperforming the single best classifier. © Springer-Verlag Berlin Heidelberg 2005.3627469482Klir, G., Yuan, B., (1995) Fuzzy Sets and Fuzzy Logic - Theory and Applications, , Prentice-HallPedrycz, W., Gomide, F., (1998) An Introduction to Fuzzy Sets: Analysis and Design, , MIT PressCordón, O., Herrera, F., Gomide, F., Hoffmann, F., Magdalena, L., Ten years of genetic-fuzzy systems: A current framework and new trends (2001) Proceedings of Joint 9th IFSA World Congress and 20th NAFIPS International Conference, pp. 1241-1246. , Vancouver, CanadaCordón, O., Herrera, F., Hoffmann, F., Magdalena, L., Genetic fuzzy systems. Evolutionary tuning and learning of fuzzy knowledge bases (2001) Advances in Fuzzy Systems: Applications and Theory, 19. , World ScientificYuan, Y., Zhuang, H., A genetic algorithm for generating fuzzy classification rules (1996) Fuzzy Sets and Systems, 84 (4), pp. 1-19Goldberg, D.E., Richardson, J.J., Genetic algorithms with sharing for multimodal function optimization (1987) Proceedings of the 2nd International Conference on Genetic Algorithms, pp. 41-49. , CambridgeBonissone, P., Eklund, N., Goebel, K., Using an ensemble of classifiers to audit a production classifier (2005) Proceedings of the 6th International Workshop on Multiple Classifier Systems, , Monterey, CA, June 13-1Dasgupta, D., (1999) Artificial Immune Systems and Their Applications, , Springer-VerlagDe Castro, L.N., Timmis, J., (2002) An Introduction to Artificial Immune Systems: A New Computational Intelligence Paradigm, , Springer-VerlagGomes, L.C.T., De Sousa, J.S., Bezerra, G.B., De Castro, L.N., Von Zuben, F.J., Copt-aiNet and the gene ordering problem (2003) Second Brazilian Workshop on Bioinformatics, , Macaé, BrazilHansen, L.K., Salamon, P., Neural network ensembles (1990) IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, pp. 993-1001Opitz, D., Shavlik, J., Generating accurate and diverse members of a neural network ensemble (1996) Advances in Neural Information Processing Systems, 8, pp. 535-541. , D. Touretsky, M. Mozer, and M. Hasselmo (Eds.), MIT PressCordon, O., Del Jesus, M.J., Herrera, F., A proposal on reasoning methods in fuzzy rule-based classification systems (2001) International Journal of Approximate Reasoning, 20, pp. 21-45De Castro, L.N., Von Zuben, F.J., aiNet: An artificial immune network for data analysis (2001) Data Mining: A Heuristic Approach, pp. 231-259. , H. A. Abbass, R. A. Sarker, and C. S. Newton (eds.), Idea Group Publishing, USA, Chapter XIIDe Castro, L.N., Timmis, J.I., An artificial immune network for multimodal function optimization (2002) Proceedings of IEEE Congress of Evolutionary Computation, 1, pp. 699-1674. , HawaiiAda, G.L., Nossal, G.J.V., The clonal selection theory (1987) Scientific American, 257 (2), pp. 50-57Jerne, N.K., Towards a network theory of the immune system (1974) Ann. Immunol. (Inst. Pasteur), 125 C, pp. 373-389Freitas, A.A., (2002) Data Mining and Knowledge Discovery with Evolutionary Algorithms, , Springer-Verlag New York, Secaucus, NJ, USAKrogh, A., Vedelsby, J., Neural network ensembles, cross validation, and active learning (1994) Advances in Neural Information Processing System, 7, pp. 231-238. , G. Tesauro, D. Touretzky, and T. Leen (Eds.), MIT PressGlover, F., Laguna, M., (1998) Tabu Search, , Kluwer Academic PublishersAlves, R.T., Delgado, M.R.B., Lopes, H.S., Freitas, A., Induction of fuzzy classification rules with an artificial immune system (2004) Proceedings of Brazilian Symposium on Neural Networks, , BrazilCamargo, H.A., Pires, M.G., Castro, P.A.D., Genetic design of fuzzy knowledge bases - A study of different approaches (2004) Proceedings of 23rd IEEE International Conference of NAFIPS, 2, pp. 954-959. , CanadaPerrone, M.P., Cooper, L.N., When networks disagree: Ensemble method for neural networks (1993) Neural Networks for Speech and Image Processing, , R.J. Mammone (ed.). Chapman-HallLima, C.A.M., Coelho, A.L.V., Von Zuben, F.J., Fuzzy systems design via ensembles of ANFIS (2002) Proceedings of the IEEE International Conference on Fuzzy Systems (FUZZ-IEEE2002), 1, pp. 506-511. , HawaiiIshibuchi, H., Yamamoto, T., Evolutionary multi-objective optimization for generating an ensemble of fuzzy rule-based classifiers (2003) Proc. of the Genetic and Evolutionary Computation Conference, pp. 1077-1088. , USAZhou, Z.H., Wu, J., Tang, W., Ensembling neural networks: Many could be better than all (2002) Artificial Intelligence, 137 (1-2), pp. 239-263Blake, C.L., Merz, C.J., (1998) UCI Repository of Machine Learning Databases, , http://www.ics.uci.edu/m~learn/MLRepository.html, Irvine, CA: University of California, Department of Information and Computer ScienceIshibuchi, H., Yamamoto, T., Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining (2004) Fuzzy Sets and Systems, 141, pp. 59-8

Bayesian Learning Of Neural Networks By Means Of Artificial Immune Systems

Once the design of Artificial Neural Networks (ANN) may require the optimization of numerical and structural parameters, bio-inspired algorithms have been successfully applied to accomplish this task, since they are population-based search strategies capable of dealing successfully with complex and large search spaces, avoiding local minima. In tills paper, we propose the use of an Artificial Immune System for learning feedforward ANN's topologies. Besides the number of neurons in the hidden layer, the algorithm also optimizes the type of activation function for each node. The use of a Bayesian framework to infer the weights and weight decay terms as well as to perform model selection allows us to find neural models with high generalization capability and low complexity, once the Occam's razor principle is incorporated into the framework. We demonstrate the applicability of the proposal on seven classification problems and promising results were obtained. © 2006 IEEE.48314838Ada, G.L., Nossal, G.J.V., The Clonal Selection Theory (1987) Scientific American, 257 (2), pp. 50-57Angeline, P.J., Sauders, G.M., Pollack, J.B., An evolutionary algorithm that constructs recurrent neural networks (1994) IEEE Trans. Neural Networks, 5, p. 5465Battiti, R., First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method (1992) Neural Computation, 4 (2), pp. 141-166Blake, C.L., Merz, C.J., (1998) UCI Repository of machine learning databases, , http://www.ics.uci.edu/~mlearn/MLRepository.html, Irvine, CA: University of California, Department of Information and Computer ScienceBishop, C.M., (1995) Neural Networks for Pattern Recognition, , Clarendon PressBuntine, W.L., Weigend, A.S., Bayesian backpropagation (1991) Complex Systems, 5, p. 603643Blum, C., Socha, K., Training feed-forward neural networks with ant colony optimization: An application to pattern classification (2005) Proc. 5th Intern. Conf. on Hybrid Intelligent Svstems(HIS-2005), pp. 10-15Castro, P.A.D., Coelho, G.P., Caetano, M.F., Von Zuben, F.J., Designing Ensembles of Fuzzy Classification Systems: An Immune-Inspired Approach (2005) Lectures Notes in Computer Science, 3627, pp. 469-482. , Springer-VerlagCastro, P.A.D., Von Zuben, F.J., An Immune-Inspired Approach to Bayesian Networks (2005) Proc. 5th Intern. Conf. on Hybrid Intelligent Systems (HIS-2005), pp. 23-28. , N. Nedjah et al, edsCoelho, G.P., Castro, P.A.D., Von Zuben, F.J., The Effective Use of Diverse Rule Bases in Fuzzy Classification (2005) VII Brazilian Congress on Neural NetworksDasgupta, D., (1999) Artificial Immune Systems and Their Applications, , Springer-Verlagde Castro, L.N., Timmis, J., (2002) An Introduction to Artificial Immune Systems: A New Computational Intelligence Paradigm, , Springer-Verlagde Castro, L.N., Timmis, J.I., An Artificial Immune Network for Multimodal Function Optimization (2002) Proceedings of IEEE Congress of Evolutionary Computation, 1, pp. 699-674de Castro, L.N., Von Zuben, F.J., An Immunological Approach to Initialize Feedforward Neural Network Weights (2001) Proc. Intern. Conf. on Artificial Neural Networks and Genetic Algorithms, pp. 126-129Gomes, L.C.T., de Sousa, J.S., Bezerra, G.B., de Castro, L.N., Von Zuben, F.J., Copt-aiNet and the Gene Ordering Problem (2003) Revista Tecnologia da Informação, 3 (2), pp. 27-33Haykin, S., (1998) Neural Networks: A Comprehensive Foundation, , 2nd edition, Prentice Hall PTRIyoda, E.M., Von Zuben, F.J., Hybrid Neural Networks: An Evolutionary Approach With Local Search (2002) Integrated Computer-Aided Engineering, 9 (1), pp. 57-72Jerne, N.K., Towards a Network Theory of the Immune System (1974) Ann. Immunol. (Inst. Pasteur), 125 C, pp. 373-389Jones, A.J., Genetic algorithms and their applications to the design of neural networks (1993) Neural Computing & Appl, 1, pp. 32-45Kwok, T.Y., Yeung, D.Y., Bayesian regularization in constructive neural networks (1996) Proc. Intern. Conf. on Artificial Neural Networks, p. 557562Kwok, T.Y., Yeung, D.Y., Constructive algorithms for structure learning in feedforward neural networks for regression problems (1997) IEEE Trans. on Neural Networks, 8 (3), pp. 630-645Liu, Y., Yao, X., Evolutionary design of artificial neural networks with different node transfer functions (1996) Proc. 3rd IEEE Intern. Conf. on Evolutionary Computation (CEC, pp. 670-675. , 96, ppMatteucci, M., Spadoni, D., Evolutionary Learning of Rich Neural Networks in the Bayesian Model Selection Framework (2004) Intern. Journal of Applied Mathematics and Computer Science, 14 (3), pp. 423-440MacKay, D.J.C., A Practical Bayesian Framework for Backpropagation Networks (1992) Neural Computation, 4 (3), pp. 448-472MacKay, D.J.C., Bayesian Interpolation (1992) Neural Computation, 4 (3), pp. 415-447MacKay, D.J.C., The Evidence Framework Applied to Classification Networks (1992) Neural Computation, 4 (5), pp. 698-714Moller, M.F., A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning (1993) Neural Networks, 6 (4), pp. 525-533Neal, R.M., (1996) Bayesian Learning for Neural Networks, , Springer-VerlagRussell, R., Pruning algorithm A survey (1993) IEEE Trans. on Neural Networks, 4 (5), pp. 740-747Thodberg, H.H., A review of Bayesian neural networks with an application to near infrared spectroscopy (1996) IEEE Trans. Neural Networks, 7, p. 5672Vertosick, F.T., Kelly, R.H., The Immune System as a Neural Network: A Multi-epitope Approach (1991) Journal of Theoretical Biology, 150, pp. 225-237Yang, J., Parekh, R., Honavar, V., DistAL: An interpattern distancebased constructive learning algorithm (1999) Intelligent Data Analysis, 3, pp. 55-73Yao, X., Evolving Artificial Neural Networks (1999) Proceedings of the IEEE, 87, pp. 1423-1447Zhang, B.T., A Bayesian Evolutionary Approach to the Design and Learning of Heterogeneous Neural Trees (2002) Integrated Computer-Aided Engineering, 9 (1), pp. 73-8