The Influence Of The Pool Of Candidates On The Performance Of Selection And Combination Techniques In Ensembles

Abstract

In this paper, we propose the use of an immune-inspired approach called opt-aiNet to generate a diverse set of high-performance candidates to compose an ensemble of neural network classifiers. Being a population-based search algorithm, the opt-aiNet is capable of maintaining diversity and finding many high-performance solutions simultaneously, which are known to be desired features when synthesizing an ensemble. Concerning the selection and combination phases, the most relevant selection and combination techniques already proposed in the literature have been considered. The main contribution of this paper is the indication that there is no pair of selection/combination technique that can be considered the best one, because the performance of the obtained ensemble varies significantly with the current composition of the pool of candidates already produced by the generation phase. Notwithstanding, this variability in performance is not restricted to the choice of opt-aiNet as the generative device. As a consequence, to overcome the performance of the best individual classifier, every possible pairs of selection and combination techniques should be tried. Only with such an exhaustive search (notice that the main computational burden is usually related to the generation phase), the performance of the ensemble was invariably superior to the performance of the best individual classifier on four benchmark classification problems. ©2006 IEEE.51325139Hansen, L., Salamon, P., Neural network ensembles (1990) IEEE Trans. on Pattern Anal. and Machine Intelligence, 12, pp. 993-1005Hashem, S., Schmeiser, B., Yih, Y., Optimal linear combinations of neural, networks: An overview (1994) Proc. IEEE Int. Conf. on Neural Networks, , OrlandoBrown, G., Wyatt, J., Harris, R., Yao, X., Diversity Creation Methods: A Survey and Categorisation (2005) Journal of Information Fusion, 6 (1), pp. 5-20. , JanuaryLiu, Y., Negative correlation learning and evolutionary neural network ensembles, (1998), Ph.D. thesis, University College, The University of New South Wales, Australian Defense Force Academy, Canberra(1999) Artificial Immune Systems and their Applications, , D. Dasgupta ed, Springer-Verlagde Castro, L.N., Timmis, J., (2002) An Introduction to Artificial Immune Systems: A New Computational Intelligence Paradigm, , Springer-Verlagde Castro, L.N., Timmis, J., An artificial immune network for multimodal function optimization (2002) Proc. 2002 Conf. on Evolutionary Computation, 1, pp. 699-704. , 12-17 MayCastro, P.A.D., Coelho, G.P., Caetano, M., Von Zuben, F.J., Ensembles of Fuzzy Classification Systems: An Immune-Inspired Approach (2005) Lecture Notes in Computer Science, 3627, pp. 469-482. , C. Jacob et al. editors, Springer-VerlagG. P. Coelho, P. A. D. Castro, and F. J. Von Zuben, The Effective use of Diverse Rule Bases in Fuzzy Classification. In A. D. D. Neto et al. eds., VII Congresso Brasileiro de Redes Neurais, Brazil, 2005Zhou, Z.H., Wu, J., Tang, W., Ensembling Neural Networks: Many Could be Better Than All (2002) Artificial Intelligence, 137 (1-2), pp. 239-263Bezerra, G.B., Barra, T.V., de Castro, L.N., Von Zuben, F.J., Adaptive Radius Immune Algorithm for Data Clustering (2005) Lecture Notes in Computer Science, 3627, pp. 290-303. , C. Jacob et al. editors, Springer-VerlagTorres-Sospedra, J., Fernández-Redondo, M., Hernández-Espinosa, C., A Research on Combination. Methods for Ensembles of Multilayer Feedforward (2005) Proc. Int. Joint Conf. on Neural Networks, pp. 1125-1130L. N. de Castro, and F. J. Von Zuben, aiNet: An Artificial Immune Network for Data Analysis. In H. A. Abbass, R. A. Sarker, and C. S. Newton, editors, Data Mining: A Heuristic Approach, Idea Group Publishing, USA, Chapter XII, pp. 231-259, 2001Attux, R.R.F., Loiola, M.B., Suyama, R., de Castro, L.N., Von Zuben, F.J., Romano, J.M.T., Blind Search for Optimal Wiener Equalizers Using an Artificial Immune Network Model (2003) EURASIP J. on Applied Signal Proc, 2003 (8), pp. 740-747Coello Coello, C.A., Cruz Cortés, N., Solving Multiobjective Optimization Problems Using an Artificial Immune System (2005) Genetic Programming and Evolvable Machines, 6 (2), pp. 163-190Ada, G.L., Nossal, G.J.V., The Clonal Selection Theory (1987) Scientific American, 257 (2), pp. 50-57N. K. Jerne, Towards a Network Theory of the Immune System. Ann. Immunol., (Inst. Pasteur) 125C, pp. 373-389, 1974Perrone, M.P., Cooper, L.N., When networks disagree: Ensemble method for neural networks (1993) Artificial Neural Networks for Speech and Vision, pp. 126-142. , R. J. Mamone, ed, Chapman & Hall, ppLiu, Y., Yao, S., Higuchi, T., Evolutionary Ensembles with Negative Correlation Learning (2000) IEEE Trans. on Evolutionary Computation, 4 (4), pp. 380-387. , NovemberBlake, C.L., Merz, C.J., (1998) UCI Repository of machine learning databases, , Irvine, CA: University of California, Department of Information and Computer ScienceInoue, H., Narihisa, H., Effective Online Pruning Method for Ensemble Self-Generating Neural Networks (2004) Proc. of IEEE Int. Midwest Symp. on Circuits and Systems, pp.III-85-III-88Lazarevic, A., Obradovic, Z., The Effective Pruning of Neural Network Ensembles (2001) Proc. IEEE/INNS Int. Conf. on Neural Neural Networks, pp. 796-801. , Washington, D.C, pp, Jul