Semantic variation operators for multidimensional genetic programming

Multidimensional genetic programming represents candidate solutions as sets of programs, and thereby provides an interesting framework for exploiting building block identification. Towards this goal, we investigate the use of machine learning as a way to bias which components of programs are promoted, and propose two semantic operators to choose where useful building blocks are placed during crossover. A forward stagewise crossover operator we propose leads to significant improvements on a set of regression problems, and produces state-of-the-art results in a large benchmark study. We discuss this architecture and others in terms of their propensity for allowing heuristic search to utilize information during the evolutionary process. Finally, we look at the collinearity and complexity of the data representations that result from these architectures, with a view towards disentangling factors of variation in application.Comment: 9 pages, 8 figures, GECCO 201

Combating catastrophic forgetting with developmental compression

Generally intelligent agents exhibit successful behavior across problems in several settings. Endemic in approaches to realize such intelligence in machines is catastrophic forgetting: sequential learning corrupts knowledge obtained earlier in the sequence, or tasks antagonistically compete for system resources. Methods for obviating catastrophic forgetting have sought to identify and preserve features of the system necessary to solve one problem when learning to solve another, or to enforce modularity such that minimally overlapping sub-functions contain task specific knowledge. While successful, both approaches scale poorly because they require larger architectures as the number of training instances grows, causing different parts of the system to specialize for separate subsets of the data. Here we present a method for addressing catastrophic forgetting called developmental compression. It exploits the mild impacts of developmental mutations to lessen adverse changes to previously-evolved capabilities and `compresses' specialized neural networks into a generalized one. In the absence of domain knowledge, developmental compression produces systems that avoid overt specialization, alleviating the need to engineer a bespoke system for every task permutation and suggesting better scalability than existing approaches. We validate this method on a robot control problem and hope to extend this approach to other machine learning domains in the future

An adaptive and modular framework for evolving deep neural networks

Santos, F. J. J. B., Gonçalves, I., & Castelli, M. (2023). Neuroevolution with box mutation: An adaptive and modular framework for evolving deep neural networks. Applied Soft Computing, 147(November), 1-15. [110767]. https://doi.org/10.1016/j.asoc.2023.110767 --- Funding: This work is funded by national funds through the FCT - Foundation for Science and Technology, I.P., within the scope of the projects CISUC - UID/CEC/00326/2020, UIDB/04152/2020 - Centro de Investigação em Gestão de Informação (MagIC)/NOVA IMS, and by European Social Fund, through the Regional Operational Program Centro 2020 .The pursuit of self-evolving neural networks has driven the emerging field of Evolutionary Deep Learning, which combines the strengths of Deep Learning and Evolutionary Computation. This work presents a novel method for evolving deep neural networks by adapting the principles of Geometric Semantic Genetic Programming, a subfield of Genetic Programming, and Semantic Learning Machine. Our approach integrates evolution seamlessly through natural selection with the optimization power of backpropagation in deep learning, enabling the incremental growth of neural networks’ neurons across generations. By evolving neural networks that achieve nearly 89% accuracy on the CIFAR-10 dataset with relatively few parameters, our method demonstrates remarkable efficiency, evolving in GPU minutes compared to the field standard of GPU days.publishersversionpublishe

Towards The Deep Semantic Learning Machine Neuroevolution Algorithm: An exploration on the CIFAR-10 problem task

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced AnalyticsSelecting the topology and parameters of Convolutional Neural Network (CNN) for a given supervised machine learning task is a non-trivial problem. The Deep Semantic Learning Machine (Deep-SLM) deals with this problem by automatically constructing CNNs without the use of the Backpropagation algorithm. The Deep-SLM is a novel neuroevolution technique and functions as stochastic semantic hill-climbing algorithm searching over the space of CNN topologies and parameters. The geometric semantic properties of the Deep-SLM induce a unimodel error space and eliminate the existence of local optimal solutions. This makes the Deep-SLM potentially favorable in terms of search efficiency and effectiveness. This thesis provides an exploration of a variant of the Deep-SLM algorithm on the CIFAR-10 problem task, and a validation of its proof of concept. This specific variant only forms mutation node ! mutation node connections in the non-convolutional part of the constructed CNNs. Furthermore, a comparative study between the Deep-SLM and the Semantic Learning Machine (SLM) algorithms was conducted. It was observed that sparse connections can be an effective way to prevent overfitting. Additionally, it was shown that a single 2D convolution layer initialized with random weights does not result in well-generalizing features for the Deep-SLM directly, but, in combination with a 2D max-pooling down sampling layer, effective improvements in performance and generalization of the Deep-SLM could be achieved. These results constitute to the hypothesis that convolution and pooling layers can improve performance and generalization of the Deep-SLM, unless the components are properly optimized.Selecionar a topologia e os parâmetros da Rede Neural Convolucional (CNN) para uma tarefa de aprendizado automático supervisionada não é um problema trivial. A Deep Semantic Learning Machine (Deep-SLM) lida com este problema construindo automaticamente CNNs sem recorrer ao uso do algoritmo de Retro-propagação. A Deep-SLM é uma nova técnica de neuroevolução que funciona enquanto um algoritmo de escalada estocástico semântico na pesquisa de topologias e de parâmetros CNN. As propriedades geométrico-semânticas da Deep-SLM induzem um unimodel error space que elimina a existência de soluções ótimas locais, favorecendo, potencialmente, a Deep-SLM em termos de eficiência e eficácia. Esta tese providencia uma exploração de uma variante do algoritmo da Deep-SLM no problemo de CIFAR-10, assim como uma validação do seu conceito de prova. Esta variante específica apenas forma conexões nó de mutação!nó de mutação na parte non convolucional da CNN construída. Mais ainda, foi conduzido um estudo comparativo entre a Deep-SLM e o algoritmo da Semantic Learning Machine (SLM). Tendo sido observado que as conexões esparsas poderão tratar-se de uma forma eficiente de prevenir o overfitting. Adicionalmente, mostrou-se que uma singular camada de convolução 2D, iniciada com valores aleatórios, não resulta, directamente, em características generalizadas para a Deep-SLM, mas, em combinação com uma camada de 2D max-pooling, melhorias efectivas na performance e na generalização da Deep-SLM poderão ser concretizadas. Estes resultados constituem, assim, a hipótese de que as camadas de convolução e pooling poderão melhorar a performance e a generalização da Deep-SLM, a não ser que os componentes sejam adequadamente otimizados

A study with the semantic learning machine

Lapa, P., Rundo, L., Gonçalves, I., & Castelli, M. (2019). Enhancing classification performance of convolutional neural networks for prostate cancer detection on magnetic resonance images: A study with the semantic learning machine. In GECCO 2019 : Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion (pp. 381-382). (GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion). Association for Computing Machinery, Inc. https://doi.org/10.1145/3319619.3322035 --- This work was partially supported by projects UID/MULTI/00308/2019 and by the European Regional Development Fund through the COMPETE 2020 Programme, FCT - Portuguese Foundation for Science and Technology and Regional Operational Program of the Center Region (CENTRO2020) within project MAnAGER (POCI-01-0145-FEDER-028040). This work was also partially supported by national funds through FCT (Fundação para a Ciência e a Tecnologia) under project DSAIPA/DS/0022/2018 (GADgET).Prostate cancer (PCa) is the most common oncological disease in Western men. Even though a significant effort has been carried out by the scientific community, accurate and reliable automated PCa detection methods are still a compelling issue. In this clinical scenario, high-resolution multiparametric Magnetic Resonance Imaging (MRI) is becoming the most used modality, also enabling quantitative studies. Recently, deep learning techniques have achieved outstanding results in prostate MRI analysis tasks, in particular with regard to image classification. This paper studies the feasibility of using the Semantic Learning Machine (SLM) neuroevolution algorithm to replace the fully-connected architecture commonly used in the last layers of Convolutional Neural Networks (CNNs). The experimental phase considered the PROSTATEx dataset composed of multispectral MRI sequences. The achieved results show that, on the same non-contrast-enhanced MRI series, SLM outperforms with statistical significance a state-of-the-art CNN trained with backpropagation. The SLM performance is achieved without pre-training the underlying CNN with backpropagation. Furthermore, on average the SLM training time is approximately 14 times faster than the backpropagation-based approach.authorsversionpublishe

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

A variety of methods have been applied to the architectural configuration and learning or training of artificial deep neural networks (DNN). These methods play a crucial role in the success or failure of the DNN for most problems and applications. Evolutionary Algorithms (EAs) are gaining momentum as a computationally feasible method for the automated optimisation and training of DNNs. Neuroevolution is a term which describes these processes of automated configuration and training of DNNs using EAs. While many works exist in the literature, no comprehensive surveys currently exist focusing exclusively on the strengths and limitations of using neuroevolution approaches in DNNs. Prolonged absence of such surveys can lead to a disjointed and fragmented field preventing DNNs researchers potentially adopting neuroevolutionary methods in their own research, resulting in lost opportunities for improving performance and wider application within real-world deep learning problems. This paper presents a comprehensive survey, discussion and evaluation of the state-of-the-art works on using EAs for architectural configuration and training of DNNs. Based on this survey, the paper highlights the most pertinent current issues and challenges in neuroevolution and identifies multiple promising future research directions.Comment: 20 pages (double column), 2 figures, 3 tables, 157 reference

Explorations of the semantic learning machine neuroevolution algorithm: dynamic training data use and ensemble construction methods

Dissertation presented as the partial requirement for obtaining a Master’s degree in Data Science and Advanced AnalyticsAs the world’s technology evolves, the power to implement new and more efficient algorithms increases but so does the complexity of the problems at hand. Neuroevolution algorithms fit in this context in the sense that they are able to evolve Artificial Neural Networks (ANNs). The recently proposed Neuroevolution algorithm called Semantic Learning Machine (SLM) has the advantage of searching over unimodal error landscapes in any Supervised Learning task where the error is measured as a distance to the known targets. The absence of local optima in the search space results in a more efficient learning when compared to other neuroevolution algorithms. This work studies how different approaches of dynamically using the training data affect the generalization of the SLM algorithm. Results show that these methods can be useful in offering different alternatives to achieve a superior generalization. These approaches are evaluated experimentally in fifteen real-world binary classification data sets. Across these fifteen data sets, results show that the SLM is able to outperform the Multilayer Perceptron (MLP) in 13 out of the 15 considered problems with statistical significance after parameter tuning was applied to both algorithms. Furthermore, this work also considers how different ensemble construction methods such as a simple averaging approach, Bagging and Boosting affect the resulting generalization of the SLM and MLP algorithms. Results suggest that the stochastic nature of the SLM offers enough diversity to the base learner in a way that a simple averaging method can be competitive when compared to more complex techniques like Bagging and Boosting.À medida que a tecnologia evolui, a possibilidade de implementar algoritmos novos e mais eficientes aumenta, no entanto, a complexidade dos problemas com que nos deparamos também se torna maior. Algoritmos de Neuroevolution encaixam-se neste contexto, na medida em que são capazes de evoluir Artificial Neural Networks (ANNs). O algoritmo de Neuroevolution recentemente proposto chamado Semantic Learning Machine (SLM) tem a vantagem de procurar sobre landscapes de erros unimodais em qualquer problema de Supervised Learning, onde o erro é medido como a distância aos alvos conhecidos. A não existência de local optima no espaço de procura resulta numa aprendizagem mais eficiente quando comparada com outros algoritmos de Neuroevolution. Este trabalho estuda como métodos diferentes de uso dinâmico de dados de treino afeta a generalização do algoritmo SLM. Os resultados mostram que estes métodos são úteis a oferecer uma alternativa que atinge uma generalização competitiva. Estes métodos são testados em quinze problemas reais de classificação binária. Nestes quinze problemas, o algoritmo SLM mostra superioridade ao Multilayer Perceptron (MLP) em treze deles com significância estatística depois de ser aplicado parameter tuning em ambos os algoritmos. Para além disso, este trabalho também considera como diferentes métodos de construção de ensembles, tal como um simples método de averaging, Bagging e Boosting afetam os valores de generalização dos algoritmos SLM e MLP. Os resultados sugerem que a natureza estocástica da SLM oferece diversidade suficiente aos base learners de maneira a que o método mais simples de construção de ensembles se torne competitivo quando comparado com técnicas mais complexas como Bagging e Boosting