196 research outputs found
Towards The Deep Semantic Learning Machine Neuroevolution Algorithm: An exploration on the CIFAR-10 problem task
Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced AnalyticsSelecting the topology and parameters of Convolutional Neural Network (CNN) for a given supervised machine learning task is a non-trivial problem. The Deep Semantic Learning Machine (Deep-SLM) deals with this problem by automatically constructing CNNs without the use of the Backpropagation algorithm. The Deep-SLM is a novel neuroevolution technique and functions as stochastic semantic hill-climbing algorithm searching over the space of CNN topologies and parameters. The geometric semantic properties of the Deep-SLM induce a unimodel error space and eliminate the existence of local optimal solutions. This makes the Deep-SLM potentially favorable in terms of search efficiency and effectiveness.
This thesis provides an exploration of a variant of the Deep-SLM algorithm on the CIFAR-10 problem task, and a validation of its proof of concept. This specific variant only forms mutation node ! mutation node connections in the non-convolutional part of the constructed CNNs. Furthermore, a comparative study between the Deep-SLM and the Semantic Learning Machine (SLM) algorithms was conducted. It was observed that sparse connections can be an effective way to prevent overfitting. Additionally, it was shown that a single 2D convolution layer initialized with random weights does not result in well-generalizing features for the Deep-SLM directly, but, in combination with a 2D max-pooling down sampling layer, effective improvements in performance and generalization of the Deep-SLM could be achieved. These results constitute to the hypothesis that convolution and pooling layers can improve performance and generalization of the Deep-SLM, unless the components are properly optimized.Selecionar a topologia e os parâmetros da Rede Neural Convolucional (CNN) para uma tarefa de aprendizado automático supervisionada não é um problema trivial. A Deep Semantic Learning Machine (Deep-SLM) lida com este problema construindo automaticamente CNNs sem recorrer ao uso do algoritmo de Retro-propagação. A Deep-SLM é uma nova técnica de neuroevolução que funciona enquanto um algoritmo de
escalada estocástico semântico na pesquisa de topologias e de parâmetros CNN. As propriedades geométrico-semânticas da Deep-SLM induzem um unimodel error space que elimina a existência de soluções ótimas locais, favorecendo, potencialmente, a Deep-SLM em termos de eficiência e eficácia.
Esta tese providencia uma exploração de uma variante do algoritmo da Deep-SLM no problemo de CIFAR-10, assim como uma validação do seu conceito de prova. Esta variante específica apenas forma conexões nó de mutação!nó de mutação na parte non convolucional da CNN construída. Mais ainda, foi conduzido um estudo comparativo entre a Deep-SLM e o algoritmo da Semantic Learning Machine (SLM). Tendo sido observado
que as conexões esparsas poderão tratar-se de uma forma eficiente de prevenir o overfitting. Adicionalmente, mostrou-se que uma singular camada de convolução 2D, iniciada com valores aleatórios, não resulta, directamente, em características generalizadas para a Deep-SLM, mas, em combinação com uma camada de 2D max-pooling, melhorias efectivas na performance e na generalização da Deep-SLM poderão ser concretizadas.
Estes resultados constituem, assim, a hipótese de que as camadas de convolução e pooling poderão melhorar a performance e a generalização da Deep-SLM, a não ser que os componentes sejam adequadamente otimizados
Analysing Symbolic Regression Benchmarks under a Meta-Learning Approach
The definition of a concise and effective testbed for Genetic Programming
(GP) is a recurrent matter in the research community. This paper takes a new
step in this direction, proposing a different approach to measure the quality
of the symbolic regression benchmarks quantitatively. The proposed approach is
based on meta-learning and uses a set of dataset meta-features---such as the
number of examples or output skewness---to describe the datasets. Our idea is
to correlate these meta-features with the errors obtained by a GP method. These
meta-features define a space of benchmarks that should, ideally, have datasets
(points) covering different regions of the space. An initial analysis of 63
datasets showed that current benchmarks are concentrated in a small region of
this benchmark space. We also found out that number of instances and output
skewness are the most relevant meta-features to GP output error. Both
conclusions can help define which datasets should compose an effective testbed
for symbolic regression methods.Comment: 8 pages, 3 Figures, Proceedings of Genetic and Evolutionary
Computation Conference Companion, Kyoto, Japa
A Study of Geometric Semantic Genetic Programming with Linear Scaling
Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced Analytics, specialization in Data ScienceMachine Learning (ML) is a scientific discipline that endeavors to enable computers
to learn without the need for explicit programming. Evolutionary Algorithms (EAs),
a subset of ML algorithms, mimic Darwin’s Theory of Evolution by using natural
selection mechanisms (i.e., survival of the fittest) to evolve a group of individuals
(i.e., possible solutions to a given problem). Genetic Programming (GP) is the most
recent type of EA and it evolves computer programs (i.e., individuals) to map a set of
input data into known expected outputs. Geometric Semantic Genetic Programming
(GSGP) extends this concept by allowing individuals to evolve and vary in the semantic
space, where the output vectors are located, rather than being constrained by syntaxbased
structures. Linear Scaling (LS) is a method that was introduced to facilitate the
task of GP of searching for the best function matching a set of known data. GSGP
and LS have both, independently, shown the ability to outperform standard GP for
symbolic regression. GSGP uses Geometric Semantic Operators (GSOs), different
from the standard ones, without altering the fitness, while LS modifies the fitness
without altering the genetic operators. To the best of our knowledge, there has been
no prior utilization of the combined methodology of GSGP and LS for classification
problems. Furthermore, despite the fact that they have been used together in one
practical regression application, a methodological evaluation of the advantages and
disadvantages of integrating these methods for regression or classification problems
has never been performed. In this dissertation, a study of a system that integrates both
GSGP and LS (GSGP-LS) is presented. The performance of the proposed method, GSGPLS,
was tested on six hand-tailored regression benchmarks, nine real-life regression
problems and three real-life classification problems. The obtained results indicate that
GSGP-LS outperforms GSGP in the majority of the cases, confirming the expected
benefit of this integration. However, for some particularly hard regression datasets,
GSGP-LS overfits training data, being outperformed by GSGP on unseen data. This
contradicts the idea that LS is always beneficial for GP, warning the practitioners about
its risk of overfitting in some specific cases.A Aprendizagem Automática (AA) é uma disciplina científica que se esforça por
permitir que os computadores aprendam sem a necessidade de programação explícita.
Algoritmos Evolutivos (AE),um subconjunto de algoritmos de ML, mimetizam a Teoria
da Evolução de Darwin, usando a seleção natural e mecanismos de "sobrevivência dos
mais aptos"para evoluir um grupo de indivíduos (ou seja, possíveis soluções para
um problema dado). A Programação Genética (PG) é um processo algorítmico que
evolui programas de computador (ou indivíduos) para ligar características de entrada e
saída. A Programação Genética em Geometria Semântica (PGGS) estende esse conceito
permitindo que os indivíduos evoluam e variem no espaço semântico, onde os vetores
de saída estão localizados, em vez de serem limitados por estruturas baseadas em
sintaxe. A Escala Linear (EL) é um método introduzido para facilitar a tarefa da PG de
procurar a melhor função que corresponda a um conjunto de dados conhecidos. Tanto
a PGGS quanto a EL demonstraram, independentemente, a capacidade de superar a
PG padrão para regressão simbólica. A PGGS usa Operadores Semânticos Geométricos
(OSGs), diferentes dos padrões, sem alterar o fitness, enquanto a EL modifica o fitness
sem alterar os operadores genéticos. Até onde sabemos, não houve utilização prévia
da metodologia combinada de PGGS e EL para problemas de classificação. Além disso,
apesar de terem sido usados juntos em uma aplicação prática de regressão, nunca foi
realizada uma avaliação metodológica das vantagens e desvantagens da integração
desses métodos para problemas de regressão ou classificação. Nesta dissertação, é
apresentado um estudo de um sistema que integra tanto a PGGS quanto a EL (PGGSEL).
O desempenho do método proposto, PGGS-EL, foi testado em seis benchmarks de
regressão personalizados, nove problemas de regressão da vida real e três problemas
de classificação da vida real. Os resultados obtidos indicam que o PGGS-EL supera
o PGGS na maioria dos casos, confirmando o benefício esperado desta integração.
No entanto, para alguns conjuntos de dados de regressão particularmente difíceis, o
PGGS-EL faz overfit aos dados de treino, obtendo piores resultados em comparação com
PGGS em dados não vistos. Isso contradiz a ideia de que a EL é sempre benéfica para
a PG, alertando os praticantes sobre o risco de overfitting em alguns casos específicos
Comparison of semantic-based local search methods for multiobjective genetic programming
We report a series of experiments that use semantic-based local search within a multiobjective genetic programming (GP) framework. We compare various ways of selecting target subtrees for local search as well as different methods for performing that search; we have also made comparison with the random desired operator of Pawlak et al. using statistical hypothesis testing. We find that a standard steady state or generational GP followed by a carefully-designed single-objective GP implementing semantic-based local search produces models that are mode accurate and with statistically smaller (or equal) tree size than those generated by the corresponding baseline GP algorithms. The depth fair selection strategy of Ito et al. is found to perform best compared with other subtree selection methods in the model refinement
Mining Explicit and Implicit Relationships in Data Using Symbolic Regression
Identification of implicit and explicit relations within observed data is a generic problem commonly encountered in several domains including science, engineering, finance, and more. It forms the core component of data analytics, a process of discovering useful information from data sets that are potentially huge and otherwise incomprehensible. In industries, such information is often instrumental for profitable decision making, whereas in science and engineering it is used to build empirical models, propose new or verify existing theories and explain natural phenomena. In recent times, digital and internet based technologies have proliferated, making it viable to generate and collect large amount of data at low cost. This inturn has resulted in an ever growing need for methods to analyse and draw interpretations from such data quickly and reliably. With this overarching goal, this thesis attempts to make contributions towards developing accurate and efficient methods for discovering such relations through evolutionary search, a method commonly referred to as Symbolic Regression (SR).
A data set of input variables x and a corresponding observed response y is given. The aim is to find an explicit function y = f (x) or an implicit function f (x, y) = 0, which represents the data set. While seemingly simple, the problem is challenging for several reasons. Some of the conventional regression methods try to “guess” a functional form such as linear/quadratic/polynomial, and attempt to do a curve-fitting of the data to the equation, which may limit the possibility of discovering more complex relations, if they exist. On the other hand, there are meta-modelling techniques such as response surface method, Kriging, etc., that model the given data accurately, but provide a “black-box” predictor instead of an expression. Such approximations convey little or no insights about how the variables and responses are dependent on each other, or their relative contribution to the output. SR attempts to alleviate the above two extremes by providing a structure which evolves mathematical expressions instead of assuming them. Thus, it is flexible enough to represent the data, but at the same time provides useful insights instead of a black-box predictor. SR can be categorized as part of Explainable Artificial Intelligence and can contribute to Trustworthy Artificial Intelligence.
The works proposed in this thesis aims to integrate the concept of “semantics” deeper into Genetic Programming (GP) and Evolutionary Feature Synthesis, which are the two algorithms usually employed for conducting SR. The semantics will be integrated into well-known components of the algorithms such as compactness, diversity, recombination, constant optimization, etc. The main contribution of this thesis is the proposal of two novel operators to generate expressions based on Linear Programming and Mixed Integer Programming with the aim of controlling the length of the discovered expressions without compromising on the accuracy. In the experiments, these operators are proven to be able to discover expressions with better accuracy and interpretability on many explicit and implicit benchmarks. Moreover, some applications of SR on real-world data sets are shown to demonstrate the practicality of the proposed approaches. Besides, in related to practical problems, how GP can be applied to effectively solve the Resource Constrained Scheduling Problems is also presented
- …