74,243 research outputs found
Unstable Dynamics, Nonequilibrium Phases and Criticality in Networked Excitable Media
Here we numerically study a model of excitable media, namely, a network with
occasionally quiet nodes and connection weights that vary with activity on a
short-time scale. Even in the absence of stimuli, this exhibits unstable
dynamics, nonequilibrium phases -including one in which the global activity
wanders irregularly among attractors- and 1/f noise while the system falls into
the most irregular behavior. A net result is resilience which results in an
efficient search in the model attractors space that can explain the origin of
certain phenomenology in neural, genetic and ill-condensed matter systems. By
extensive computer simulation we also address a relation previously conjectured
between observed power-law distributions and the occurrence of a "critical
state" during functionality of (e.g.) cortical networks, and describe the
precise nature of such criticality in the model.Comment: 18 pages, 9 figure
Mining continuous classes using evolutionary computing
Data mining is the term given to knowledge discovery paradigms that attempt to infer knowledge, in the form of rules, from structured data using machine learning algorithms. Specifically, data mining attempts to infer rules that are accurate, crisp, comprehensible and interesting. There are not many data mining algorithms for mining continuous classes. This thesis develops a new approach for mining continuous classes. The approach is based on a genetic program, which utilises an efficient genetic algorithm approach to evolve the non-linear regressions described by the leaf nodes of individuals in the genetic program's population. The approach also optimises the learning process by using an efficient, fast data clustering algo¬rithm to reduce the training pattern search space. Experimental results from both algorithms are compared with results obtained from a neural network. The experimental results of the genetic program is also compared against a commercial data mining package (Cubist). These results indicate that the genetic algorithm technique is substantially faster than the neural network, and produces comparable accuracy. The genetic program produces substantially less complex rules than that of both the neural network and Cubist.Dissertation (MSc)--University of Pretoria, 2006.Computer Scienceunrestricte
EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
Designing the structure of neural networks is considered one of the most
challenging tasks in deep learning, especially when there is few prior
knowledge about the task domain. In this paper, we propose an
Ecologically-Inspired GENetic (EIGEN) approach that uses the concept of
succession, extinction, mimicry, and gene duplication to search neural network
structure from scratch with poorly initialized simple network and few
constraints forced during the evolution, as we assume no prior knowledge about
the task domain. Specifically, we first use primary succession to rapidly
evolve a population of poorly initialized neural network structures into a more
diverse population, followed by a secondary succession stage for fine-grained
searching based on the networks from the primary succession. Extinction is
applied in both stages to reduce computational cost. Mimicry is employed during
the entire evolution process to help the inferior networks imitate the behavior
of a superior network and gene duplication is utilized to duplicate the learned
blocks of novel structures, both of which help to find better network
structures. Experimental results show that our proposed approach can achieve
similar or better performance compared to the existing genetic approaches with
dramatically reduced computation cost. For example, the network discovered by
our approach on CIFAR-100 dataset achieves 78.1% test accuracy under 120 GPU
hours, compared to 77.0% test accuracy in more than 65, 536 GPU hours in [35].Comment: CVPR 201
Automated Feature Engineering for Deep Neural Networks with Genetic Programming
Feature engineering is a process that augments the feature vector of a machine learning model with calculated values that are designed to enhance the accuracy of a model’s predictions. Research has shown that the accuracy of models such as deep neural networks, support vector machines, and tree/forest-based algorithms sometimes benefit from feature engineering. Expressions that combine one or more of the original features usually create these engineered features. The choice of the exact structure of an engineered feature is dependent on the type of machine learning model in use. Previous research demonstrated that various model families benefit from different types of engineered feature. Random forests, gradient-boosting machines, or other tree-based models might not see the same accuracy gain that an engineered feature allowed neural networks, generalized linear models, or other dot-product based models to achieve on the same data set. This dissertation presents a genetic programming-based algorithm that automatically engineers features that increase the accuracy of deep neural networks for some data sets. For a genetic programming algorithm to be effective, it must prioritize the search space and efficiently evaluate what it finds. This dissertation algorithm faced a potential search space composed of all possible mathematical combinations of the original feature vector. Five experiments were designed to guide the search process to efficiently evolve good engineered features. The result of this dissertation is an automated feature engineering (AFE) algorithm that is computationally efficient, even though a neural network is used to evaluate each candidate feature. This approach gave the algorithm a greater opportunity to specifically target deep neural networks in its search for engineered features that improve accuracy. Finally, a sixth experiment empirically demonstrated the degree to which this algorithm improved the accuracy of neural networks on data sets augmented by the algorithm’s engineered features
Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
Efficient identification of people and objects, segmentation of regions of
interest and extraction of relevant data in images, texts, audios and videos
are evolving considerably in these past years, which deep learning methods,
combined with recent improvements in computational resources, contributed
greatly for this achievement. Although its outstanding potential, development
of efficient architectures and modules requires expert knowledge and amount of
resource time available. In this paper, we propose an evolutionary-based neural
architecture search approach for efficient discovery of convolutional models in
a dynamic search space, within only 24 GPU hours. With its efficient search
environment and phenotype representation, Gene Expression Programming is
adapted for network's cell generation. Despite having limited GPU resource time
and broad search space, our proposal achieved similar state-of-the-art to
manually-designed convolutional networks and also NAS-generated ones, even
beating similar constrained evolutionary-based NAS works. The best cells in
different runs achieved stable results, with a mean error of 2.82% in CIFAR-10
dataset (which the best model achieved an error of 2.67%) and 18.83% for
CIFAR-100 (best model with 18.16%). For ImageNet in the mobile setting, our
best model achieved top-1 and top-5 errors of 29.51% and 10.37%, respectively.
Although evolutionary-based NAS works were reported to require a considerable
amount of GPU time for architecture search, our approach obtained promising
results in little time, encouraging further experiments in evolutionary-based
NAS, for search and network representation improvements.Comment: Accepted for presentation at the IEEE Congress on Evolutionary
Computation (IEEE CEC) 202
Metaheuristic design of feedforward neural networks: a review of two decades of research
Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era
Genetic algorithms with DNN-based trainable crossover as an example of partial specialization of general search
Universal induction relies on some general search procedure that is doomed to
be inefficient. One possibility to achieve both generality and efficiency is to
specialize this procedure w.r.t. any given narrow task. However, complete
specialization that implies direct mapping from the task parameters to
solutions (discriminative models) without search is not always possible. In
this paper, partial specialization of general search is considered in the form
of genetic algorithms (GAs) with a specialized crossover operator. We perform a
feasibility study of this idea implementing such an operator in the form of a
deep feedforward neural network. GAs with trainable crossover operators are
compared with the result of complete specialization, which is also represented
as a deep neural network. Experimental results show that specialized GAs can be
more efficient than both general GAs and discriminative models.Comment: AGI 2017 procedding, The final publication is available at
link.springer.co
- …