Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Optimizing Weights And Biases in MLP Using Whale Optimization Algorithm

Artificial Neural Networks are intelligent and non-parametric mathematical models inspired by the human nervous system. They have been widely studied and applied for classification, pattern recognition and forecasting problems. The main challenge of training an Artificial Neural network is its learning process, the nonlinear nature and the unknown best set of main controlling parameters (weights and biases). When the Artificial Neural Networks are trained using the conventional training algorithm, they get caught in the local optima stagnation and slow convergence speed; this makes the stochastic optimization algorithm a definitive alternative to alleviate the drawbacks. This thesis proposes an algorithm based on the recently proposed Whale Optimization Algorithm(WOA). The algorithm has proven to solve a wide range of optimization problems and outperform existing algorithms. The successful implementation of this algorithm motivated our attempts to benchmark its performance in training feed-forward neural networks. We have taken a set of 20 datasets with different difficulty levels and tested the proposed WOA-MLP based trainer. Further, the results are verified by comparing WOA-MLP with the back propagation algorithms and six evolutionary techniques. The results have proved that the proposed trainer can outperform the current algorithms on the majority of datasets in terms of local optima avoidance and convergence speed

Deep Neuroevolution: Smart City Applications

Particularmente, la contribución de esta tesis se centra en cuatro aspectos: Primero, proponemos la técnica Mean Absolute Error Random Sampling (MRS) para estimar el rendimiento de una RNN, la cual se basa en la distribución del error observado en un muestreo aleatorio. Nuestros resultados muestran que MRS es una estimación fiable y de bajo coste computacional para predecir el rendimiento de una RNN. Segundo, diseñamos un algoritmo evolutivo (RESN) que explota MRS para optimizar la arquitectura de una RNN. RESN muestra resultados competitivos a la vez que reduce significativamente el tiempo. Tercero, en el contexto de la aplicación, proponemos soluciones para problemas de movilidad, electricidad y gestión de residuos inteligente, y hemos revisado el estado del arte de la ciudad inteligente y su relación con la informática. Cuarto, hemos desarrollado la biblioteca de software Deep Learning OPTimization (DLOPT), la cual está disponible bajo la licencia GNU GPL v3. Ésta contiene la mayor parte del trabajo realizado en esta tesis.El interés por desarrollar redes neuronales artificiales ha resurgido de la mano del Aprendizaje Profundo. En términos simples, el aprendizaje profundo consiste en diseñar y entrenar una red neuronal de gran complejidad y tamaño con una inmensa cantidad de datos. Esta creciente complejidad propone nuevos desafíos, siendo de especial relevancia la optimización del diseño dado un problema. Tradicionalmente, este problema ha sido resuelto en una combinación de conocimiento experto (humano) con prueba y error. Sin embargo, conforme la complejidad aumenta, este acercamiento se vuelve ineficiente (e impracticable). Esta tesis doctoral aborda el diseño de redes neuronales recurrentes (RNN), un tipo de red neuronal profunda, desde la neuroevolución. Concretamente, se combinan técnicas de aprendizaje automático con metaheurísticas avanzadas, con el fin de proveer una solución eficaz y eficiente. Por otra parte, se aplican las técnicas desarrolladas a problemas de la ciudad inteligente

An Improved Bees Algorithm for Training Deep Recurrent Networks for Sentiment Classification

Recurrent neural networks (RNNs) are powerful tools for learning information from temporal sequences. Designing an optimum deep RNN is difficult due to configuration and training issues, such as vanishing and exploding gradients. In this paper, a novel metaheuristic optimisation approach is proposed for training deep RNNs for the sentiment classification task. The approach employs an enhanced Ternary Bees Algorithm (BA-3+), which operates for large dataset classification problems by considering only three individual solutions in each iteration. BA-3+ combines the collaborative search of three bees to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. Local learning with exploitative search utilises the greedy selection strategy. Stochastic gradient descent (SGD) learning with singular value decomposition (SVD) aims to handle vanishing and exploding gradients of the decision parameters with the stabilisation strategy of SVD. Global learning with explorative search achieves faster convergence without getting trapped at local optima to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. BA-3+ has been tested on the sentiment classification task to classify symmetric and asymmetric distribution of the datasets from different domains, including Twitter, product reviews, and movie reviews. Comparative results have been obtained for advanced deep language models and Differential Evolution (DE) and Particle Swarm Optimization (PSO) algorithms. BA-3+ converged to the global minimum faster than the DE and PSO algorithms, and it outperformed the SGD, DE, and PSO algorithms for the Turkish and English datasets. The accuracy value and F1 measure have improved at least with a 30–40% improvement than the standard SGD algorithm for all classification datasets. Accuracy rates in the RNN model trained with BA-3+ ranged from 80% to 90%, while the RNN trained with SGD was able to achieve between 50% and 60% for most datasets. The performance of the RNN model with BA-3+ has as good as for Tree-LSTMs and Recursive Neural Tensor Networks (RNTNs) language models, which achieved accuracy results of up to 90% for some datasets. The improved accuracy and convergence results show that BA-3+ is an efficient, stable algorithm for the complex classification task, and it can handle the vanishing and exploding gradients problem of deep RNNs

Soft Computing Techiniques for the Protein Folding Problem on High Performance Computing Architectures

The protein-folding problem has been extensively studied during the last fifty years. The understanding of the dynamics of global shape of a protein and the influence on its biological function can help us to discover new and more effective drugs to deal with diseases of pharmacological relevance. Different computational approaches have been developed by different researchers in order to foresee the threedimensional arrangement of atoms of proteins from their sequences. However, the computational complexity of this problem makes mandatory the search for new models, novel algorithmic strategies and hardware platforms that provide solutions in a reasonable time frame. We present in this revision work the past and last tendencies regarding protein folding simulations from both perspectives; hardware and software. Of particular interest to us are both the use of inexact solutions to this computationally hard problem as well as which hardware platforms have been used for running this kind of Soft Computing techniques.This work is jointly supported by the FundaciónSéneca (Agencia Regional de Ciencia y Tecnología, Región de Murcia) under grants 15290/PI/2010 and 18946/JLI/13, by the Spanish MEC and European Commission FEDER under grant with reference TEC2012-37945-C02-02 and TIN2012-31345, by the Nils Coordinated Mobility under grant 012-ABEL-CM-2014A, in part financed by the European Regional Development Fund (ERDF). We also thank NVIDIA for hardware donation within UCAM GPU educational and research centers.Ingeniería, Industria y Construcció

Design Analysis and Implementation of Stock Market Forecasting System using Improved Soft Computing Technique

In this paper, a stock market prediction model was created utilizing artificial neural networks. Many people nowadays are attempting to predict future trends in bonds, currencies, equities, and stock markets. It is quite challenging for a capitalist and an industry to forecast changes in stock market prices. Due to the numerous economic, political, and psychological aspects at play, forecasting future value changes on the stock markets is quite challenging. In addition, stock market forecasting is a difficult endeavor because it relies on a wide range of known and unknown variables. Many approaches, including technical analysis, fundamental analysis, time series analysis, and statistical analysis are used to attempt to predict the share price; however, none of these methods has been demonstrated to be a consistently effective prediction tool. Artificial neural networks (ANNs), a subfield of artificial intelligence, are one of the most modern and promising methods for resolving financial issues, such as categorizing corporate bonds and anticipating stock market indexes and bankruptcy (AI). Artificial neural networks (ANN) are a prominent technology used to forecast the future of the stock market. In order to understand financial time series, it is often essential to extract relevant information from enormous data sets using artificial neural networks. An outcome prediction neural network with three layers is trained using the back propagation method. Analysis shows that ANN outperforms every other prediction technique now available to academics in terms of stock market price predictions. It is concluded that ANN is a useful technique for predicting stock market movements globally

An optimized deep learning model for optical character recognition applications

The convolutional neural networks (CNN) are among the most utilized neural networks in various applications, including deep learning. In recent years, the continuing extension of CNN into increasingly complicated domains has made its training process more difficult. Thus, researchers adopted optimized hybrid algorithms to address this problem. In this work, a novel chaotic black hole algorithm-based approach was created for the training of CNN to optimize its performance via avoidance of entrapment in the local minima. The logistic chaotic map was used to initialize the population instead of using the uniform distribution. The proposed training algorithm was developed based on a specific benchmark problem for optical character recognition applications; the proposed method was evaluated for performance in terms of computational accuracy, convergence analysis, and cost

Optimization of a Steam Reforming Plant Modeled with Artificial Neural Networks

The objective of this research is to improve the hydrogen production and total profit of a real Steam Reforming plant. Given the impossibility of tuning the real factory to optimize its operation, we propose modelling the plant using Artificial Neural Networks (ANNs). Particularly, we combine a set of independent ANNs into a single model. Each ANN uses different sets of inputs depending on the physical processes simulated. The model is then optimized as a black-box system using metaheuristics (Genetic and Memetic Algorithms). We demonstrate that the proposed ANN model presents a high correlation between the real output and the predicted one. Additionally, the performance of the proposed optimization techniques has been validated by the engineers of the plant, who reported a significant increase in the benefit that was obtained after optimization. Furthermore, this approach has been favorably compared with the results that were provided by a general black-box solver. All methods were tested over real data that were provided by the factory.Ministerio de Ciencia, Innovación y Universidades PGC2018-095322-B-C22Comunidad de Madrid P2018/TCS-4566Unión Europea P2018/TCS-456

Enhanced Deep Network Designs Using Mitochondrial DNA Based Genetic Algorithm And Importance Sampling

Machine learning (ML) is playing an increasingly important role in our lives. It has already made huge impact in areas such as cancer diagnosis, precision medicine, self-driving cars, natural disasters predictions, speech recognition, etc. The painstakingly handcrafted feature extractors used in the traditional learning, classification and pattern recognition systems are not scalable for large-sized datasets or adaptable to different classes of problems or domains. Machine learning resurgence in the form of Deep Learning (DL) in the last decade after multiple AI (artificial intelligence) winters and hype cycles is a result of the convergence of advancements in training algorithms, availability of massive data (big data) and innovation in compute resources (GPUs and cloud). If we want to solve more complex problems with machine learning, we need to optimize all three of these areas, i.e., algorithms, dataset and compute. Our dissertation research work presents the original application of nature-inspired idea of mitochondrial DNA (mtDNA) to improve deep learning network design. Additional fine-tuning is provided with Monte Carlo based method called importance sampling (IS). The primary performance indicators for machine learning are model accuracy, loss and training time. The goal of our dissertation is to provide a framework to address all these areas by optimizing network designs (in the form of hyperparameter optimization) and dataset using enhanced Genetic Algorithm (GA) and importance sampling. Algorithms are by far the most important aspect of machine learning. We demonstrate the application of mitochondrial DNA to complement the standard genetic algorithm for architecture optimization of deep Convolution Neural Network (CNN). We use importance sampling to reduce the dataset variance and sample more often from the instances that add greater value from the training outcome perspective. And finally, we leverage massive parallel and distributed processing of GPUs in the cloud to speed up training. Thus, our multi-approach method for enhancing deep learning combines architecture optimization, dataset optimization and the power of the cloud to drive better model accuracy and reduce training time