An Improved Bees Algorithm for Training Deep Recurrent Networks for Sentiment Classification

Recurrent neural networks (RNNs) are powerful tools for learning information from temporal sequences. Designing an optimum deep RNN is difficult due to configuration and training issues, such as vanishing and exploding gradients. In this paper, a novel metaheuristic optimisation approach is proposed for training deep RNNs for the sentiment classification task. The approach employs an enhanced Ternary Bees Algorithm (BA-3+), which operates for large dataset classification problems by considering only three individual solutions in each iteration. BA-3+ combines the collaborative search of three bees to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. Local learning with exploitative search utilises the greedy selection strategy. Stochastic gradient descent (SGD) learning with singular value decomposition (SVD) aims to handle vanishing and exploding gradients of the decision parameters with the stabilisation strategy of SVD. Global learning with explorative search achieves faster convergence without getting trapped at local optima to find the optimal set of trainable parameters of the proposed deep recurrent learning architecture. BA-3+ has been tested on the sentiment classification task to classify symmetric and asymmetric distribution of the datasets from different domains, including Twitter, product reviews, and movie reviews. Comparative results have been obtained for advanced deep language models and Differential Evolution (DE) and Particle Swarm Optimization (PSO) algorithms. BA-3+ converged to the global minimum faster than the DE and PSO algorithms, and it outperformed the SGD, DE, and PSO algorithms for the Turkish and English datasets. The accuracy value and F1 measure have improved at least with a 30–40% improvement than the standard SGD algorithm for all classification datasets. Accuracy rates in the RNN model trained with BA-3+ ranged from 80% to 90%, while the RNN trained with SGD was able to achieve between 50% and 60% for most datasets. The performance of the RNN model with BA-3+ has as good as for Tree-LSTMs and Recursive Neural Tensor Networks (RNTNs) language models, which achieved accuracy results of up to 90% for some datasets. The improved accuracy and convergence results show that BA-3+ is an efficient, stable algorithm for the complex classification task, and it can handle the vanishing and exploding gradients problem of deep RNNs

Automatic design of deep neural network architectures with evolutionary computation

Deep Neural Networks (DNNs) are algorithms with widespread use in the extraction of knowledge from raw data. DNNs are used to solve problems in the fields of computer vision, natural language understanding, signal processing, and others. DNNs are state-of-the-art machine learning models capable of achieving better results than humans in many tasks. However, their application in fields outside computer science and engineering has been hindered due to the tedious process of trial and error multiple computationally intensive models. Thus, the development of algorithms that could allow for the automatic development of DNNs would further advance the field. Two central problems need to be addressed to allow the automatic design of DNN models: generation and pruning. The automatic generation of DNN architectures would allow for the creation of state-of-the-art models without relying on knowledge from human experts. In contrast, the automatic pruning of DNN architectures would reduce the computational complexity of such models for use in less powerful hardware. The generation and pruning of DNN models can be seen as a combinatorial optimization problem, which can be solved with the tools from the Evolutionary Computation (EC) field. This Ph.D. work proposes the use of Particle Swarm Optimization (PSO) for DNN architecture searching with competitive results and fast convergence, called psoCNN. Another algorithm based on Evolution Strategy (ES) is used for the pruning of DNN architectures, called DeepPruningES. The proposed psoCNN algorithm is capable of finding CNN architectures, a particular type of DNN, for image classification tasks with comparable results to human-crafted DNN models. Likewise, the DeepPruningES algorithm is capable of reducing the number of floating operations of a given DNN model up to 80 percent, and it uses the principles of Multi-Criteria Decision Making (MCDM) to output three pruned model with different trade-offs between computational complexity and classification accuracy. These ideas are then applied to the creation of a unified framework for searching highly accurate, and compact DNN applied for Medical Imaging Diagnostics, and the pruning of Generative Adversarial Networks (GANs) for Medical Imaging Synthesis with competitive results

Performance analysis of biological resource allocation algorithms for next generation networks.

Masters Degree. University of KwaZulu-Natal, Durban.Abstract available in PDF.Publications listed on page iii

Tracking the Temporal-Evolution of Supernova Bubbles in Numerical Simulations

The study of low-dimensional, noisy manifolds embedded in a higher dimensional space has been extremely useful in many applications, from the chemical analysis of multi-phase flows to simulations of galactic mergers. Building a probabilistic model of the manifolds has helped in describing their essential properties and how they vary in space. However, when the manifold is evolving through time, a joint spatio-temporal modelling is needed, in order to fully comprehend its nature. We propose a first-order Markovian process that propagates the spatial probabilistic model of a manifold at fixed time, to its adjacent temporal stages. The proposed methodology is demonstrated using a particle simulation of an interacting dwarf galaxy to describe the evolution of a cavity generated by a Supernov

Multiple Objective Fitness Functions for Cognitive Radio Adaptation

This thesis explores genetic algorithm and rule-based optimization techniques used by cognitive radios to make operating parameter decisions. Cognitive radios take advantage of intelligent control methods by using sensed information to determine the optimal set of transmission parameters for a given situation. We have chosen to explore and compare two control methods. A biologically-inspired genetic algorithm (GA) and a rule-based expert system are proposed, analyzed and tested using simulations. We define a common set of eight transmission parameters and six environment parameters used by cognitive radios, and develop a set of preliminary fitness functions that encompass the relationships between a small set of these input and output parameters. Five primary communication objectives are also defined and used in conjunction with the fitness functions to direct the cognitive radio to a solution. These fitness functions are used to implement the two cognitive control methods selected. The hardware resources needed to practically implement each technique are studied. It is observed, through simulations, that several trade offs exist between both the accuracy and speed of the final decision and the size of the parameter sets used to determine the decision. Sensitivity analysis is done on each parameter in order to determine the impact on the decision making process each parameter has on the cognitive engine. This analysis quantifies the usefulness of each parameter