Evaluation of Neuro-Evolution Algorithms for Tactic Volatility Aware Processes

Our society is increasingly evolving to rely on computer mechanisms that perform a variety of tasks. From a self-driving car to a satellite in space relaying data from Mars rovers, we need these systems to perform optimally and without failure. One such point of failure these systems can encounter is tactic volatility of an adaptation tactic. Adaptation tactics are defined workflows that allow systems to navigate their environment. Tactic volatility is the variance in the behavior in the attribute of a tactic, such as cost and latency and/or the combination of the two. Current systems consider these tactic attributes to be static. Studies have shown that not accounting for tactic volatility can adversely affect a system\u27s ability to operate effectively and resiliently. To support self-adaptive systems and address their limitations, this paper proposes a Tactic Volatility Aware solution that utilizes eRNN (TVA-E) and addresses the limitations of current self-adaptive systems. For this research, we used real-world data that has been made available for use by researchers and academics. This data contains real-world volatility and helps us demonstrate the positive impact TVA-E when used in self-adaptive systems. We also employ the use of uncertainty reduction tactics and how they can assist in accounting for tactic volatility. This work will serve as an evaluation and a comparison of using different machine learning methods to predict and account for tactic volatility. We will study different predictive mechanisms in this paper: Auto-Regressive Moving Average(ARIMA), Evolving Recurrent Neural Network(eRNN), Multi-Layer Perceptron(MLP), and Support Vector Regression(SVR). These methods will be studied with our TVA-E process and we will analyze how they can enhance a self-adaptive system’s performance when it accounts for tactic volatility

Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field

Neuroevolutionary Training of Deep Convolutional Generative Adversarial Networks

Recent developments in Deep Learning are noteworthy when it comes to learning the probability distribution of points through neural networks, and one of the crucial parts for such progress is because of Generative Adversarial Networks (GANs). In GANs, two neural networks, Generator and Discriminator, compete amongst each other to learn the probability distribution of points in visual pictures. A lot of research has been conducted to overcome the challenges of GANs which include training instability, mode collapse and vanishing gradient. However, there was no significant proof found on whether modern techniques consistently outperform vanilla GANs, and it turns out that different advanced techniques distinctively perform on different datasets. In this thesis, we propose two neuroevolutionary training techniques for deep convolutional GANs. We evolve the deep GANs architecture in low data regime. Using Fréchet Inception Distance (FID) score as the fitness function, we select the best deep convolutional topography generated by the evolutionary algorithm. The parameters of the best-selected individuals are maintained throughout the generations, and we continue to train the population until individuals demonstrate convergence. We compare our approach with the Vanilla GANs, Deep Convolutional GANs and COEGAN. Our experiments show that an evolutionary algorithm-based training technique gives a lower FID score than those of benchmark models. A lower FID score results in better image quality and diversity in the generated images

Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

A variety of methods have been applied to the architectural configuration and learning or training of artificial deep neural networks (DNN). These methods play a crucial role in the success or failure of the DNN for most problems and applications. Evolutionary Algorithms (EAs) are gaining momentum as a computationally feasible method for the automated optimisation and training of DNNs. Neuroevolution is a term which describes these processes of automated configuration and training of DNNs using EAs. While many works exist in the literature, no comprehensive surveys currently exist focusing exclusively on the strengths and limitations of using neuroevolution approaches in DNNs. Prolonged absence of such surveys can lead to a disjointed and fragmented field preventing DNNs researchers potentially adopting neuroevolutionary methods in their own research, resulting in lost opportunities for improving performance and wider application within real-world deep learning problems. This paper presents a comprehensive survey, discussion and evaluation of the state-of-the-art works on using EAs for architectural configuration and training of DNNs. Based on this survey, the paper highlights the most pertinent current issues and challenges in neuroevolution and identifies multiple promising future research directions.Comment: 20 pages (double column), 2 figures, 3 tables, 157 reference

Du placement des services à la surveillance des services dans les réseaux 5G et post-5G

5G and beyond 5G (B5G) networks are expected to accommodate a plethora of network services with diverse requirements using a single physical infrastructure. Hence, the ``one-size fits all'' paradigm that characterized the 4th generation of wireless networks is no longer suitable. By leveraging the last advent of Network Function Virtualization (NFV) and Software-Defined Networking (SDN), Network Slicing (NS) is considered as one of the key enablers of this paradigm shift. NS will enable the coexistence of heterogeneous services by partitioning the physical infrastructure into a set of virtual networks ''(the slices)'', each running a particular service. Besides, NS offers more flexibility and agility in business operations.Despite the advantages it brings, NS raises some technical challenges. The placement of network slices is one of them, it is known in the literature as the Virtual Network Embedding Problem (VNEP), and it is an NP-Hard problem. Therefore, the first part of this thesis focuses on unveiling the potential of Deep Reinforcement Learning (DRL) and Graph Neural Networks (GNNs) to solve the network slice placement problem and overcome the limitations of existing methods. Two approaches are considered: The first one aims to learn automatically how to solve the VNEP. Instead of putting any constraint on the topology of the physical infrastructure or extracting features manually, we formulate the task as a reinforcement problem, and we use a graph convolutional-based neural architecture to learn how to find an optimal solution. Next, instead of training a DRL agent from scratch to find the optimal solution, a process that may result in unsafe training, we train it to reduce the optimality gap of existing heuristics. The motivation behind this contribution is to ensure safety during the training of the DRL agent.The placement of the slices is not the only challenge raised by NS. Once the slices are placed, monitoring the status of network slices becomes a priority for both network slices' tenants and providers in order to ensure that Service Level Agreements (SLAs) are not violated. In the second part of this thesis, we propose to leverage machine learning techniques and network tomography to monitor the network slices. Network Tomography (NT) is defined as a set of methods that aim to infer unmeasured network metrics using an end-to-end measurement between monitors.We focus on two main challenges. First, on the inference of slices metrics based on some end-to-end measurements between monitors, as well as on the efficient monitor placement. For the inference, we model the task as a multi-output regression problem, which we solve using neural networks. We propose to train on synthetic data to augment the diversity of the training data and avoid the overfitting issue. Moreover, to deal with the changes that may occur either on the slices we monitor or the topology on top of which they are placed, we use transfer learning techniques.Regarding the monitor's placement problem, we consider a special case where only cycles' probes are allowed. The probing cycle schemes have a significant advantage compared to regular paths since the source probe is actually the destination, which reduces the synchronization problems. We formulate the problem as a variant of the Minimum Set Cover problem. Owing to its complexity, we introduce a standalone solution based on GNNs and genetic algorithms to find a trade-off between the quality of monitors placement and the cost to achieve it.Les réseaux 5G et au-delà sont destinés à servir un large éventail de services réseau aux besoins très disparates tout en utilisant la même infrastructure physique. En scindant l'infrastructure physique en un ensemble de réseaux virtuels, chacun exploitant un service spécifique, le Network Slicing (NS) permettra la coexistence de ces services. En dépit de ses avantages, le NS est complexe d'un point de vue technique puisqu'il s'agit d'un problème NP-hard. La première section de la thèse explore le potentiel de l'apprentissage par renforcement profond (DRL) basé sur des graphes de réseaux neuronaux pour résoudre le problème du placement des tranches de réseau et remédier aux limites des techniques existantes. Deux approches sont proposées : la première consiste à apprendre à résoudre automatiquement le problème du placement. Plutôt que de se limiter à la topologie de l'infrastructure physique ou à extraire manuellement des caractéristiques, le problème est formulé sous la forme d'un processus de décision markovien qui est résolu à l'aide d’un réseau de neurones convolutif à base de graphes pour apprendre à découvrir une solution optimale. Ensuite, plutôt que de former un agent DRL de zéro pour identifier la meilleure solution, ce qui pourrait entraîner un défaut de fiabilité, un agent est présenté pour réduire l'écart d'optimalité des heuristiques existantes. Une fois les tranches placées, la surveillance de l'état des tranches de réseau devient une priorité pour s'assurer que les SLAs sont respectés. Ainsi, dans la deuxième partie de la thèse, il est proposé d'utiliser des techniques d'apprentissage automatique et la tomographie réseau (NT) pour surveiller les tranches de réseau. Il y a deux problèmes majeurs à prendre en compte. Premièrement, les métriques de slices sont déduites sur la base de diverses mesures de bout en bout entre les moniteurs, ainsi que du placement efficace des moniteurs. Des réseaux neuronaux sont utilisés pour traiter l'inférence des métriques. Une approche d'apprentissage par transfert est également utilisée pour faire face aux changements qui peuvent se produire sur les slices surveillés ou sur la topologie physique sur laquelle elles sont placées. Des sondes cycliques sont envisagées pour le problème du placement des moniteurs. Le problème est formulé comme une variante du problème de couverture par ensembles. En raison de sa complexité, il est proposé d'introduire une solution autonome basée sur des réseaux neuronaux à base de graphes (GNN) et des algorithmes génétiques pour trouver un compromis entre la qualité du placement des moniteurs et le coût pour y parvenir

Evolutionary design of deep neural networks

Mención Internacional en el título de doctorFor three decades, neuroevolution has applied evolutionary computation to the optimization of the topology of artificial neural networks, with most works focusing on very simple architectures. However, times have changed, and nowadays convolutional neural networks are the industry and academia standard for solving a variety of problems, many of which remained unsolved before the discovery of this kind of networks. Convolutional neural networks involve complex topologies, and the manual design of these topologies for solving a problem at hand is expensive and inefficient. In this thesis, our aim is to use neuroevolution in order to evolve the architecture of convolutional neural networks. To do so, we have decided to try two different techniques: genetic algorithms and grammatical evolution. We have implemented a niching scheme for preserving the genetic diversity, in order to ease the construction of ensembles of neural networks. These techniques have been validated against the MNIST database for handwritten digit recognition, achieving a test error rate of 0.28%, and the OPPORTUNITY data set for human activity recognition, attaining an F1 score of 0.9275. Both results have proven very competitive when compared with the state of the art. Also, in all cases, ensembles have proven to perform better than individual models. Later, the topologies learned for MNIST were tested on EMNIST, a database recently introduced in 2017, which includes more samples and a set of letters for character recognition. Results have shown that the topologies optimized for MNIST perform well on EMNIST, proving that architectures can be reused across domains with similar characteristics. In summary, neuroevolution is an effective approach for automatically designing topologies for convolutional neural networks. However, it still remains as an unexplored field due to hardware limitations. Current advances, however, should constitute the fuel that empowers the emergence of this field, and further research should start as of today.This Ph.D. dissertation has been partially supported by the Spanish Ministry of Education, Culture and Sports under FPU fellowship with identifier FPU13/03917. This research stay has been partially co-funded by the Spanish Ministry of Education, Culture and Sports under FPU short stay grant with identifier EST15/00260.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: María Araceli Sanchís de Miguel.- Secretario: Francisco Javier Segovia Pérez.- Vocal: Simon Luca