Self adaptation in evolutionary algorithms

Evolutionary Algorithms are search algorithms based on the Darwinian metaphor of “Natural Selection”. Typically these algorithms maintain a population of individual solutions, each of which has a fitness attached to it, which in some way reflects the quality of the solution. The searchproceeds via the iterative generation, evaluation and possible incorporation of new individuals based on the current population, using a number of parameterisedgenetic operators. In this thesis the phenomenon of Self Adaptation of the genetic operators is investigated.A new framework for classifying adaptive algorithms is proposed, based on the scope of the adaptation, and on the nature of the transition function guiding the search through the space of possible configurations of the algorithm. Mechanisms are investigated for achieving the self adaptation of recombination and mutation operators within a genetic algorithm, and means of combining them are investigated. These are shown to produce significantly better results than any of the combinations of fixed operators tested, across a range of problem types. These new operators reduce the need for the designer of an algorithm to select appropriate choices of operators and parameters, thus aiding the implementation of geneticalgorithms. The nature of the evolving search strategies are investigated and explained in terms of the known properties of the landscapes used, and it is suggested how observations of evolving strategies on unknown landscapes may be used to categorise them, and guide further changes in other facets of the genetic algorithm.This work provides a contribution towards the study of adaptation in Evolutionary Algorithms, and towards the design of robust search algorithms for “real world” problems

Evolutionary Reinforcement Learning: A Survey

Reinforcement learning (RL) is a machine learning approach that trains agents to maximize cumulative rewards through interactions with environments. The integration of RL with deep learning has recently resulted in impressive achievements in a wide range of challenging tasks, including board games, arcade games, and robot control. Despite these successes, there remain several crucial challenges, including brittle convergence properties caused by sensitive hyperparameters, difficulties in temporal credit assignment with long time horizons and sparse rewards, a lack of diverse exploration, especially in continuous search space scenarios, difficulties in credit assignment in multi-agent reinforcement learning, and conflicting objectives for rewards. Evolutionary computation (EC), which maintains a population of learning agents, has demonstrated promising performance in addressing these limitations. This article presents a comprehensive survey of state-of-the-art methods for integrating EC into RL, referred to as evolutionary reinforcement learning (EvoRL). We categorize EvoRL methods according to key research fields in RL, including hyperparameter optimization, policy search, exploration, reward shaping, meta-RL, and multi-objective RL. We then discuss future research directions in terms of efficient methods, benchmarks, and scalable platforms. This survey serves as a resource for researchers and practitioners interested in the field of EvoRL, highlighting the important challenges and opportunities for future research. With the help of this survey, researchers and practitioners can develop more efficient methods and tailored benchmarks for EvoRL, further advancing this promising cross-disciplinary research field

Meta-heuristic Solution Methods for Rich Vehicle Routing Problems

Le problème de tournées de véhicules (VRP), introduit par Dantzig and Ramser en 1959, est devenu l'un des problèmes les plus étudiés en recherche opérationnelle, et ce, en raison de son intérêt méthodologique et de ses retombées pratiques dans de nombreux domaines tels que le transport, la logistique, les télécommunications et la production. L'objectif général du VRP est d'optimiser l'utilisation des ressources de transport afin de répondre aux besoins des clients tout en respectant les contraintes découlant des exigences du contexte d’application. Les applications réelles du VRP doivent tenir compte d’une grande variété de contraintes et plus ces contraintes sont nombreuse, plus le problème est difficile à résoudre. Les VRPs qui tiennent compte de l’ensemble de ces contraintes rencontrées en pratique et qui se rapprochent des applications réelles forment la classe des problèmes ‘riches’ de tournées de véhicules. Résoudre ces problèmes de manière efficiente pose des défis considérables pour la communauté de chercheurs qui se penchent sur les VRPs. Cette thèse, composée de deux parties, explore certaines extensions du VRP vers ces problèmes. La première partie de cette thèse porte sur le VRP périodique avec des contraintes de fenêtres de temps (PVRPTW). Celui-ci est une extension du VRP classique avec fenêtres de temps (VRPTW) puisqu’il considère un horizon de planification de plusieurs jours pendant lesquels les clients n'ont généralement pas besoin d’être desservi à tous les jours, mais plutôt peuvent être visités selon un certain nombre de combinaisons possibles de jours de livraison. Cette généralisation étend l'éventail d'applications de ce problème à diverses activités de distributions commerciales, telle la collecte des déchets, le balayage des rues, la distribution de produits alimentaires, la livraison du courrier, etc. La principale contribution scientifique de la première partie de cette thèse est le développement d'une méta-heuristique hybride dans la quelle un ensemble de procédures de recherche locales et de méta-heuristiques basées sur les principes de voisinages coopèrent avec un algorithme génétique afin d’améliorer la qualité des solutions et de promouvoir la diversité de la population. Les résultats obtenus montrent que la méthode proposée est très performante et donne de nouvelles meilleures solutions pour certains grands exemplaires du problème. La deuxième partie de cette étude a pour but de présenter, modéliser et résoudre deux problèmes riches de tournées de véhicules, qui sont des extensions du VRPTW en ce sens qu'ils incluent des demandes dépendantes du temps de ramassage et de livraison avec des restrictions au niveau de la synchronization temporelle. Ces problèmes sont connus respectivement sous le nom de Time-dependent Multi-zone Multi-Trip Vehicle Routing Problem with Time Windows (TMZT-VRPTW) et de Multi-zone Mult-Trip Pickup and Delivery Problem with Time Windows and Synchronization (MZT-PDTWS). Ces deux problèmes proviennent de la planification des opérations de systèmes logistiques urbains à deux niveaux. La difficulté de ces problèmes réside dans la manipulation de deux ensembles entrelacés de décisions: la composante des tournées de véhicules qui vise à déterminer les séquences de clients visités par chaque véhicule, et la composante de planification qui vise à faciliter l'arrivée des véhicules selon des restrictions au niveau de la synchronisation temporelle. Auparavant, ces questions ont été abordées séparément. La combinaison de ces types de décisions dans une seule formulation mathématique et dans une même méthode de résolution devrait donc donner de meilleurs résultats que de considérer ces décisions séparément. Dans cette étude, nous proposons des solutions heuristiques qui tiennent compte de ces deux types de décisions simultanément, et ce, d'une manière complète et efficace. Les résultats de tests expérimentaux confirment la performance de la méthode proposée lorsqu’on la compare aux autres méthodes présentées dans la littérature. En effet, la méthode développée propose des solutions nécessitant moins de véhicules et engendrant de moindres frais de déplacement pour effectuer efficacement la même quantité de travail. Dans le contexte des systèmes logistiques urbains, nos résultats impliquent une réduction de la présence de véhicules dans les rues de la ville et, par conséquent, de leur impact négatif sur la congestion et sur l’environnement.For more than half of century, since the paper of Dantzig and Ramser (1959) was introduced, the Vehicle Routing Problem (VRP) has been one of the most extensively studied problems in operations research due to its methodological interest and practical relevance in many fields such as transportation, logistics, telecommunications, and production. The general goal of the VRP is to optimize the use of transportation resources to service customers with respect to side-constraints deriving from real-world applications. The practical applications of the VRP may have a variety of constraints, and obviously, the larger the set of constraints that need to be considered, i.e., corresponding to `richer' VRPs, the more difficult the task of problem solving. The needs to study closer representations of actual applications and methodologies producing high-quality solutions quickly to larger-sized application problems have increased steadily, providing significant challenges for the VRP research community. This dissertation explores these extensional issues of the VRP. The first part of the dissertation addresses the Periodic Vehicle Routing Problem with Time Windows (PVRPTW) which generalizes the classical Vehicle Routing Problem with Time Windows (VRPTW) by extending the planning horizon to several days where customers generally do not require delivery on every day, but rather according to one of a limited number of possible combinations of visit days. This generalization extends the scope of applications to many commercial distribution activities such as waste collection, street sweeping, grocery distribution, mail delivery, etc. The major contribution of this part is the development of a population-based hybrid meta-heuristic in which a set of local search procedures and neighborhood-based meta-heuristics cooperate with the genetic algorithm population evolution mechanism to enhance the solution quality as well as to promote diversity of the genetic algorithm population. The results show that the proposed methodology is highly competitive, providing new best solutions in some large instances. The second part of the dissertation aims to present, model and solve two rich vehicle routing problems which further extend the VRPTW with time-dependent demands of pickup and delivery, and hard time synchronization restrictions. They are called Time-dependent Multi-zone Multi-Trip Vehicle Routing Problem with Time Windows (TMZT-VRPTW), and Multi-zone Mult-Trip Pickup and Delivery Problem with Time Windows and Synchronization (MZT-PDTWS), respectively. These two problems originate from planning the operations of two-tiered City Logistics systems. The difficulty of these problems lies in handling two intertwined sets of decisions: the routing component which aims to determine the sequences of customers visited by each vehicle, and the scheduling component which consists in planning arrivals of vehicles at facilities within hard time synchronization restrictions. Previously, these issues have been addressed separately. Combining these decisions into one formulation and solution method should yield better results. In this dissertation we propose meta-heuristics that address the two decisions simultaneously, in a comprehensive and efficient way. Experiments confirm the good performance of the proposed methodology compared to the literature, providing system managers with solution requiring less vehicles and travel costs to perform efficiently the same amount of work. In the context of City Logistics systems, our results indicate a reduction in the presence of vehicles on the streets of the city and, thus, in their negative impact on congestion and environment

Hybrid genetic algorithms in agent-based artificial market model for simulating fan tokens trading

In recent years cryptographic tokens have gained popularity as they can be used as a form of emerging alter- native financing and as a means of building platforms. The token markets innovate quickly through technology and decentralization, and they are constantly changing, and they have a high risk. Negotiation strategies must therefore be suited to these new circumstances. The genetic algorithm offers a very appropriate approach to resolving these complex issues. However, very little is known about genetic algorithm methods in cryptographic tokens. Accordingly, this paper presents a case study of the simulation of Fan Tokens trading by implementing selected best trading rule sets by a genetic algorithm that simulates a negotiation system through the Monte Carlo method. We have applied Adaptive Boosting and Genetic Algorithms, Deep Learning Neural Network-Genetic Algorithms, Adaptive Genetic Algorithms with Fuzzy Logic, and Quantum Genetic Algorithm techniques. The period selected is from December 1, 2021 to August 25, 2022, and we have used data from the Fan Tokens of Paris Saint-Germain, Manchester City, and Barcelona, leaders in the market. Our results conclude that the Hybrid and Quantum Genetic algorithm display a good execution during the training and testing period. Our study has a major impact on the current decentralized markets and future business opportunitiesThis research was funded by the Universitat de Barcelona, under the grant UB-AE-AS017634

Active Processor Scheduling Using Evolution Algorithms

The allocation of processes to processors has long been of interest to engineers. The processor allocation problem considered here assigns multiple applications onto a computing system. With this algorithm researchers could more efficiently examine real-time sensor data like that used by United States Air Force digital signal processing efforts or real-time aerosol hazard detection as examined by the Department of Homeland Security. Different choices for the design of a load balancing algorithm are examined in both the problem and algorithm domains. Evolutionary algorithms are used to find near-optimal solutions. These algorithms incorporate multiobjective coevolutionary and parallel principles to create an effective and efficient algorithm for real-world allocation problems. Three evolutionary algorithms (EA) are developed. The primary algorithm generates a solution to the processor allocation problem. This allocation EA is capable of evaluating objectives in both an aggregate single objective and a Pareto multiobjective manner. The other two EAs are designed for fine turning returned allocation EA solutions. One coevolutionary algorithm is used to optimize the parameters of the allocation algorithm. This meta-EA is parallelized using a coarse-grain approach to improve performance. Experiments are conducted that validate the improved effectiveness of the parallelized algorithm. Pareto multiobjective approach is used to optimize both effectiveness and efficiency objectives. The other coevolutionary algorithm generates difficult allocation problems for testing the capabilities of the allocation EA. The effectiveness of both coevolutionary algorithms for optimizing the allocation EA is examined quantitatively using standard statistical methods. Also the allocation EAs objective tradeoffs are analyzed and compared

Evolution through reputation: noise-resistant selection in evolutionary multi-agent systems

Little attention has been paid, in depth, to the relationship between fitness evaluation in evolutionary algorithms and reputation mechanisms in multi-agent systems, but if these could be related it opens the way for implementation of distributed evolutionary systems via multi-agent architectures. Our investigation concentrates on the effectiveness with which social selection, in the form of reputation, can replace direct fitness observation as the selection bias in an evolutionary multi-agent system. We do this in two stages: In the first, we implement a peer-to-peer, adaptive Genetic Algorithm (GA), in which agents act as individual GAs that, in turn, evolve dynamically themselves in real-time, using the traditional evolutionary operators of fitness-based selection, crossover and mutation. In the second stage, we replace the fitness-based selection operator with a reputation-based one, in which agents choose their mates based on the collective past experiences of themselves and their peers. Our investigation shows that this simple model of distributed reputation can be successful as the evolutionary drive in such a system, exhibiting practically identical performance and scalability to direct fitness observation. Further, we discuss the effect of noise (in the form of “defective” agents) in both models. We show that the reputation-based model is significantly better at identifying the defective agents, thus showing an increased level of resistance to noise

Parallel Multi-Objective Evolutionary Algorithms: A Comprehensive Survey

Multi-Objective Evolutionary Algorithms (MOEAs) are powerful search techniques that have been extensively used to solve difficult problems in a wide variety of disciplines. However, they can be very demanding in terms of computational resources. Parallel implementations of MOEAs (pMOEAs) provide considerable gains regarding performance and scalability and, therefore, their relevance in tackling computationally expensive applications. This paper presents a survey of pMOEAs, describing a refined taxonomy, an up-to-date review of methods and the key contributions to the field. Furthermore, some of the open questions that require further research are also briefly discussed

DNAgents: Genetically Engineered Intelligent Mobile Agents

Mobile agents are a useful paradigm for network coding providing many advantages and disadvantages. Unfortunately, widespread adoption of mobile agents has been hampered by the disadvantages, which could be said to outweigh the advantages. There is a variety of ongoing work to address these issues, and this is discussed. Ultimately, genetic algorithms are selected as the most interesting potential avenue. Genetic algorithms have many potential benefits for mobile agents. The primary benefit is the potential for agents to become even more adaptive to situational changes in the environment and/or emergent security risks. There are secondary benefits such as the natural obfuscation of functions inherent to genetic algorithms. Pitfalls also exist, namely the difficulty of defining a satisfactory fitness function and the variable execution time of mobile agents arising from the fact that it exists on a network. DNAgents 1.0, an original application of genetic algorithms to mobile agents is implemented and discussed, and serves to highlight these difficulties. Modifications of traditional genetic algorithms are also discussed. Ultimately, a combination of genetic algorithms and artificial life is considered to be the most appropriate approach to mobile agents. This allows the consideration of agents to be organisms, and the network to be their environment. Towards this end, a novel framework called DNAgents 2.0 is designed and implemented. This framework allows the continual evolution of agents in a network without having a seperate training and deployment phase. Parameters for this new framework were defined and explored. Lastly, an experiment similar to DNAgents 1.0 is performed for comparative purposes against DNAgents 1.0 and to prove the viability of this new framework

Advances in Evolutionary Algorithms

With the recent trends towards massive data sets and significant computational power, combined with evolutionary algorithmic advances evolutionary computation is becoming much more relevant to practice. Aim of the book is to present recent improvements, innovative ideas and concepts in a part of a huge EA field