Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Population-based algorithms for improved history matching and uncertainty quantification of Petroleum reservoirs

In modern field management practices, there are two important steps that shed light on a multimillion dollar investment. The first step is history matching where the simulation model is calibrated to reproduce the historical observations from the field. In this inverse problem, different geological and petrophysical properties may provide equally good history matches. Such diverse models are likely to show different production behaviors in future. This ties the history matching with the second step, uncertainty quantification of predictions. Multiple history matched models are essential for a realistic uncertainty estimate of the future field behavior. These two steps facilitate decision making and have a direct impact on technical and financial performance of oil and gas companies. Population-based optimization algorithms have been recently enjoyed growing popularity for solving engineering problems. Population-based systems work with a group of individuals that cooperate and communicate to accomplish a task that is normally beyond the capabilities of each individual. These individuals are deployed with the aim to solve the problem with maximum efficiency. This thesis introduces the application of two novel population-based algorithms for history matching and uncertainty quantification of petroleum reservoir models. Ant colony optimization and differential evolution algorithms are used to search the space of parameters to find multiple history matched models and, using a Bayesian framework, the posterior probability of the models are evaluated for prediction of reservoir performance. It is demonstrated that by bringing latest developments in computer science such as ant colony, differential evolution and multiobjective optimization, we can improve the history matching and uncertainty quantification frameworks. This thesis provides insights into performance of these algorithms in history matching and prediction and develops an understanding of their tuning parameters. The research also brings a comparative study of these methods with a benchmark technique called Neighbourhood Algorithms. This comparison reveals the superiority of the proposed methodologies in various areas such as computational efficiency and match quality

Mobile Ad Hoc Networks

Guiding readers through the basics of these rapidly emerging networks to more advanced concepts and future expectations, Mobile Ad hoc Networks: Current Status and Future Trends identifies and examines the most pressing research issues in Mobile Ad hoc Networks (MANETs). Containing the contributions of leading researchers, industry professionals, and academics, this forward-looking reference provides an authoritative perspective of the state of the art in MANETs. The book includes surveys of recent publications that investigate key areas of interest such as limited resources and the mobility of mobile nodes. It considers routing, multicast, energy, security, channel assignment, and ensuring quality of service. Also suitable as a text for graduate students, the book is organized into three sections: Fundamentals of MANET Modeling and Simulation—Describes how MANETs operate and perform through simulations and models Communication Protocols of MANETs—Presents cutting-edge research on key issues, including MAC layer issues and routing in high mobility Future Networks Inspired By MANETs—Tackles open research issues and emerging trends Illustrating the role MANETs are likely to play in future networks, this book supplies the foundation and insight you will need to make your own contributions to the field. It includes coverage of routing protocols, modeling and simulations tools, intelligent optimization techniques to multicriteria routing, security issues in FHAMIPv6, connecting moving smart objects to the Internet, underwater sensor networks, wireless mesh network architecture and protocols, adaptive routing provision using Bayesian inference, and adaptive flow control in transport layer using genetic algorithms

Multi-objective Digital VLSI Design Optimisation

Modern VLSI design's complexity and density has been exponentially increasing over the past 50 years and recently reached a stage within its development, allowing heterogeneous, many-core systems and numerous functions to be integrated into a tiny silicon die. These advancements have revealed intrinsic physical limits of process technologies in advanced silicon technology nodes. Designers and EDA vendors have to handle these challenges which may otherwise result in inferior design quality, even failures, and lower design yields under time-to-market pressure. Multiple or many design objectives and constraints are emerging during the design process and often need to be dealt with simultaneously. Multi-objective evolutionary algorithms show flexible capabilities in maintaining multiple variable components and factors in uncertain environments. The VLSI design process involves a large number of available parameters both from designs and EDA tools. This provides many potential optimisation avenues where evolutionary algorithms can excel. This PhD work investigates the application of evolutionary techniques for digital VLSI design optimisation. Automated multi-objective optimisation frameworks, compatible with industrial design flows and foundry technologies, are proposed to improve solution performance, expand feasible design space, and handle complex physical floorplan constraints through tuning designs at gate-level. Methodologies for enriching standard cell libraries regarding drive strength are also introduced to cooperate with multi-objective optimisation frameworks, e.g., subsequent hill-climbing, providing a richer pool of solutions optimised for different trade-offs. The experiments of this thesis demonstrate that multi-objective evolutionary algorithms, derived from biological inspirations, can assist the digital VLSI design process, in an industrial design context, to more efficiently search for well-balanced trade-off solutions as well as optimised design space coverage. The expanded drive granularity of standard cells can push the performance of silicon technologies with offering improved solutions regarding critical objectives. The achieved optimisation results can better deliver trade-off solutions regarding power, performance and area metrics than using standard EDA tools alone. This has been not only shown for a single circuit solution but also covered the entire standard-tool-produced design space

Mobile Ad Hoc Networks

Guiding readers through the basics of these rapidly emerging networks to more advanced concepts and future expectations, Mobile Ad hoc Networks: Current Status and Future Trends identifies and examines the most pressing research issues in Mobile Ad hoc Networks (MANETs). Containing the contributions of leading researchers, industry professionals, and academics, this forward-looking reference provides an authoritative perspective of the state of the art in MANETs. The book includes surveys of recent publications that investigate key areas of interest such as limited resources and the mobility of mobile nodes. It considers routing, multicast, energy, security, channel assignment, and ensuring quality of service. Also suitable as a text for graduate students, the book is organized into three sections: Fundamentals of MANET Modeling and Simulation—Describes how MANETs operate and perform through simulations and models Communication Protocols of MANETs—Presents cutting-edge research on key issues, including MAC layer issues and routing in high mobility Future Networks Inspired By MANETs—Tackles open research issues and emerging trends Illustrating the role MANETs are likely to play in future networks, this book supplies the foundation and insight you will need to make your own contributions to the field. It includes coverage of routing protocols, modeling and simulations tools, intelligent optimization techniques to multicriteria routing, security issues in FHAMIPv6, connecting moving smart objects to the Internet, underwater sensor networks, wireless mesh network architecture and protocols, adaptive routing provision using Bayesian inference, and adaptive flow control in transport layer using genetic algorithms

Telecommunications Networks

This book guides readers through the basics of rapidly emerging networks to more advanced concepts and future expectations of Telecommunications Networks. It identifies and examines the most pressing research issues in Telecommunications and it contains chapters written by leading researchers, academics and industry professionals. Telecommunications Networks - Current Status and Future Trends covers surveys of recent publications that investigate key areas of interest such as: IMS, eTOM, 3G/4G, optimization problems, modeling, simulation, quality of service, etc. This book, that is suitable for both PhD and master students, is organized into six sections: New Generation Networks, Quality of Services, Sensor Networks, Telecommunications, Traffic Engineering and Routing

Parallel optimization algorithms for high performance computing : application to thermal systems

The need of optimization is present in every field of engineering. Moreover, applications requiring a multidisciplinary approach in order to make a step forward are increasing. This leads to the need of solving complex optimization problems that exceed the capacity of human brain or intuition. A standard way of proceeding is to use evolutionary algorithms, among which genetic algorithms hold a prominent place. These are characterized by their robustness and versatility, as well as their high computational cost and low convergence speed. Many optimization packages are available under free software licenses and are representative of the current state of the art in optimization technology. However, the ability of optimization algorithms to adapt to massively parallel computers reaching satisfactory efficiency levels is still an open issue. Even packages suited for multilevel parallelism encounter difficulties when dealing with objective functions involving long and variable simulation times. This variability is common in Computational Fluid Dynamics and Heat Transfer (CFD & HT), nonlinear mechanics, etc. and is nowadays a dominant concern for large scale applications. Current research in improving the performance of evolutionary algorithms is mainly focused on developing new search algorithms. Nevertheless, there is a vast knowledge of sequential well-performing algorithmic suitable for being implemented in parallel computers. The gap to be covered is efficient parallelization. Moreover, advances in the research of both new search algorithms and efficient parallelization are additive, so that the enhancement of current state of the art optimization software can be accelerated if both fronts are tackled simultaneously. The motivation of this Doctoral Thesis is to make a step forward towards the successful integration of Optimization and High Performance Computing capabilities, which has the potential to boost technological development by providing better designs, shortening product development times and minimizing the required resources. After conducting a thorough state of the art study of the mathematical optimization techniques available to date, a generic mathematical optimization tool has been developed putting a special focus on the application of the library to the field of Computational Fluid Dynamics and Heat Transfer (CFD & HT). Then the main shortcomings of the standard parallelization strategies available for genetic algorithms and similar population-based optimization methods have been analyzed. Computational load imbalance has been identified to be the key point causing the degradation of the optimization algorithm¿s scalability (i.e. parallel efficiency) in case the average makespan of the batch of individuals is greater than the average time required by the optimizer for performing inter-processor communications. It occurs because processors are often unable to finish the evaluation of their queue of individuals simultaneously and need to be synchronized before the next batch of individuals is created. Consequently, the computational load imbalance is translated into idle time in some processors. Several load balancing algorithms have been proposed and exhaustively tested, being extendable to any other population-based optimization method that needs to synchronize all processors after the evaluation of each batch of individuals. Finally, a real-world engineering application that consists on optimizing the refrigeration system of a power electronic device has been presented as an illustrative example in which the use of the proposed load balancing algorithms is able to reduce the simulation time required by the optimization tool.El aumento de las aplicaciones que requieren de una aproximación multidisciplinar para poder avanzar se constata en todos los campos de la ingeniería, lo cual conlleva la necesidad de resolver problemas de optimización complejos que exceden la capacidad del cerebro humano o de la intuición. En estos casos es habitual el uso de algoritmos evolutivos, principalmente de los algoritmos genéticos, caracterizados por su robustez y versatilidad, así como por su gran coste computacional y baja velocidad de convergencia. La multitud de paquetes de optimización disponibles con licencias de software libre representan el estado del arte actual en tecnología de optimización. Sin embargo, la capacidad de adaptación de los algoritmos de optimización a ordenadores masivamente paralelos alcanzando niveles de eficiencia satisfactorios es todavía una tarea pendiente. Incluso los paquetes adaptados al paralelismo multinivel tienen dificultades para gestionar funciones objetivo que requieren de tiempos de simulación largos y variables. Esta variabilidad es común en la Dinámica de Fluidos Computacional y la Transferencia de Calor (CFD & HT), mecánica no lineal, etc. y es una de las principales preocupaciones en aplicaciones a gran escala a día de hoy. La investigación actual que tiene por objetivo la mejora del rendimiento de los algoritmos evolutivos está enfocada principalmente al desarrollo de nuevos algoritmos de búsqueda. Sin embargo, ya se conoce una gran variedad de algoritmos secuenciales apropiados para su implementación en ordenadores paralelos. La tarea pendiente es conseguir una paralelización eficiente. Además, los avances en la investigación de nuevos algoritmos de búsqueda y la paralelización son aditivos, por lo que el proceso de mejora del software de optimización actual se verá incrementada si se atacan ambos frentes simultáneamente. La motivación de esta Tesis Doctoral es avanzar hacia una integración completa de las capacidades de Optimización y Computación de Alto Rendimiento para así impulsar el desarrollo tecnológico proporcionando mejores diseños, acortando los tiempos de desarrollo del producto y minimizando los recursos necesarios. Tras un exhaustivo estudio del estado del arte de las técnicas de optimización matemática disponibles a día de hoy, se ha diseñado una librería de optimización orientada al campo de la Dinámica de Fluidos Computacional y la Transferencia de Calor (CFD & HT). A continuación se han analizado las principales limitaciones de las estrategias de paralelización disponibles para algoritmos genéticos y otros métodos de optimización basados en poblaciones. En el caso en que el tiempo de evaluación medio de la tanda de individuos sea mayor que el tiempo medio que necesita el optimizador para llevar a cabo comunicaciones entre procesadores, se ha detectado que la causa principal de la degradación de la escalabilidad o eficiencia paralela del algoritmo de optimización es el desequilibrio de la carga computacional. El motivo es que a menudo los procesadores no terminan de evaluar su cola de individuos simultáneamente y deben sincronizarse antes de que se cree la siguiente tanda de individuos. Por consiguiente, el desequilibrio de la carga computacional se convierte en tiempo de inactividad en algunos procesadores. Se han propuesto y testado exhaustivamente varios algoritmos de equilibrado de carga aplicables a cualquier método de optimización basado en una población que necesite sincronizar los procesadores tras cada tanda de evaluaciones. Finalmente, se ha presentado como ejemplo ilustrativo un caso real de ingeniería que consiste en optimizar el sistema de refrigeración de un dispositivo de electrónica de potencia. En él queda demostrado que el uso de los algoritmos de equilibrado de carga computacional propuestos es capaz de reducir el tiempo de simulación que necesita la herramienta de optimización

New computational techniques for detecting, learning and managing criteria in design problems

Los problemas de diseño suelen involucrar la consideración de criterios de diferente naturaleza, incluyendo necesidades técnicas, económicas, sociales y medioambientales, entre otras. Las herramientas CAD tradicionales ayudan a los diseñadores en la representación, modificación, análisis, documentación y evaluación de sus diseños. Sin embargo, los ordenadores pueden cumplir un papel más complejo: el del diseño computacional, consistente en la síntesis de nuevas soluciones de diseño. Esta tesis se ha enfocado en este rol no trivial, siendo su objetivo general el desarrollo de nuevas técnicas de diseño computacional capaces de considerar criterios de diseño. Algunos aspectos del proceso de diseño pueden entenderse como una búsqueda o exploración en un espacio de alternativas de diseño. Esta perspectiva facilita la explotación de técnicas computacionales para implementar métodos que sinteticen soluciones de acuerdo al propósito de un problema de diseño dado. Para el desarrollo de metodologías de diseño computacional, es necesario un sistema generativo capaz de representar y generar el espacio de diseño. En esta tesis hemos considerado el formalismo de las gramáticas de formas dado su uso intensivo en la literatura de diseño computacional y dada también su versatilidad. El enfoque tradicional a la hora de usar gramáticas de formas consiste en codificar el conjunto completo de criterios de diseño en las mismas reglas, dando lugar a gramáticas expertas que son difíciles de crear, modificar y mantener. Este tipo de gramáticas también promueve soluciones previsibles, dado que las reglas han sido creadas con el conocimiento previo de los requisitos que las formas han de cumplir. En esta tesis hemos considerado otra alternativa, que consiste en reducir o minimizar el número de criterios codificados en las reglas. Así, tratamos con gramáticas más ingenuas que no pueden producir soluciones factibles y necesitan de un mecanismo de control que guíe la derivación hacia buenos diseños. Concretamente, hemos utilizado algoritmos de búsqueda y métodos de aprendizaje por refuerzo para llevar a cabo dicho control. Las principales conclusiones de esta tesis pueden ser resumidas como sigue: 1. Se ha propuesto un esquema de clasificación para posibles enfoques al diseño computacional basados en gramáticas de formas. Concretamente, consideramos dos aspectos: el primero considera la cantidad de criterios de diseño codificados en las reglas, siendo las gramáticas puramente expertas aquellas en las que la totalidad de los criterios han sido codificados de esta manera. Cuantos menos criterios sean codificados en las reglas, más ingenua puede ser considerada la gramática. El segundo considera la complejidad del método de control empleado, desde sistemas que carecen de dicho sistema de control hasta sistemas que emplean mecanismos complejos. 2. Se ha desarrollado una metodología de diseño computacional basada en gramáticas de formas expertas y un mecanismo de control complejo. Dicha metodología está basada en la idea de codificar algunos requisitos de diseño en las reglas y utilizar el resto de manera explícita para evaluar las formas producidas a lo largo del proceso de generación, guiando dicho proceso hacia buenos diseños. Las gramáticas de formas involucradas son por tanto menos expertas que en el enfoque tradicional. En esta configuración distinguimos entre criterios que se especifican mejor geométricamente (dentro de las reglas) y criterios que se expresan mejor como predicados lógicos (restricciones y objetivos usados en un algoritmo de búsqueda). 3. Se ha desarrollado una herramienta software (ShaDe) para editar y ejecutar gramáticas de formas con capas, restricciones y objetivos. 4. Se ha desarrollado una metodología de diseño computacional basada en gramáticas de formas ingenuas y un mecanismo de control complejo. En esta metodología se usa el conjunto completo de criterios de diseño como recompensas en un proceso de aprendizaje por refuerzo, con el objetivo de aprender un heurístico que determine cómo aplicar las reglas del sistema generativo. Se han presentado dos alternativas para aprender las políticas de aplicación de reglas, dependiendo de la manera de tratar la naturaleza multi-objetivo del diseño. En la primera, las recompensas son escalarizadas. La segunda alternativa no escalariza las recompensas; han de aprenderse múltiples políticas que pueden ser utilizadas para producir un conjunto de soluciones óptimas. 5. Se ha propuesto un nuevo algoritmo de aprendizaje por refuerzo multiobjetivo (PQ-learning). En el contexto de la metodología en la que no se escalarizan las recompensas, hemos propuesto una nueva técnica de aprendizaje por refuerzo, basada en una extensión directa del algoritmo Q-learning, que trabaja con recompensas vectoriales. Este nuevo método ha sido probado en dos problemas pertenecientes a un benchmark de aprendizaje por refuerzo multi-objetivo. 6. Las metodologías desarrolladas han sido puestas en práctica en diferentes escenarios relacionados con la arquitectura. 7. Se han llevado a cabo dos estudios empíricos con estudiantes de arquitectura. Particularmente, fueron asociados con las metodologías correspondientes a gramáticas de formas ingenuas con y sin control, para estudiar diversos aspectos como la reacción de los alumnos y la viabilidad de los sistemas propuestos. A continuación detallamos los aspectos de esta tesis que merecen una investigación más profunda: -La extensión de las metodologías propuestas a tipos más complejos de gramáticas de formas, tales como gramáticas tridimensionales o incluso paramétricas. -Los casos de aplicación que involucran gramáticas ingenuas y aprendizaje por refuerzo escalarizado se basan en una división en fases del problema de diseño considerado. Esta división reduce el conjunto de criterios que han de tenerse en cuenta en cada paso. Sin embargo, también introduce una limitación importante que puede afectarnos en el caso de problemas de diseño más complejos: el proceso de aprendizaje sólo trata con los criterios locales de cada fase, y por tanto las políticas no pueden reflejar aspectos globales. Una posibilidad para afrontar dicho problema es la integración con técnicas más potentes como la generalización no lineal ofrecida por las redes neuronales. -Hemos mostrado cómo la metodología basada en gramáticas ingenuas y PQ-learning puede ser utilizada para abordar problemas geométricos, pero es necesaria más investigación para aplicar dicha metodología en escenarios reales. Creemos que esto puede conseguirse por medio de la integración de PQ-learning con técnicas de generalización. -Finalmente, la aplicación de las metodologías propuestas a otros ámbitos de diseño es también una línea importante de investigación

Machine learning for network based intrusion detection: an investigation into discrepancies in findings with the KDD cup '99 data set and multi-objective evolution of neural network classifier ensembles from imbalanced data.

For the last decade it has become commonplace to evaluate machine learning techniques for network based intrusion detection on the KDD Cup '99 data set. This data set has served well to demonstrate that machine learning can be useful in intrusion detection. However, it has undergone some criticism in the literature, and it is out of date. Therefore, some researchers question the validity of the findings reported based on this data set. Furthermore, as identified in this thesis, there are also discrepancies in the findings reported in the literature. In some cases the results are contradictory. Consequently, it is difficult to analyse the current body of research to determine the value in the findings. This thesis reports on an empirical investigation to determine the underlying causes of the discrepancies. Several methodological factors, such as choice of data subset, validation method and data preprocessing, are identified and are found to affect the results significantly. These findings have also enabled a better interpretation of the current body of research. Furthermore, the criticisms in the literature are addressed and future use of the data set is discussed, which is important since researchers continue to use it due to a lack of better publicly available alternatives. Due to the nature of the intrusion detection domain, there is an extreme imbalance among the classes in the KDD Cup '99 data set, which poses a significant challenge to machine learning. In other domains, researchers have demonstrated that well known techniques such as Artificial Neural Networks (ANNs) and Decision Trees (DTs) often fail to learn the minor class(es) due to class imbalance. However, this has not been recognized as an issue in intrusion detection previously. This thesis reports on an empirical investigation that demonstrates that it is the class imbalance that causes the poor detection of some classes of intrusion reported in the literature. An alternative approach to training ANNs is proposed in this thesis, using Genetic Algorithms (GAs) to evolve the weights of the ANNs, referred to as an Evolutionary Neural Network (ENN). When employing evaluation functions that calculate the fitness proportionally to the instances of each class, thereby avoiding a bias towards the major class(es) in the data set, significantly improved true positive rates are obtained whilst maintaining a low false positive rate. These findings demonstrate that the issues of learning from imbalanced data are not due to limitations of the ANNs; rather the training algorithm. Moreover, the ENN is capable of detecting a class of intrusion that has been reported in the literature to be undetectable by ANNs. One limitation of the ENN is a lack of control of the classification trade-off the ANNs obtain. This is identified as a general issue with current approaches to creating classifiers. Striving to create a single best classifier that obtains the highest accuracy may give an unfruitful classification trade-off, which is demonstrated clearly in this thesis. Therefore, an extension of the ENN is proposed, using a Multi-Objective GA (MOGA), which treats the classification rate on each class as a separate objective. This approach produces a Pareto front of non-dominated solutions that exhibit different classification trade-offs, from which the user can select one with the desired properties. The multi-objective approach is also utilised to evolve classifier ensembles, which yields an improved Pareto front of solutions. Furthermore, the selection of classifier members for the ensembles is investigated, demonstrating how this affects the performance of the resultant ensembles. This is a key to explaining why some classifier combinations fail to give fruitful solutions