Bio-inspired Approaches for Engineering Adaptive Systems

AbstractAdaptive systems are composed of different heterogeneous parts or entities that interact and perform actions favouring the emer- gence of global desired behavior. In this type of systems entities might join or leave without disturbing the collective, and the system should self-organize and continue performing their goals. Furthermore, entities must self-evolve and self-improve by learn- ing from their interactions with the environment. The main challenge for engineering these systems is to design and develop distributed and adaptive algorithms that allow system entities to select the best suitable strategy/action and drive the system to the best suitable behavior according to the current state of the system and environment changes. This paper describes existing work related to the development of adaptive systems and approaches and shed light on how features from natural and biological systems could be exploited for engineering adaptive approaches

Algoritmos inspirados en Swarm intelligence para el enrutamieto en redes de telecomunicaciones.

En las últimas décadas hemos visto un rápido desarrollo de las redes de telecomunicación llegando a todos los rincones de la sociedad, bien a través de cable o bien de forma inalámbrica. Dichas redes, que cada vez son más grandes, dinámicas y complejas, integrando un mayor número de servicios y protocolos, requieren de un componente central que es el enrutamiento. El enrutamiento determina las estrategias a utilizar por los nodos de una red para encontrar las rutas óptimas entre un origen y un destino en el envío de información. Resulta difícil conseguir una estrategia que se adapte a este tipo de entornos altamente dinámicos, complejos y con un alto grado de heterogeneidad. Los algoritmos clásicos propuestos hasta la fecha suelen ser algoritmos centralizados que tratan de gestionar una arquitectura claramente distribuida, que en escenarios estacionarios pueden mantener un buen rendimiento, pero que no funcionan bien en escenarios donde se dan continuos cambios en la topología de red o en los patrones de tráfico. Es necesario proponer nuevos algoritmos que permitan el enrutamiento de forma distribuida, más adaptables a los cambios, robustos y escalables. Aquí vamos a tratar de hacer una revisión de los algoritmos propuestos inspirados en la naturaleza, particularmente en los comportamientos colectivos de sociedades de insectos. Veremos cómo de una forma descentralizada y auto-organizada, mediante agentes simples e interacciones locales, podemos alcanzar un comportamiento global "inteligente" que cumpla dichas cualidades. Por último proponemos Abira, un algoritmo ACO basado en AntNet-FA que trata de mejorar el rendimiento y la convergencia introduciendo mecanismos de exploración, de feedback negativo como la penalización y de comunicación de de las mejores rutas. Tras realizar una simulación y comparar los resultados con el algoritmo original, vemos que Abira muestra un mejor rendimiento

Modeling and simulation of routing protocol for ad hoc networks combining queuing network analysis and ANT colony algorithms

The field of Mobile Ad hoc Networks (MANETs) has gained an important part of the interest of researchers and become very popular in last few years. MANETs can operate without fixed infrastructure and can survive rapid changes in the network topology. They can be studied formally as graphs in which the set of edges varies in time. The main method for evaluating the performance of MANETs is simulation. Our thesis presents a new adaptive and dynamic routing algorithm for MANETs inspired by the Ant Colony Optimization (ACO) algorithms in combination with network delay analysis. Ant colony optimization algorithms have all been inspired by a specific foraging behavior of ant colonies which are able to find, if not the shortest, at least a very good path connecting the colony’s nest with a source of food. Our evaluation of MANETs is based on the evaluation of the mean End-to-End delay to send a packet from source to destination node through a MANET. We evaluated the mean End-to-End delay as one of the most important performance evaluation metrics in computer networks. Finally, we evaluate our proposed ant algorithm by a comparative study with respect to one of the famous On-Demand (reactive) routing protocols called Ad hoc On-Demand Distance Vector (AODV) protocol. The evaluation shows that, the ant algorithm provides a better performance by reducing the mean End-to-End delay than the AODV algorithm. We investigated various simulation scenarios with different node density and pause times. Our new algorithm gives good results under certain conditions such as, increasing the pause time and decreasing node density. The scenarios that are applied for evaluating our routing algorithm have the following assumptions: 2-D rectangular area, no obstacles, bi-directional links, fixed number of nodes operate for the whole simulation time and nodes movements are performed according to the Random Waypoint Mobility (RWM) or the Boundless Simulation Area Mobility (BSAM) model. KEYWORDS: Ant Colony Optimization (ACO), Mobile Ad hoc Network (MANET), Queuing Network Analysis, Routing Algorithms, Mobility Models, Hybrid Simulation

Reinforcement learning for routing in communication networks

Thesis (MSc)--Stellenbosch University, 2003.ENGLISH ABSTRACT: Routing policies for packet-switched communication networks must be able to adapt to changing traffic patterns and topologies. We study the feasibility of implementing an adaptive routing policy using the Q-Learning algorithm which learns sequences of actions from delayed rewards. The Q-Routing algorithm adapts a network's routing policy based on local information alone and converges toward an optimal solution. We demonstrate that Q-Routing is a viable alternative to other adaptive routing methods such as Bellman-Ford. We also study variations of Q-Routing designed to better explore possible routes and to take into consideration limited buffer size and optimize multiple objectives.AFRIKAANSE OPSOMMING:Die roetering in kommunikasienetwerke moet kan aanpas by veranderings in netwerktopologie en verkeersverspreidings. Ons bestudeer die bruikbaarheid van 'n aanpasbare roeteringsalgoritme gebaseer op die "Q-Learning"-algoritme wat dit moontlik maak om 'n reeks besluite te kan neem gebaseer op vertraagde vergoedings. Die roeteringsalgoritme gebruik slegs nabygelee inligting om roeteringsbesluite te maak en konvergeer na 'n optimale oplossing. Ons demonstreer dat die roeteringsalgoritme 'n goeie alternatief vir aanpasbare roetering is, aangesien dit in baie opsigte beter vaar as die Bellman-Ford algoritme. Ons bestudeer ook variasies van die roeteringsalgoritme wat beter paaie kan ontdek, minder geheue gebruik by netwerkelemente, en wat meer as een doelfunksie kan optimeer

Optimizing the Replication of Multi-Quality Web Applications Using ACO and WoLF

This thesis presents the adaptation of Ant Colony Optimization to a new NP-hard problem involving the replication of multi-quality database-driven web applications (DAs) by a large application service provider (ASP). The ASP must assign DA replicas to its network of heterogeneous servers so that user demand is satisfied and replica update loads are minimized. The algorithm proposed, AntDA, for solving this problem is novel in several respects: ants traverse a bipartite graph in both directions as they construct solutions, pheromone is used for traversing from one side of the bipartite graph to the other and back again, heuristic edge values change as ants construct solutions, and ants may sometimes produce infeasible solutions. Experiments show that AntDA outperforms several other solution methods, but there was room for improvement in the convergence rates of the ants. Therefore, in an attempt to achieve the goals of faster convergence and better solution values for larger problems, AntDA was combined with the variable-step policy hill-climbing algorithm called Win or Learn Fast (WoLF). In experimentation, the addition of this learning algorithm in AntDA provided for faster convergence while outperforming other solution methods

On thermal sensor calibration and software techniques for many-core thermal management

The high power density of a many-core processor results in increased temperature which negatively impacts system reliability and performance. Dynamic thermal management applies thermal-aware techniques at run time to avoid overheating using temperature information collected from on-chip thermal sensors. Temperature sensing and thermal control schemes are two critical technologies for successfully maintaining thermal safety. In this dissertation, on-line thermal sensor calibration schemes are developed to provide accurate temperature information. Software-based dynamic thermal management techniques are proposed using calibrated thermal sensors. Due to process variation and silicon aging, on-chip thermal sensors require periodic calibration before use in DTM. However, the calibration cost for thermal sensors can be prohibitively high as the number of on-chip sensors increases. Linear models which are suitable for on-line calculation are employed to estimate temperatures at multiple sensor locations using performance counters. The estimated temperature and the actual sensor thermal profile show a very high similarity with correlation coefficient ~0.9 for SPLASH2 and SPEC2000 benchmarks. A calibration approach is proposed to combine potentially inaccurate temperature values obtained from two sources: thermal sensor readings and temperature estimations. A data fusion strategy based on Bayesian inference, which combines information from these two sources, is demonstrated. The result shows the strategy can effectively recalibrate sensor readings in response to inaccuracies caused by process variation and environmental noise. The average absolute error of the corrected sensor temperature readings is A dynamic task allocation strategy is proposed to address localized overheating in many-core systems. Our approach employs reinforcement learning, a dynamic machine learning algorithm that performs task allocation based on current temperatures and a prediction regarding which assignment will minimize the peak temperature. Our results show that the proposed technique is fast (scheduling performed in \u3c1 \u3ems) and can efficiently reduce peak temperature by up to 8 degree C in a 49-core processor (6% on average) versus a leading competing task allocation approach for a series of SPLASH-2 benchmarks. Reinforcement learning has also been applied to 3D integrated circuits to allocate tasks with thermal awareness

On Novel Variants of the Hierarchical Stochastic Searching on the Line

Master's thesis Mechatronics MAS500 - University of Agder, 2012Konfidensiell til / confidential until 01.07.201

Learning algorithms for the control of routing in integrated service communication networks

There is a high degree of uncertainty regarding the nature of traffic on future integrated service networks. This uncertainty motivates the use of adaptive resource allocation policies that can take advantage of the statistical fluctuations in the traffic demands. The adaptive control mechanisms must be 'lightweight', in terms of their overheads, and scale to potentially large networks with many traffic flows. Adaptive routing is one form of adaptive resource allocation, and this thesis considers the application of Stochastic Learning Automata (SLA) for distributed, lightweight adaptive routing in future integrated service communication networks. The thesis begins with a broad critical review of the use of Artificial Intelligence (AI) techniques applied to the control of communication networks. Detailed simulation models of integrated service networks are then constructed, and learning automata based routing is compared with traditional techniques on large scale networks. Learning automata are examined for the 'Quality-of-Service' (QoS) routing problem in realistic network topologies, where flows may be routed in the network subject to multiple QoS metrics, such as bandwidth and delay. It is found that learning automata based routing gives considerable blocking probability improvements over shortest path routing, despite only using local connectivity information and a simple probabilistic updating strategy. Furthermore, automata are considered for routing in more complex environments spanning issues such as multi-rate traffic, trunk reservation, routing over multiple domains, routing in high bandwidth-delay product networks and the use of learning automata as a background learning process. Automata are also examined for routing of both 'real-time' and 'non-real-time' traffics in an integrated traffic environment, where the non-real-time traffic has access to the bandwidth 'left over' by the real-time traffic. It is found that adopting learning automata for the routing of the real-time traffic may improve the performance to both real and non-real-time traffics under certain conditions. In addition, it is found that one set of learning automata may route both traffic types satisfactorily. Automata are considered for the routing of multicast connections in receiver-oriented, dynamic environments, where receivers may join and leave the multicast sessions dynamically. Automata are shown to be able to minimise the average delay or the total cost of the resulting trees using the appropriate feedback from the environment. Automata provide a distributed solution to the dynamic multicast problem, requiring purely local connectivity information and a simple updating strategy. Finally, automata are considered for the routing of multicast connections that require QoS guarantees, again in receiver-oriented dynamic environments. It is found that the distributed application of learning automata leads to considerably lower blocking probabilities than a shortest path tree approach, due to a combination of load balancing and minimum cost behaviour

Smart data packet ad hoc routing protocol

This paper introduces a smart data packet routing protocol (SMART) based on swarm technology for mobile ad hoc networks. The main challenge facing a routing protocol is to cope with the dynamic environment of mobile ad hoc networks. The problem of finding best route between communication end points in such networks is an NP problem. Swarm algorithm is one of the methods used solve such a problem. However, copping with the dynamic environment will demand the use of a lot of training iterations. We present a new infrastructure where data packets are smart enough to guide themselves through best available route in the network. This approach uses distributed swarm learning approach which will minimize convergence time by using smart data packets. This will decrease the number of control packets in the network as well as it provides continues learning which in turn provides better reaction to changes in the network environment. The learning information is distributed throughout the nodes of the network. This information can be used and updated by successive packets in order to maintain and find better routes. This protocol is a hybrid Ant Colony Optimization (ACO) and river formation dynamics (RFD) swarm algorithms protocol. ACO is used to set up multi-path routes to destination at the initialization, while RFD mainly used as a base algorithm for the routing protocol. RFD offers many advantages toward implementing this approach. The main two reasons of using RFD are the small amount of information that required to be added to the packets (12 bytes in our approach) and the main idea of the RFD algorithm which is based on one kind of agent called drop that moves from source to destination only. This will eliminate the need of feedback packets to update the network and offers a suitable solution to change data packet into smart packets. Simulation results shows improvement in the throughput and reduction in end to end delay and jitter compared to AODV and AntHocNet protocols. © 2013 Elsevier B.V. All rights reserved

Reinforcement Learning

Brains rule the world, and brain-like computation is increasingly used in computers and electronic devices. Brain-like computation is about processing and interpreting data or directly putting forward and performing actions. Learning is a very important aspect. This book is on reinforcement learning which involves performing actions to achieve a goal. The first 11 chapters of this book describe and extend the scope of reinforcement learning. The remaining 11 chapters show that there is already wide usage in numerous fields. Reinforcement learning can tackle control tasks that are too complex for traditional, hand-designed, non-learning controllers. As learning computers can deal with technical complexities, the tasks of human operators remain to specify goals on increasingly higher levels. This book shows that reinforcement learning is a very dynamic area in terms of theory and applications and it shall stimulate and encourage new research in this field