A deep recurrent Q network towards self-adapting distributed microservice architecture

One desired aspect of microservice architecture is the ability to self-adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self-adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE-K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q-learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE-K model offers distributed microservice architecture with self-adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q-learning and policy gradient algorithms, including (1) a deep Q-learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services-oriented architecture and offers better performance than other dynamic decision-making algorithms

10021 Abstracts Collection -- Service-Oriented Architecture and (Multi-)Agent Systems Technology

From 10.01. to 15.01.2010, Dagstuhl Seminar 10021 ``Service-Oriented Architecture and (Multi-)Agent Systems Technology \u27\u27 was held in Schloss Dagstuhl~--~Leibniz Center for Informatics. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. The first section describes the seminar topics and goals in general. Links to extended abstracts or full papers are provided, if available

RAMARL: Robustness Analysis with Multi-Agent Reinforcement Learning - Robust Reasoning in Autonomous Cyber-Physical Systems

A key driver to offering smart services is an infrastructure of Cyber-Physical systems (CPS)s. By definition, CPSs are intertwined physical and computational components that integrate physical behaviour with computation. The reason is to autonomously execute a task or a set of tasks providing a service or a list of end-users services. In real-life applications, CPSs operate in dynamically changing surroundings characterized by unexpected or unpredictable situations. Such operations involve complex interactions between multiple intelligent agents in a highly non-stationary environment. For safety reasons, a CPS should withstand a certain amount of disruption and exert the operations in a stable and robust manner when performing complex tasks. Recent advances in reinforcement learning have proven suitable for enabling multi-agents to robustly adapt to their environment, yet they often depend on a massive amount of training data and experiences. In these cases, robustness analysis outlines necessary components and specifications in a framework, ensuring reliable and stable behaviour while considering the dynamicity of the environment. This paper presents a combination of multi-agent reinforcement learning with robustness analysis shaping a cyber-physical system infrastructure that reasons robustly in a dynamically changing environment. The combination strengthens the reinforcement learning, increasing the reliability and flexibility of the system by applying robustness analysis. Robustness analysis identifies vulnerability issues when the system interacts within a dynamically changing environment. Based on this identification, when incorporated into the system, robustness analysis suggests robust solutions and actions rather than optimal ones provided by reinforcement learning alone. Results from the combination show that this infrastructure can enable reliable operations with the flexibility to adapt to the changing environment dynamics.publishedVersio

Visual Execution Analysis for Multiagent Systems

Multiagent systems have become increasingly important in developing complex software systems. Multiagent systems introduce collective intelligence and provide benefits such as flexibility, scalability, decentralization, and increased reliability. A software agent is a high-level software abstraction that is capable of performing given tasks in an environment without human intervention. Although multiagent systems provide a convenient and powerful way to organize complex software systems, developing such system is very complicated. To help manage this complexity this research develops a methodology and technique for analyzing, monitoring and troubleshooting multiagent systems execution. This is accomplished by visualizing a multiagent system at multiple levels of abstraction to capture the relationships and dependencies among the agents

Enabling Cyber Physical Systems with Wireless Sensor Networking Technologies, Multiagent System Paradigm, and Natural Ecosystems

Wireless sensor networks (WSNs) are key components in the emergent cyber physical systems (CPSs). They may include hundreds of spatially distributed sensors which interact to solve complex tasks going beyond their individual capabilities. Due to the limited capabilities of sensors, sensor actions cannot meet CPS requirements while controlling and coordinating the operations of physical and engineered systems. To overcome these constraints, we explore the ecosystem metaphor for WSNs with the aim of taking advantage of the efficient adaptation behavior and communication mechanisms of living organisms. By mapping these organisms onto sensors and ecosystems onto WSNs, we highlight shortcomings that prevent WSNs from delivering the capabilities of ecosystems at several levels, including structure, topology, goals, communications, and functions. We then propose an agent-based architecture that migrates complex processing tasks outside the physical sensor network while incorporating missing characteristics of autonomy, intelligence, and context awareness to the WSN. Unlike existing works, we use software agents to map WSNs to natural ecosystems and enhance WSN capabilities to take advantage of bioinspired algorithms. We extend our architecture and propose a new intelligent CPS framework where several control levels are embedded in the physical system, thereby allowing agents to support WSNs technologies in enabling CPSs

Learning and Reasoning Strategies for User Association in Ultra-dense Small Cell Vehicular Networks

Recent vehicular ad hoc networks research has been focusing on providing intelligent transportation services by employing information and communication technologies on road transport. It has been understood that advanced demands such as reliable connectivity, high user throughput, and ultra-low latency required by these services cannot be met using traditional communication technologies. Consequently, this thesis reports on the application of artificial intelligence to user association as a technology enabler in ultra-dense small cell vehicular networks. In particular, the work focuses on mitigating mobility-related concerns and networking issues at different mobility levels by employing diverse heuristic as well as reinforcement learning (RL) methods. Firstly, driven by rapid fluctuations in the network topology and the radio environment, a conventional, three-step sequence user association policy is designed to highlight and explore the impact of vehicle speed and different performance indicators on network quality of service (QoS) and user experience. Secondly, inspired by control-theoretic models and dynamic programming, a real-time controlled feedback user association approach is proposed. The algorithm adapts to the changing vehicular environment by employing derived network performance information as a heuristic, resulting in improved network performance. Thirdly, a sequence of novel RL based user association algorithms are developed that employ variable learning rate, variable rewards function and adaptation of the control feedback framework to improve the initial and steady-state learning performance. Furthermore, to accelerate the learning process and enhance the adaptability and robustness of the developed RL algorithms, heuristically accelerated RL and case-based transfer learning methods are employed. A comprehensive, two-tier, event-based, system level simulator which is an integration of a dynamic vehicular network, a highway, and an ultra-dense small cell network is developed. The model has enabled the analysis of user mobility effects on the network performance across different mobility levels as well as served as a firm foundation for the evaluation of the empirical properties of the investigated approaches

Semantic resource allocation with historical data based predictions

One of the most important issues for Service Providers in Cloud Computing is delivering a good quality of service. This is achieved by means of the adaptation to a changing environment where different failures can occur during the execution of different services and tasks. Some of these failures can be predicted taking into account the information obtained from previous executions. The results of these predictions will help the schedulers to improve the allocation of resources to the different tasks. In this paper, we present a framework which uses semantically enhanced historical data for predicting the behavior of tasks and resources in the system, and allocating the resources according to these predictions

Design of an intelligent waterway ambient infrastructure based on Multiagent Systems and Wireless Sensor Networks

Lately Maritime research areas have moved their interests from traditional ship studies and traffic systems to new areas that confer a more general character to them as, for example, environmental monitoring. BOYAS project is proposed including these new perspectives as well as more classical ones. Trying to get this integral character for the waterway ambient and its activities management, the confluence between two recent research areas is studied. The convergence of Multiagent Systems and Wireless Sensor Networks constitutes a good framework and scenario in which this new research activities may be studied and develop.Ministerio de Industria, Turismo y Comercio FIT-340000-2006-2