A deep recurrent Q network towards self-adapting distributed microservice architecture

One desired aspect of microservice architecture is the ability to self-adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self-adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE-K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q-learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE-K model offers distributed microservice architecture with self-adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q-learning and policy gradient algorithms, including (1) a deep Q-learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services-oriented architecture and offers better performance than other dynamic decision-making algorithms

Multi-Agent Systems and Complex Networks: Review and Applications in Systems Engineering

Systems engineering is an ubiquitous discipline of Engineering overlapping industrial, chemical, mechanical, manufacturing, control, software, electrical, and civil engineering. It provides tools for dealing with the complexity and dynamics related to the optimisation of physical, natural, and virtual systems management. This paper presents a review of how multi-agent systems and complex networks theory are brought together to address systems engineering and management problems. The review also encompasses current and future research directions both for theoretical fundamentals and applications in the industry. This is made by considering trends such as mesoscale, multiscale, and multilayer networks along with the state-of-art analysis on network dynamics and intelligent networks. Critical and smart infrastructure, manufacturing processes, and supply chain networks are instances of research topics for which this literature review is highly relevant

Analysis and evaluation of multi-agent systems for digital production planning and control

Industrial manufacturing companies have different IT control functions that can be represented with a so-called hierarchical automation pyramid. While these conventional software systems especially support the mass production with consistent demand, the future project “Industry 4.0” focuses on customer-oriented and adaptable production processes. In order to move from conventional production systems to a factory of the future, the control levels must be redistributed. With the help of cyber-physical production systems, an interoperable architecture must be, implemented which removes the hierarchical connection of the former control levels. The accompanied digitalisation of industrial companies makes the transition to modular production possible. At the same time, the requirements for production planning and control are increasing, which can be solved with approaches such as multi-agent systems (MASs). These software solutions are autonomous and intelligent objects with a distinct collaborative ability. There are different modelling methods, communication and interaction structures, as well as different development frameworks for these new systems. Since multi-agent systems have not yet been established as an industrial standard due to their high complexity, they are usually only tested in simulations. In this bachelor thesis, a detailed literature review on the topic of MASs in the field of production planning and control is presented. In addition, selected multi-agent approaches are evaluated and compared using specific classification criteria. In addition, the applicability of using these systems in digital and modular production is assessed

A Deep Recurrent Q Network Towards Self-adapting Distributed Microservices Architecture (in press)

One desired aspect of microservices architecture is the ability to self-adapt its own architecture and behaviour in response to changes in the operational environment. To achieve the desired high levels of self-adaptability, this research implements the distributed microservices architectures model, as informed by the MAPE-K model. The proposed architecture employs a multi adaptation agents supported by a centralised controller, that can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q-network (DRQN). It is argued that such integration between DRQN and MDP agents in a MAPE-K model offers distributed microservice architecture with self-adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resources over-provisioning and thrashing. The performance of DRQN is evaluated against deep Q-learning and policy gradient algorithms including: i) deep q-network (DQN), ii) dulling deep Q-network (DDQN), iii) a policy gradient neural network (PGNN), and iv) deep deterministic policy gradient (DDPG). The DRQN implementation in this paper manages to outperform the above mentioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training times. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services-oriented architecture and offers better performance than other dynamic decision-making algorithms

Learning to Self-Manage by Intelligent Monitoring, Prediction and Intervention

Despite the growing prevalence of multimorbidities, current digital self-management approaches still prioritise single conditions. The future of out-of-hospital care requires researchers to expand their horizons; integrated assistive technologies should enable people to live their life well regardless of their chronic conditions. Yet, many of the current digital self-management technologies are not equipped to handle this problem. In this position paper, we suggest the solution for these issues is a model-aware and data-agnostic platform formed on the basis of a tailored self-management plan and three integral concepts - Monitoring (M) multiple information sources to empower Predictions (P) and trigger intelligent Interventions (I). Here we present our ideas for the formation of such a platform, and its potential impact on quality of life for sufferers of chronic conditions

Agents for educational games and simulations

This book consists mainly of revised papers that were presented at the Agents for Educational Games and Simulation (AEGS) workshop held on May 2, 2011, as part of the Autonomous Agents and MultiAgent Systems (AAMAS) conference in Taipei, Taiwan. The 12 full papers presented were carefully reviewed and selected from various submissions. The papers are organized topical sections on middleware applications, dialogues and learning, adaption and convergence, and agent applications