Contrastive learning-based agent modeling for deep reinforcement learning

Multi-agent systems often require agents to collaborate with or compete against other agents with diverse goals, behaviors, or strategies. Agent modeling is essential when designing adaptive policies for intelligent machine agents in multiagent systems, as this is the means by which the ego agent understands other agents' behavior and extracts their meaningful policy representations. These representations can be used to enhance the ego agent's adaptive policy which is trained by reinforcement learning. However, existing agent modeling approaches typically assume the availability of local observations from other agents (modeled agents) during training or a long observation trajectory for policy adaption. To remove these constrictive assumptions and improve agent modeling performance, we devised a Contrastive Learning-based Agent Modeling (CLAM) method that relies only on the local observations from the ego agent during training and execution. With these observations, CLAM is capable of generating consistent high-quality policy representations in real-time right from the beginning of each episode. We evaluated the efficacy of our approach in both cooperative and competitive multi-agent environments. Our experiments demonstrate that our approach achieves state-of-the-art on both cooperative and competitive tasks, highlighting the potential of contrastive learning-based agent modeling for enhancing reinforcement learning.Comment: 8 pages, 6 figure

Transparent modelling of finite stochastic processes for multiple agents

Stochastic Processes are ubiquitous, from automated engineering, through financial markets, to space exploration. These systems are typically highly dynamic, unpredictable and resistant to analytic methods; coupled with a need to orchestrate long control sequences which are both highly complex and uncertain. This report examines some existing single- and multi-agent modelling frameworks, details their strengths and weaknesses, and uses the experience to identify some fundamental tenets of good practice in modelling stochastic processes. It goes on to develop a new family of frameworks based on these tenets, which can model single- and multi-agent domains with equal clarity and flexibility, while remaining close enough to the existing frameworks that existing analytic and learning tools can be applied with little or no adaption. Some simple and larger examples illustrate the similarities and differences of this approach, and a discussion of the challenges inherent in developing more flexible tools to exploit these new frameworks concludes matters

Multi-Agent Modelling of Industrial Cyber-Physical Systems for IEC 61499 Based Distributed Intelligent Automation

Traditional industrial automation systems developed under IEC 61131-3 in centralized architectures are statically programmed with determined procedures to perform predefined tasks in structured environments. Major challenges are that these systems designed under traditional engineering techniques and running on legacy automation platforms are unable to automatically discover alternative solutions, flexibly coordinate reconfigurable modules, and actively deploy corresponding functions, to quickly respond to frequent changes and intelligently adapt to evolving requirements in dynamic environments. The core objective of this research is to explore the design of multi-layer automation architectures to enable real-time adaptation at the device level and run-time intelligence throughout the whole system under a well-integrated modelling framework. Central to this goal is the research on the integration of multi-agent modelling and IEC 61499 function block modelling to form a new automation infrastructure for industrial cyber-physical systems. Multi-agent modelling uses autonomous and cooperative agents to achieve run-time intelligence in system design and module reconfiguration. IEC 61499 function block modelling applies object-oriented and event-driven function blocks to realize real-time adaption of automation logic and control algorithms. In this thesis, the design focuses on a two-layer self-manageable architecture modelling: a) the high-level cyber module designed as multi-agent computing model consisting of Monitoring Agent, Analysis Agent, Self-Learning Agent, Planning Agent, Execution Agent, and Knowledge Agent; and b) the low-level physical module designed as agent-embedded IEC 61499 function block model with Self-Manageable Service Execution Agent, Self-Configuration Agent, Self-Healing Agent, Self-Optimization Agent, and Self-Protection Agent. The design results in a new computing module for high-level multi-agent based automation architectures and a new design pattern for low-level function block modelled control solutions. The architecture modelling framework is demonstrated through various tests on the multi-agent simulation model developed in the agent modelling environment NetLogo and the experimental testbed designed on the Jetson Nano and Raspberry Pi platforms. The performance evaluation of regular execution time and adaptation time in two typical conditions for systems designed under three different architectures are also analyzed. The results demonstrate the ability of the proposed architecture to respond to major challenges in Industry 4.0