Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) discovers policies that maximize reward but do not have safety guarantees during the learning and deployment phases. Although shielding with Linear Temporal Logic (LTL) is a promising formal method to ensure safety in single-agent Reinforcement Learning (RL), it results in conservative behaviors when scaling to multi-agent scenarios. Additionally, it poses computational challenges for synthesizing shields in complex multi-agent environments. This work introduces Model-based Dynamic Shielding (MBDS) to support MARL algorithm design. Our algorithm synthesizes distributive shields, which are reactive systems running in parallel with each MARL agent, to monitor and rectify unsafe behaviors. The shields can dynamically split, merge, and recompute based on agents' states. This design enables efficient synthesis of shields to monitor agents in complex environments without coordination overheads. We also propose an algorithm to synthesize shields without prior knowledge of the dynamics model. The proposed algorithm obtains an approximate world model by interacting with the environment during the early stage of exploration, making our MBDS enjoy formal safety guarantees with high probability. We demonstrate in simulations that our framework can surpass existing baselines in terms of safety guarantees and learning performance.Comment: Accepted in AAMAS 202

Reasoning over the Air: A Reasoning-based Implicit Semantic-Aware Communication Framework

Semantic-aware communication is a novel paradigm that draws inspiration from human communication focusing on the delivery of the meaning of messages. It has attracted significant interest recently due to its potential to improve the efficiency and reliability of communication and enhance users' QoE. Most existing works focus on transmitting and delivering the explicit semantic meaning that can be directly identified from the source signal. This paper investigates the implicit semantic-aware communication in which the hidden information that cannot be directly observed from the source signal must be recognized and interpreted by the intended users. To this end, a novel implicit semantic-aware communication (iSAC) architecture is proposed for representing, communicating, and interpreting the implicit semantic meaning between source and destination users. A projection-based semantic encoder is proposed to convert the high-dimensional graphical representation of explicit semantics into a low-dimensional semantic constellation space for efficient physical channel transmission. To enable the destination user to learn and imitate the implicit semantic reasoning process of source user, a generative adversarial imitation learning-based solution, called G-RML, is proposed. Different from existing communication solutions, the source user in G-RML does not focus only on sending as much of the useful messages as possible; but, instead, it tries to guide the destination user to learn a reasoning mechanism to map any observed explicit semantics to the corresponding implicit semantics that are most relevant to the semantic meaning. Compared to the existing solutions, our proposed G-RML requires much less communication and computational resources and scales well to the scenarios involving the communication of rich semantic meanings consisting of a large number of concepts and relations.Comment: accepted at IEEE Transactions on Wireless Communication

Exogenous Application of a Plant Elicitor Induces Volatile Emission in Wheat and Enhances the Attraction of an Aphid Parasitoid Aphidius gifuensis.

peer reviewedIt is well known that plant elicitors can induce plant defense against pests. The herbivore-induced plant volatile (HIPV) methyl salicylate (MeSA), as a signaling hormone involved in plant pathogen defense, is used to recruit natural enemies to protect wheat and other crops. However, the defense mechanism remains largely unknown. Here, the headspace volatiles of wheat plants were collected and analyzed by gas chromatography-mass spectrometry (GC-MS), gas chromatography with electroantennographic detection (GC-EAD) and principal component analysis (PCA). The results showed that exogenous application of MeSA induced qualitative and quantitative changes in the volatiles emitted from wheat plants, and these changes were mainly related to Carveol, Linalool, m-Diethyl-benzene, p-Cymene, Nonanal, D-limonene and 6-methyl-5-Hepten-2-one. Then, the electroantennogram (EAG) and Y-tube bioassay were performed to test the physiological and behavioral responses of Aphidius gifuensis Ashmesd to the active volatile compounds (p-Cymene, m-Diethyl-benzene, Carveol) that identified by using GC-EAD. The female A. gifuensis showed strong physiological responses to 1 μg/μL p-Cymene and 1 μg/μL m-Diethyl-benzene. Moreover, a mixture blend was more attractive to female A. gifuensis than a single compound. These findings suggested that MeSA could induce wheat plant indirect defense against wheat aphids through attracting parasitoid in the wheat agro-ecosystem