Model-based Dynamic Shielding for Safe and Efficient Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) discovers policies that maximize reward but do not have safety guarantees during the learning and deployment phases. Although shielding with Linear Temporal Logic (LTL) is a promising formal method to ensure safety in single-agent Reinforcement Learning (RL), it results in conservative behaviors when scaling to multi-agent scenarios. Additionally, it poses computational challenges for synthesizing shields in complex multi-agent environments. This work introduces Model-based Dynamic Shielding (MBDS) to support MARL algorithm design. Our algorithm synthesizes distributive shields, which are reactive systems running in parallel with each MARL agent, to monitor and rectify unsafe behaviors. The shields can dynamically split, merge, and recompute based on agents' states. This design enables efficient synthesis of shields to monitor agents in complex environments without coordination overheads. We also propose an algorithm to synthesize shields without prior knowledge of the dynamics model. The proposed algorithm obtains an approximate world model by interacting with the environment during the early stage of exploration, making our MBDS enjoy formal safety guarantees with high probability. We demonstrate in simulations that our framework can surpass existing baselines in terms of safety guarantees and learning performance.Comment: Accepted in AAMAS 202

Reasoning over the Air: A Reasoning-based Implicit Semantic-Aware Communication Framework

Semantic-aware communication is a novel paradigm that draws inspiration from human communication focusing on the delivery of the meaning of messages. It has attracted significant interest recently due to its potential to improve the efficiency and reliability of communication and enhance users' QoE. Most existing works focus on transmitting and delivering the explicit semantic meaning that can be directly identified from the source signal. This paper investigates the implicit semantic-aware communication in which the hidden information that cannot be directly observed from the source signal must be recognized and interpreted by the intended users. To this end, a novel implicit semantic-aware communication (iSAC) architecture is proposed for representing, communicating, and interpreting the implicit semantic meaning between source and destination users. A projection-based semantic encoder is proposed to convert the high-dimensional graphical representation of explicit semantics into a low-dimensional semantic constellation space for efficient physical channel transmission. To enable the destination user to learn and imitate the implicit semantic reasoning process of source user, a generative adversarial imitation learning-based solution, called G-RML, is proposed. Different from existing communication solutions, the source user in G-RML does not focus only on sending as much of the useful messages as possible; but, instead, it tries to guide the destination user to learn a reasoning mechanism to map any observed explicit semantics to the corresponding implicit semantics that are most relevant to the semantic meaning. Compared to the existing solutions, our proposed G-RML requires much less communication and computational resources and scales well to the scenarios involving the communication of rich semantic meanings consisting of a large number of concepts and relations.Comment: accepted at IEEE Transactions on Wireless Communication

Exogenous Application of a Plant Elicitor Induces Volatile Emission in Wheat and Enhances the Attraction of an Aphid Parasitoid Aphidius gifuensis.

peer reviewedIt is well known that plant elicitors can induce plant defense against pests. The herbivore-induced plant volatile (HIPV) methyl salicylate (MeSA), as a signaling hormone involved in plant pathogen defense, is used to recruit natural enemies to protect wheat and other crops. However, the defense mechanism remains largely unknown. Here, the headspace volatiles of wheat plants were collected and analyzed by gas chromatography-mass spectrometry (GC-MS), gas chromatography with electroantennographic detection (GC-EAD) and principal component analysis (PCA). The results showed that exogenous application of MeSA induced qualitative and quantitative changes in the volatiles emitted from wheat plants, and these changes were mainly related to Carveol, Linalool, m-Diethyl-benzene, p-Cymene, Nonanal, D-limonene and 6-methyl-5-Hepten-2-one. Then, the electroantennogram (EAG) and Y-tube bioassay were performed to test the physiological and behavioral responses of Aphidius gifuensis Ashmesd to the active volatile compounds (p-Cymene, m-Diethyl-benzene, Carveol) that identified by using GC-EAD. The female A. gifuensis showed strong physiological responses to 1 μg/μL p-Cymene and 1 μg/μL m-Diethyl-benzene. Moreover, a mixture blend was more attractive to female A. gifuensis than a single compound. These findings suggested that MeSA could induce wheat plant indirect defense against wheat aphids through attracting parasitoid in the wheat agro-ecosystem

Sodium butyrate ameliorates gut dysfunction and motor deficits in a mouse model of Parkinson’s disease by regulating gut microbiota

BackgroundA growing body of evidence showed that gut microbiota dysbiosis might be associated with the pathogenesis of Parkinson’s disease (PD). Microbiota-targeted interventions could play a protective role in PD by regulating the gut microbiota-gut-brain axis. Sodium butyrate (NaB) could improve gut microbiota dysbiosis in PD and other neuropsychiatric disorders. However, the potential mechanism associated with the complex interaction between NaB and gut microbiota-gut-brain communication in PD needs further investigation.MethodsC57BL/6 mice were subjected to a rotenone-induced PD model and were treated intragastrically with NaB for 4 weeks. The gut function and motor function were evaluated. The α-synuclein expression in colon and substantia nigra were detected by western blotting. Tyrosine hydroxylase (TH)-positive neurons in substantia nigra were measured by immunofluorescence. Moreover, gut microbiota composition was analyzed by 16S rRNA sequencing. Fecal short chain fatty acids (SCFAs) levels were determined by liquid chromatography tandem mass spectrometry (LC–MS). The levels of glucagon like peptide-1 (GLP-1) in tissues and serum were evaluated using enzyme-linked immunosorbent assay (ELISA).ResultsNaB ameliorated gut dysfunction and motor deficits in rotenone-induced mice. Meanwhile, NaB protected against rotenone-induced α-synuclein expression in colon and substantia nigra, and prevented the loss of TH-positive neurons. In addition, NaB could remodel gut microbiota composition, and regulate gut SCFAs metabolism, and restore GLP-1 levels in colon, serum, and substantia nigra in PD mice.ConclusionNaB could ameliorate gut dysfunction and motor deficits in rotenone-induced PD mice, and the mechanism might be associated with the regulation of gut microbiota dysbiosis

Sacroiliac screws fixation navigated with three-dimensional printing personalized guide template for the treatment of posterior pelvic ring injury: A case report

ObjectivePelvic injuries refer to the disruption of the inherent structural and mechanical integrity of the pelvic ring. Sacroiliac screw fixation technique is often used for the treatment of posterior pelvic ring injury, which is prone to the iatrogenic injury. Various attempts were proposed to avoid iatrogenic injuries, while the executing processes are usually too cumbersome. The patient-personalized guide template based on 3D printing technology has been considered as a promising method, which can achieve lower deviation and higher accuracy in a simple and convenient way. We reported the first case of posterior pelvic ring injury using 3D printing personalized guide template with the verification of intraoperative CT.MethodsThe subject was a 74-year-old female with posterior pelvic ring injury. Two patient-specific guide templates were customized based on 3D printing technology, one for S1 and the other for S2. We used the guide templates for navigation to place the sacroiliac screws. The placement of screws was verified by intraoperative CT. Intraoperative and postoperative variables were collected.ResultsThe technique helped us successfully insert the sacroiliac screws into the safe zone. The intraoperative blood loss was 23.03 ml, and the duration of operation was 62 min. The exposure dose during CT scanning was 7.025 mSv. The assessment of screws position was excellent. Furthermore, there was no sign of any functional impairment postoperatively.ConclusionSacroiliac screws fixation with the assistance of 3D printing personalized guide template under the verification of intraoperative CT may be a promising method to treat posterior pelvic ring injuries