Discrete Message via Online Clustering Labels in Decentralized POMDP

Communication is crucial for solving cooperative Multi-Agent Reinforcement Learning tasks in Partially-Observable Markov Decision Processes. Existing works often rely on black-box methods to encode local information/features into messages shared with other agents. However, such black-box approaches are unable to provide any quantitative guarantees on the expected return and often lead to the generation of continuous messages with high communication overhead and poor interpretability. In this paper, we establish an upper bound on the return gap between an ideal policy with full observability and an optimal partially-observable policy with discrete communication. This result enables us to recast multi-agent communication into a novel online clustering problem over the local observations at each agent, with messages as cluster labels and the upper bound on the return gap as clustering loss. By minimizing the upper bound, we propose a surprisingly simple design of message generation functions in multi-agent communication and integrate it with reinforcement learning using a Regularized Information Maximization loss function. Evaluations show that the proposed discrete communication significantly outperforms state-of-the-art multi-agent communication baselines and can achieve nearly-optimal returns with few-bit messages that are naturally interpretable

Scalable Multi-agent Skill Discovery based on Kronecker Graphs

Covering skill (a.k.a., option) discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent option discovery either become prohibitive or fail to directly discover joint options that improve the connectivity of the joint state space. In this paper, we show how to directly compute multi-agent options with collaborative exploratory behaviors while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents' transition graphs. Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques. The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms the state-of-the-art. Codes are available at: https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP.Comment: Accepted to NeurIPS 2022. arXiv admin note: substantial text overlap with arXiv:2201.0822

Covering the Cover

Background/Aims: Endoscopic submucosal dissection has been widely accepted. At present, the number of antiplatelet (APT) users has been growing. Moreover, because of high risks of thromboembolism, some patients need to continuously receive APT agents. The relationship between hemorrhage and continuous therapy with low-dose aspirin (LDA) remains controversial. Materials and Methods: A systematic search was conducted; studies were screened out- if data of no-anticoagulant/APT drugs use and interrupted and continued-LDA use were reported separately. The Newcastle-scale was chosen to assess the quality of the included studies. Review Manager 5.2 was used for quality assessment statistical analysis, and the odd ratio (OR) and 95% confidence interval (CI) were calculated. Results: Pooled data suggested a significantly higher bleeding ratio in the LDA-continued group compared to both the LDA-interrupted group (OR=2.05, 95% CI=1.05-3.99) and no-anticoagulant/APT group (OR=2.89, 95% CI=1.86-4.47). However, the LDA-interrupted group did not differ significantly from the no-anticoagulant/APT group. The en bloc resection rates of the LDA-continued group versus the LDA-interrupted group, the LDAcontinued group versus no-anticoagulant/APT group, and the LDA-interrupted group versus the no-anticoagulant/APT group were similar (OR=0.82, 95% CI=0.21-3.24, p=0.78; OR=0.80, 95% CI=0.24-2.65, p=0.71; OR=1.41, 95% CI=0.38-5.24, p=0.60, respectively). Conclusion: There is an extremely high ratio of bleeding in the LDA-continued group compared to both the LDA-interrupted group and no-anticoagulant/APT group. All groups had similar ratios of en bloc resection

RIDE: Real-time Intrusion Detection via Explainable Machine Learning Implemented in a Memristor Hardware Architecture

Deep Learning (DL) based methods have shown great promise in network intrusion detection by identifying malicious network traffic behavior patterns with high accuracy, but their applications to real-time, packet-level detections in high-speed communication networks are challenging due to the high computation time and resource requirements of Deep Neural Networks (DNNs), as well as lack of explainability. To this end, we propose a packet-level network intrusion detection solution that makes novel use of Recurrent Autoencoders to integrate an arbitrary-length sequence of packets into a more compact joint feature embedding, which is fed into a DNN-based classifier. To enable explainability and support real-time detections at micro-second speed, we further develop a Software-Hardware Co-Design approach to efficiently realize the proposed solution by converting the learned detection policies into decision trees and implementing them using an emerging architecture based on memristor devices. By jointly optimizing associated software and hardware constraints, we show that our approach leads to an extremely efficient, real-time solution with high detection accuracy at the packet level. Evaluation results on real-world datasets (e.g., UNSW and CIC-IDS datasets) demonstrate nearly three-nines detection accuracy with a substantial speedup of nearly four orders of magnitude

Ru doping induced spin frustration and enhancement of the room-temperature anomalous Hall effect in La2/3Sr1/3MnO3 films

In transition-metal-oxide heterostructures, the anomalous Hall effect (AHE) is a powerful tool for detecting the magnetic state and revealing intriguing interfacial magnetic orderings. However, achieving a larger AHE at room temperature in oxide heterostructures is still challenging due to the dilemma of mutually strong spin-orbit coupling and magnetic exchange interactions. Here, we exploit the Ru doping-enhanced AHE in LSMRO epitaxial films. As the B-site Ru doping level increases up to 20 percent, the anomalous Hall resistivity at room temperature can be enhanced from nOhmcm to uOhmcm scale. Ru doping leads to strong competition between ferromagnetic double-exchange interaction and antiferromagnetic super-exchange interaction. The resultant spin frustration and spin-glass state facilitate a strong skew-scattering process, thus significantly enhancing the extrinsic AHE. Our findings could pave a feasible approach for boosting the controllability and reliability of oxide-based spintronic devices

Super-tetragonal Sr4Al2O7: a versatile sacrificial layer for high-integrity freestanding oxide membranes

Releasing the epitaxial oxide heterostructures from substrate constraints leads to the emergence of various correlated electronic phases and paves the way for integrations with advanced semiconductor technologies. Identifying a suitable water-soluble sacrificial layer, compatible with the high-quality epitaxial growth of oxide heterostructures, is currently the key to the development of large-scale freestanding oxide membranes. In this study, we unveil the super-tetragonal Sr4Al2O7 (SAOT) as a promising water-soluble sacrificial layer. The distinct low-symmetric crystal structure of SAOT enables a superior capability to sustain epitaxial strain, thus allowing for broad tunability in lattice constants. The resultant structural coherency and defect-free interface in perovskite ABO3/SAOT heterostructures effectively restrain crack formations during the water-assisted release of freestanding oxide membranes. For a variety of non-ferroelectric oxide membranes, the crack-free areas can span up to a few millimeters in length scale. These compelling features, combined with the inherent high-water solubility, make SAOT a versatile and feasible sacrificial layer for producing high-quality freestanding oxide membranes, thereby boosting their potential for innovative oxide electronics and flexible device designs.Comment: 5 figures and SI, it is the second version of this manuscrip

Gut microbe and hepatic macrophage polarization in non-alcoholic fatty liver disease

Non-alcoholic fatty liver disease (NAFLD) is a common chronic hepatic disorder with the potential to progress to hepatic fibrosis, hepatic cirrhosis, and even hepatocellular carcinoma. Activation of hepatic macrophages, important innate immune cells predominantly composed of Kupffer cells, plays a pivotal role in NAFLD initiation and progression. Recent findings have underscored the regulatory role of microbes in both local and distal immune responses, including in the liver, emphasizing their contribution to NAFLD initiation and progression. Key studies have further revealed that gut microbes can penetrate the intestinal mucosa and translocate to the liver, thereby directly influencing hepatic macrophage polarization and NAFLD progression. In this review, we discuss recent evidence regarding the translocation of intestinal microbes into the liver, as well as their impact on hepatic macrophage polarization and associated cellular and molecular signaling pathways. Additionally, we summarize the potential mechanisms by which translocated microbes may activate hepatic macrophages and accelerate NAFLD progression

Learning Multi-agent Options for Tabular Reinforcement Learning using Factor Graphs

Covering option discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. However, these option discovery methods cannot be directly extended to multi-agent scenarios, since the joint state space grows exponentially with the number of agents in the system. Thus, existing researches on adopting options in multi-agent scenarios still rely on single-agent option discovery and fail to directly discover the joint options that can improve the connectivity of the joint state space of agents. In this paper, we show that it is indeed possible to directly compute multi-agent options with collaborative exploratory behaviors among the agents, while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph -- the Kronecker product of individual agents' state transition graphs, based on which we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual agents' transition graphs. This decomposition enables us to efficiently construct multi-agent joint options by encouraging agents to connect the sub-goal joint states which are corresponding to the minimum or maximum values of the estimated joint Fiedler vector. The evaluation based on multi-agent collaborative tasks shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options, in terms of both faster exploration and higher cumulative rewards

Research progress on dental materials for preventing root caries

The high incidence and untreated rate of root caries, a common and frequently occurring oral disease with challenging treatment in elderly individuals, is the main cause of tooth loss among elderly people, as rapid development results in pulpitis and periapical periodontitis or residual crown and root, which has been regarded as one of the common chronic oral diseases seriously affecting the quality of life of elderly people. Thus, early intervention and prevention are important. Traditional dental materials for preventing root caries have been widely used in clinical practice; however, they have the disadvantages of tooth coloring, remineralization and low sterilization efficiency. A series of new dental materials for preventing root caries have gradually become a research hotspot recently, which have the advantages of promoting the mineralization of deep dental tissue, prolonging the action time and enhancing adhesion. Future caries prevention materials should be designed according to the characteristics of root surface caries and the application population and should be developed toward simplicity, high efficiency and low toxicity. This review describes current research regarding anti-caries prevention material application, serving as a theoretical underpinning for the research of root caries prevention materials, which is important for both promotion in the effective prevention of root caries and improvement in the status of oral health and the quality of life among old people

The development of collagen based composite scaffolds for bone regeneration

Bone is consisted of bone matrix, cells and bioactive factors, and bone matrix is the combination of inorganic minerals and organic polymers. Type I collagen fibril made of five triple-helical collagen chains is the main organic polymer in bone matrix. It plays an important role in the bone formation and remodeling process. Moreover, collagen is one of the most commonly used scaffold materials for bone tissue engineering due to its excellent biocompatibility and biodegradability. However, the low mechanical strength and osteoinductivity of collagen limit its wider applications in bone regeneration field. By incorporating different biomaterials, the properties such as porosity, structural stability, osteoinductivity, osteogenicity of collagen matrixes can be largely improved. This review summarizes and categorizes different kinds of biomaterials including bioceramic, carbon and polymer materials used as components to fabricate collagen based composite scaffolds for bone regeneration. Moreover, the possible directions of future research and development in this field are also proposed