Learning Adaptable Risk-Sensitive Policies to Coordinate in Multi-Agent General-Sum Games

In general-sum games, the interaction of self-interested learning agents commonly leads to socially worse outcomes, such as defect-defect in the iterated stag hunt (ISH). Previous works address this challenge by sharing rewards or shaping their opponents' learning process, which require too strong assumptions. In this paper, we demonstrate that agents trained to optimize expected returns are more likely to choose a safe action that leads to guaranteed but lower rewards. However, there typically exists a risky action that leads to higher rewards in the long run only if agents cooperate, e.g., cooperate-cooperate in ISH. To overcome this, we propose using action value distribution to characterize the decision's risk and corresponding potential payoffs. Specifically, we present Adaptable Risk-Sensitive Policy (ARSP). ARSP learns the distributions over agent's return and estimates a dynamic risk-seeking bonus to discover risky coordination strategies. Furthermore, to avoid overfitting training opponents, ARSP learns an auxiliary opponent modeling task to infer opponents' types and dynamically alter corresponding strategies during execution. Empirically, agents trained via ARSP can achieve stable coordination during training without accessing opponent's rewards or learning process, and can adapt to non-cooperative opponents during execution. To the best of our knowledge, it is the first method to learn coordination strategies between agents both in iterated prisoner's dilemma (IPD) and iterated stag hunt (ISH) without shaping opponents or rewards, and can adapt to opponents with distinct strategies during execution. Furthermore, we show that ARSP can be scaled to high-dimensional settings.Comment: arXiv admin note: substantial text overlap with arXiv:2205.1585

Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network

The recent discovered spatial-temporal information processing capability of bio-inspired Spiking neural networks (SNN) has enabled some interesting models and applications. However designing large-scale and high-performance model is yet a challenge due to the lack of robust training algorithms. A bio-plausible SNN model with spatial-temporal property is a complex dynamic system. Each synapse and neuron behave as filters capable of preserving temporal information. As such neuron dynamics and filter effects are ignored in existing training algorithms, the SNN downgrades into a memoryless system and loses the ability of temporal signal processing. Furthermore, spike timing plays an important role in information representation, but conventional rate-based spike coding models only consider spike trains statistically, and discard information carried by its temporal structures. To address the above issues, and exploit the temporal dynamics of SNNs, we formulate SNN as a network of infinite impulse response (IIR) filters with neuron nonlinearity. We proposed a training algorithm that is capable to learn spatial-temporal patterns by searching for the optimal synapse filter kernels and weights. The proposed model and training algorithm are applied to construct associative memories and classifiers for synthetic and public datasets including MNIST, NMNIST, DVS 128 etc.; and their accuracy outperforms state-of-art approaches

SAP: An IoT Application Module Placement Strategy Based on Simulated Annealing Algorithm in Edge-Cloud Computing

The Internet of Things (IoT) is rapidly growing and provides the foundation for the development of smart cities, smart home, and health care. With more and more devices connecting to the Internet, huge amounts of data are produced, creating a great challenge for data processing. Traditional cloud computing has the problems of long delays. Edge computing is an extension of cloud computing, processing data at the edge of the network can reduce the long processing delay of cloud computing. Due to the limited computing resources of edge servers, resource management of edge servers has become a critical research problem. However, the structural characteristics of the subtask chain between each pair of sensors and actuators are not considered to address the task scheduling problem in most existing research. To reduce processing latency and energy consumption of the edge-cloud system, we propose a multilayer edge computing system. The application deployed in the system is based on directed digraph. To fully use the edge servers, we proposed an application module placement strategy using Simulated Annealing module Placement (SAP) algorithm. The modules in an application are bounded to each sensor. The SAP algorithm is designed to find a module placement scheme for each sensor and to generate a module chain including the mapping of the module and servers for each sensor. Thus, the edge servers can transmit the tuples in the network with the module chain. To evaluate the efficacy of our algorithm, we simulate the strategy in iFogSim. Results show the scheme is able to achieve significant reductions in latency and energy consumption

Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning

Federated learning (FL), as an emerging edge artificial intelligence paradigm, enables many edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order to second-order methods. However, these algorithms cannot be applied in scenarios where the gradient information is not available, e.g., federated black-box attack and federated hyperparameter tuning. To address this issue, in this paper we propose a derivative-free federated zeroth-order optimization (FedZO) algorithm featured by performing multiple local updates based on stochastic gradient estimators in each communication round and enabling partial device participation. Under non-convex settings, we derive the convergence performance of the FedZO algorithm on non-independent and identically distributed data and characterize the impact of the numbers of local iterates and participating edge devices on the convergence. To enable communication-efficient FedZO over wireless networks, we further propose an over-the-air computation (AirComp) assisted FedZO algorithm. With an appropriate transceiver design, we show that the convergence of AirComp-assisted FedZO can still be preserved under certain signal-to-noise ratio conditions. Simulation results demonstrate the effectiveness of the FedZO algorithm and validate the theoretical observations.Comment: This work was accepted to Transaction on Signal Processin

Unifying Vision, Text, and Layout for Universal Document Processing

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 8 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark.Comment: CVPR 202

Endonuclease increases efficiency of osteoblast isolation from murine calvariae

Bone is a highly dynamic organ that undergoes remodeling equally regulated by osteoblast-mediated bone formation and osteoclast-mediated bone resorption. To clarify the regulation of osteoblastogenesis, primary murine osteoblasts are required for an in vitro study. Primary osteoblasts are isolated from neonatal calvariae through digestion with collagenase. However, the number of cells collected from one pup is not sufficient for further in vitro experiments, leading to an increase in the use of euthanized pups. We hypothesized that the viscosity of digested calvariae and digestion solution supplemented with collagenase results in cell clumping and reduction of isolated cells from bones. We simply added Benzonase, a genetically engineered endonuclease that shears all forms of DNAs/RNAs, in order to reduce nucleic acid-mediated viscosity. We found that addition of Benzonase increased the number of collected osteoblasts by three fold compared to that without Benzonase through reduction of viscosity. Additionally, Benzonase has no effect on cellular identity and function. The new osteoblast isolation protocol with Benzonase minimizes the number of neonatal pups required for an in vitro study and expands the concept that isolation of other populations of cells including osteocytes that are difficult to be purified could be modified by Benzonase

Insights into the genetic influences of the microbiota on the life span of a host

Escherichia coli (E. coli) mutant strains have been reported to extend the life span of Caenorhabditis elegans (C. elegans). However, the specific mechanisms through which the genes and pathways affect aging are not yet clear. In this study, we fed Drosophila melanogaster (fruit fly) various E. coli single-gene knockout strains to screen mutant strains with an extended lifespan. The results showed that D. melanogaster fed with E. coli purE had the longest mean lifespan, which was verified by C. elegans. We conducted RNA-sequencing and analysis of C. elegans fed with E. coli purE (a single-gene knockout mutant) to further explore the underlying molecular mechanism. We used differential gene expression (DGE) analysis, enrichment analysis, and gene set enrichment analysis (GSEA) to screen vital genes and modules with significant changes in overall expression. Our results suggest that E. coli mutant strains may affect the host lifespan by regulating the protein synthesis rate (cfz-2) and ATP level (catp-4). To conclude, our study could provide new insights into the genetic influences of the microbiota on the life span of a host and a basis for developing anti-aging probiotics and drugs

Correlation-aided method for identification and gradation of periodicities in hydrologic time series

Identification of periodicities in hydrological time series and evaluation of their statistical significance are not only important for water-related studies, but also challenging issues due to the complex variability of hydrological processes. In this article, we develop a “Moving Correlation Coefficient Analysis” (MCCA) method for identifying periodicities of a time series. In the method, the correlation between the original time series and the periodic fluctuation is used as a criterion, aiming to seek out the periodic fluctuation that fits the original time series best, and to evaluate its statistical significance. Consequently, we take periodic components consisting of simple sinusoidal variation as an example, and do statistical experiments to verify the applicability and reliability of the developed method by considering various parameters changing. Three other methods commonly used, harmonic analysis method (HAM), power spectrum method (PSM) and maximum entropy method (MEM) are also applied for comparison. The results indicate that the efficiency of each method is positively connected to the length and amplitude of samples, but negatively correlated with the mean value, variation coefficient and length of periodicity, without relationship with the initial phase of periodicity. For those time series with higher noise component, the developed MCCA method performs best among the four methods. Results from the hydrological case studies in the Yangtze River basin further verify the better performances of the MCCA method compared to other three methods for the identification of periodicities in hydrologic time series

Metabolomics reveals the response of hydroprimed maize to mitigate the impact of soil salinization

Soil salinization is a major environmental stressor hindering global crop production. Hydropriming has emerged as a promising approach to reduce salt stress and enhance crop yields on salinized land. However, a better mechanisitic understanding is required to improve salt stress tolerance. We used a biochemical and metabolomics approach to study the effect of salt stress of hydroprimed maize to identify the types and variation of differentially accumulated metabolites. Here we show that hydropriming significantly increased catalase (CAT) activity, soluble sugar and proline content, decreased superoxide dismutase (SOD) activity and peroxide (H2O2) content. Conversely, hydropriming had no significant effect on POD activity, soluble protein and MDA content under salt stress. The Metabolite analysis indicated that salt stress significantly increased the content of 1278 metabolites and decreased the content of 1044 metabolites. Ethisterone (progesterone) was the most important metabolite produced in the roots of unprimed samples in response to salt s tress. Pathway enrichment analysis indicated that flavone and flavonol biosynthesis, which relate to scavenging reactive oxygen species (ROS), was the most significant metabolic pathway related to salt stress. Hydropriming significantly increased the content of 873 metabolites and significantly decreased the content of 1313 metabolites. 5-Methyltetrahydrofolate, a methyl donor for methionine, was the most important metabolite produced in the roots of hydroprimed samples in response to salt stress. Plant growth regulator, such as melatonin, gibberellin A8, estrone, abscisic acid and brassinolide involved in both treatment. Our results not only verify the roles of key metabolites in resisting salt stress, but also further evidence that flavone and flavonol biosynthesis and plant growth regulator relate to salt tolerance