Settling the Variance of Multi-Agent Policy Gradients

Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents’ explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin. Code is released at https://github.com/morning9393/Optimal-Baseline-for-Multi-agent-Policy-Gradients

Descent directions of quasi-Newton methods for symmetric nonlinear equations

2002-2003 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe

An incremental dual nu-support vector regression algorithm

© 2018, Springer International Publishing AG, part of Springer Nature. Support vector regression (SVR) has been a hot research topic for several years as it is an effective regression learning algorithm. Early studies on SVR mostly focus on solving large-scale problems. Nowadays, an increasing number of researchers are focusing on incremental SVR algorithms. However, these incremental SVR algorithms cannot handle uncertain data, which are very common in real life because the data in the training example must be precise. Therefore, to handle the incremental regression problem with uncertain data, an incremental dual nu-support vector regression algorithm (dual-v-SVR) is proposed. In the algorithm, a dual-v-SVR formulation is designed to handle the uncertain data at first, then we design two special adjustments to enable the dual-v-SVR model to learn incrementally: incremental adjustment and decremental adjustment. Finally, the experiment results demonstrate that the incremental dual-v-SVR algorithm is an efficient incremental algorithm which is not only capable of solving the incremental regression problem with uncertain data, it is also faster than batch or other incremental SVR algorithms

Hybrid plasma-catalytic steam reforming of toluene as a biomass tar model compound over Ni/Al₂O₃ catalysts

In this study, plasma-catalytic steam reforming of toluene as a biomass tar model compound was carried out in a coaxial dielectric barrier discharge (DBD) plasma reactor. The effect of Ni/Al2O3 catalysts with different nickel loadings (5–20 wt%) on the plasma-catalytic gas cleaning process was evaluated in terms of toluene conversion, gas yield, by-products formation and energy efficiency of the plasma-catalytic process. Compared to the plasma reaction without a catalyst, the combination of DBD with the Ni/Al2O3 catalysts significantly enhanced the toluene conversion, hydrogen yield and energy efficiency of the hybrid plasma process, while significantly reduced the production of organic by-products. Increasing Ni loading of the catalyst improved the performance of the plasma-catalytic processing of toluene, with the highest toluene conversion of 52% and energy efficiency of 2.6 g/kWh when placing the 20 wt% Ni/Al2O3 catalyst in the plasma. The possible reaction pathways in the hybrid plasma-catalytic process were proposed through the combined analysis of both gas and liquid products

3D time series analysis of cell shape using Laplacian approaches

Background: Fundamental cellular processes such as cell movement, division or food uptake critically depend on cells being able to change shape. Fast acquisition of three-dimensional image time series has now become possible, but we lack efficient tools for analysing shape deformations in order to understand the real three-dimensional nature of shape changes. Results: We present a framework for 3D+time cell shape analysis. The main contribution is three-fold: First, we develop a fast, automatic random walker method for cell segmentation. Second, a novel topology fixing method is proposed to fix segmented binary volumes without spherical topology. Third, we show that algorithms used for each individual step of the analysis pipeline (cell segmentation, topology fixing, spherical parameterization, and shape representation) are closely related to the Laplacian operator. The framework is applied to the shape analysis of neutrophil cells. Conclusions: The method we propose for cell segmentation is faster than the traditional random walker method or the level set method, and performs better on 3D time-series of neutrophil cells, which are comparatively noisy as stacks have to be acquired fast enough to account for cell motion. Our method for topology fixing outperforms the tools provided by SPHARM-MAT and SPHARM-PDM in terms of their successful fixing rates. The different tasks in the presented pipeline for 3D+time shape analysis of cells can be solved using Laplacian approaches, opening the possibility of eventually combining individual steps in order to speed up computations

Reduced stability of mRNA secondary structure near the translation-initiation site in dsDNA viruses

<p>Abstract</p> <p>Background</p> <p>Recent studies have demonstrated a selection pressure for reduced mRNA secondary-structure stability near the start codon of coding sequences. This selection pressure can be observed in bacteria, archaea, and eukaryotes, and is likely caused by the requirement of efficient translation initiation in cellular organism.</p> <p>Results</p> <p>Here, we surveyed the complete genomes of 650 dsDNA virus strains for signals of reduced stability of mRNA secondary structure near the start codon. Our analysis included viruses infecting eukaryotic, prokaryotic, and archaeic hosts. We found that many viruses showed evidence for reduced mRNA secondary-structure stability near the start codon. The effect was most pronounced in viruses infecting prokaryotes, but was also observed in viruses infecting eukaryotes and archaea. The reduction in stability generally increased with increasing genomic GC content. For bacteriophage, the reduction was correlated with a corresponding reduction of stability in the phage hosts.</p> <p>Conclusions</p> <p>We conclude that reduced stability of the mRNA secondary structure near the start codon is a common feature for dsDNA viruses, likely driven by the same selective pressures that cause it in cellular organisms.</p

Of Toasters and Molecular Ticker Tapes

Experiments in systems neuroscience can be seen as consisting of three steps: (1) selecting the signals we are interested in, (2) probing the system with carefully chosen stimuli, and (3) getting data out of the brain. Here I discuss how emerging techniques in molecular biology are starting to improve these three steps. To estimate its future impact on experimental neuroscience, I will stress the analogy of ongoing progress with that of microprocessor production techniques. These techniques have allowed computers to simplify countless problems; because they are easier to use than mechanical timers, they are even built into toasters. Molecular biology may advance even faster than computer speeds and has made immense progress in understanding and designing molecules. These advancements may in turn produce impressive improvements to each of the three steps, ultimately shifting the bottleneck from obtaining data to interpreting it

Scaling properties of protein family phylogenies

One of the classical questions in evolutionary biology is how evolutionary processes are coupled at the gene and species level. With this motivation, we compare the topological properties (mainly the depth scaling, as a characterization of balance) of a large set of protein phylogenies with a set of species phylogenies. The comparative analysis shows that both sets of phylogenies share remarkably similar scaling behavior, suggesting the universality of branching rules and of the evolutionary processes that drive biological diversification from gene to species level. In order to explain such generality, we propose a simple model which allows us to estimate the proportion of evolvability/robustness needed to approximate the scaling behavior observed in the phylogenies, highlighting the relevance of the robustness of a biological system (species or protein) in the scaling properties of the phylogenetic trees. Thus, the rules that govern the incapability of a biological system to diversify are equally relevant both at the gene and at the species level.Comment: Replaced with final published versio

Observation of a ppb mass threshoud enhancement in \psi^\prime\to\pi^+\pi^-J/\psi(J/\psi\to\gamma p\bar{p}) decay

The decay channel $\psi^\prime\to\pi^+\pi^-J/\psi(J/\psi\to\gamma p\bar{p})$ is studied using a sample of $1.06\times 10^8$ $\psi^\prime$ events collected by the BESIII experiment at BEPCII. A strong enhancement at threshold is observed in the $p\bar{p}$ invariant mass spectrum. The enhancement can be fit with an $S$-wave Breit-Wigner resonance function with a resulting peak mass of $M=1861^{+6}_{-13} {\rm (stat)}^{+7}_{-26} {\rm (syst)} {\rm MeV/}c^2$ and a narrow width that is $\Gamma<38 {\rm MeV/}c^2$ at the 90% confidence level. These results are consistent with published BESII results. These mass and width values do not match with those of any known meson resonance.Comment: 5 pages, 3 figures, submitted to Chinese Physics

Generalization and fine mapping of European ancestry-based central adiposity variants in African ancestry populations

Central adiposity measures such as waist circumference (WC) and waist-to-hip ratio (WHR) are associated with cardiometabolic disorders independently of BMI and are gaining clinically utility. Several studies report genetic variants associated with central adiposity, but most utilize only European ancestry populations. Understanding whether the genetic associations discovered among mainly European descendants are shared with African ancestry populations will help elucidate the biological underpinnings of abdominal fat deposition