On Efficient Reinforcement Learning for Full-length Game of StarCraft II

StarCraft II (SC2) poses a grand challenge for reinforcement learning (RL), of which the main difficulties include huge state space, varying action space, and a long time horizon. In this work, we investigate a set of RL techniques for the full-length game of StarCraft II. We investigate a hierarchical RL approach involving extracted macro-actions and a hierarchical architecture of neural networks. We investigate a curriculum transfer training procedure and train the agent on a single machine with 4 GPUs and 48 CPU threads. On a 64x64 map and using restrictive units, we achieve a win rate of 99% against the level-1 built-in AI. Through the curriculum transfer learning algorithm and a mixture of combat models, we achieve a 93% win rate against the most difficult non-cheating level built-in AI (level-7). In this extended version of the paper, we improve our architecture to train the agent against the cheating level AIs and achieve the win rate against the level-8, level-9, and level-10 AIs as 96%, 97%, and 94%, respectively. Our codes are at https://github.com/liuruoze/HierNet-SC2. To provide a baseline referring the AlphaStar for our work as well as the research and open-source community, we reproduce a scaled-down version of it, mini-AlphaStar (mAS). The latest version of mAS is 1.07, which can be trained on the raw action space which has 564 actions. It is designed to run training on a single common machine, by making the hyper-parameters adjustable. We then compare our work with mAS using the same resources and show that our method is more effective. The codes of mini-AlphaStar are at https://github.com/liuruoze/mini-AlphaStar. We hope our study could shed some light on the future research of efficient reinforcement learning on SC2 and other large-scale games.Comment: 48 pages,21 figure

Genomic and oncogenic preference of HBV integration in hepatocellular carcinoma

Hepatitis B virus (HBV) can integrate into the human genome, contributing to genomic instability and hepatocarcinogenesis. Here by conducting high-throughput viral integration detection and RNA sequencing, we identify 4,225 HBV integration events in tumour and adjacent non-tumour samples from 426 patients with HCC. We show that HBV is prone to integrate into rare fragile sites and functional genomic regions including CpG islands. We observe a distinct pattern in the preferential sites of HBV integration between tumour and non-tumour tissues. HBV insertional sites are significantly enriched in the proximity of telomeres in tumours. Recurrent HBV target genes are identified with few that overlap. The overall HBV integration frequency is much higher in tumour genomes of males than in females, with a significant enrichment of integration into chromosome 17. Furthermore, a cirrhosis-dependent HBV integration pattern is observed, affecting distinct targeted genes. Our data suggest that HBV integration has a high potential to drive oncogenic transformation

On the Hadronic Origin of High-energy Emission of γ-Ray-loud Narrow-line Seyfert 1 PKS 1502+036

The radiation mechanism of Radio-Loud Narrow-Line Seyfert 1 (RL-NLS1) galaxies from X-ray to γ -ray bands remains an open question. While the leptonic model has been employed to explain the spectral energy distribution (SED), the hadronic process may potentially account for the high-energy radiation of some γ -ray-loud Narrow-Line Seyfert 1 (NLS1) galaxies as well. We study one of such RL-NLS1s, PKS 1502+036, comparing the theoretical SEDs predicted by the leptonic model and the lepto-hadronic model to the observed one. For the hadronic processes, we take into account the proton synchrotron radiation and proton–photon interactions (including the Bethe–Heitler process and the photopion process) including the emission of pairs generated in the electromagnetic cascade initiated by these processes. Our results show that the leptonic model can reproduce the SED of this source, in which the X-ray to γ -ray radiation can be interpreted as the inverse Compton scattering. On the other hand, the proton synchrotron radiation can also explain the high-energy component of SED although extreme parameters are needed. We also demonstrate that the p γ interactions as well as the cascade process cannot explain SED. Our results imply that a leptonic origin is favored for the multiwavelength emission of PKS 1502+036