Macro action selection with deep reinforcement learning in StarCraft

StarCraft (SC) is one of the most popular and successful Real Time Strategy (RTS) games. In recent years, SC is also widely accepted as a challenging testbed for AI research because of its enormous state space, partially observed information, multi-agent collaboration, and so on. With the help of annual AIIDE and CIG competitions, a growing number of SC bots are proposed and continuously improved. However, a large gap remains between the top-level bot and the professional human player. One vital reason is that current SC bots mainly rely on predefined rules to select macro actions during their games. These rules are not scalable and efficient enough to cope with the enormous yet partially observed state space in the game. In this paper, we propose a deep reinforcement learning (DRL) framework to improve the selection of macro actions. Our framework is based on the combination of the Ape-X DQN and the Long-Short-Term-Memory (LSTM). We use this framework to build our bot, named as LastOrder. Our evaluation, based on training against all bots from the AIIDE 2017 StarCraft AI competition set, shows that LastOrder achieves an 83% winning rate, outperforming 26 bots in total 28 entrants

The origin and properties of massive prolate galaxies in the Illustris simulation

We study galaxy shapes in the Illustris cosmological hydrodynamic simulation. We find that massive galaxies have a higher probability of being prolate. For galaxies with stellar mass larger than $10^{11}\rm M_{\odot}$, 35 out of total 839 galaxies are prolate. For 21 galaxies with stellar mass larger than $10^{12}\rm M_{\odot}$, 9 are prolate, 4 are triaxial while the others are close to being oblate. There are almost no prolate galaxies with stellar mass smaller than $3\times10^{11}\rm M_{\odot}$. We check the merger history of the prolate galaxies, and find that they are formed by major dry mergers. All the prolate galaxies have at least one such merger, with most having mass ratios between $1:1$ and $1:3$. The gas fraction (gas mass to total baryon mass) of the progenitors is 0-3 percent for nearly all these mergers, except for one whose second progenitor contains $\sim 15\%$ gas mass, while its main progenitor still contains less than $5\%$. For the 35 massive prolate galaxies that we find, 18 of them have minor axis rotation, and their angular momenta mostly come from the spin angular momenta of the progenitors (usually that of the main progenitor). We analyse the merger orbits of these prolate galaxies and find that most of them experienced a nearly radial merger orbit. Oblate galaxies with major dry mergers can have either radial or circular merger orbits. We further discuss various properties of these prolate galaxies, such as spin parameter $\lambda_{\rm R}$, spherical anisotropy parameter $\beta$, dark matter fraction, as well as inner density slopes for the stellar, dark matter and total mass distributions.Comment: Accepted for publication in MNRAS. 24 pages, 14 figure

Efficient Online Learning with Memory via Frank-Wolfe Optimization: Algorithms with Bounded Dynamic Regret and Applications to Control

Projection operations are a typical computation bottleneck in online learning. In this paper, we enable projection-free online learning within the framework of Online Convex Optimization with Memory (OCO-M) -- OCO-M captures how the history of decisions affects the current outcome by allowing the online learning loss functions to depend on both current and past decisions. Particularly, we introduce the first projection-free meta-base learning algorithm with memory that minimizes dynamic regret, i.e., that minimizes the suboptimality against any sequence of time-varying decisions. We are motivated by artificial intelligence applications where autonomous agents need to adapt to time-varying environments in real-time, accounting for how past decisions affect the present. Examples of such applications are: online control of dynamical systems; statistical arbitrage; and time series prediction. The algorithm builds on the Online Frank-Wolfe (OFW) and Hedge algorithms. We demonstrate how our algorithm can be applied to the online control of linear time-varying systems in the presence of unpredictable process noise. To this end, we develop a controller with memory and bounded dynamic regret against any optimal time-varying linear feedback control policy. We validate our algorithm in simulated scenarios of online control of linear time-invariant systems.Comment: The version corrects proofs and updates presentatio

Panel Data Visualization in R (panelView) and Stata (panelview)

We develop an R package panelView and a Stata package panelview for panel data visualization. They are designed to assist causal analysis with panel data and have three main functionalities: (1) They plot the treatment status and missing values in a panel dataset; (2) they visualize the temporal dynamics of the main variables of interest; and (3) they depict the bivariate relationships between a treatment variable and an outcome variable either by unit or in aggregate. These tools can help researchers better understand their panel datasets before conducting statistical analysis

Propagation Characteristics of Seismic Wave While Transiting Interlining in Soil Medium

Based on the theory of wave and the Snell theorem, propagation characteristics of seismic wave, three kinds of elastic body waves, is studied when they pass through soil/elastic solid medium including elastic thin plan intercalation in half-space. The analytic solution of the amplitude rate is the rate of the amplitude of the wave behind the intercalation to that of the incident wave. Compared with the field data, the analytic results are beneficial, affected by soft clay stratum, at the Nankai District of Tianjin Municipality caused by the 1976 Tangshan earthquake (had a magnitude of 7.8). In this area, however, damage of the buildings was generally minor, about 50%, with the Heping District near Nankai

Sparse and Deep Representations for Face Recognition and Object Detection

Face recognition and object detection are two very fundamental visual recognition applications in computer vision. How to learn “good” feature representations using machine learning has become the cornerstone of perception-based systems. A good feature representation is often the one that is robust and discriminative to multiple instances of the same category. Starting from features such as intensity, histogram etc. in the image, followed by hand-crafted features, to the most recent sophisticated deep feature representations, we have witnessed the remarkable improvement in the ability of a feature learning algorithm to perform pattern recognition tasks such as face recognition and object detection. One of the conventional feature learning methods, dictionary learning has been proposed to learn discriminative and sparse representations for visual recognition. These dictionary learning methods can learn both representative and discriminative dictionaries, and the associated sparse representations are effective for vision tasks such as face recognition. More recently, deep features have been widely adopted by the computer vision community owing to the powerful deep neural network, which is capable of distilling information from high dimensional input spaces to a low dimensional semantic space. The research problems which comprise this dissertation lie at the cross section of conventional feature and deep feature learning approaches. Thus, in this dissertation, we study both sparse and deep representations for face recognition and object detection. First, we begin by studying the topic of spare representations. We present a simple thresholded feature learning algorithm under sparse support recovery. We show that under certain conditions, the thresholded feature exactly recovers the nonzero support of the sparse code. Secondly, based on the theoretical guarantees, we derive the model and algorithm named Dictionary Learning for Thresholded Features (DLTF), to learn the dictionary that is optimized for the thresholded feature. The DLTF dictionaries are specifically designed for using the thresholded feature at inference, which prioritize simplicity, efficiency, general usability and theoretical guarantees. Both synthetic simulations and real-data experiments (i.e. image clustering and unsupervised hashing) verify the competitive quantitative results and remarkable efficiency of applying thresholded features with DLTF dictionaries. Continuing our focus on investigating the sparse representation and its application to computer vision tasks, we address the sparse representations for unconstrained face verification/recognition problem. In the first part, we address the video-based face recognition problem since it brings more challenges due to the fact that the videos are often acquired under significant variations in poses, expressions, lighting conditions and backgrounds. In order to extract representations that are robust to these variations, we propose a structured dictionary learning framework. Specifically, we employ dictionary learning and low-rank approximation methods to preserve the invariant structure of face images in videos. The learned structured dictionary is both discriminative and reconstructive. We demonstrate the effectiveness of our approach through extensive experiments on three video-based face recognition datasets. Recently, template-based face verification has gained more popularity. Unlike traditional verification tasks, which evaluate on image-to-image or video-to-video pairs, template-based face verification/recognition methods can exploit training and/or gallery data containing a mixture of both images or videos from the person of interest. In the second part, we propose a regularized sparse coding approach for template-based face verification. First, we construct a reference dictionary, which represents the training set. Then we learn the discriminative sparse codes of the templates for verification through the proposed template regularized sparse coding approach. Finally, we measure the similarity between templates. However, in real world scenarios, training and test data are sampled from different distributions. Therefore, we also extend the dictionary learning techniques to tackle the domain adaptation problem, where the data from the training set (source domain) and test set (target domain) have different underlying distributions (domain shift). We propose a domain-adaptive dictionary learning framework to model the domain shift by generating a set of intermediate domains. These intermediate domains bridge the gap between the source and target domains. Specifically, we not only learn a common dictionary to encode the domain-shared features but also learn a set of domain specific dictionaries to model the domain shift. This separation enables us to learn more compact and reconstructive dictionaries for domain adaptation. The domain-adaptive features for recognition are finally derived by aligning all the recovered feature representations of both source and target along the domain path. We evaluate our approach on both cross-domain face recognition and object classification tasks. Finally, we study another fundamental problem in computer vision: generic object detection. Object detection has become one of the most valuable pattern recognition tasks, with great benefits in scene understanding, face recognition, action recognition, robotics and self-driving vehicles, etc. We propose a novel object detector named "Deep Regionlets" by blending deep learning and the traditional regionlet method. The proposed framework "Deep Regionlets" is able to address the limitations of traditional regionlet methods, leading to significant precision improvement by exploiting the power of deep convolutional neural networks. Furthermore, we conduct a detailed analysis of our approach to understand its merits and properties. Extensive experiments on two detection benchmark datasets show that the proposed deep regionlet approach outperforms several state-of-the-art competitors

Mass Dependence of Galaxy-Halo Alignment in LOWZ and CMASS

We measure the galaxy-ellipticity (GI) correlations for the Slogan Digital Sky Survey DR12 LOWZ and CMASS samples with the shape measurements from the DESI Legacy Imaging Surveys. We model the GI correlations in an N-body simulation with our recent accurate stellar-halo mass relation from the Photometric object Around Cosmic webs (PAC) method. The large data set and our accurate modeling turns out an accurate measurement of the alignment angle between central galaxies and their host halos. We find that the alignment of central {\textit {elliptical}} galaxies with their host halos increases monotonically with galaxy stellar mass or host halo mass, which can be well described by a power law for the massive galaxies. We also find that central elliptical galaxies are more aligned with their host halos in LOWZ than in CMASS, which might indicate an evolution of galaxy-halo alignment, though future studies are needed to verify this is not induced by the sample selections. In contrast, central {\textit {disk}} galaxies are aligned with their host halos about 10 times more weakly in the GI correlation. These results have important implications for intrinsic alignment (IA) correction in weak lensing studies, IA cosmology, and theory of massive galaxy formation.Comment: 9 pages, 4 figures. Accepted for publication in Ap