Reinforcement Learning in Multiple-UAV Networks: Deployment and Movement Design

A novel framework is proposed for quality of experience (QoE)-driven deployment and dynamic movement of multiple unmanned aerial vehicles (UAVs). The problem of joint non-convex three-dimensional (3D) deployment and dynamic movement of the UAVs is formulated for maximizing the sum mean opinion score (MOS) of ground users, which is proved to be NP-hard. In the aim of solving this pertinent problem, a three-step approach is proposed for attaining 3D deployment and dynamic movement of multiple UAVs. Firstly, genetic algorithm based K-means (GAK-means) algorithm is utilized for obtaining the cell partition of the users. Secondly, Q-learning based deployment algorithm is proposed, in which each UAV acts as an agent, making their own decision for attaining 3D position by learning from trial and mistake. In contrast to conventional genetic algorithm based learning algorithms, the proposed algorithm is capable of training the direction selection strategy offline. Thirdly, Q-learning based movement algorithm is proposed in the scenario that the users are roaming. The proposed algorithm is capable of converging to an optimal state. Numerical results reveal that the proposed algorithms show a fast convergence rate after a small number of iterations. Additionally, the proposed Q-learning based deployment algorithm outperforms K-means algorithms and Iterative-GAKmean (IGK) algorithms with a low complexity

Trajectory Design and Power Control for Multi-UAV Assisted Wireless Networks: A Machine Learning Approach

A novel framework is proposed for the trajectory design of multiple unmanned aerial vehicles (UAVs) based on the prediction of users' mobility information. The problem of joint trajectory design and power control is formulated for maximizing the instantaneous sum transmit rate while satisfying the rate requirement of users. In an effort to solve this pertinent problem, a three-step approach is proposed which is based on machine learning techniques to obtain both the position information of users and the trajectory design of UAVs. Firstly, a multi-agent Q-learning based placement algorithm is proposed for determining the optimal positions of the UAVs based on the initial location of the users. Secondly, in an effort to determine the mobility information of users based on a real dataset, their position data is collected from Twitter to describe the anonymous user-trajectories in the physical world. In the meantime, an echo state network (ESN) based prediction algorithm is proposed for predicting the future positions of users based on the real dataset. Thirdly, a multi-agent Q-learning based algorithm is conceived for predicting the position of UAVs in each time slot based on the movement of users. In this algorithm, multiple UAVs act as agents to find optimal actions by interacting with their environment and learn from their mistakes. Additionally, we also prove that the proposed multi-agent Q-learning based trajectory design and power control algorithm can converge under mild conditions. Numerical results are provided to demonstrate that as the size of the reservoir increases, the proposed ESN approach improves the prediction accuracy. Finally, we demonstrate that throughput gains of about 17% are achieved

Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networks

Unmanned aerial vehicles (UAVs) are capable of serving as aerial base stations (BSs) for providing both cost-effective and on-demand wireless communications. This article investigates dynamic resource allocation of multiple UAVs enabled communication networks with the goal of maximizing long-term rewards. More particularly, each UAV communicates with a ground user by automatically selecting its communicating users, power levels and subchannels without any information exchange among UAVs. To model the uncertainty of environments, we formulate the long-term resource allocation problem as a stochastic game for maximizing the expected rewards, where each UAV becomes a learning agent and each resource allocation solution corresponds to an action taken by the UAVs. Afterwards, we develop a multi-agent reinforcement learning (MARL) framework that each agent discovers its best strategy according to its local observations using learning. More specifically, we propose an agent-independent method, for which all agents conduct a decision algorithm independently but share a common structure based on Q-learning. Finally, simulation results reveal that: 1) appropriate parameters for exploitation and exploration are capable of enhancing the performance of the proposed MARL based resource allocation algorithm; 2) the proposed MARL algorithm provides acceptable performance compared to the case with complete information exchanges among UAVs. By doing so, it strikes a good tradeoff between performance gains and information exchange overheads.Comment: 30 pages, 8 figure

Exploiting Multiple Access in Clustered Millimeter Wave Networks: NOMA or OMA?

In this paper, we introduce a clustered millimeter wave network with non-orthogonal multiple access (NOMA), where the base station (BS) is located at the center of each cluster and all users follow a Poisson Cluster Process. To provide a realistic directional beamforming, an actual antenna pattern is deployed at all BSs. We provide a nearest-random scheme, in which near user is the closest node to the corresponding BS and far user is selected at random, to appraise the coverage performance and universal throughput of our system. Novel closed-form expressions are derived under a loose network assumption. Moreover, we present several Monte Carlo simulations and numerical results, which show that: 1) NOMA outperforms orthogonal multiple access regarding the system rate; 2) the coverage probability is proportional to the number of possible NOMA users and a negative relationship with the variance of intra-cluster receivers; and 3) an optimal number of the antenna elements is existed for maximizing the system throughput.Comment: This paper has been accepted by IEEE International Conference on Communications (ICC), May, USA, 2018. Please cite the format version of this pape

Modeling and Analysis of MmWave Communications in Cache-enabled HetNets

In this paper, we consider a novel cache-enabled heterogeneous network (HetNet), where macro base stations (BSs) with traditional sub-6 GHz are overlaid by dense millimeter wave (mmWave) pico BSs. These two-tier BSs, which are modeled as two independent homogeneous Poisson Point Processes, cache multimedia contents following the popularity rank. High-capacity backhauls are utilized between macro BSs and the core server. A maximum received power strategy is introduced for deducing novel algorithms of the success probability and area spectral efficiency (ASE). Moreover, Monte Carlo simulations are presented to verify the analytical conclusions and numerical results demonstrate that: 1) the proposed HetNet is an interference-limited system and it outperforms the traditional HetNets; 2) there exists an optimal pre-decided rate threshold that contributes to the maximum ASE; and 3) 73 GHz is the best mmWave carrier frequency regarding ASE due to the large antenna scale.Comment: This paper has been accepted by IEEE International Conference on Communications (ICC), May, USA, 2018. Please cite the format version of this pape

Comparative Studies of 10 Programming Languages within 10 Diverse Criteria -- a Team 7 COMP6411-S10 Term Report

There are many programming languages in the world today.Each language has their advantage and disavantage. In this paper, we will discuss ten programming languages: C++, C#, Java, Groovy, JavaScript, PHP, Schalar, Scheme, Haskell and AspectJ. We summarize and compare these ten languages on ten different criterion. For example, Default more secure programming practices, Web applications development, OO-based abstraction and etc. At the end, we will give our conclusion that which languages are suitable and which are not for using in some cases. We will also provide evidence and our analysis on why some language are better than other or have advantages over the other on some criterion.Comment: 139 pages, programming languages comparison table

When Machine Learning Meets Big Data: A Wireless Communication Perspective

We have witnessed an exponential growth in commercial data services, which has lead to the 'big data era'. Machine learning, as one of the most promising artificial intelligence tools of analyzing the deluge of data, has been invoked in many research areas both in academia and industry. The aim of this article is twin-fold. Firstly, we briefly review big data analysis and machine learning, along with their potential applications in next-generation wireless networks. The second goal is to invoke big data analysis to predict the requirements of mobile users and to exploit it for improving the performance of "social network-aware wireless". More particularly, a unified big data aided machine learning framework is proposed, which consists of feature extraction, data modeling and prediction/online refinement. The main benefits of the proposed framework are that by relying on big data which reflects both the spectral and other challenging requirements of the users, we can refine the motivation, problem formulations and methodology of powerful machine learning algorithms in the context of wireless networks. In order to characterize the efficiency of the proposed framework, a pair of intelligent practical applications are provided as case studies: 1) To predict the positioning of drone-mounted areal base stations (BSs) according to the specific tele-traffic requirements by gleaning valuable data from social networks. 2) To predict the content caching requirements of BSs according to the users' preferences by mining data from social networks. Finally, open research opportunities are identified for motivating future investigations.Comment: This article has been accepted by IEEE Vehicular Technology Magazin

Non-Orthogonal Multiple Access for Air-to-Ground Communication

This paper investigates ground-aerial uplink non-orthogonal multiple access (NOMA) cellular networks. A rotary-wing unmanned aerial vehicle (UAV) user and multiple ground users (GUEs) are served by ground base stations (GBSs) by utilizing the uplink NOMA protocol. The UAV is dispatched to upload specific information bits to each target GBSs. Specifically, our goal is to minimize the UAV mission completion time by jointly optimizing the UAV trajectory and UAV-GBS association order while taking into account the UAV's interference to non-associated GBSs. The formulated problem is a mixed integer non-convex problem and involves infinite variables. To tackle this problem, we efficiently check the feasibility of the formulated problem by utilizing graph theory and topology theory. Next, we prove that the optimal UAV trajectory needs to satisfy the \emph{fly-hover-fly} structure. With this insight, we first design an efficient solution with predefined hovering locations by leveraging graph theory techniques. Furthermore, we propose an iterative UAV trajectory design by applying successive convex approximation (SCA) technique, which is guaranteed to coverage to a locally optimal solution. We demonstrate that the two proposed designs exhibit polynomial time complexity. Finally, numerical results show that: 1) the SCA based design outperforms the fly-hover-fly based design; 2) the UAV mission completion time is significantly minimized with proposed NOMA schemes compared with the orthogonal multiple access (OMA) scheme; 3) the increase of GUEs' quality of service (QoS) requirements will increase the UAV mission completion time

Secure Communications in a Unified Non-Orthogonal Multiple Access Framework

This paper investigates the impact of physical layer secrecy on the performance of a unified non-orthogonal multiple access (NOMA) framework, where both external and internal eavesdropping scenarios are examined. The spatial locations of legitimate users (LUs) and eavesdroppers are modeled by invoking stochastic geometry. To characterize the security performance, new exact and asymptotic expressions of secrecy outage probability (SOP) are derived for both code-domain NOMA (CD-NOMA) and power-domain NOMA (PD-NOMA), in which imperfect successive interference cancellation (ipSIC) and perfect SIC (pSIC) are taken into account. For the external eavesdropping scenario, the secrecy diversity orders by a pair of LUs (the n-th user and m-th user) for CD/PD-NOMA are obtained. Analytical results make known that the diversity orders of the $n$-th user with ipSIC/pSIC for CD-NOMA and PD-NOMA are equal to zero/K and zero/one, respectively. The diversity orders of the m-th user are equal to K/one for CD/PD-NOMA. For the internal eavesdropping scenario, we examine the analysis of secrecy diversity order and observe that the m-th user to wiretap the n-th user with ipSIC/pSIC for CD-NOMA and PD-NOMA provide the diversity orders of zero/K and zero/one, respectively, which is consistent with external eavesdropping scenario. Numerical results are present to confirm the accuracy of the analytical results developed and show that: i) The secrecy outage behavior of the $n$-th user is superior to that of the m-th user; ii) By increasing the number of subcarriers, CD-NOMA is capable of achieving a larger secrecy diversity gain compared to PD-NOMA.Comment: 15 pages,10 figure

Reconfigurable Intelligent Surface-assisted Networks: Phase Alignment Categories

The reconfigurable intelligent surface (RIS) is one of the promising technology contributing to the next generation smart radio environment. The application scenarios include massive connectivity support, signal enhancement, and security protection. One crucial difficulty of analyzing the RIS-assisted networks is that the channel performance is sensitive to the change of user receiving direction. This paper tackles the problem by categorizing the RIS illuminated space into four categories: perfect alignment, coherent alignment, random alignment, and destructive alignment. These four categories cover all the possible phase alignment conditions that a user could experience within the overall $2$ pi solid angle of RIS-illuminated space. We perform analysis for each of these categories, deriving analytical expressions for the outage probability and diversity order. Simulation results are presented to confirm the effectiveness of the proposed analytical results.Comment: 6 pages, submitted to IEEE ICC'21 - MWN Symposiu