Reinforcement Learning Based Handoff for Millimeter Wave Heterogeneous Cellular Networks

The millimeter wave (mmWave) radio band is promising for the next-generation heterogeneous cellular networks (HetNets) due to its large bandwidth available for meeting the increasing demand of mobile traffic. However, the unique propagation characteristics at mmWave band cause huge redundant handoffs in mmWave HetNets if conventional Reference Signal Received Power (RSRP) based handoff mechanism is used. In this paper, we propose a reinforcement learning based handoff policy named LESH to reduce the number of handoffs while maintaining user Quality of Service (QoS) requirements in mmWave HetNets. In LESH, we determine handoff trigger conditions by taking into account both mmWave channel characteristics and QoS requirements of UEs. Furthermore, we propose reinforcement-learning based BS selection algorithms for different UE densities. Numerical results show that in typical scenarios, LESH can significantly reduce the number of handoffs when compared with traditional handoff policies

User Behavior Aware Cell Association in Heterogeneous Cellular Networks

In heterogeneous cellular networks (HetNets), cell association of User Equipment (UE) affects UE transmit rate and network throughput. Conventional cell association rules are usually based on UE received Signal-to-Interference-and-Noise-Ratio (SINR) without taking into account user behaviors, which can indeed be exploited for improving network performance. In this paper, we investigate UE cell association in HetNets based on individual user behavior characteristics with aim to maximize long- term expected system throughput. We model the problem as a stochastic optimization model Restless Multi-Armed Bandit (RMAB). As it is a PSPACE-hard problem, we develop a primal-dual heuristic index algorithm and the solution specifies the rule that determines which arms in the RMAB model to be selected at each decision time. According to the solution of RMAB, we propose a new cell association strategy called Index Enabled Association (IDEA). We also conduct simulation experiments to compare IDEA with conventional max-SINR cell association strategy and an existing game-based RAT selection scheme. Numerical results demonstrate the advantages of IDEA in typical scenarios

Taming and Leveraging Directionality and Blockage in Millimeter Wave Communications

To cope with the challenge for high-rate data transmission, Millimeter Wave(mmWave) is one potential solution. The short wavelength unlatched the era of directional mobile communication. The semi-optical communication requires revolutionary thinking. To assist the research and evaluate various algorithms, we build a motion-sensitive mmWave testbed with two degrees of freedom for environmental sensing and general wireless communication.The first part of this thesis contains two approaches to maintain the connection in mmWave mobile communication. The first one seeks to solve the beam tracking problem using motion sensor within the mobile device. A tracking algorithm is given and integrated into the tracking protocol. Detailed experiments and numerical simulations compared several compensation schemes with optical benchmark and demonstrated the efficiency of overhead reduction. The second strategy attempts to mitigate intermittent connections during roaming is multi-connectivity. Taking advantage of properties of rateless erasure code, a fountain code type multi-connectivity mechanism is proposed to increase the link reliability with simplified backhaul mechanism. The simulation demonstrates the efficiency and robustness of our system design with a multi-link channel record.The second topic in this thesis explores various techniques in blockage mitigation. A fast hear-beat like channel with heavy blockage loss is identified in the mmWave Unmanned Aerial Vehicle (UAV) communication experiment due to the propeller blockage. These blockage patterns are detected through Holm\u27s procedure as a problem of multi-time series edge detection. To reduce the blockage effect, an adaptive modulation and coding scheme is designed. The simulation results show that it could greatly improve the throughput given appropriately predicted patterns. The last but not the least, the blockage of directional communication also appears as a blessing because the geometrical information and blockage event of ancillary signal paths can be utilized to predict the blockage timing for the current transmission path. A geometrical model and prediction algorithm are derived to resolve the blockage time and initiate active handovers. An experiment provides solid proof of multi-paths properties and the numeral simulation demonstrates the efficiency of the proposed algorithm

Seamless coverage for the next generation wireless communication networks

Data demand has exponentially increased due to the rapid growth of wireless and mobile devices traffic in recent years. With the advent of the fifth generation, 5G, and beyond networks, users will be able to take advantage of additional services beyond the capability of current wireless networks while maintaining a highquality experience. The exploitation of millimeter-wave (mm-wave) frequency in 5G promises to meet the demands of future networks with the motto of providing high data rate coverage with low latency to its users, which will allow future networks to function more efficiently. However, while planning a network using mm-wave frequencies, it is important to consider their small coverage footprints and weak penetration resistance. Heterogeneous network planning with the dense deployment of the small cells is one way of overcoming these issues, yet, without proper planning of the integrated network within the same or different frequencies could lead to other problems such as coverage gaps and frequent handovers; due to the natural physics of mm-wave frequencies. Therefore this thesis focuses on bringing ultra-reliable low-latency communication for mm-wave indoor users by increasing the indoor coverage and reducing the frequency of handovers. Towards achieving this thesis’s aim, a detailed literature review of mm-wave coverage is provided in Chapter 2. Moreover, a table that highlights the penetration loss of materials at various frequencies is provided as a result of thorough research in this field, which will be helpful to the researchers investigating this subject. According to our knowledge, this is the first table presenting the most studies that have been conducted in this field. Chapter 3 examines the interference effect of the outdoor base station (BS) inside the building in the context of a heterogeneous network environment. A single building model scenario is created, and the interference analysis is performed to observe the effects of different building materials used as walls. The results reveal the importance of choosing the material type when outdoor BS is close to the building. Moreover, the interference effect of outdoor BS should be minimized when the frequency re-use technique is deployed over very short distances. Chapter 4 presents two-fold contributions, in addition to providing a comprehensive handover study of mm-wave technology. The first study starts with addressing the problem of modelling users’ movement in the indoor environment. Therefore, a user-based indoor mobility prediction via Markov chain with an initial transition matrix is proposed, acquired from Q-learning algorithms. Based on the acquired knowledge of the user’s mobility in the indoor environment, the second contribution of this chapter provides a pre-emptive handover algorithm to provide seamless connection while the user moves within the heterogeneous network. The implementation and evaluation of the proposed algorithm show a reduction in the handover signalling costs by more than 50%, outperforming conventional handover algorithms. Lastly, Chapter 5 contributes to providing robust signal coverage for coverage blind areas and implementing and evaluating the proposed handover algorithm with the intelligent reflective surface. The results show a reduction in the handover signalling costs by more than 33%, outperforming conventional handover algorithms with the pre-emptive handover initiation

QoS-aware User Association and Transmission Scheduling for Millimeter-Wave Train-ground Communications

With the development of wireless communication, people have put forward higher requirements for train-ground communications in the high-speed railway (HSR) scenarios. With the help of mobile relays (MRs) installed on the roof of the train, the application of Millimeter-Wave (mm-wave) communication which has rich spectrum resources to the train-ground communication system can realize high data rate, so as to meet users' increasing demand for broad-band multimedia access. Also, full-duplex (FD) technology can theoretically double the spectral efficiency. In this paper, we formulate the user association and transmission scheduling problem in the mm-wave train-ground communication system with MR operating in the FD mode as a nonlinear programming problem. In order to maximize the system throughput and the number of users meeting quality of service (QoS) requirements, we propose an algorithm based on coalition game to solve the challenging NP-hard problem, and also prove the convergence and Nash-stable structure of the proposed algorithm. Extensive simulation results demonstrate that the proposed coalition game based algorithm can effectively improve the system throughput and meet the QoS requirements of as many users as possible, so that the communication system has a certain QoS awareness.Comment: 14 page

Machine learning enabled millimeter wave cellular system and beyond

Millimeter-wave (mmWave) communication with advantages of abundant bandwidth and immunity to interference has been deemed a promising technology for the next generation network and beyond. With the help of mmWave, the requirements envisioned of the future mobile network could be met, such as addressing the massive growth required in coverage, capacity as well as traffic, providing a better quality of service and experience to users, supporting ultra-high data rates and reliability, and ensuring ultra-low latency. However, due to the characteristics of mmWave, such as short transmission distance, high sensitivity to the blockage, and large propagation path loss, there are some challenges for mmWave cellular network design. In this context, to enjoy the benefits from the mmWave networks, the architecture of next generation cellular network will be more complex. With a more complex network, it comes more complex problems. The plethora of possibilities makes planning and managing a complex network system more difficult. Specifically, to provide better Quality of Service and Quality of Experience for users in the such network, how to provide efficient and effective handover for mobile users is important. The probability of handover trigger will significantly increase in the next generation network, due to the dense small cell deployment. Since the resources in the base station (BS) is limited, the handover management will be a great challenge. Further, to generate the maximum transmission rate for the users, Line-of-sight (LOS) channel would be the main transmission channel. However, due to the characteristics of mmWave and the complexity of the environment, LOS channel is not feasible always. Non-line-of-sight channel should be explored and used as the backup link to serve the users. With all the problems trending to be complex and nonlinear, and the data traffic dramatically increasing, the conventional method is not effective and efficiency any more. In this case, how to solve the problems in the most efficient manner becomes important. Therefore, some new concepts, as well as novel technologies, require to be explored. Among them, one promising solution is the utilization of machine learning (ML) in the mmWave cellular network. On the one hand, with the aid of ML approaches, the network could learn from the mobile data and it allows the system to use adaptable strategies while avoiding unnecessary human intervention. On the other hand, when ML is integrated in the network, the complexity and workload could be reduced, meanwhile, the huge number of devices and data could be efficiently managed. Therefore, in this thesis, different ML techniques that assist in optimizing different areas in the mmWave cellular network are explored, in terms of non-line-of-sight (NLOS) beam tracking, handover management, and beam management. To be specific, first of all, a procedure to predict the angle of arrival (AOA) and angle of departure (AOD) both in azimuth and elevation in non-line-of-sight mmWave communications based on a deep neural network is proposed. Moreover, along with the AOA and AOD prediction, a trajectory prediction is employed based on the dynamic window approach (DWA). The simulation scenario is built with ray tracing technology and generate data. Based on the generated data, there are two deep neural networks (DNNs) to predict AOA/AOD in the azimuth (AAOA/AAOD) and AOA/AOD in the elevation (EAOA/EAOD). Furthermore, under an assumption that the UE mobility and the precise location is unknown, UE trajectory is predicted and input into the trained DNNs as a parameter to predict the AAOA/AAOD and EAOA/EAOD to show the performance under a realistic assumption. The robustness of both procedures is evaluated in the presence of errors and conclude that DNN is a promising tool to predict AOA and AOD in a NLOS scenario. Second, a novel handover scheme is designed aiming to optimize the overall system throughput and the total system delay while guaranteeing the quality of service (QoS) of each user equipment (UE). Specifically, the proposed handover scheme called O-MAPPO integrates the reinforcement learning (RL) algorithm and optimization theory. An RL algorithm known as multi-agent proximal policy optimization (MAPPO) plays a role in determining handover trigger conditions. Further, an optimization problem is proposed in conjunction with MAPPO to select the target base station and determine beam selection. It aims to evaluate and optimize the system performance of total throughput and delay while guaranteeing the QoS of each UE after the handover decision is made. Third, a multi-agent RL-based beam management scheme is proposed, where multiagent deep deterministic policy gradient (MADDPG) is applied on each small-cell base station (SCBS) to maximize the system throughput while guaranteeing the quality of service. With MADDPG, smart beam management methods can serve the UEs more efficiently and accurately. Specifically, the mobility of UEs causes the dynamic changes of the network environment, the MADDPG algorithm learns the experience of these changes. Based on that, the beam management in the SCBS is optimized according the reward or penalty when severing different UEs. The approach could improve the overall system throughput and delay performance compared with traditional beam management methods. The works presented in this thesis demonstrate the potentiality of ML when addressing the problem from the mmWave cellular network. Moreover, it provides specific solutions for optimizing NLOS beam tracking, handover management and beam management. For NLOS beam tracking part, simulation results show that the prediction errors of the AOA and AOD can be maintained within an acceptable range of ±2. Further, when it comes to the handover optimization part, the numerical results show the system throughput and delay are improved by 10% and 25%, respectively, when compared with two typical RL algorithms, Deep Deterministic Policy Gradient (DDPG) and Deep Q-learning (DQL). Lastly, when it considers the intelligent beam management part, numerical results reveal the convergence performance of the MADDPG and the superiority in improving the system throughput compared with other typical RL algorithms and the traditional beam management method