Accelerating Reinforcement Learning for Dynamic Spectrum Access in Cognitive Wireless Networks

This thesis studies the applications of distributed reinforcement learning (RL) based machine intelligence to dynamic spectrum access (DSA) in future cognitive wireless networks. In particular, this work focuses on ways of accelerating distributed RL based DSA algorithms in order to improve their adaptability in terms of the initial and steady-state performance, and the quality of service (QoS) convergence behaviour. The performance of the DSA schemes proposed in this thesis is empirically evaluated using large-scale system-level simulations of a temporary event scenario which involves a cognitive small cell network installed in a densely populated stadium, and in some cases a base station on an aerial platform and a number of local primary LTE base stations, all sharing the same spectrum. Some of the algorithms are also theoretically evaluated using a Bayesian network based probabilistic convergence analysis method proposed by the author. The thesis presents novel distributed RL based DSA algorithms that employ a Win-or-Learn-Fast (WoLF) variable learning rate and an adaptation of the heuristically accelerated RL (HARL) framework in order to significantly improve the initial performance and the convergence speed of classical RL algorithms and, thus, increase their adaptability in challenging DSA environments. Furthermore, a distributed case-based RL approach to DSA is proposed. It combines RL and case-based reasoning to increase the robustness and adaptability of distributed RL based DSA schemes in dynamically changing wireless environments

TDA-MAC : TDMA without clock synchronization in underwater acoustic networks

This paper investigates the application of underwater acoustic sensor networks for large scale monitoring of the ocean environment. The low propagation speed of acoustic signals presents a fundamental challenge in coordinating the access to the shared communication medium in such networks. In this paper, we propose two medium access control (MAC) protocols, namely, Transmit Delay Allocation MAC (TDA-MAC) and Accelerated TDA-MAC, that are capable of providing time division multiple access (TDMA) to sensor nodes without the need for centralized clock synchronization. A comprehensive simulation study of a network deployed on the sea bed shows that the proposed protocols are capable of closely matching the throughput and packet delay performance of ideal synchronized TDMA. The TDA-MAC protocols also significantly outperform T-Lohi, a classical contention-based MAC protocol for underwater acoustic networks, in terms of network throughput and, in many cases, end-To-end packet delay. Furthermore, the assumption of no clock synchronization among different devices in the network is a major advantage of TDA-MAC over other TDMA-based MAC protocols in the literature. Therefore, it is a feasible networking solution for real-world underwater sensor network deployments

Scalable adaptive networking for the Internet of Underwater Things

Internet of Underwater Things (IoUT) systems comprising tens or hundreds of underwater acoustic communication nodes will become feasible in the near future. The development of scalable networking protocols is a key enabling technology for such IoUT systems, but this task is challenging due to the fundamental limitations of the underwater acoustic communication channel: extremely slow propagation and limited bandwidth. The aim of this paper is to propose the JOIN protocol to enable the integration of new nodes into an existing IoUT network without the control overhead of typical state-of-the-art solutions. The proposed solution is based on the capability of a joining node to incorporate local topology and schedule information into a probabilistic model that allows it to choose when to join the network to minimize the expected number of collisions. The proposed approach is tested in numerical simulations and validated in two sea trials. The simulations show that the JOIN protocol achieves fast convergence to a collision-free solution, fast network adaptation to new nodes, and negligible network disruption due to collisions caused by a joining node. The sea trials demonstrate the practical feasibility of this protocol in real UAN deployments and provide valuable insight for future work on the trade-off between control overhead and reliability of the JOIN protocol in a harsh acoustic communication environment

Interference Cancellation for UWA Random Access Data Packet Transmission

In underwater acoustic (UWA) random access communication networks with multiple users and data packet transmissions, the packet collisions are the main cause of the network performance degradation. The aim of this paper is to investigate interference cancellation (IC) techniques capable of resolving such collisions in a low-complexity modem with single-carrier modulation and single transducer. More specifically, in this modem, the IC is used at multiple stages of the receiver. Firstly, the IC is performed for cancelling the multipath interference to improve the equalization performance in comparison with the linear equalization and Rake combining. Secondly, the IC removes the interference from collided data packets within extracted signal segments after identifying the collisions. Finally, the IC is applied to the received baseband signal to improve the data packet detection. The modem performance is investigated in a lake experiment with intensive multipath channels. The experimental results demonstrate high detection performance of the proposed modem design and show that the proposed IC techniques can significantly improve the throughput of random access UWA networks.Comment: 13 pages, 13 figure