Accelerating Reinforcement Learning for Dynamic Spectrum Access in Cognitive Wireless Networks

This thesis studies the applications of distributed reinforcement learning (RL) based machine intelligence to dynamic spectrum access (DSA) in future cognitive wireless networks. In particular, this work focuses on ways of accelerating distributed RL based DSA algorithms in order to improve their adaptability in terms of the initial and steady-state performance, and the quality of service (QoS) convergence behaviour. The performance of the DSA schemes proposed in this thesis is empirically evaluated using large-scale system-level simulations of a temporary event scenario which involves a cognitive small cell network installed in a densely populated stadium, and in some cases a base station on an aerial platform and a number of local primary LTE base stations, all sharing the same spectrum. Some of the algorithms are also theoretically evaluated using a Bayesian network based probabilistic convergence analysis method proposed by the author. The thesis presents novel distributed RL based DSA algorithms that employ a Win-or-Learn-Fast (WoLF) variable learning rate and an adaptation of the heuristically accelerated RL (HARL) framework in order to significantly improve the initial performance and the convergence speed of classical RL algorithms and, thus, increase their adaptability in challenging DSA environments. Furthermore, a distributed case-based RL approach to DSA is proposed. It combines RL and case-based reasoning to increase the robustness and adaptability of distributed RL based DSA schemes in dynamically changing wireless environments

TDA-MAC : TDMA without clock synchronization in underwater acoustic networks

This paper investigates the application of underwater acoustic sensor networks for large scale monitoring of the ocean environment. The low propagation speed of acoustic signals presents a fundamental challenge in coordinating the access to the shared communication medium in such networks. In this paper, we propose two medium access control (MAC) protocols, namely, Transmit Delay Allocation MAC (TDA-MAC) and Accelerated TDA-MAC, that are capable of providing time division multiple access (TDMA) to sensor nodes without the need for centralized clock synchronization. A comprehensive simulation study of a network deployed on the sea bed shows that the proposed protocols are capable of closely matching the throughput and packet delay performance of ideal synchronized TDMA. The TDA-MAC protocols also significantly outperform T-Lohi, a classical contention-based MAC protocol for underwater acoustic networks, in terms of network throughput and, in many cases, end-To-end packet delay. Furthermore, the assumption of no clock synchronization among different devices in the network is a major advantage of TDA-MAC over other TDMA-based MAC protocols in the literature. Therefore, it is a feasible networking solution for real-world underwater sensor network deployments