Performance analysis of Reinforcement Learning for achieving context-awareness and intelligence in cognitive radio networks

Cognitive Radio (CR) is a novel and promising paradigm for next-generation wireless communication. It is able to sense and change its transmission and reception parameters adaptively according to spectrum availability at different channels. The Cognition Cycle (CC) is a state machine that is embodied in each CR that defines the mechanisms related to achieving context-awareness and intelligence including observation, orientation, learning, planning, decision making, and action selection. The CC is the key element in the design of various schemes in CR networks such as Dynamic Channel Selection (DCS), scheduling and congestion control. Hence, a good implementation of the CC is of paramount importance. In this paper, Reinforcement Learning (RL) is employed to implement the CC. The main focus is to analyze the performance of RL as an approach to achieving context-awareness and intelligence in regard to DCS. The contributions of this paper are twofold. Firstly, we seek to justify whether RL is an appropriate tool to implement the CC. Secondly, we seek to understand the effects of changes on RL parameters on network performance. In addition, we propose solutions for the problems associated with the application of RL in DCS. The results presented in this paper show that RL is a promising approach

Context Awareness and Intelligence in Cognitive Radio Networks: Design and Applications

CR technology, which is the next-generation wireless communication system, improves the utilization of the overall radio spectrum through dynamic adaptation to local spectrum availability. In CR networks, unlicensed or Secondary Users (SUs) may operate in underutilized spectrum (called white spaces) owned by the licensed or Primary Users (PUs) conditional upon PUs encountering acceptably low interference levels. Ideally, the PUs are oblivious to the presence of the SUs. Context awareness enables an SU to sense and observe its operating environment, which is complex and dynamic in nature; while intelligence enables the SU to learn knowledge, which can be acquired through observing the consequences of its prior action, about its operating environment so that it carries out the appropriate action to achieve optimum network performance in an efficient manner without following a strict and static predefined set of policies. Traditionally, without the application of intelligence, each wireless host adheres to a strict and static predefined set of policies, which may not be optimum in many kinds of operating environment. With the application of intelligence, the knowledge changes in line with the dynamic operating environment. This thesis investigates the application of an artificial intelligence approach called reinforcement learning to achieve context awareness and intelligence in order to enable the SUs to sense and utilize the high quality white spaces. To date, the research focus of the CR research community has been primarily on the physical layer of the open system interconnection model. The research into the data link layer is still in its infancy, and our research work focusing on this layer has been pioneering in this field and has attacted considerable international interest. There are four major outcomes in this thesis. Firstly, various types of multi-channel medium access control protocols are reviewed, followed by discussion of their merits and demerits. The purpose is to show the additional functionalities and challenges that each multi-channel medium access control protocol has to offer and address in order to operate in CR networks. Secondly, a novel cross-layer based quality of service architecture called C2net for CR networks is proposed to provide service prioritization and tackle the issues associated with CR networks. Thirdly, reinforcement learning is applied to pursue context awareness and intelligence in both centralized and distributed CR networks. Analysis and simulation results show that reinforcement learning is a promising mechanism to achieve context awareness and intelligence. Fourthly, the versatile reinforcement learning approach is applied in various schemes for performance enhancement in CR networks