Analysis and Implementation of Reinforcement Learning on a GNU Radio Cognitive Radio Platform

Spectrum today is regulated based on fixed licensees. In the past radio operators have been allocated a frequency band for exclusive use. This has become problem for new users and the modern explosion in wireless services that, having arrived late find there is a scarcity in the remaining available spectrum. Cognitive radio (CR) presents a solution. CRs combine intelligence, spectrum sensing and software reconfigurable radio capabilities. This allows them to opportunistically transmit among several licensed bands for seamless communications, switching to another channel when a licensee is sensed in the original band without causing interference. Enabling this is an intelligent dynamic channel selection strategy capable of finding the best quality channel to transmit on that suffers from the least licensee interruption. This thesis evaluates a Q-learning channel selection scheme using an experimental approach. A cognitive radio deploying the scheme is implemented on GNU Radio and its performance is measured among channels with different utilizations in terms of its packet transmission success rate, goodput and interference caused. We derive similar analytical expressions in the general case of large-scale networks. Our results show that using the Q-learning scheme for channel selection significantly improves the goodput and packet transmission success rate of the system

Route selection for multi-hop cognitive radio networks using reinforcement learning: an experimental study

Cognitive radio (CR) enables unlicensed users to explore and exploit underutilized licensed channels (or white spaces). While multi-hop CR network has drawn significant research interest in recent years, majority work has been validated through simulation. A key challenge in multi-hop CR network is to select a route with high quality of service (QoS) and lesser number of route breakages. In this paper, we propose three route selection schemes to enhance the network performance of CR networks, and investigate them using a real testbed environment, which consists of universal software radio peripheral and GNU radio units. Two schemes are based on reinforcement learning (RL), while a scheme is based on spectrum leasing (SL). RL is an artificial intelligence technique, whereas SL is a new paradigm that allows communication between licensed and unlicensed users in CR networks. We compare the route selection schemes with an existing route selection scheme in the literature, called highest-channel (HC), in a multi-hop CR network. With respect to the QoS parameters (i.e., throughput, packet delivery ratio, and the number of route breakages), the experimental results show that RL approaches achieve a better performance in comparison with the HC approach, and also achieve close to the performance achieved by the SL approach

Implementation of a Space Communications Cognitive Engine

Although communications-based cognitive engines have been proposed, very few have been implemented in a full system, especially in a space communications system. In this paper, we detail the implementation of a multi-objective reinforcement-learning algorithm and deep artificial neural networks for the use as a radio-resource-allocation controller. The modular software architecture presented encourages re-use and easy modification for trying different algorithms. Various trade studies involved with the system implementation and integration are discussed. These include the choice of software libraries that provide platform flexibility and promote reusability, choices regarding the deployment of this cognitive engine within a system architecture using the DVB-S2 standard and commercial hardware, and constraints placed on the cognitive engine caused by real-world radio constraints. The implemented radio-resource allocation-management controller was then integrated with the larger spaceground system developed by NASA Glenn Research Center (GRC)

Cognition-inspired 5G cellular networks: a review and the road ahead

Despite the evolution of cellular networks, spectrum scarcity and the lack of intelligent and autonomous capabilities remain a cause for concern. These problems have resulted in low network capacity, high signaling overhead, inefficient data forwarding, and low scalability, which are expected to persist as the stumbling blocks to deploy, support and scale next-generation applications, including smart city and virtual reality. Fifth-generation (5G) cellular networking, along with its salient operational characteristics - including the cognitive and cooperative capabilities, network virtualization, and traffic offload - can address these limitations to cater to future scenarios characterized by highly heterogeneous, ultra-dense, and highly variable environments. Cognitive radio (CR) and cognition cycle (CC) are key enabling technologies for 5G. CR enables nodes to explore and use underutilized licensed channels; while CC has been embedded in CR nodes to learn new knowledge and adapt to network dynamics. CR and CC have brought advantages to a cognition-inspired 5G cellular network, including addressing the spectrum scarcity problem, promoting interoperation among heterogeneous entities, and providing intelligence and autonomous capabilities to support 5G core operations, such as smart beamforming. In this paper, we present the attributes of 5G and existing state of the art focusing on how CR and CC have been adopted in 5G to provide spectral efficiency, energy efficiency, improved quality of service and experience, and cost efficiency. This main contribution of this paper is to complement recent work by focusing on the networking aspect of CR and CC applied to 5G due to the urgent need to investigate, as well as to further enhance, CR and CC as core mechanisms to support 5G. This paper is aspired to establish a foundation and to spark new research interest in this topic. Open research opportunities and platform implementation are also presented to stimulate new research initiatives in this exciting area

RFRL Gym: A Reinforcement Learning Testbed for Cognitive Radio Applications

Radio Frequency Reinforcement Learning (RFRL) is anticipated to be a widely applicable technology in the next generation of wireless communication systems, particularly 6G and next-gen military communications. Given this, our research is focused on developing a tool to promote the development of RFRL techniques that leverage spectrum sensing. In particular, the tool was designed to address two cognitive radio applications, specifically dynamic spectrum access and jamming. In order to train and test reinforcement learning (RL) algorithms for these applications, a simulation environment is necessary to simulate the conditions that an agent will encounter within the Radio Frequency (RF) spectrum. In this paper, such an environment has been developed, herein referred to as the RFRL Gym. Through the RFRL Gym, users can design their own scenarios to model what an RL agent may encounter within the RF spectrum as well as experiment with different spectrum sensing techniques. Additionally, the RFRL Gym is a subclass of OpenAI gym, enabling the use of third-party ML/RL Libraries. We plan to open-source this codebase to enable other researchers to utilize the RFRL Gym to test their own scenarios and RL algorithms, ultimately leading to the advancement of RL research in the wireless communications domain. This paper describes in further detail the components of the Gym, results from example scenarios, and plans for future additions. Index Terms-machine learning, reinforcement learning, wireless communications, dynamic spectrum access, OpenAI gy

Improving the Efficiency of UAV Communication Channels in the Context of Electronic Warfare

The article is devoted to the development of a method for increasing the efficiency of communication channels of unmanned aerial vehicles (UAVs) in the conditions of electronic warfare (EW). The author analyses the threats that may be caused by the use of electronic warfare against autonomous UAVs. A review of some technologies that can be used to create original algorithms for countering electronic warfare and increasing the autonomy of UAVs on the battlefield is carried out. The structure of modern digital communication systems is considered. The requirements of unmanned aerial vehicle manufacturers for onboard electronic equipment are analyzed, and the choice of the hardware platform of the target radio system is justified. The main idea and novelty of the proposed method are highlighted. The creation of a model of a cognitive radio channel for UAVs is considered step by step. The main steps of modeling the spectral activity of electronic warfare equipment are proposed. The main criteria for choosing a free spectral range are determined. The type of neural network for use in the target cognitive radio system is substantiated. The idea of applying adaptive coding in UAV communication channels using multicomponent turbo codes in combination with neural networks, which are simultaneously used for cognitive radio, has been further developed

Improving the Efficiency of UAV Communication Channels in the Context of Electronic Warfare

The article is devoted to the development of a method for increasing the efficiency of communication channels of unmanned aerial vehicles (UAVs) in the conditions of electronic warfare (EW). The author analyses the threats that may be caused by the use of electronic warfare against autonomous UAVs. A review of some technologies that can be used to create original algorithms for countering electronic warfare and increasing the autonomy of UAVs on the battlefield is carried out. The structure of modern digital communication systems is considered. The requirements of unmanned aerial vehicle manufacturers for onboard electronic equipment are analyzed, and the choice of the hardware platform of the target radio system is justified. The main idea and novelty of the proposed method are highlighted. The creation of a model of a cognitive radio channel for UAVs is considered step by step. The main steps of modeling the spectral activity of electronic warfare equipment are proposed. The main criteria for choosing a free spectral range are determined. The type of neural network for use in the target cognitive radio system is substantiated. The idea of applying adaptive coding in UAV communication channels using multicomponent turbo codes in combination with neural networks, which are simultaneously used for cognitive radio, has been further developed

An RL approach to radio resource management in heterogeneous virtual RANs

Proceedings of: 2021 16th Annual Conference on Wireless On-demand Network Systems and Services Conference (WONS), 9-11 March 2021, Klosters, Switzerland.5G networks are primarily designed to support a wide range of services characterized by diverse key performance indicators (KPIs). A fundamental component of 5G networks, and a pivotal factor to the fulfillment of the services KPIs, is the virtual radio access network (RAN), which allows high flexibility on the control of the radio link. However, to fully exploit the potentiality of virtual RANs in non-stationary environments, an efficient mapping of the rapidly varying context to radio control decisions is not only essential, but also challenging owing to the non-trivial interdependence of network and channel conditions. In this paper, we propose CAREM, an RL framework for dynamic radio resource allocation, which selects the best link and modulation and coding scheme (MCS) for packet transmission, so as to meet the KPI requirements in heterogeneous virtual RANs. To show its effectiveness in real-world conditions, we provide a proof-of-concept through actual testbed implementation. Experimental results demonstrate that CAREM enables an efficient radio resource allocation, for any of the considered time periodicity of the decision-making process.This work has been supported by the EC H2020 5GPPP 5GROWTH project (Grant No. 856709.

Decentralized Spectrum Learning for IoT Wireless Networks Collision Mitigation

This paper describes the principles and implementation results of reinforcement learning algorithms on IoT devices for radio collision mitigation in ISM unlicensed bands. Learning is here used to improve both the IoT network capability to support a larger number of objects as well as the autonomy of IoT devices. We first illustrate the efficiency of the proposed approach in a proof-of-concept based on USRP software radio platforms operating on real radio signals. It shows how collisions with other RF signals present in the ISM band are diminished for a given IoT device. Then we describe the first implementation of learning algorithms on LoRa devices operating in a real LoRaWAN network, that we named IoTligent. The proposed solution adds neither processing overhead so that it can be ran in the IoT devices, nor network overhead so that no change is required to LoRaWAN. Real life experiments have been done in a realistic LoRa network and they show that IoTligent device battery life can be extended by a factor 2 in the scenarios we faced during our experiment