Trajectory Optimization for UAV Emergency Communication with Limited User Equipment Energy: A safe-DQN Approach

In post-disaster scenarios, it is challenging to provide reliable and flexible emergency communications, especially when the mobile infrastructure is seriously damaged. This article investigates the unmanned aerial vehicle (UAV)-based emergency communication networks, in which UAV is used as the mobile aerial base station for collecting information from ground users in affected areas. Due to the breakdown of ground power system after disasters, the available energy of affected user equipment (UE) is limited. Meanwhile, with the complex geographical conditions after disasters, there are obstacles affecting the flight of UAV. Aiming at maximizing the uplink throughput of UAV networks during the flying time, we formulate the UAV trajectory optimization problem considering UE energy limitation and location of obstacles on the ground. Since the constraint on UE energy is dynamic and long-term cumulative, it is hard to be solved directly. We transform the problem into a constrained Markov decision-making process (CMDP) with UAV as agent. To tackle the CMDP, we propose a safe-deep-Q-network (safe-DQN) based UAV trajectory design algorithm, where the UAV learns to selects the optimal action in reasonable policy sets. Simulation results reveal that: i) the uplink throughput of the proposed algorithm converges within multiple iterations; and ii) compared with the benchmark algorithms, the proposed algorithm performs better in terms of uplink throughput and UE energy efficiency, achieving a good trade-off between UE energy consumption and uplink throughput

RLOps:Development Life-cycle of Reinforcement Learning Aided Open RAN

Radio access network (RAN) technologies continue to witness massive growth, with Open RAN gaining the most recent momentum. In the O-RAN specifications, the RAN intelligent controller (RIC) serves as an automation host. This article introduces principles for machine learning (ML), in particular, reinforcement learning (RL) relevant for the O-RAN stack. Furthermore, we review state-of-the-art research in wireless networks and cast it onto the RAN framework and the hierarchy of the O-RAN architecture. We provide a taxonomy of the challenges faced by ML/RL models throughout the development life-cycle: from the system specification to production deployment (data acquisition, model design, testing and management, etc.). To address the challenges, we integrate a set of existing MLOps principles with unique characteristics when RL agents are considered. This paper discusses a systematic life-cycle model development, testing and validation pipeline, termed: RLOps. We discuss all fundamental parts of RLOps, which include: model specification, development and distillation, production environment serving, operations monitoring, safety/security and data engineering platform. Based on these principles, we propose the best practices for RLOps to achieve an automated and reproducible model development process.Comment: 17 pages, 6 figrue

Guest Editorial: Design and Analysis of Communication Interfaces for Industry 4.0

This special issue (SI) aims to present recent advances in the design and analysis of communication interfaces for Industry 4.0. The Industry 4.0 paradigm aims to integrate advanced manufacturing techniques with Industrial Internet-of-Things (IIoT) to create an agile digital manufacturing ecosystem. The main goal is to instrument production processes by embedding sensors, actuators and other control devices which autonomously communicate with each other throughout the value-chain [1]

Dynamic Aerial Base Station Placement for Minimum-Delay Communications

Queuing delay is of essential importance in the Internet-of-Things scenarios where the buffer sizes of devices are limited. The existing cross-layer research contributions aiming at minimizing the queuing delay usually rely on either transmit power control or dynamic spectrum allocation. Bearing in mind that the transmission throughput is dependent on the distance between the transmitter and the receiver, in this context we exploit the agility of the unmanned-aerial-vehicle (UAV)-mounted base stations (BSs) for proactively adjusting the aerial BS (ABS)'s placement in accordance with wireless teletraffic dynamics. Specifically, we formulate a minimum-delay ABS placement problem for UAV-enabled networks, subject to realistic constraints on the ABS's battery life and velocity. Its solutions are technically realized under three different assumptions in regard to the wireless teletraffic dynamics. The backward induction technique is invoked for both the scenario where the full knowledge of the wireless teletraffic dynamics is available, and for the case where only their statistical knowledge is available. In contrast, a reinforcement learning aided approach is invoked for the case when neither the exact number of arriving packets nor that of their statistical knowledge is available. The numerical results demonstrate that our proposed algorithms are capable of improving the system's performance compared to the benchmark schemes in terms of both the average delay and of the buffer overflow probability

Resource Management and Backhaul Routing in Millimeter-Wave IAB Networks Using Deep Reinforcement Learning

Thesis (PhD (Electronic Engineering))--University of Pretoria, 2023..The increased densification of wireless networks has led to the development of integrated access and backhaul (IAB) networks. In this thesis, deep reinforcement learning was applied to solve resource management and backhaul routing problems in millimeter-wave IAB networks. In the research work, a resource management solution that aims to avoid congestion for access users in an IAB network was proposed and implemented. The proposed solution applies deep reinforcement learning to learn an optimized policy that aims to achieve effective resource allocation whilst minimizing congestion and satisfying the user requirements. In addition, a deep reinforcement learning-based backhaul adaptation strategy that leverages a recursive discrete choice model was implemented in simulation. Simulation results where the proposed algorithms were compared with two baseline methods showed that the proposed scheme provides better throughput and delay performance.Sentech Chair in Broadband Wireless Multimedia Communications.Electrical, Electronic and Computer EngineeringPhD (Electronic Engineering)Unrestricte

A backhaul adaptation scheme for IAB networks using deep reinforcement learning with recursive discrete choice model

Challenges such as backhaul availability and backhaul scalability have continued to outweigh the progress of integrated access and backhaul (IAB) networks that enable multi-hop backhauling in 5G networks. These challenges, which are predominant in poor wireless channel conditions such as foliage, may lead to high energy consumption and packet losses. It is essential that the IAB topology enables efficient traffic flow by minimizing congestion and increasing robustness to backhaul failure. This article proposes a backhaul adaptation scheme that is controlled by the load on the access side of the network. The routing problem is formulated as a constrained Markov decision process and solved using a dual decomposition approach due to the existence of explicit and implicit constraints. A deep reinforcement learning (DRL) strategy that takes advantage of a recursive discrete choice model (RDCM) was proposed and implemented in a knowledge-defined networking architecture of an IAB network. The incorporation of the RDCM was shown to improve robustness to backhaul failure in IAB networks. The performance of the proposed algorithm was compared to that of conventional DRL, i.e., without RDCM, and generative model-based learning (GMBL) algorithms. The simulation results of the proposed approach reveal risk perception by introducing certain biases on alternative choices and the results showed that the proposed algorithm provides better throughput and delay performance over the two baselines.The Sentech Chair in Broadband Wireless Multimedia Communications (BWMC) and the University of Pretoria.https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=6287639Electrical, Electronic and Computer Engineerin