Optimal Network Control in Partially-Controllable Networks

The effectiveness of many optimal network control algorithms (e.g., BackPressure) relies on the premise that all of the nodes are fully controllable. However, these algorithms may yield poor performance in a partially-controllable network where a subset of nodes are uncontrollable and use some unknown policy. Such a partially-controllable model is of increasing importance in real-world networked systems such as overlay-underlay networks. In this paper, we design optimal network control algorithms that can stabilize a partially-controllable network. We first study the scenario where uncontrollable nodes use a queue-agnostic policy, and propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm with an explicit learning of the policy used by uncontrollable nodes. Next, we investigate the scenario where uncontrollable nodes use a queue-dependent policy and the problem is formulated as an MDP with unknown queueing dynamics. We propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), and prove that TUCRL achieves tunable three-way tradeoffs between throughput, delay and convergence rate

RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systems

With the rapid advance of information technology, network systems have become increasingly complex and hence the underlying system dynamics are often unknown or difficult to characterize. Finding a good network control policy is of significant importance to achieve desirable network performance (e.g., high throughput or low delay). In this work, we consider using model-based reinforcement learning (RL) to learn the optimal control policy for queueing networks so that the average job delay (or equivalently the average queue backlog) is minimized. Traditional approaches in RL, however, cannot handle the unbounded state spaces of the network control problem. To overcome this difficulty, we propose a new algorithm, called Reinforcement Learning for Queueing Networks (RL-QN), which applies model-based RL methods over a finite subset of the state space, while applying a known stabilizing policy for the rest of the states. We establish that the average queue backlog under RL-QN with an appropriately constructed subset can be arbitrarily close to the optimal result. We evaluate RL-QN in dynamic server allocation, routing and switching problems. Simulation results show that RL-QN minimizes the average queue backlog effectively

A backhaul adaptation scheme for IAB networks using deep reinforcement learning with recursive discrete choice model

Challenges such as backhaul availability and backhaul scalability have continued to outweigh the progress of integrated access and backhaul (IAB) networks that enable multi-hop backhauling in 5G networks. These challenges, which are predominant in poor wireless channel conditions such as foliage, may lead to high energy consumption and packet losses. It is essential that the IAB topology enables efficient traffic flow by minimizing congestion and increasing robustness to backhaul failure. This article proposes a backhaul adaptation scheme that is controlled by the load on the access side of the network. The routing problem is formulated as a constrained Markov decision process and solved using a dual decomposition approach due to the existence of explicit and implicit constraints. A deep reinforcement learning (DRL) strategy that takes advantage of a recursive discrete choice model (RDCM) was proposed and implemented in a knowledge-defined networking architecture of an IAB network. The incorporation of the RDCM was shown to improve robustness to backhaul failure in IAB networks. The performance of the proposed algorithm was compared to that of conventional DRL, i.e., without RDCM, and generative model-based learning (GMBL) algorithms. The simulation results of the proposed approach reveal risk perception by introducing certain biases on alternative choices and the results showed that the proposed algorithm provides better throughput and delay performance over the two baselines.The Sentech Chair in Broadband Wireless Multimedia Communications (BWMC) and the University of Pretoria.https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=6287639Electrical, Electronic and Computer Engineerin

Optimal Network Control in Partially-Controllable Networks

The effectiveness of many optimal network control algorithms (e.g., BackPressure) relies on the premise that all of the nodes are fully controllable. However, these algorithms may yield poor performance in a partially-controllable network where a subset of nodes are uncontrollable and use some unknown policy. Such a partially-controllable model is of increasing importance in real-world networked systems such as overlay-underlay networks. In this paper, we design optimal network control algorithms that can stabilize a partially-controllable network. We first study the scenario where uncontrollable nodes use a queue-agnostic policy, and propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm with an explicit learning of the policy used by uncontrollable nodes. Next, we investigate the scenario where uncontrollable nodes use a queue-dependent policy and the problem is formulated as an MDP with unknown queueing dynamics. We propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), and prove that TUCRL achieves tunable three-way tradeoffs between throughput, delay and convergence rate.National Science Foundation (U.S.) (Grant CNS-1524317)United States. Defense Advanced Research Projects Agency (Contract HROO l l-l 5-C-0097