Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first investigate the use of \emph{offline} RL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms, including behavior constrained Q-learning (BCQ), conservative Q-learning (CQL), and implicit Q-learning (IQL), for a specific RRM problem that aims at maximizing a linear combination {of sum and} 5-percentile rates via user scheduling. We observe that the performance of offline RL for the RRM problem depends critically on the behavior policy used for data collection, and further propose a novel offline RL solution that leverages heterogeneous datasets collected by different behavior policies. We show that with a proper mixture of the datasets, offline RL can produce a near-optimal RL policy even when all involved behavior policies are highly suboptimal.Comment: This paper is the camera ready version for Asilomar 202

Modeling Applications for Adaptive QoS-based Resource Management

This paper describes two innovative models that facilitate adaptive QoS-driven resource management in distributed systems comprising heterogeneous computing, storage, and communication resources. The first model, denoted the Logical Application Stream Model (LASM), recursively captures a distributed application's structure, resource requirements, and relevant end-to-end quality-of-service (QoS) parameters. Upon invocation of the application by a user, the resource manager can use this model to initially structure the end-to-end application, allocate resources to this application, and schedule this application on these resources, so as to provide QoS to all applications and to efficiently utilize system resources; later, when the system state changes, the resource manager can use this application model to dynamically reallocate, reschedule, and restructure applications. The recursive nature of the model enables application developers to easily model large-scale applications. We also describe a model, denoted the Benefit Function (BF), that captures user QoS preferences and enables the resource manager to gracefully degrade application QoS under certain conditions