Queue-Learning: A Reinforcement Learning Approach for Providing Quality of Service

End-to-end delay is a critical attribute of quality of service (QoS) in application domains such as cloud computing and computer networks. This metric is particularly important in tandem service systems, where the end-to-end service is provided through a chain of services. Service-rate control is a common mechanism for providing QoS guarantees in service systems. In this paper, we introduce a reinforcement learning-based (RL-based) service-rate controller that provides probabilistic upper-bounds on the end-to-end delay of the system, while preventing the overuse of service resources. In order to have a general framework, we use queueing theory to model the service systems. However, we adopt an RL-based approach to avoid the limitations of queueing-theoretic methods. In particular, we use Deep Deterministic Policy Gradient (DDPG) to learn the service rates (action) as a function of the queue lengths (state) in tandem service systems. In contrast to existing RL-based methods that quantify their performance by the achieved overall reward, which could be hard to interpret or even misleading, our proposed controller provides explicit probabilistic guarantees on the end-to-end delay of the system. The evaluations are presented for a tandem queueing system with non-exponential inter-arrival and service times, the results of which validate our controller's capability in meeting QoS constraints.Comment: 8 pages, Accepted to AAAI 202

Real-time Delay Prediction and Quality of Service Assurance for Traffic Management in Service Systems

Ensuring quality of service (QoS) is one of the important challenges facing service providers. A key metric of the QoS is the experienced delay by the customers, which can depend on various factors such as the demand patterns, service capacity and the scheduling discipline. In this thesis, we study delay distribution prediction in service networks and propose novel QoS assurance methods which target the mentioned factors. In particular, the proposed methods require no knowledge of the system model or parameters, which is an important feature for real-world applications. We first consider the delay distribution prediction problem in tandem and acyclic queueing networks. Our analytical results suggest that the Gaussian mixture model can be a good candidate for estimating the end-to-end delay distribution in these systems. Motivated by this result, we use mixture density networks to propose a delay distribution predictor based on queue length information, which requires no knowledge of the system model or parameters. As the next step, we take a reinforcement learning (RL) approach and propose an admission controller with the goal of providing probabilistic upper-bounds on the end-to-end delay of the accepted jobs, while minimizing the probability of unnecessary rejections. Since admission control might not be a viable solution in some applications, we propose a deep RL-based service-rate controller as an alternative, which is capable of providing probabilistic upper-bounds on the end-to-end delay of the system by dynamic adjustment of the service rates. In the last part of this thesis, we study urban traffic management, as another important example of service systems. Specifically, we introduce two notions of fairness in this context, which are concerned with the delay of the vehicles and throughput of the traffic flows at an intersection. This is particularly important asneglecting fairness can lead to situations where some vehicles experience unacceptable long delays, or where the throughput of a particular traffic flow is highly impacted by the fluctuations of another conflicting flow. Finally, we propose two traffic signal control methods for implementing these fairness notions, which also consider the efficiency of the system at the same time.Ph.D

Distributed Fair Scheduling for Information Exchange in Multi-Agent Systems

Information exchange is a crucial component of many real-world multi-agent systems. However, the communication between the agents involves two major challenges: the limited bandwidth, and the shared communication medium between the agents, which restricts the number of agents that can simultaneously exchange information. While both of these issues need to be addressed in practice, the impact of the latter problem on the performance of the multi-agent systems has often been neglected. This becomes even more important when the agents' information or observations have different importance, in which case the agents require different priorities for accessing the medium and sharing their information. Representing the agents' priorities by fairness weights and normalizing each agent's share by the assigned fairness weight, the goal can be expressed as equalizing the agents' normalized shares of the communication medium. To achieve this goal, we adopt a queueing theoretic approach and propose a distributed fair scheduling algorithm for providing weighted fairness in single-hop networks. Our proposed algorithm guarantees an upper-bound on the normalized share disparity among any pair of agents. This can particularly improve the short-term fairness, which is important in real-time applications. Moreover, our scheduling algorithm adjusts itself dynamically to achieve a high throughput at the same time. The simulation results validate our claims and comparisons with the existing methods show our algorithm's superiority in providing short-term fairness, while achieving a high throughput