Towards 5G: A reinforcement learning-based scheduling solution for data traffic management

Dominated by delay-sensitive and massive data applications, radio resource management in 5G access networks is expected to satisfy very stringent delay and packet loss requirements. In this context, the packet scheduler plays a central role by allocating user data packets in the frequency domain at each predefined time interval. Standard scheduling rules are known limited in satisfying higher quality of service (QoS) demands when facing unpredictable network conditions and dynamic traffic circumstances. This paper proposes an innovative scheduling framework able to select different scheduling rules according to instantaneous scheduler states in order to minimize the packet delays and packet drop rates for strict QoS requirements applications. To deal with real-time scheduling, the reinforcement learning (RL) principles are used to map the scheduling rules to each state and to learn when to apply each. Additionally, neural networks are used as function approximation to cope with the RL complexity and very large representations of the scheduler state space. Simulation results demonstrate that the proposed framework outperforms the conventional scheduling strategies in terms of delay and packet drop rate requirements

Sustainable scheduling policies for radio access networks based on LTE technology

A thesis submitted to the University of Bedfordshire in partial fulfilment of the requirements for the degree of Doctor of PhilosophyIn the LTE access networks, the Radio Resource Management (RRM) is one of the most important modules which is responsible for handling the overall management of radio resources. The packet scheduler is a particular sub-module which assigns the existing radio resources to each user in order to deliver the requested services in the most efficient manner. Data packets are scheduled dynamically at every Transmission Time Interval (TTI), a time window used to take the user’s requests and to respond them accordingly. The scheduling procedure is conducted by using scheduling rules which select different users to be scheduled at each TTI based on some priority metrics. Various scheduling rules exist and they behave differently by balancing the scheduler performance in the direction imposed by one of the following objectives: increasing the system throughput, maintaining the user fairness, respecting the Guaranteed Bit Rate (GBR), Head of Line (HoL) packet delay, packet loss rate and queue stability requirements. Most of the static scheduling rules follow the sequential multi-objective optimization in the sense that when the first targeted objective is satisfied, then other objectives can be prioritized. When the targeted scheduling objective(s) can be satisfied at each TTI, the LTE scheduler is considered to be optimal or feasible. So, the scheduling performance depends on the exploited rule being focused on particular objectives. This study aims to increase the percentage of feasible TTIs for a given downlink transmission by applying a mixture of scheduling rules instead of using one discipline adopted across the entire scheduling session. Two types of optimization problems are proposed in this sense: Dynamic Scheduling Rule based Sequential Multi-Objective Optimization (DSR-SMOO) when the applied scheduling rules address the same objective and Dynamic Scheduling Rule based Concurrent Multi-Objective Optimization (DSR-CMOO) if the pool of rules addresses different scheduling objectives. The best way of solving such complex optimization problems is to adapt and to refine scheduling policies which are able to call different rules at each TTI based on the best matching scheduler conditions (states). The idea is to develop a set of non-linear functions which maps the scheduler state at each TTI in optimal distribution probabilities of selecting the best scheduling rule. Due to the multi-dimensional and continuous characteristics of the scheduler state space, the scheduling functions should be approximated. Moreover, the function approximations are learned through the interaction with the RRM environment. The Reinforcement Learning (RL) algorithms are used in this sense in order to evaluate and to refine the scheduling policies for the considered DSR-SMOO/CMOO optimization problems. The neural networks are used to train the non-linear mapping functions based on the interaction among the intelligent controller, the LTE packet scheduler and the RRM environment. In order to enhance the convergence in the feasible state and to reduce the scheduler state space dimension, meta-heuristic approaches are used for the channel statement aggregation. Simulation results show that the proposed aggregation scheme is able to outperform other heuristic methods. When the aggregation scheme of the channel statements is exploited, the proposed DSR-SMOO/CMOO problems focusing on different objectives which are solved by using various RL approaches are able to: increase the mean percentage of feasible TTIs, minimize the number of TTIs when the RL approaches punish the actions taken TTI-by-TTI, and minimize the variation of the performance indicators when different simulations are launched in parallel. This way, the obtained scheduling policies being focused on the multi-objective criteria are sustainable. Keywords: LTE, packet scheduling, scheduling rules, multi-objective optimization, reinforcement learning, channel, aggregation, scheduling policies, sustainable

Quality of service optimization of multimedia traffic in mobile networks

Mobile communication systems have continued to evolve beyond the currently deployed Third Generation (3G) systems with the main goal of providing higher capacity. Systems beyond 3G are expected to cater for a wide variety of services such as speech, data, image transmission, video, as well as multimedia services consisting of a combination of these. With the air interface being the bottleneck in mobile networks, recent enhancing technologies such as the High Speed Downlink Packet Access (HSDPA), incorporate major changes to the radio access segment of 3G Universal Mobile Telecommunications System (UMTS). HSDPA introduces new features such as fast link adaptation mechanisms, fast packet scheduling, and physical layer retransmissions in the base stations, necessitating buffering of data at the air interface which presents a bottleneck to end-to-end communication. Hence, in order to provide end-to-end Quality of Service (QoS) guarantees to multimedia services in wireless networks such as HSDPA, efficient buffer management schemes are required at the air interface. The main objective of this thesis is to propose and evaluate solutions that will address the QoS optimization of multimedia traffic at the radio link interface of HSDPA systems. In the thesis, a novel queuing system known as the Time-Space Priority (TSP) scheme is proposed for multimedia traffic QoS control. TSP provides customized preferential treatment to the constituent flows in the multimedia traffic to suit their diverse QoS requirements. With TSP queuing, the real-time component of the multimedia traffic, being delay sensitive and loss tolerant, is given transmission priority; while the non-real-time component, being loss sensitive and delay tolerant, enjoys space priority. Hence, based on the TSP queuing paradigm, new buffer managementalgorithms are designed for joint QoS control of the diverse components in a multimedia session of the same HSDPA user. In the thesis, a TSP based buffer management algorithm known as the Enhanced Time Space Priority (E-TSP) is proposed for HSDPA. E-TSP incorporates flow control mechanisms to mitigate congestion in the air interface buffer of a user with multimedia session comprising real-time and non-real-time flows. Thus, E-TSP is designed to provide efficient network and radio resource utilization to improve end-to-end multimedia traffic performance. In order to allow real-time optimization of the QoS control between the real-time and non-real-time flows of the HSDPA multimedia session, another TSP based buffer management algorithm known as the Dynamic Time Space Priority (D-TSP) is proposed. D-TSP incorporates dynamic priority switching between the real-time and non-real-time flows. D-TSP is designed to allow optimum QoS trade-off between the flows whilst still guaranteeing the stringent real-time component’s QoS requirements. The thesis presents results of extensive performance studies undertaken via analytical modelling and dynamic network-level HSDPA simulations demonstrating the effectiveness of the proposed TSP queuing system and the TSP based buffer management schemes

Adaptive Video Streaming for Wireless Networks with Multiple Users and Helpers

We consider the optimal design of a scheduling policy for adaptive video streaming in a wireless network formed by several users and helpers. A feature of such networks is that any user is typically in the range of multiple helpers. Hence, in order to cope with user-helper association, load balancing and inter-cell interference, an efficient streaming policy should allow the users to dynamically select the helper node to download from, and determine adaptively the video quality level of the download. In order to obtain a tractable formulation, we follow a "divide and conquer" approach: i) Assuming that each video packet (chunk) is delivered within its playback delay ("smooth streaming regime"), the problem is formulated as a network utility maximization (NUM), subject to queue stability, where the network utility function is a concave and componentwise non-decreasing function of the users' video quality measure. ii) We solve the NUM problem by using a Lyapunov Drift Plus Penalty approach, obtaining a scheme that naturally decomposes into two sub-policies referred to as "congestion control" (adaptive video quality and helper station selection) and "transmission scheduling" (dynamic allocation of the helper-user physical layer transmission rates).Our solution is provably optimal with respect to the proposed NUM problem, in a strong per-sample path sense. iii) Finally, we propose a method to adaptively estimate the maximum queuing delays, such that each user can calculate its pre-buffering and re-buffering time in order to cope with the fluctuations of the queuing delays. Through simulations, we evaluate the performance of the proposed algorithm under realistic assumptions of a network with densely deployed helper nodes, and demonstrate the per-sample path optimality of the proposed solution by considering a non-stationary non-ergodic scenario with user mobility, VBR video coding.Comment: final version to appear in IEEE Transactions on Communication

Optimization and Performance Analysis of High Speed Mobile Access Networks

The end-to-end performance evaluation of high speed broadband mobile access networks is the main focus of this work. Novel transport network adaptive flow control and enhanced congestion control algorithms are proposed, implemented, tested and validated using a comprehensive High speed packet Access (HSPA) system simulator. The simulation analysis confirms that the aforementioned algorithms are able to provide reliable and guaranteed services for both network operators and end users cost-effectively. Further, two novel analytical models one for congestion control and the other for the combined flow control and congestion control which are based on Markov chains are designed and developed to perform the aforementioned analysis efficiently compared to time consuming detailed system simulations. In addition, the effects of the Long Term Evolution (LTE) transport network (S1and X2 interfaces) on the end user performance are investigated and analysed by introducing a novel comprehensive MAC scheduling scheme and a novel transport service differentiation model

Design and Analysis of Green Mission-Critical Fiber-Wireless Broadband Access Networks

In recent years, the ever-increasing environmental friendliness concern has made energy efficiency in telecom networks as an important theme in their operations. Meanwhile, mission-critical (MC) services and systems (such as healthcare, police, and firefighting) have been acquiring special attention from telecom designers and operators. The currently deployed MC network technologies are indigent in terms of bandwidth capacity, and thus they are not able to support the emerging MC multimedia applications. Therefore in this thesis, we first explore the possibility of provisioning the MC services over the integration of fiber-wireless (FiWi) technologies, which has been considered as a promising candidate for the deployment of high-speed and mobile broadband access networks. We then investigate the energy efficiency problem in the FiWi integration, which consists of WiMAX in the wireless plane, and of Ethernet Passive Optical Network (EPON) - the most popular variant of the next-generation PON (NG-PON) technology, in the optical plane. In WiMAX, the energy saving protocol has been extensively investigated and standardized. Conversely, it has been recently studied in NG-PON, which currently consumes the least power among all the high-speed access networks. However, NG-PON has notably matured in the past few years and is envisioned to massively evolve in the near future. This trend will increase the power requirements of NG-PON and make it no longer coveted. Therefore we address the energy efficiency problem in NG-PON. For each of our contributions, we conduct extensive simulations to demonstrate the effectiveness and advantages of the proposed solutions