Deep Reinforcement Learning for Smart Queue Management

With the goal of meeting the stringent throughput and delay requirements of classified network flows, we propose a Deep Q-learning Network (DQN) for optimal weight selection in an active queue management system based on Weighted Fair Queuing (WFQ). Our system schedules flows belonging to different priority classes (Gold, Silver, and Bronze) into separate queues, and learns how and when to dequeue from each queue. The neural network implements deep reinforcement learning tools such as target networks and replay buffers to help learn the best weights depending on the network state. We show, via simulations, that our algorithm converges to an efficient model capable of adapting to the flow demands, producing thus lower delays with respect to traditional WFQ

Renegotiation based dynamic bandwidth allocation for selfsimilar VBR traffic

The provision of QoS to applications traffic depends heavily on how different traffic types are categorized and classified, and how the prioritization of these applications are managed. Bandwidth is the most scarce network resource. Therefore, there is a need for a method or system that distributes an available bandwidth in a network among different applications in such a way that each class or type of traffic receives their constraint QoS requirements. In this dissertation, a new renegotiation based dynamic resource allocation method for variable bit rate (VBR) traffic is presented. First, pros and cons of available off-line methods that are used to estimate selfsimilarity level (represented by Hurst parameter) of a VBR traffic trace are empirically investigated, and criteria to select measurement parameters for online resource management are developed. It is shown that wavelet analysis based methods are the strongest tools in estimation of Hurst parameter with their low computational complexities, compared to the variance-time method and R/S pox plot. Therefore, a temporal energy distribution of a traffic data arrival counting process among different frequency sub-bands is considered as a traffic descriptor, and then a robust traffic rate predictor is developed by using the Haar wavelet analysis. The empirical results show that the new on-line dynamic bandwidth allocation scheme for VBR traffic is superior to traditional dynamic bandwidth allocation methods that are based on adaptive algorithms such as Least Mean Square, Recursive Least Square, and Mean Square Error etc. in terms of high utilization and low queuing delay. Also a method is developed to minimize the number of bandwidth renegotiations to decrease signaling costs on traffic schedulers (e.g. WFQ) and networks (e.g. ATM). It is also quantified that the introduced renegotiation based bandwidth management scheme decreases heavytailedness of queue size distributions, which is an inherent impact of traffic self similarity. The new design increases the achieved utilization levels in the literature, provisions given queue size constraints and minimizes the number of renegotiations simultaneously. This renegotiation -based design is online and practically embeddable into QoS management blocks, edge routers and Digital Subscriber Lines Access Multiplexers (DSLAM) and rate adaptive DSL modems

Estimation-Based Queue Scheduling Model to Improve QoS for End Users in MANETs

Using MANETs for real time applications is always a challenge as the network is extremely dynamic with brisk topology changes. Despite this, several real time schedulers have been developed that aimed at providing QoS to ad hoc nodes. The quality of service (QoS) is standardized in terms of capacity, reliability, link quality, delays/jitters, and network cost. Thus, for QoS, the better transmission should be maintained at end user as well as at the transmitting unit. QoS of a network is affected by delays and bandwidth allocated for transmission. For an efficient network, it is required to predict these metrics during transmission. For this, in this paper, integration of quaternion-based Kalman filter is performed that predicts the required bandwidth and the network delays with higher accuracy. From the analysis, it is shown that bandwidth can be optimized but it is not possible to aloof delays in the network. Thus, while implementing such admission control procedures, estimation process allows control over delays and sustain them from going beyond a certain threshold value. The model proposed is a novel approach and has not been formulated in any of previous work related to QoS in MANETs. The effectiveness of model is demonstrated using both simulation and real time results

Energy and power aware scheduling algorithm modeling for battery powered embedded systems

Energy management has always been recognized as a challenge in mobile systems, especially in modern OS-based mobile systems where multi-functioning are widely supported. Nowadays, it is common for a mobile system user to run multiple applications simultaneously while having a target battery lifetime in mind for a specific application. Traditional OS-level power management (PM) policies make their best effort to save energy under performance constraint, but fail to guarantee a target lifetime, leaving the painful trading off between the total performance of applications and the target lifetime to the user itself. This thesis provides a new way to deal with the problem. It is advocated that a strong energy-aware PM scheme should first guarantee a user-specified battery lifetime to a target application by restricting the average power of those less important applications, and in addition to that, maximize the total performance of applications without harming the lifetime guarantee. As a support, energy, instead of CPU or transmission bandwidth, should be globally managed as the first-class resource by the OS. As the first-stage work of a complete PM scheme, this thesis presents the energy-based fair queuing scheduling, a novel class of energy-aware scheduling algorithms which, in combination with a mechanism of battery discharge rate restricting, systematically manage energy as the first-class resource with the objective of guaranteeing a user-specified battery lifetime for a target application in OS-based mobile systems. Energy-based fair queuing is a cross-application of the traditional fair queuing in the energy management domain. It assigns a power share to each task, and manages energy by proportionally serving energy to tasks according to their assigned power shares. The proportional energy use establishes proportional share of the system power among tasks, which guarantees a minimum power for each task and thus, avoids energy starvation on any task. Energy-based fair queuing treats all tasks equally as one type and supports periodical time-sensitive tasks by allocating each of them a share of system power that is adequate to meet the highest energy demand in all periods. However, an overly conservative power share is usually required to guarantee the meeting of all time constraints. To provide more effective and flexible support for various types of time-sensitive tasks in general purpose operating systems, an extra real-time friendly mechanism is introduced to combine priority-based scheduling into the energy-based fair queuing. Since a method is available to control the maximum time one time-sensitive task can run with priority, the power control and time-constraint meeting can be flexibly traded off. A SystemC-based test-bench is designed to assess the algorithms. Simulation results show the success of the energy-based fair queuing in achieving proportional energy use, time-constraint meeting, and a proper trading off between them. La gestión de energía en los sistema móviles está considerada hoy en día como un reto fundamental, notándose, especialmente, en aquellos terminales que utilizando un sistema operativo implementan múltiples funciones. Es común en los sistemas móviles actuales ejecutar simultaneamente diferentes aplicaciones y tener, para una de ellas, un objetivo de tiempo de uso de la batería. Tradicionalmente, las políticas de gestión de consumo de potencia de los sistemas operativos hacen lo que está en sus manos para ahorrar energía y satisfacer sus requisitos de prestaciones, pero no son capaces de proporcionar un objetivo de tiempo de utilización del sistema, dejando al usuario la difícil tarea de buscar un compromiso entre prestaciones y tiempo de utilización del sistema. Esta tesis, como contribución, proporciona una nueva manera de afrontar el problema. En ella se establece que un esquema de gestión de consumo de energía debería, en primer lugar, garantizar, para una aplicación dada, un tiempo mínimo de utilización de la batería que estuviera especificado por el usuario, restringiendo la potencia media consumida por las aplicaciones que se puedan considerar menos importantes y, en segundo lugar, maximizar las prestaciones globales sin comprometer la garantía de utilización de la batería. Como soporte de lo anterior, la energía, en lugar del tiempo de CPU o el ancho de banda, debería gestionarse globalmente por el sistema operativo como recurso de primera clase. Como primera fase en el desarrollo completo de un esquema de gestión de consumo, esta tesis presenta un algoritmo de planificación de encolado equitativo (fair queueing) basado en el consumo de energía, es decir, una nueva clase de algoritmos de planificación que, en combinación con mecanismos que restrinjan la tasa de descarga de una batería, gestionen de forma sistemática la energía como recurso de primera clase, con el objetivo de garantizar, para una aplicación dada, un tiempo de uso de la batería, definido por el usuario, en sistemas móviles empotrados. El encolado equitativo de energía es una extensión al dominio de la energía del encolado equitativo tradicional. Esta clase de algoritmos asigna una reserva de potencia a cada tarea y gestiona la energía sirviéndola de manera proporcional a su reserva. Este uso proporcional de la energía garantiza que cada tarea reciba una porción de potencia y evita que haya tareas que se vean privadas de recibir energía por otras con un comportamiento más ambicioso. Esta clase de algoritmos trata a todas las tareas por igual y puede planificar tareas periódicas en tiempo real asignando a cada una de ellas una reserva de potencia que es adecuada para proporcionar la mayor de las cantidades de energía demandadas por período. Sin embargo, es posible demostrar que sólo se consigue cumplir con los requisitos impuestos por todos los plazos temporales con reservas de potencia extremadamente conservadoras. En esta tesis, para proporcionar un soporte más flexible y eficiente para diferentes tipos de tareas de tiempo real junto con el resto de tareas, se combina un mecanismo de planificación basado en prioridades con el encolado equitativo basado en energía. En esta clase de algoritmos, gracias al método introducido, que controla el tiempo que se ejecuta con prioridad una tarea de tiempo real, se puede establecer un compromiso entre el cumplimiento de los requisitos de tiempo real y el consumo de potencia. Para evaluar los algoritmos, se ha diseñado en SystemC un banco de pruebas. Los resultados muestran que el algoritmo de encolado equitativo basado en el consumo de energía consigue el balance entre el uso proporcional a la energía reservada y el cumplimiento de los requisitos de tiempo real

DOWNSTREAM RESOURCE ALLOCATION IN DOCSIS 3.0 CHANNEL BONDED NETWORKS

Modern broadband internet access cable systems follow the Data Over Cable System Interface Specification (DOCSIS) for data transfer between the individual cable modem (CM) and the Internet. The newest version of DOCSIS, version 3.0, provides an abstraction referred to as bonding groups to help manage bandwidth and to increase bandwidth to each user beyond that available within a single 6MHz. television channel. Channel bonding allows more than one channel to be used by a CM to provide a virtual channel of much greater bandwidth. This combining of channels into bonding groups, especially when channels overlap between more than one bonding group, complicates the resource allocation problem within these networks. The goal of resource allocation in this research is twofold, to provide for fairness among users while at the same time making maximum possible utilization of the available system bandwidth. The problem of resource allocation in computer networks has been widely studied by the academic community. Past work has studied resource allocation in many network types, however application in a DOCSIS channel bonded network has not been explored. This research begins by first developing a definition of fairness in a channel bonded system. After providing a theoretical definition of fairness we implement simulations of different scheduling disciplines and evaluate their performance against this theoretical ideal. The complexity caused by overlapped channels requires even the simplest scheduling algorithms to be modified to work correctly. We then develop an algorithm to maximize the use of the available system bandwidth. The approach involves using competitive analysis techniques and an online algorithm to dynamically reassign flows among the available channels. Bandwidth usage and demand requests are monitored for bandwidth that is underutilized, and demand that is unsatisfied, and real time changes are made to the flow-to-channel mappings to improve the utilization of the total available bandwidth. The contribution of this research is to provide a working definition of fairness in a channel bonded environment, the implementation of several scheduling disciplines and evaluation of their adherence to that definition, and development of an algorithm to improve overall bandwidth utilization of the system

Opportunistic transmission scheduling for next generation wireless communication systems with multimedia services

The explosive growth of the Internet and the continued dramatic increase for all wireless services are fueling the demand for increased capacity, data rates, and support of different quality of service (QoS) requirements for different classes of services. Since in the current and future wireless communication infrastructures, the performances of the various services are strongly correlated, as the resources are shared among them, dynamic resource allocation methods should be employed. With the demand for high data rate and support of multiple QoS, the transmission scheduling plays a key role in the efficient resource allocation process in wireless systems. The fundamental problem of scheduling the users\u27 transmissions and allocating the available resources in a realistic CDMA wireless system that supports multi-rate multimedia services, with efficiency and fairness, is investigated and analyzed in this dissertation. Our proposed approach adopts the use of dynamically assigned data rates that match the channel capacity in order to improve the system throughput and overcome the problems associated with the location-dependent and time-dependent errors and channel conditions, the variable system capacity and the transmission power limitation. We first introduce and describe two new scheduling algorithms, namely the Channel Adaptive Rate Scheduling (CARS) and Fair Channel Adaptive Rate Scheduling (FCARS). CARS exploits the channel variations to reach high throughput, by adjusting the transmission rates according to the varying channel conditions and by performing an iterative procedure to determine the power index that a user can accept by its current channel condition and transmission power. Based on the assignment of CARS and to overcome potential unfair service allocation, FCARS implements a compensation algorithm, in which the lagging users can receive compensation service when the corresponding channel conditions improve, in order to achieve asymptotic throughput fairness, while still maintaining all the constraints imposed by the system. Furthermore the problem of opportunistic fair scheduling in the uplink transmission of CDMA systems, with the objective of maximizing the uplink system throughput, while satisfying the users\u27 QoS requirements and maintaining the long-term fairness among the various users despite their different varying channel conditions, is rigorously formulated, and a throughput optimal fair scheduling policy is obtained. The corresponding problem is expressed as a weighted throughput maximization problem, under certain power and QoS constraints, where the weights are the control parameters that reflect the fairness constraints. With the introduction of the power index capacity it is shown that this optimization problem can be converted into a binary knapsack problem, where all the corresponding constraints are replaced by the users\u27 power index capacities at some certain system power index. It is then argued that the optimal solution can be obtained as a global search within a certain range, while a stochastic approximation method is presented in order to effectively identify the required control parameters. Finally, since some real-time services may demand certain amount of service within specific short span of time in order to avoid service delays, the problem of designing policies that can achieve high throughput while at the same time maintain short term fairness, is also considered and investigated. To this end a new Credit-based Short-term Fairness Scheduling (CSFS) algorithm, which achieves to provide short-term fairness to the delay-sensitive users while still schedules opportunistically the non-delay-sensitive users to obtain high system throughput, is proposed and evaluated

Q-Andrew: a consolidated QOS management framework

Tese de mestrado em Segurança Informática, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2008As redes IP convergentes são compostas por uma diversidade de tecnologias que suportam múltiplos tipos de serviços com diferentes características. Cada fabricante de equipamento activo de rede usa sistemas de manutenção proprietários, incompatíveis com equipamentos de outros fabricantes. Para um operador de telecomunicações a gestão da Qualidade de Serviço, numa rede composta por vários fabricantes, é uma tarefa complexa e dispendiosa. Algumas tarefas requerem configuração manual para garantir a compatibilidade entre configurações de equipamentos de fabricantes diferentes. Melhorar a resposta operacional e reduzir os custos de operação nestas circunstâncias é apenas possível com a consolidação da gestão de rede. Para responder a este desafio, propomos: Um conjunto de mecanismos geradores de configurações de Qualidade de Serviço, consistentes entre equipamentos de diversos fabricantes; A definição de um modelo abstracto de representação destas configurações, reutilizável em futuras aproximações de gestão consolidada de rede; Por fim, descrevemos uma aplicação de demonstração onde algumas das propostas apresentadas são concretizadas, tendo como objectivo futuro a sua utilização numa rede real de um operador de telecomunicações nacional, onde são utilizados equipamentos de diversos fabricantes.Converged IP networks consist of diverse technologies and support both legacy and emerging services. Different vendors use separate management systems to achieve similar goals. Manual provisioning today represents a large portion of the total effort required to manage a complex IP network. A consolidated Quality-of-Service policy is difficult to implement in heterogeneous networks. Creating and maintaining such policies is very demanding in terms of operations. For this reason, reducing operational costs while improving Quality-of-Service Management is only possible through a consolidated approach to network management. To leverage operations in converged IP networks, we propose the following: A mechanism to automatically generate consistent configurations across a network with equipment from different vendors; A framework definition such that network element configurations can be specified using a common model; Applying some of the methods proposed to an application that can be used in a real network with diverse technologies and equipment vendors

A VOICE PRIORITY QUEUE (VPQ) SCHEDULER FOR VOIP OVER WLANs

The Voice over Internet Protocol (VoIP) application has observed the fastest growth in the world of telecommunication. The Wireless Local Area Network (WLAN) is the most assuring of technologies among the wireless networks, which has facilitated high-rate voice services at low cost and good flexibility. In a voice conversation, each client works as a sender and as a receiver depending on the direction of traffic flow over the network. A VoIP application requires a higher throughput, less packet loss and a higher fairness index over the network. The packets of VoIP streaming may experience drops because of the competition among the different kinds of traffic flow over the network. A VoIP application is also sensitive to delay and requires the voice packets to arrive on time from the sender to the receiver side without any delay over WLANs. The scheduling system model for VoIP traffic is still an unresolved problem. A new traffic scheduler is necessary to offer higher throughput and a higher fairness index for a VoIP application. The objectives of this thesis are to propose a new scheduler and algorithms that support the VoIP application and to evaluate, validate and verify the newly proposed scheduler and algorithms with the existing scheduling algorithms over WLANs through simulation and experimental environment. We proposed a new Voice Priority Queue (VPQ) scheduling system model and algorithms to solve scheduling issues. VPQ system model is implemented in three stages. The first stage of the model is to ensure efficiency by producing a higher throughput and fairness for VoIP packets. The second stage will be designed for bursty Virtual-VoIP Flow (Virtual-VF) while the third stage is a Switch Movement (SM) technique. Furthermore, we compared the VPQ scheduler with other well known schedulers and algorithms. We observed in our simulation and experimental environment that the VPQ provides better results for the VoIP over WLANs

Radio Resource Management for Satellite UMTS. Dynamic scheduling algorithm for a UMTS-compatible satellite network.

The third generation of mobile communication systems introduce interactive Multicast and Unicast multimedia services at a fast data rate of up to 2 Mbps and is expected to complete the globalization of the mobile telecommunication systems. The implementation of these services on satellite systems, particularly for broadcast and multicast applications to complement terrestrial services is ideal since satellite systems are capable of providing global coverage in areas not served by terrestrial telecommunication services. However, the main bottleneck of such systems is the scarcity of radio resources for supporting multimedia applications which has resulted in the rapid growth in research efforts for deriving efficient radio resource management techniques. This issue is addressed in this thesis, where the main emphasis is to design a dynamic scheduling framework and algorithm that can improve the overall performance of the radio resource management strategy of a UMTS compatible satellite network, taking into account the unique characteristics of wireless channel conditions. This thesis will initially be focused on the design of the network and functional architecture of a UMTS -compatible satellite network. Based on this architecture, an effective scheduling framework is designed, which can provide different types of resource assigning strategies. A functional model of scheduler is defined to describe the behaviours and interactions between different functional entities. An OPNET simulation model with a complete network protocol stack is developed to validate the performance of the scheduling algorithms implemented in the satellite network. Different types of traffic are considered for the OPNET simulation, such as the Poisson Process, ONOFF Source and Self Similar Process, so that the performance of scheduling algorithm can be analyzed for different types of services. A novel scheduling algorithm is proposed to optimise the channel utilisation by considering the characteristics of the wireless channel, which are bursty and location dependent. In order to overcome the channel errors, different code rates are applied for the user under different channel conditions. The proposed scheduling algorithm is designed to give higher priority to users with higher code rate, so that the throughput of network is optimized and at the same time, maintaining the end users¿ service level agreements. The fairness of the proposed scheduling algorithm is validated using OPNET simulation. The simulation results show that the algorithm can fairly allocate resource to different connections not only among different service classes but also within the same service class depending on their QoS attributes.Inmarsat Global Ltd. BGAN and the European Space Agency (ESA

A Method with Node Autonomy to Preserve QoS in Networks with Per- Route Admission

This paper presents a method called the Short Term Protection (STP) which is applied to fixed networks whereflows1 are constrained to follow specific routes, where the admission-process is distributed, per-route, and wherethe traffic is sensitive to delay (like MPEG4 traffic). The share of bandwidth of any route is protected, in the shortterm,against the traffic-increment of its intersecting-routes. In the long term, in every congested link, the share ofbandwidth between the routes that intersect at that link is proportional to the average relative measured demandsof those routes. The nodes act autonomously, without central administration. This method is expected to: 1) helpnetwork administrators to have confidence, within a time-frame , about the amount of bandwidth they count on forevery route; 2) allow a simple bandwidth management in the network, with prospect to be scalable to at least tensof nodes. This paper presents the general and detailed operation of the method, the evaluation of the method, bysimulations, including a comparison with other method, a discussion of a possible scenario of application,conclusions and possible paths for further research