GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing

Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations

Evaluation of a multi-cell and multi-tenant capacity sharing solution under heterogeneous traffic distributions

One of the key features of the 5G architecture is network slicing, which allows the simultaneous support of diverse service types with heterogeneous requirements over a common network infrastructure. In order to support this feature in the Radio Access Network (RAN), it is required to have capacity sharing mechanisms that distribute the available capacity in each cell among the existing RAN slices while satisfying their requirements and efficiently using the available resources. Deep Reinforcement Learning (DRL) techniques are good candidates to deal with the complexity of capacity sharing in multi-cell scenarios where the traffic in the different cells can be heterogeneously distributed in the time and space domains. In this paper, a multi-agent reinforcement learning-based solution for capacity sharing in multi-cell scenarios is discussed and assessed under heterogeneous traffic conditions. Results show the capability of the solution to satisfy the requirements of the RAN slices while using the resources in the different cells efficiently.This work has been supported by the Spanish Research Council and FEDER funds under SONAR 5G grant (ref.TEC2017-82651-R), by the European Commission’s Horizon 2020 5G-CLARITY project under grant agreement 871428 and by the Secretariat for Universities and Research of the Ministry of Business and Knowledge of the Government of Catalonia under grant 2020FI_B2 00075.Peer ReviewedPostprint (author's final draft

Deep Reinforcement Learning for Artificial Upwelling Energy Management

The potential of artificial upwelling (AU) as a means of lifting nutrient-rich bottom water to the surface, stimulating seaweed growth, and consequently enhancing ocean carbon sequestration, has been gaining increasing attention in recent years. This has led to the development of the first solar-powered and air-lifted AU system (AUS) in China. However, efficient scheduling of air injection systems remains a crucial challenge in operating AUS, as it holds the potential to significantly improve system efficiency. Conventional approaches based on rules or models are often impractical due to the complex and heterogeneous nature of the marine environment and its associated disturbances. To address this challenge, we propose a novel energy management approach that utilizes deep reinforcement learning (DRL) algorithm to develop efficient strategies for operating AUS. Through extensive simulations, we evaluate the performance of our algorithm and demonstrate its superior effectiveness over traditional rule-based approaches and other DRL algorithms in reducing energy wastage while ensuring the stable and efficient operation of AUS. Our findings suggest that a DRL-based approach offers a promising way for improving the efficiency of AUS and enhancing the sustainability of seaweed cultivation and carbon sequestration in the ocean.Comment: 31 pages, 13 figure

Fairness-Oriented User Scheduling for Bursty Downlink Transmission Using Multi-Agent Reinforcement Learning

In this work, we develop practical user scheduling algorithms for downlink bursty traffic with emphasis on user fairness. In contrast to the conventional scheduling algorithms that either equally divides the transmission time slots among users or maximizing some ratios without physcial meanings, we propose to use the 5%-tile user data rate (5TUDR) as the metric to evaluate user fairness. Since it is difficult to directly optimize 5TUDR, we first cast the problem into the stochastic game framework and subsequently propose a Multi-Agent Reinforcement Learning (MARL)-based algorithm to perform distributed optimization on the resource block group (RBG) allocation. Furthermore, each MARL agent is designed to take information measured by network counters from multiple network layers (e.g. Channel Quality Indicator, Buffer size) as the input states while the RBG allocation as action with a proposed reward function designed to maximize 5TUDR. Extensive simulation is performed to show that the proposed MARL-based scheduler can achieve fair scheduling while maintaining good average network throughput as compared to conventional schedulers.Comment: 30 pages, 13 figure

Multi-Agent Deep Reinforcement Learning for Request Dispatching in Distributed-Controller Software-Defined Networking

Recently, distributed controller architectures have been quickly gaining popularity in Software-Defined Networking (SDN). However, the use of distributed controllers introduces a new and important Request Dispatching (RD) problem with the goal for every SDN switch to properly dispatch their requests among all controllers so as to optimize network performance. This goal can be fulfilled by designing an RD policy to guide distribution of requests at each switch. In this paper, we propose a Multi-Agent Deep Reinforcement Learning (MA-DRL) approach to automatically design RD policies with high adaptability and performance. This is achieved through a new problem formulation in the form of a Multi-Agent Markov Decision Process (MA-MDP), a new adaptive RD policy design and a new MA-DRL algorithm called MA-PPO. Extensive simulation studies show that our MA-DRL technique can effectively train RD policies to significantly outperform man-made policies, model-based policies, as well as RD policies learned via single-agent DRL algorithms

Contribution to the modelling and evaluation of radio network slicing solutions in 5G

Network slicing is a key feature of 5G architecture that allows the partitioning of the network into multiple logical networks, known as network slices, where each of them is customised according to the specific needs of a service or application. Thus, network slicing allows the materialisation of multi-tenant networks, in which a common network infrastructure is shared among multiple communication providers, acting as tenants and each of them using a different network slice. The support of multi-tenancy through slicing in the Radio Access Network (RAN), known as RAN slicing, is particularly challenging because it involves the configuration and operation of multiple and diverse RAN behaviours over the common pool of radio resources available at each of the RAN nodes. Moreover, this configuration needs to be performed in such a way that the specific requirements of each tenant are satisfied and, at the same time, the available radio resources are efficiently used. Therefore, new functionalities that allow the deployment of RAN slices are needed to be introduced at different levels, ranging from Radio Resource Management (RRM) functionalities that incorporate RAN slicing parameters to functionalities that support the lifecycle management of RAN slices. This thesis has addressed this need by proposing, developing and assessing diverse solutions for the support RAN slicing, which has allowed evaluating the capacities, requirements and limitations of network slicing in the RAN from diverse perspectives. Specifically, this thesis is firstly focused on the analytical assessment of RRM functionalities that support multi-tenant and multi-services scenarios, where services are defined according to their 5G QoS requirements. This assessment is conducted through the Markov modelling of admission control policies and the statistical modelling of the resourc allocation, both supporting multiple tenants and multiple services. Secondly, the thesis addresses the problem of slice admission control by proposing a methodology for the estimation of the radio resources required by a RAN slice based on data analytics. This methodology supports the decision on the admission or rejection of new RAN slice creation requests. Thirdly, the thesis explores the potential of artificial intelligence, and specifically, of Deep Reinforcement Learning (DRL) to deal with the capacity sharing problem in RAN slicing scenarios. To this end, a DRL-based capacity sharing solution that distributes the available capacity of a multi-cell scenario among multiple tenants is proposed and assessed. The solution consists in a Multi-Agent Reinforcement Learning (MARL) approach based on Deep Q-Network. Finally, this thesis discuses diverse implementation aspects of the DRL-based capacity sharing solution, including considerations on its compatibility with the standards, the impact of the training on the achieved performance, as well as the tools and technologies required for the deployment of the solution in the real network environment.El Network Slicing és una tecnologia clau de l’arquitectura del 5G que permet dividir la xarxa en múltiples xarxes lògiques, conegudes com a network slices, on cada una es configura d’acord a les necessitats d’un servei o aplicació específic. Així, el network slicing permet la materialització de les xarxes amb múltiples inquilins, on una infraestructura de xarxa comuna es comparteix entre diferents proveïdors de comunicacions, que actuen com a inquilins i utilitzen network slices diferents. El suport de múltiples inquilins mitjançant l’ús del network slicing a la xarxa d’accés ràdio (RAN), que es coneix com a RAN slicing, és un gran repte tecnològic, ja que comporta la configuració i operació de múltiples i diversos comportaments sobre els recursos ràdio disponibles a cadascun dels nodes de la xarxa d’accés. A més a més, aquesta configuració s’ha de portar a terme de forma que els requisits específics de cada inquilí es satisfacin i, al mateix temps, els recursos ràdio disponibles s’utilitzin eficientment. Per tant, és necessari introduir noves funcionalitats a diferents nivells que permetin el desplegament de les RAN slices, des de funcionalitats relacionades amb la gestió dels recursos ràdio (RRM) que incorporin paràmetres per al RAN slicing a funcionalitats que proporcionin suport a la gestió del cicle de vida de les RAN slices. Aquesta tesi ha adreçat aquesta necessitat proposant, desenvolupant i avaluant diverses solucions pel suport del RAN slicing, que han permès analitzar les capacitats, requisits i limitacions del RAN slicing des de diferents perspectives. Específicament, aquesta tesi es centra, en primer lloc, en realitzar una anàlisi de les funcionalitats de RRM que suporten escenaris amb múltiples inquilins i múltiples serveis, on els serveis es defineixen d’acord amb els seus requisits de 5G QoS. Aquesta anàlisi es porta a terme mitjançant la caracterització de polítiques de control d’admissió amb un model de Markov i el modelat estadístic de l’assignació de recursos, ambdós suportant múltiples inquilins i múltiples serveis. En segon lloc, la tesi aborda el problema del control d’admissió de network slices proposant una metodologia per l¿estimació dels recursos requerits per una RAN slice, que es basa en la anàlisi de dades. Aquesta metodologia dona suport a la decisió sobre l’admissió o rebuig de noves sol·licituds de creació de RAN slices. En tercer lloc, la tesi explora el potencial de la intel·ligència artificial, concretament, de les tècniques de Deep Reinforcement Learning (DRL) per a tractar el problema de la compartició de capacitat en escenaris amb RAN slicing. Amb aquest objectiu, es proposa i s’avalua una solució de compartició de capacitat basada en DRL que distribueix la capacitat disponible en un escenari multicel·lular entre múltiples inquilins. Aquesta solució es planteja com una solución de Multi-Agent Reinforcement Learning (MARL) basat en Deep Q-Network. Finalment, aquesta tesi tracta diversos aspectes relacionats amb la implementació de la solució de compartició de capacitat basada en DRL, incloent-hi consideracions sobre la compatibilitat de la solució amb els estàndards, l’impacte de l’entrenament de la solució al seu comportament i rendiment, així com les eines i tecnologies necessàries per al desplegament de la solució en un entorn de xarxa real.El Network Slicing es una tecnología clave de la arquitectura del 5G que permite dividir la red en múltiples redes lógicas, conocidas como network slices, que se configuran de acuerdo a las necesidades de servicios y aplicaciones específicas. Así, el network slicing permite la materialización de las redes con múltiples inquilinos, donde una infraestructura de red común se comparte entre diferentes proveedores de comunicaciones, que actúan como inquilinos y que usan network slices diferentes. El soporte de múltiples inquilinos mediante el uso del network slicing en la red de acceso radio (RAN), que se conoce como RAN slicing, es un gran reto tecnológico, ya que comporta la configuración y operación de múltiples y diversos comportamientos sobre los recursos radio disponibles en cada uno de los nodos de la red de acceso. Además, esta configuración debe realizarse de tal manera que los requisitos específicos de cada inquilino se satisfagan y, al mismo tiempo, los recursos radio disponibles se utilicen eficazmente. Por lo tanto, es necesario introducir nuevas funcionalidades a diferentes niveles que permitan el despliegue de las RAN slices, desde funcionalidades relacionadas con la gestión de recursos radio (RRM) que incorporen parámetros para el RAN slicing a funcionalidades que proporcionen soporte a la gestión del ciclo de vida de las RAN slices. Esta tesis ha abordado esta necesidad proponiendo, desarrollando y evaluando diversas soluciones para el soporte del RAN slicing, lo que ha permitido analizar las capacidades, requisitos y limitaciones del RAN slicing desde diversas perspectivas. Específicamente, esta tesis se centra, en primer lugar, en realizar un análisis de funcionalidades de RRM que soportan escenarios con múltiples inquilinos y múltiples servicios, donde los servicios se definen según sus requisitos de 5G QoS. Este análisis se lleva a cabo mediante la caracterización de políticas de control de admisión mediante un modelo de Markov y el modelado a nivel estadístico de la asignación de recursos, ambos soportando múltiples inquilinos y múltiples servicios. En segundo lugar, la tesis aborda el problema del control de admisión de network slices proponiendo una metodología para la estimación de los recursos radio requeridos por una RAN slice que se basa en análisis de datos. Esta metodología da soporte a la decisión sobre la admisión o el rechazo de nuevas solicitudes de creación de RAN slice. En tercer lugar, la tesis explora el potencial de la inteligencia artificial, y en concreto, de las técnicas de Deep Reinforcement Learning (DRL) para tratar el problema de compartición de capacidad en escenarios de RAN slicing. Para ello, se propone y evalúa una solución de compartición de capacidad basada en DRL que distribuye la capacidad disponible de un escenario multicelular entre múltiples inquilinos. Esta solución se plantea como una solución de Multi-Agent Reinforcement Learning (MARL) basado en Deep Q-Network. Finalmente, en esta tesis se tratan diversos aspectos relacionados con la implementación de la solución de reparto de capacidad basada en DRL, incluyendo consideraciones sobre su compatibilidad con los estándares, el impacto del entrenamiento en el comportamiento y rendimiento conseguido, así como las herramientas y tecnologías necesarias para su despliegue en un entorno de red real.Postprint (published version