Deep Reinforcement Learning for Resource Management in Network Slicing

Network slicing is born as an emerging business to operators, by allowing them to sell the customized slices to various tenants at different prices. In order to provide better-performing and cost-efficient services, network slicing involves challenging technical issues and urgently looks forward to intelligent innovations to make the resource management consistent with users' activities per slice. In that regard, deep reinforcement learning (DRL), which focuses on how to interact with the environment by trying alternative actions and reinforcing the tendency actions producing more rewarding consequences, is assumed to be a promising solution. In this paper, after briefly reviewing the fundamental concepts of DRL, we investigate the application of DRL in solving some typical resource management for network slicing scenarios, which include radio resource slicing and priority-based core network slicing, and demonstrate the advantage of DRL over several competing schemes through extensive simulations. Finally, we also discuss the possible challenges to apply DRL in network slicing from a general perspective.Comment: The manuscript has been accepted by IEEE Access in Nov. 201

A machine learning approach to 5G infrastructure market optimization

It is now commonly agreed that future 5G Networks will build upon the network slicing concept. The ability to provide virtual, logically independent "slices" of the network will also have an impact on the models that will sustain the business ecosystem. Network slicing will open the door to new players: the infrastructure provider, which is the owner of the infrastructure, and the tenants, which may acquire a network slice from the infrastructure provider to deliver a specific service to their customers. In this new context, how to correctly handle resource allocation among tenants and how to maximize the monetization of the infrastructure become fundamental problems that need to be solved. In this paper, we address this issue by designing a network slice admission control algorithm that (i) autonomously learns the best acceptance policy while (ii) it ensures that the service guarantees provided to tenants are always satisfied. The contributions of this paper include: (i) an analytical model for the admissibility region of a network slicing-capable 5G Network, (ii) the analysis of the system (modeled as a Semi-Markov Decision Process) and the optimization of the infrastructure providers revenue, and (iii) the design of a machine learning algorithm that can be deployed in practical settings and achieves close to optimal performance.The work of University Carlos III of Madrid was supported by the H2020 5G-MoNArch project (Grant Agreement No. 761445) and the 5GCity project of the Spanish Ministry of Economy and Competitiveness (TEC2016-76795-C6-3-R). The work of NEC Laboratories Europe was supported by the 5G-Transformer project (Grant Agreement No. 761536)

Slice-Aware Radio Resource Management for Future Mobile Networks

The concept of network slicing has been introduced in order to enable mobile networks to accommodate multiple heterogeneous use cases that are anticipated to be served within a single physical infrastructure. The slices are end-to-end virtual networks that share the resources of a physical network, spanning the core network (CN) and the radio access network (RAN). RAN slicing can be more challenging than CN slicing as the former deals with the distribution of radio resources, where the capacity is not constant over time and is hard to extend. The main challenge in RAN slicing is to simultaneously improve multiplexing gains while assuring enough isolation between slices, meaning one of the slices cannot negatively influence other slices. In this work, a flexible and configurable framework for RAN slicing is provided, where diverse requirements of slices are taken into account, and slice management algorithms adjust the control parameters of different radio resource management (RRM) mechanisms to satisfy the slices' service level agreements (SLAs). A new entity that translates the key performance indicator (KPI) targets of the SLAs to the control parameters is introduced and is called RAN slice orchestrator. Diverse algorithms governing this entity are introduced, which range from heuristics-based to model-free methods. Besides, a protection mechanism is constructed to prevent the negative influences of slices on each other's performances. The simulation-based analysis demonstrates the feasibility of slicing the RAN with multiplexing gains and slice isolation

Machine Learning-based Orchestration Solutions for Future Slicing-Enabled Mobile Networks

The fifth generation mobile networks (5G) will incorporate novel technologies such as network programmability and virtualization enabled by Software-Defined Networking (SDN) and Network Function Virtualization (NFV) paradigms, which have recently attracted major interest from both academic and industrial stakeholders. Building on these concepts, Network Slicing raised as the main driver of a novel business model where mobile operators may open, i.e., “slice”, their infrastructure to new business players and offer independent, isolated and self-contained sets of network functions and physical/virtual resources tailored to specific services requirements. While Network Slicing has the potential to increase the revenue sources of service providers, it involves a number of technical challenges that must be carefully addressed. End-to-end (E2E) network slices encompass time and spectrum resources in the radio access network (RAN), transport resources on the fronthauling/backhauling links, and computing and storage resources at core and edge data centers. Additionally, the vertical service requirements’ heterogeneity (e.g., high throughput, low latency, high reliability) exacerbates the need for novel orchestration solutions able to manage end-to-end network slice resources across different domains, while satisfying stringent service level agreements and specific traffic requirements. An end-to-end network slicing orchestration solution shall i) admit network slice requests such that the overall system revenues are maximized, ii) provide the required resources across different network domains to fulfill the Service Level Agreements (SLAs) iii) dynamically adapt the resource allocation based on the real-time traffic load, endusers’ mobility and instantaneous wireless channel statistics. Certainly, a mobile network represents a fast-changing scenario characterized by complex spatio-temporal relationship connecting end-users’ traffic demand with social activities and economy. Legacy models that aim at providing dynamic resource allocation based on traditional traffic demand forecasting techniques fail to capture these important aspects. To close this gap, machine learning-aided solutions are quickly arising as promising technologies to sustain, in a scalable manner, the set of operations required by the network slicing context. How to implement such resource allocation schemes among slices, while trying to make the most efficient use of the networking resources composing the mobile infrastructure, are key problems underlying the network slicing paradigm, which will be addressed in this thesis

Contribution to the modelling and evaluation of radio network slicing solutions in 5G

Network slicing is a key feature of 5G architecture that allows the partitioning of the network into multiple logical networks, known as network slices, where each of them is customised according to the specific needs of a service or application. Thus, network slicing allows the materialisation of multi-tenant networks, in which a common network infrastructure is shared among multiple communication providers, acting as tenants and each of them using a different network slice. The support of multi-tenancy through slicing in the Radio Access Network (RAN), known as RAN slicing, is particularly challenging because it involves the configuration and operation of multiple and diverse RAN behaviours over the common pool of radio resources available at each of the RAN nodes. Moreover, this configuration needs to be performed in such a way that the specific requirements of each tenant are satisfied and, at the same time, the available radio resources are efficiently used. Therefore, new functionalities that allow the deployment of RAN slices are needed to be introduced at different levels, ranging from Radio Resource Management (RRM) functionalities that incorporate RAN slicing parameters to functionalities that support the lifecycle management of RAN slices. This thesis has addressed this need by proposing, developing and assessing diverse solutions for the support RAN slicing, which has allowed evaluating the capacities, requirements and limitations of network slicing in the RAN from diverse perspectives. Specifically, this thesis is firstly focused on the analytical assessment of RRM functionalities that support multi-tenant and multi-services scenarios, where services are defined according to their 5G QoS requirements. This assessment is conducted through the Markov modelling of admission control policies and the statistical modelling of the resourc allocation, both supporting multiple tenants and multiple services. Secondly, the thesis addresses the problem of slice admission control by proposing a methodology for the estimation of the radio resources required by a RAN slice based on data analytics. This methodology supports the decision on the admission or rejection of new RAN slice creation requests. Thirdly, the thesis explores the potential of artificial intelligence, and specifically, of Deep Reinforcement Learning (DRL) to deal with the capacity sharing problem in RAN slicing scenarios. To this end, a DRL-based capacity sharing solution that distributes the available capacity of a multi-cell scenario among multiple tenants is proposed and assessed. The solution consists in a Multi-Agent Reinforcement Learning (MARL) approach based on Deep Q-Network. Finally, this thesis discuses diverse implementation aspects of the DRL-based capacity sharing solution, including considerations on its compatibility with the standards, the impact of the training on the achieved performance, as well as the tools and technologies required for the deployment of the solution in the real network environment.El Network Slicing és una tecnologia clau de l’arquitectura del 5G que permet dividir la xarxa en múltiples xarxes lògiques, conegudes com a network slices, on cada una es configura d’acord a les necessitats d’un servei o aplicació específic. Així, el network slicing permet la materialització de les xarxes amb múltiples inquilins, on una infraestructura de xarxa comuna es comparteix entre diferents proveïdors de comunicacions, que actuen com a inquilins i utilitzen network slices diferents. El suport de múltiples inquilins mitjançant l’ús del network slicing a la xarxa d’accés ràdio (RAN), que es coneix com a RAN slicing, és un gran repte tecnològic, ja que comporta la configuració i operació de múltiples i diversos comportaments sobre els recursos ràdio disponibles a cadascun dels nodes de la xarxa d’accés. A més a més, aquesta configuració s’ha de portar a terme de forma que els requisits específics de cada inquilí es satisfacin i, al mateix temps, els recursos ràdio disponibles s’utilitzin eficientment. Per tant, és necessari introduir noves funcionalitats a diferents nivells que permetin el desplegament de les RAN slices, des de funcionalitats relacionades amb la gestió dels recursos ràdio (RRM) que incorporin paràmetres per al RAN slicing a funcionalitats que proporcionin suport a la gestió del cicle de vida de les RAN slices. Aquesta tesi ha adreçat aquesta necessitat proposant, desenvolupant i avaluant diverses solucions pel suport del RAN slicing, que han permès analitzar les capacitats, requisits i limitacions del RAN slicing des de diferents perspectives. Específicament, aquesta tesi es centra, en primer lloc, en realitzar una anàlisi de les funcionalitats de RRM que suporten escenaris amb múltiples inquilins i múltiples serveis, on els serveis es defineixen d’acord amb els seus requisits de 5G QoS. Aquesta anàlisi es porta a terme mitjançant la caracterització de polítiques de control d’admissió amb un model de Markov i el modelat estadístic de l’assignació de recursos, ambdós suportant múltiples inquilins i múltiples serveis. En segon lloc, la tesi aborda el problema del control d’admissió de network slices proposant una metodologia per l¿estimació dels recursos requerits per una RAN slice, que es basa en la anàlisi de dades. Aquesta metodologia dona suport a la decisió sobre l’admissió o rebuig de noves sol·licituds de creació de RAN slices. En tercer lloc, la tesi explora el potencial de la intel·ligència artificial, concretament, de les tècniques de Deep Reinforcement Learning (DRL) per a tractar el problema de la compartició de capacitat en escenaris amb RAN slicing. Amb aquest objectiu, es proposa i s’avalua una solució de compartició de capacitat basada en DRL que distribueix la capacitat disponible en un escenari multicel·lular entre múltiples inquilins. Aquesta solució es planteja com una solución de Multi-Agent Reinforcement Learning (MARL) basat en Deep Q-Network. Finalment, aquesta tesi tracta diversos aspectes relacionats amb la implementació de la solució de compartició de capacitat basada en DRL, incloent-hi consideracions sobre la compatibilitat de la solució amb els estàndards, l’impacte de l’entrenament de la solució al seu comportament i rendiment, així com les eines i tecnologies necessàries per al desplegament de la solució en un entorn de xarxa real.El Network Slicing es una tecnología clave de la arquitectura del 5G que permite dividir la red en múltiples redes lógicas, conocidas como network slices, que se configuran de acuerdo a las necesidades de servicios y aplicaciones específicas. Así, el network slicing permite la materialización de las redes con múltiples inquilinos, donde una infraestructura de red común se comparte entre diferentes proveedores de comunicaciones, que actúan como inquilinos y que usan network slices diferentes. El soporte de múltiples inquilinos mediante el uso del network slicing en la red de acceso radio (RAN), que se conoce como RAN slicing, es un gran reto tecnológico, ya que comporta la configuración y operación de múltiples y diversos comportamientos sobre los recursos radio disponibles en cada uno de los nodos de la red de acceso. Además, esta configuración debe realizarse de tal manera que los requisitos específicos de cada inquilino se satisfagan y, al mismo tiempo, los recursos radio disponibles se utilicen eficazmente. Por lo tanto, es necesario introducir nuevas funcionalidades a diferentes niveles que permitan el despliegue de las RAN slices, desde funcionalidades relacionadas con la gestión de recursos radio (RRM) que incorporen parámetros para el RAN slicing a funcionalidades que proporcionen soporte a la gestión del ciclo de vida de las RAN slices. Esta tesis ha abordado esta necesidad proponiendo, desarrollando y evaluando diversas soluciones para el soporte del RAN slicing, lo que ha permitido analizar las capacidades, requisitos y limitaciones del RAN slicing desde diversas perspectivas. Específicamente, esta tesis se centra, en primer lugar, en realizar un análisis de funcionalidades de RRM que soportan escenarios con múltiples inquilinos y múltiples servicios, donde los servicios se definen según sus requisitos de 5G QoS. Este análisis se lleva a cabo mediante la caracterización de políticas de control de admisión mediante un modelo de Markov y el modelado a nivel estadístico de la asignación de recursos, ambos soportando múltiples inquilinos y múltiples servicios. En segundo lugar, la tesis aborda el problema del control de admisión de network slices proponiendo una metodología para la estimación de los recursos radio requeridos por una RAN slice que se basa en análisis de datos. Esta metodología da soporte a la decisión sobre la admisión o el rechazo de nuevas solicitudes de creación de RAN slice. En tercer lugar, la tesis explora el potencial de la inteligencia artificial, y en concreto, de las técnicas de Deep Reinforcement Learning (DRL) para tratar el problema de compartición de capacidad en escenarios de RAN slicing. Para ello, se propone y evalúa una solución de compartición de capacidad basada en DRL que distribuye la capacidad disponible de un escenario multicelular entre múltiples inquilinos. Esta solución se plantea como una solución de Multi-Agent Reinforcement Learning (MARL) basado en Deep Q-Network. Finalmente, en esta tesis se tratan diversos aspectos relacionados con la implementación de la solución de reparto de capacidad basada en DRL, incluyendo consideraciones sobre su compatibilidad con los estándares, el impacto del entrenamiento en el comportamiento y rendimiento conseguido, así como las herramientas y tecnologías necesarias para su despliegue en un entorno de red real.Postprint (published version

Deep reinforcement learning based approaches for capacity sharing in radio access network slicing

Network slicing has become a fundamental capability for 5G networks to support the expected high variety of service requirements over a common physical network infrastructure. Each network slice can be customized for a specific application, making that the radio resources have to be accordingly managed by the Radio Access Network (RAN) part of the slice. In this thesis, three different Deep Reinforcement Learning (DRL) based approaches are presented to optimize the resource allocation among slices. A RAN slicing simulator scenario is developed, where the DRL mechanisms build knowledge about the network and learn how to optimize the capacity allocation for each tenant at every moment of time. The performance of each approach is studied based on simulation results, and before the comparison between the algorithms, the set of hyperparameters of each approach is tuned to optimize the learning process

AI gym for Networks

5G Networks are delivering better services and connecting more devices, but at the same time are becoming more complex. Problems like resource management and control optimization are increasingly dynamic and difficult to model making it very hard to use traditional model-based optimization techniques. Artificial Intelligence (AI) explores techniques such as Deep Reinforcement Learning (DRL), which uses the interaction between the agent and the environment to learn what action to take to obtain the best possible result. Researchers usually need to create and develop a simulation environment for their scenario of interest to be able to experiment with DRL algorithms. This takes a large amount of time from the research process, while the lack of a common environment makes it difficult to compare algorithms. The proposed solution aims to fill this gap by creating a tool that facilitates the setting up of DRL training environments for network scenarios. The developed tool uses three open source software, the Containernet to simulate the connections between devices, the Ryu Controller as the Software Defined Network Controller, and OpenAI Gym which is responsible for setting up the communication between the environment and the DRL agent. With the project developed during the thesis, the users will be capable of creating more scenarios in a short period, opening space to set up different environments, solving various problems as well as providing a common environment where other Agents can be compared. The developed software is used to compare the performance of several DRL agents in two different network control problems: routing and network slice admission control. A novel DRL based solution is used in the case of network slice admission control that jointly optimizes the admission and the placement of traffic of a network slice in the physical resources.As redes 5G oferecem melhores serviços e conectam mais dispositivos, fazendo com que se tornem mais complexas e difíceis de gerir. Problemas como a gestão de recursos e a otimização de controlo são cada vez mais dinâmicos e difíceis de modelar, o que torna difícil usar soluções de optimização basea- das em modelos tradicionais. A Inteligência Artificial (IA) explora técnicas como Deep Reinforcement Learning que utiliza a interação entre o agente e o ambiente para aprender qual a ação a ter para obter o melhor resultado possível. Normalmente, os investigadores precisam de criar e desenvolver um ambiente de simulação para poder estudar os algoritmos DRL e a sua interação com o cenário de interesse. A criação de ambientes a partir do zero retira tempo indispensável para a pesquisa em si, e a falta de ambientes de treino comuns torna difícil a comparação dos algoritmos. A solução proposta foca-se em preencher esta lacuna criando uma ferramenta que facilite a configuração de ambientes de treino DRL para cenários de rede. A ferramenta desenvolvida utiliza três softwares open source, o Containernet para simular as conexões entre os dispositivos, o Ryu Controller como Software Defined Network Controller e o OpenAI Gym que é responsável por configurar a comunicação entre o ambiente e o agente DRL. Através do projeto desenvolvido, os utilizadores serão capazes de criar mais cenários em um curto período, abrindo espaço para configurar diferentes ambientes e resolver diferentes problemas, bem como fornecer um ambiente comum onde diferentes Agentes podem ser comparados. O software desenvolvido foi usado para comparar o desempenho de vários agentes DRL em dois problemas diferentes de controlo de rede, nomeadamente, roteamento e controlo de admissão de slices na rede. Uma solução baseada em DRL é usada no caso do controlo de admissão de slices na rede que otimiza conjuntamente a admissão e a colocação de tráfego de uma slice na rede nos recursos físicos da mesma