Reconfiguring Network Slices at the Best Time With Deep Reinforcement Learning

International audienceThe emerging 5G induces a great diversity of use cases, a multiplication of the number of connections, an increase in throughput as well as stronger constraints in terms of quality of service such as low latency and isolation of requests. To support these new constraints, Network Function Virtualization (NFV) and Software Defined Network (SDN) technologies have been coupled to introduce the network slicing paradigm. Due to the high dynamicity of the demands, it is crucial to regularly reconfigure the network slices in order to maintain an efficient provisioning of the network. A major concern is to find the best frequency to carry out these reconfigurations, as there is a tradeoff between a reduced network congestion and the additional costs induced by the reconfiguration. In this paper, we tackle the problem of deciding the best moment to reconfigure by taking into account this trade-off. By coupling Deep Reinforcement Learning for decision and a Column Generation algorithm to compute the reconfiguration, we propose Deep-REC and show that choosing the best time during the day to reconfigure allows to maximize the profit of the network operator while minimizing the use of network resources and the congestion of the network. Moreover, by selecting the best moment to reconfigure, our approach allows to decrease the number of needed reconfigurations compared to an algorithm doing periodic reconfigurations during the day

Impact Evaluation of Control Signalling onto Distributed Learning-based Packet Routing

International audienceIn recent years, several works have studied Multi-Agent Deep Reinforcement Learning for the Distributed Packet Routing problem, with promising results in various scenarios where network status changes dynamically, is uncertain, or is partially hidden (e.g., wireless ad hoc networks or wired multidomain networks). Unfortunately, these previous works focus on an ideal scenario where the impact of control signalling is neglected, and network simulation is tailored to simplistic assumptions. In this article, we present the first experimental investigation of control signalling mechanisms for distributed learning-based packet routing. We rely on PRISMA, our opensource simulation ns-3-based module. We formulate two signalling mechanisms between agents (value sharing and model sharing). We investigate the net gains considering in-band signalling and show that routing policies close to those provided by an oracle with full knowledge of traffic and network topology can be discovered with a control overhead of 150 % with respect to injected data packets, if neighboring agents share their Deep Neural Network models. We discuss the generality of our results to underline the importance of assessing net gains of Multi-Agent Deep Reinforcement Learning (MA-DRL)-based routing

Impact Evaluation of Control Signalling onto Distributed Learning-based Packet Routing

International audienceIn recent years, several works have studied Multi-Agent Deep Reinforcement Learning for the Distributed Packet Routing problem, with promising results in various scenarios where network status changes dynamically, is uncertain, or is partially hidden (e.g., wireless ad hoc networks or wired multidomain networks). Unfortunately, these previous works focus on an ideal scenario where the impact of control signalling is neglected, and network simulation is tailored to simplistic assumptions. In this article, we present the first experimental investigation of control signalling mechanisms for distributed learning-based packet routing. We rely on PRISMA, our opensource simulation ns-3-based module. We formulate two signalling mechanisms between agents (value sharing and model sharing). We investigate the net gains considering in-band signalling and show that routing policies close to those provided by an oracle with full knowledge of traffic and network topology can be discovered with a control overhead of 150 % with respect to injected data packets, if neighboring agents share their Deep Neural Network models. We discuss the generality of our results to underline the importance of assessing net gains of Multi-Agent Deep Reinforcement Learning (MA-DRL)-based routing

PRISMA: A Packet Routing Simulator for Multi-Agent Reinforcement Learning

International audienceIn this paper, we present PRISMA: Packet Routing Simulator for Multi-Agent Reinforcement Learning. To the best of our knowledge, this is the first tool specifically conceived to develop and test Reinforcement Learning (RL) algorithms for the Distributed Packet Routing (DPR) problem. In this problem, where a communication node selects the outgoing port to forward a packet using local information, distance-vector routing protocol (e.g., RIP) are traditionally applied. However, when network status changes very dynamically, is uncertain, or is partially hidden (e.g., wireless ad hoc networks or wired multi-domain networks), RL is an alternate solution to discover routing policies better fitted to these cases. Unfortunately, no RL tools have been developed to tackle the DPR problem, forcing the researchers to implement their own simplified RL simulation environments, complicating reproducibility and reducing realism. To overcome these issues, we present PRISMA, which offers to the community a standardized framework where: (i) communication process is realistically modelled (thanks to ns3); (ii) distributed nature is explicitly considered (nodes are implemented as separated threads); (iii) and, RL proposals can be easily developed (thanks to a modular code design and real-time training visualization interfaces) and fairly compared them

prisma-v2: Extension to Cloud Overlay Networks

International audienceIn this paper, we present prisma-v2, a new release of prisma, a Packet Routing Simulator for Multi-Agent Reinforcement Learning. prisma-v2 brings a new set of features. First, it allows simulating overlay network topologies, by integrating virtual links. Second, this release offers the possibility to simulate control packets, which allows to better evaluate the overhead of the network protocol. Last, we integrate the modules along with the core (ns-3) to a docker container, so that it can be run in any machine or platform. prisma-v2 is, to the best of our knowledge, the first realistic overlay network simulation playground that offers to the community the possibility to test and evaluate new network protocols

Packet Routing Simulator for Multi-Agent Reinforcement Learning (PRISMA) (Version v0.1)

PRISMA (Packet Routing Simulator for Multi-Agent Reinforcement Learning) is a network simulation playground for developing and testing Multi-Agent Reinforcement Learning (MARL) solutions for dynamic packet routing (DPR). This framework is based on the OpenAI Gym toolkit and the ns-3 simulator.The OpenAI Gym is a toolkit for RL widely used in research. The network simulator ns–3 is a standard library, which may provide useful simulation tools. It generates discrete events and provides several protocol implementations.Moreover, the NetSim implementation is based on ns3-gym, which integrates OpenAI Gym and ns-3.The main contributions of this framework: 1) A RL framework designed for specifically the DPR problem, serving as a playground where the community can easily validate their own RL approaches and compare them. 2) A more realistic modelling based on: (i) the well-known ns-3 network simulator, and (ii) a multi-threaded implementation for each agent. 3) A modular code design, which allows a researcher to test their own RL algorithm for the DPR problem, without needing to work on the implementation of the environment

Estimation of the Wind Energy Potential in Various North Algerian Regions

This investigation aims to model and assess the wind potential available in seven specific regions of North Algeria. These regions, i.e., Batna, Guelma, Medea, Meliana, Chlef, Tiaret, and Tlemcen, are known for their traditional agriculture. The wind data are obtained from the National Agency of Meteorology (NAM), and a Weibull distribution is applied. In the first part of this study, the wind potential available in these sites is assessed. Then, different models are used to estimate the wind system’s annual recoverable energy for these regions. We are interested in wind pumping for possible use to meet the needs of irrigation water in rural areas. Four kinds of wind turbines are explored to determine the possibility of wind energy conversion. In addition, the effects of the heights of the pylon holding the turbines are inspected by considering four cases (10, 20, 40, and 60 m). This estimation showed that the annual mean wind velocity varies from 2.48 to 5.60 m/s at a level of 10 m. The yearly values of Weibull parameters (k and c) at the studied sites varied within 1.61–2.43 and 3.32–6.20 m/s, respectively. The average wind power density ranged from 11.48 (at Chlef) to 238.43 W/m2 (at Tiaret), and the monthly wind recoverable potential varied from 16.64 to 138 W/m2