An Overview on Application of Machine Learning Techniques in Optical Networks

Today's telecommunication networks have become sources of enormous amounts of widely heterogeneous data. This information can be retrieved from network traffic traces, network alarms, signal quality indicators, users' behavioral data, etc. Advanced mathematical tools are required to extract meaningful information from these data and take decisions pertaining to the proper functioning of the networks from the network-generated data. Among these mathematical tools, Machine Learning (ML) is regarded as one of the most promising methodological approaches to perform network-data analysis and enable automated network self-configuration and fault management. The adoption of ML techniques in the field of optical communication networks is motivated by the unprecedented growth of network complexity faced by optical networks in the last few years. Such complexity increase is due to the introduction of a huge number of adjustable and interdependent system parameters (e.g., routing configurations, modulation format, symbol rate, coding schemes, etc.) that are enabled by the usage of coherent transmission/reception technologies, advanced digital signal processing and compensation of nonlinear effects in optical fiber propagation. In this paper we provide an overview of the application of ML to optical communications and networking. We classify and survey relevant literature dealing with the topic, and we also provide an introductory tutorial on ML for researchers and practitioners interested in this field. Although a good number of research papers have recently appeared, the application of ML to optical networks is still in its infancy: to stimulate further work in this area, we conclude the paper proposing new possible research directions

Measurement-Driven Algorithm and System Design for Wireless and Datacenter Networks

The growing number of mobile devices and data-intensive applications pose unique challenges for wireless access networks as well as datacenter networks that enable modern cloud-based services. With the enormous increase in volume and complexity of traffic from applications such as video streaming and cloud computing, the interconnection networks have become a major performance bottleneck. In this thesis, we study algorithms and architectures spanning several layers of the networking protocol stack that enable and accelerate novel applications and that are easily deployable and scalable. The design of these algorithms and architectures is motivated by measurements and observations in real world or experimental testbeds. In the first part of this thesis, we address the challenge of wireless content delivery in crowded areas. We present the AMuSe system, whose objective is to enable scalable and adaptive WiFi multicast. AMuSe is based on accurate receiver feedback and incurs a small control overhead. This feedback information can be used by the multicast sender to optimize multicast service quality, e.g., by dynamically adjusting transmission bitrate. Specifically, we develop an algorithm for dynamic selection of a subset of the multicast receivers as feedback nodes which periodically send information about the channel quality to the multicast sender. Further, we describe the Multicast Dynamic Rate Adaptation (MuDRA) algorithm that utilizes AMuSe's feedback to optimally tune the physical layer multicast rate. MuDRA balances fast adaptation to channel conditions and stability, which is essential for multimedia applications. We implemented the AMuSe system on the ORBIT testbed and evaluated its performance in large groups with approximately 200 WiFi nodes. Our extensive experiments demonstrate that AMuSe can provide accurate feedback in a dense multicast environment. It outperforms several alternatives even in the case of external interference and changing network conditions. Further, our experimental evaluation of MuDRA on the ORBIT testbed shows that MuDRA outperforms other schemes and supports high throughput multicast flows to hundreds of nodes while meeting quality requirements. As an example application, MuDRA can support multiple high quality video streams, where 90% of the nodes report excellent or very good video quality. Next, we specifically focus on ensuring high Quality of Experience (QoE) for video streaming over WiFi multicast. We formulate the problem of joint adaptation of multicast transmission rate and video rate for ensuring high video QoE as a utility maximization problem and propose an online control algorithm called DYVR which is based on Lyapunov optimization techniques. We evaluated the performance of DYVR through analysis, simulations, and experiments using a testbed composed of Android devices and o the shelf APs. Our evaluation shows that DYVR can ensure high video rates while guaranteeing a low but acceptable number of segment losses, buffer underflows, and video rate switches. We leverage the lessons learnt from AMuSe for WiFi to address the performance issues with LTE evolved Multimedia Broadcast/Multicast Service (eMBMS). We present the Dynamic Monitoring (DyMo) system which provides low-overhead and real-time feedback about eMBMS performance. DyMo employs eMBMS for broadcasting instructions which indicate the reporting rates as a function of the observed Quality of Service (QoS) for each UE. This simple feedback mechanism collects very limited QoS reports which can be used for network optimization. We evaluated the performance of DyMo analytically and via simulations. DyMo infers the optimal eMBMS settings with extremely low overhead, while meeting strict QoS requirements under different UE mobility patterns and presence of network component failures. In the second part of the thesis, we study datacenter networks which are key enablers of the end-user applications such as video streaming and storage. Datacenter applications such as distributed file systems, one-to-many virtual machine migrations, and large-scale data processing involve bulk multicast flows. We propose a hardware and software system for enabling physical layer optical multicast in datacenter networks using passive optical splitters. We built a prototype and developed a simulation environment to evaluate the performance of the system for bulk multicasting. Our evaluation shows that the optical multicast architecture can achieve higher throughput and lower latency than IP multicast and peer-to-peer multicast schemes with lower switching energy consumption. Finally, we study the problem of congestion control in datacenter networks. Quantized Congestion Control (QCN), a switch-supported standard, utilizes direct multi-bit feedback from the network for hardware rate limiting. Although QCN has been shown to be fast-reacting and effective, being a Layer-2 technology limits its adoption in IP-routed Layer 3 datacenters. We address several design challenges to overcome QCN feedback's Layer- 2 limitation and use it to design window-based congestion control (QCN-CC) and load balancing (QCN-LB) schemes. Our extensive simulations, based on real world workloads, demonstrate the advantages of explicit, multi-bit congestion feedback, especially in a typical environment where intra-datacenter traffic with short Round Trip Times (RTT: tens of s) run in conjunction with web-facing traffic with long RTTs (tens of milliseconds)

Supporting Diverse Customers and Prioritized Traffic in Next-Generation Passive Optical Networks

The already high demand for more bandwidth usage has been growing rapidly. Access network traffic is usually bursty in nature and the present traffic trend is mostly video-dominant. This motivates the need for higher transmission rates in the system. At the same time, the deployment costs and maintenance expenditures have to be reasonable. Therefore, Passive Optical Networks (PON) are considered promising next-generation access technologies. As the existing PON standards are not suitable to support future-PON services and applications, the FSAN (Full Service Access Network) group and the ITU-T (Telecommunication Standardization Sector of the International Telecommunication Union) have worked on developing the NG- PON2 (Next Generation PON 2) standard. Resource allocation is a fundamental task in any PON and it is necessary to have an efficient scheme that reduces delay, maximizes bandwidth usage, and minimizes the resource wastage. A variety of DBA (Dynamic Bandwidth Allocation) and DWBA (Dynamic Wavelength and Bandwidth Allocation) algorithms have been proposed which are based on different PONs (e.g. EPON, GPON, XG-PON, 10G- EPON, etc.). But to our knowledge, no DWBA scheme for NG-PON2 system, with diverse customers and prioritized traffic, has been proposed yet. In this work, this problem is addressed and five different dynamic wavelength and bandwidth allocation (DWBA) schemes are proposed. First, mixed integer linear programming (MILP) models are developed to minimize the total delay of the high priority data. Due to the MILP’s high computational complexity, heuristic algorithms are developed based on the MILP model insights. The five heuristics algorithms are: No Block-Split Heuristic (NBH), Equal Block-Split Heuristic (EBH), Priority Based No Block-Split Heuristic (P-NBH), Priority Based Equal Block-Split Heuristic (P-EBH), and Priority Based Decider Block-Split Heuristic (P-DBH). Six priority classes of requests are introduced with the goal of minimizing the total delay for the high priority data and to lessen the bandwidth wastage of the system. Finally, experiments for the performance evaluation of the five DWBA schemes are conducted. The results show that P-NBH, P-EBH, P-DBH schemes show a 47.63% less delay and 30% of less bandwidth wastage on average for the highest priority data transmission than the schemes without priority support (NBH and EBH). Among these five schemes, NBH method has the highest delay, whereas EBH and P-EBH waste more bandwidth than the other schemes. P-DBH is the most efficient among the five because this scheme offers the lowest delay for high priority data and the minimum bandwidth wastage for lower priority ones. Adviser: Byrav Ramamurth

Learning for Network Applications and Control

The emergence of new Internet applications and technologies have resulted in an increased complexity as well as a need for lower latency, higher bandwidth, and increased reliability. This ultimately results in an increased complexity of network operation and management. Manual management is not sufficient to meet these new requirements. There is a need for data driven techniques to advance from manual management to autonomous management of network systems. One such technique, Machine Learning (ML), can use data to create models from hidden patterns in the data and make autonomous modifications. This approach has shown significant improvements in other domains (e.g., image recognition and natural language processing). The use of ML, along with advances in programmable control of Software- Defined Networks (SDNs), will alleviate manual network intervention and ultimately aid in autonomous network operations. However, realizing a data driven system that can not only understand what is happening in the network but also operate autonomously requires advances in the networking domain, as well as in ML algorithms. In this thesis, we focus on developing ML-based network architectures and data driven net- working algorithms whose objective is to improve the performance and management of future networks and network applications. We focus on problems spanning across the network protocol stack from the application layer to the physical layer. We design algorithms and architectures that are motivated by measurements and observations in real world or experimental testbeds. In Part I we focus on the challenge of monitoring and estimating user video quality of experience (QoE) of encrypted video traffic for network operators. We develop a system for REal-time QUality of experience metric detection for Encrypted Traffic, Requet. Requet uses a detection algorithm to identify video and audio chunks from the IP headers of encrypted traffic. Features extracted from the chunk statistics are used as input to a random forest ML model to predict QoE metrics. We evaluate Requet on a YouTube dataset we collected, consisting of diverse video assets delivered over various WiFi and LTE network conditions. We then extend Requet, and present a study on YouTube TV live streaming traffic behavior over WiFi and cellular networks covering a 9-month period. We observed pipelined chunk requests, a reduced buffer capacity, and a more stable chunk duration across various video resolutions compared to prior studies of on-demand streaming services. We develop a YouTube TV analysis tool using chunks statistics detected from the extracted data as input to a ML model to infer user QoE metrics. In Part II we consider allocating end-to-end resources in cellular networks. Future cellular networks will utilize SDN and Network Function Virtualization (NFV) to offer increased flexibility for network infrastructure operators to utilize network resources. Combining these technologies with real-time network load prediction will enable efficient use of network resources. Specifically, we leverage a type of recurrent neural network, Long Short-Term Memory (LSTM) neural networks, for (i) service specific traffic load prediction for network slicing, and (ii) Baseband Unit (BBU) pool traffic load prediction in a 5G cloud Radio Access Network (RAN). We show that leveraging a system with better accuracy to predict service requirements results in a reduction of operation costs. We focus on addressing the optical physical layer in Part III. Greater network flexibility through SDN and the growth of high bandwidth services are motivating faster service provisioning and capacity management in the optical layer. These functionalities require increased capacity along with rapid reconfiguration of network resources. Recent advances in optical hardware can enable a dramatic reduction in wavelength provisioning times in optical circuit switched networks. To support such operations, it is imperative to reconfigure the network without causing a drop in service quality to existing users. Therefore, we present a ML system that uses feedforward neural networks to predict the dynamic response of an optically circuit-switched 90-channel multi-hop Reconfigurable Optical Add-Drop Multiplexer (ROADM) network. We show that the trained deep neural network can recommend wavelength assignments for wavelength switching with minimal power excursions. We extend the performance of the ML system by implementing and testing a Hybrid Machine Learning (HML) model, which combines an analytical model with a neural network machine learning model to achieve higher prediction accuracy. In Part IV, we use a data-driven approach to address the challenge of wireless content delivery in crowded areas. We present the Adaptive Multicast Services (AMuSe) system, whose objective is to enable scalable and adaptive WiFi multicast. Specifically, we develop an algorithm for dynamic selection of a subset of the multicast receivers as feedback nodes. Further, we describe the Multicast Dynamic Rate Adaptation (MuDRA) algorithm that utilizes AMuSe’s feedback to optimally tune the physical layer multicast rate. Our experimental evaluation of MuDRA on the ORBIT testbed shows that MuDRA outperforms other schemes and supports high throughput multicast flows to hundreds of nodes while meeting quality requirements. We leverage the lessons learned from AMuSe for WiFi and use order statistics to address the performance issues with LTE evolved Multimedia Broadcast/Multicast Service (eMBMS). We present the Dynamic Monitoring (DyMo) system which provides low-overhead and real-time feedback about eMBMS performance to be used for network optimization. We focus on the Quality of Service (QoS) Evaluation module and develop a Two-step estimation algorithm which can efficiently identify the SNR Threshold as a one time estimation. DyMo significantly outperforms alternative schemes based on the Order-Statistics estimation method which relies on random or periodic sampling

Characterization, design and re-optimization on multi-layer optical networks

L'augment de volum de tràfic IP provocat per l'increment de serveis multimèdia com HDTV o vídeo conferència planteja nous reptes als operadors de xarxa per tal de proveir transmissió de dades eficient. Tot i que les xarxes mallades amb multiplexació per divisió de longitud d'ona (DWDM) suporten connexions òptiques de gran velocitat, aquestes xarxes manquen de flexibilitat per suportar tràfic d’inferior granularitat, fet que provoca un pobre ús d'ample de banda. Per fer front al transport d'aquest tràfic heterogeni, les xarxes multicapa representen la millor solució. Les xarxes òptiques multicapa permeten optimitzar la capacitat mitjançant l'empaquetament de connexions de baixa velocitat dins de connexions òptiques de gran velocitat. Durant aquesta operació, es crea i modifica constantment una topologia virtual dinàmica gràcies al pla de control responsable d’aquestes operacions. Donada aquesta dinamicitat, un ús sub-òptim de recursos pot existir a la xarxa en un moment donat. En aquest context, una re-optimizació periòdica dels recursos utilitzats pot ser aplicada, millorant així l'ús de recursos. Aquesta tesi està dedicada a la caracterització, planificació, i re-optimització de xarxes òptiques multicapa de nova generació des d’un punt de vista unificat incloent optimització als nivells de capa física, capa òptica, capa virtual i pla de control. Concretament s'han desenvolupat models estadístics i de programació matemàtica i meta-heurístiques. Aquest objectiu principal s'ha assolit mitjançant cinc objectius concrets cobrint diversos temes oberts de recerca. En primer lloc, proposem una metodologia estadística per millorar el càlcul del factor Q en problemes d'assignació de ruta i longitud d'ona considerant interaccions físiques (IA-RWA). Amb aquest objectiu, proposem dos models estadístics per computar l'efecte XPM (el coll d'ampolla en termes de computació i complexitat) per problemes IA-RWA, demostrant la precisió d’ambdós models en el càlcul del factor Q en escenaris reals de tràfic. En segon lloc i fixant-nos a la capa òptica, presentem un nou particionament del conjunt de longituds d'ona que permet maximitzar, respecte el cas habitual, la quantitat de tràfic extra proveït en entorns de protecció compartida. Concretament, definim diversos models estadístics per estimar la quantitat de tràfic donat un grau de servei objectiu, i diferents models de planificació de xarxa amb l'objectiu de maximitzar els ingressos previstos i el valor actual net de la xarxa. Després de resoldre aquests problemes per xarxes reals, concloem que la nostra proposta maximitza ambdós objectius. En tercer lloc, afrontem el disseny de xarxes multicapa robustes davant de fallida simple a la capa IP/MPLS i als enllaços de fibra. Per resoldre aquest problema eficientment, proposem un enfocament basat en sobre-dimensionar l'equipament de la capa IP/MPLS i recuperar la connectivitat i el comparem amb la solució convencional basada en duplicar la capa IP/MPLS. Després de comparar solucions mitjançant models ILP i heurístiques, concloem que la nostra solució permet obtenir un estalvi significatiu en termes de costos de desplegament. Com a quart objectiu, introduïm un mecanisme adaptatiu per reduir l'ús de ports opto-electrònics (O/E) en xarxes multicapa sota escenaris de tràfic dinàmic. Una formulació ILP i diverses heurístiques són desenvolupades per resoldre aquest problema, que permet reduir significativament l’ús de ports O/E en temps molt curts. Finalment, adrecem el problema de disseny resilient del pla de control GMPLS. Després de proposar un nou model analític per quantificar la resiliència en topologies mallades de pla de control, usem aquest model per proposar un problema de disseny de pla de control. Proposem un procediment iteratiu lineal i una heurística i els usem per resoldre instàncies reals, arribant a la conclusió que es pot reduir significativament la quantitat d'enllaços del pla de control sense afectar la qualitat de servei a la xarxa.The explosion of IP traffic due to the increase of IP-based multimedia services such as HDTV or video conferencing poses new challenges to network operators to provide a cost-effective data transmission. Although Dense Wavelength Division Multiplexing (DWDM) meshed transport networks support high-speed optical connections, these networks lack the flexibility to support sub-wavelength traffic leading to poor bandwidth usage. To cope with the transport of that huge and heterogeneous amount of traffic, multilayer networks represent the most accepted architectural solution. Multilayer optical networks allow optimizing network capacity by means of packing several low-speed traffic streams into higher-speed optical connections (lightpaths). During this operation, a dynamic virtual topology is created and modified the whole time thanks to a control plane responsible for the establishment, maintenance, and release of connections. Because of this dynamicity, a suboptimal allocation of resources may exist at any time. In this context, a periodically resource reallocation could be deployed in the network, thus improving network resource utilization. This thesis is devoted to the characterization, planning, and re-optimization of next-generation multilayer networks from an integral perspective including physical layer, optical layer, virtual layer, and control plane optimization. To this aim, statistical models, mathematical programming models and meta-heuristics are developed. More specifically, this main objective has been attained by developing five goals covering different open issues. First, we provide a statistical methodology to improve the computation of the Q-factor for impairment-aware routing and wavelength assignment problems (IA-RWA). To this aim we propose two statistical models to compute the Cross-Phase Modulation variance (which represents the bottleneck in terms of computation time and complexity) in off-line and on-line IA-RWA problems, proving the accuracy of both models when computing Q-factor values in real traffic scenarios. Second and moving to the optical layer, we present a new wavelength partitioning scheme that allows maximizing the amount of extra traffic provided in shared path protected environments compared with current solutions. Specifically, we define several statistical models to estimate the traffic intensity given a target grade of service, and different network planning problems for maximizing the expected revenues and net present value. After solving these problems for real networks, we conclude that our proposed scheme maximizes both revenues and NPV. Third, we tackle the design of survivable multilayer networks against single failures at the IP/MPLS layer and WSON links. To efficiently solve this problem, we propose a new approach based on over-dimensioning IP/MPLS devices and lightpath connectivity and recovery and we compare it against the conventional solution based on duplicating backbone IP/MPLS nodes. After evaluating both approaches by means of ILP models and heuristic algorithms, we conclude that our proposed approach leads to significant CAPEX savings. Fourth, we introduce an adaptive mechanism to reduce the usage of opto-electronic (O/E) ports of IP/MPLS-over-WSON multilayer networks in dynamic scenarios. A ILP formulation and several heuristics are developed to solve this problem, which allows significantly reducing the usage of O/E ports in very short running times. Finally, we address the design of resilient control plane topologies in GMPLS-enabled transport networks. After proposing a novel analytical model to quantify the resilience in mesh control plane topologies, we use this model to propose a problem to design the control plane topology. An iterative model and a heuristic are proposed and used to solve real instances, concluding that a significant reduction in the number of control plane links can be performed without affecting the quality of service of the network

An Overview on Application of Machine Learning Techniques in Optical Networks

Today's telecommunication networks have become sources of enormous amounts of widely heterogeneous data. This information can be retrieved from network traffic traces, network alarms, signal quality indicators, users' behavioral data, etc. Advanced mathematical tools are required to extract meaningful information from these data and take decisions pertaining to the proper functioning of the networks from the network-generated data. Among these mathematical tools, machine learning (ML) is regarded as one of the most promising methodological approaches to perform network-data analysis and enable automated network self-configuration and fault management. The adoption of ML techniques in the field of optical communication networks is motivated by the unprecedented growth of network complexity faced by optical networks in the last few years. Such complexity increase is due to the introduction of a huge number of adjustable and interdependent system parameters (e.g., routing configurations, modulation format, symbol rate, coding schemes, etc.) that are enabled by the usage of coherent transmission/reception technologies, advanced digital signal processing, and compensation of nonlinear effects in optical fiber propagation. In this paper we provide an overview of the application of ML to optical communications and networking. We classify and survey relevant literature dealing with the topic, and we also provide an introductory tutorial on ML for researchers and practitioners interested in this field. Although a good number of research papers have recently appeared, the application of ML to optical networks is still in its infancy: to stimulate further work in this area, we conclude this paper proposing new possible research directions