Improving Performance of Feedback-Based Real-Time Networks using Model Checking and Reinforcement Learning

Traditionally, automatic control techniques arose due to need for automation in mechanical systems. These techniques rely on robust mathematical modelling of physical systems with the goal to drive their behaviour to desired set-points. Decades of research have successfully automated, optimized, and ensured safety of a wide variety of mechanical systems. Recent advancement in digital technology has made computers pervasive into every facet of life. As such, there have been many recent attempts to incorporate control techniques into digital technology. This thesis investigates the intersection and co-application of control theory and computer science to evaluate and improve performance of time-critical systems. The thesis applies two different research areas, namely, model checking and reinforcement learning to design and evaluate two unique real-time networks in conjunction with control technologies. The first is a camera surveillance system with the goal of constrained resource allocation to self-adaptive cameras. The second is a dual-delay real-time communication network with the goal of safe packet routing with minimal delays.The camera surveillance system consists of self-adaptive cameras and a centralized manager, in which the cameras capture a stream of images and transmit them to a central manager over a shared constrained communication channel. The event-based manager allocates fractions of the shared bandwidth to all cameras in the network. The thesis provides guarantees on the behaviour of the camera surveillance network through model checking. Disturbances that arise during image capture due to variations in capture scenes are modelled using probabilistic and non-deterministic Markov Decision Processes (MDPs). The different properties of the camera network such as the number of frame drops and bandwidth reallocations are evaluated through formal verification.The second part of the thesis explores packet routing for real-time networks constructed with nodes and directed edges. Each edge in the network consists of two different delays, a worst-case delay that captures high load characteristics, and a typical delay that captures the current network load. Each node in the network takes safe routing decisions by considering delays already encountered and the amount of remaining time. The thesis applies reinforcement learning to route packets through the network with minimal delays while ensuring the total path delay from source to destination does not exceed the pre-determined deadline of the packet. The reinforcement learning algorithm explores new edges to find optimal routing paths while ensuring safety through a simple pre-processing algorithm. The thesis shows that it is possible to apply powerful reinforcement learning techniques to time-critical systems with expert knowledge about the system

Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Intelligent Video Ingestion for Real-time Traffic Monitoring

This is the author accepted manuscript. The final version is available from ACM via the DOI in this recordAs an indispensable part of modern critical infrastructures, cameras deployed at strategic places and prime junctions in an intelligent transportation system (ITS), can help operators in observing traffic flow, identifying any emergency situation, or making decisions regarding road congestion without arriving on the scene. However, these cameras are usually equipped with heterogeneous and turbulent networks, making the realtime smooth playback of traffic monitoring videos with high quality a grand challenge. In this paper, we propose a light-weight Deep Reinforcement Learning (DRL) based approach, namely sRC-C (smart bitRate Control with a Continuous action space), to enhance the quality of realtime traffic monitoring by adjusting the video bitrate adaptively. Distinguished from the existing bitrate adjusting approaches, sRC-C can overcome the bias incurred by deterministic discretization of candidate bitrates by adjusting the video bitrate with more f ine-grained control from a continuous action space, thus significantly improving the Quality-of-Service (QoS). With carefully designed state space and neural network model, sRC-C can be implemented on cameras with scarce resources to support real-time live video streaming with low inference time. Extensive experiments show that sRC-C can reduce the frame loss counts and hold time by 24% and 15.5%, respectively, even with comparable bandwidth utilization. Meanwhile, compared to the-state-of-art approaches, sRC-C can improve the QoS by 30.4%.National Key Research and Development Program of ChinaEuropean Union Horizon 2020Leading Technology of Jiangsu Basic Research PlanNational Natural Science Foundation of ChinaChongqing Key Laboratory of Digital Cinema Art Theory and Technolog

Intelligent multimedia flow transmission through heterogeneous networks using cognitive software defined networks

[ES] La presente tesis aborda el problema del encaminamiento en las redes definidas por software (SDN). Específicamente, aborda el problema del diseño de un protocolo de encaminamiento basado en inteligencia artificial (AI) para garantizar la calidad de servicio (QoS) en transmisiones multimedia. En la primera parte del trabajo, el concepto de SDN es introducido. Su arquitectura, protocolos y ventajas son comentados. A continuación, el estado del arte es presentado, donde diversos trabajos acerca de QoS, encaminamiento, SDN y AI son detallados. En el siguiente capítulo, el controlador SDN, el cual juega un papel central en la arquitectura propuesta, es presentado. Se detalla el diseño del controlador y se compara su rendimiento con otro controlador comúnmente utilizado. Más tarde, se describe las propuestas de encaminamiento. Primero, se aborda la modificación de un protocolo de encaminamiento tradicional. Esta modificación tiene como objetivo adaptar el protocolo de encaminamiento tradicional a las redes SDN, centrado en las transmisiones multimedia. A continuación, la propuesta final es descrita. Sus mensajes, arquitectura y algoritmos son mostrados. Referente a la AI, el capítulo 5 detalla el módulo de la arquitectura que la implementa, junto con los métodos inteligentes usados en la propuesta de encaminamiento. Además, el algoritmo inteligente de decisión de rutas es descrito y la propuesta es comparada con el protocolo de encaminamiento tradicional y con su adaptación a las redes SDN, mostrando un incremento de la calidad final de la transmisión. Finalmente, se muestra y se describe algunas aplicaciones basadas en la propuesta. Las aplicaciones son presentadas para demostrar que la solución presentada en la tesis está diseñada para trabajar en redes heterogéneas.[CA] La present tesi tracta el problema de l'encaminament en les xarxes definides per programari (SDN). Específicament, tracta el problema del disseny d'un protocol d'encaminament basat en intel·ligència artificial (AI) per a garantir la qualitat de servici (QoS) en les transmissions multimèdia. En la primera part del treball, s'introdueix les xarxes SDN. Es comenten la seva arquitectura, els protocols i els avantatges. A continuació, l'estat de l'art és presentat, on es detellen els diversos treballs al voltant de QoS, encaminament, SDN i AI. Al següent capítol, el controlador SDN, el qual juga un paper central a l'arquitectura proposta, és presentat. Es detalla el disseny del controlador i es compara el seu rendiment amb altre controlador utilitzat comunament. Més endavant, es descriuen les propostes d'encaminament. Primer, s'aborda la modificació d'un protocol d'encaminament tradicional. Aquesta modificació té com a objectiu adaptar el protocol d'encaminament tradicional a les xarxes SDN, centrat a les transmissions multimèdia. A continuació, la proposta final és descrita. Els seus missatges, arquitectura i algoritmes són mostrats. Pel que fa a l'AI, el capítol 5 detalla el mòdul de l'arquitectura que la implementa, junt amb els mètodes intel·ligents usats en la proposta d'encaminament. A més a més, l'algoritme intel·ligent de decisió de rutes és descrit i la proposta és comparada amb el protocol d'encaminament tradicional i amb la seva adaptació a les xarxes SDN, mostrant un increment de la qualitat final de la transmissió. Finalment, es mostra i es descriuen algunes aplicacions basades en la proposta. Les aplicacions són presentades per a demostrar que la solució presentada en la tesi és dissenyada per a treballar en xarxes heterogènies.[EN] This thesis addresses the problem of routing in Software Defined Networks (SDN). Specifically, the problem of designing a routing protocol based on Artificial Intelligence (AI) for ensuring Quality of Service (QoS) in multimedia transmissions. In the first part of the work, SDN is introduced. Its architecture, protocols and advantages are discussed. Then, the state of the art is presented, where several works regarding QoS, routing, SDN and AI are detailed. In the next chapter, the SDN controller, which plays the central role in the proposed architecture, is presented. The design of the controller is detailed and its performance compared to another common controller. Later, the routing proposals are described. First, a modification of a traditional routing protocol is discussed. This modification intends to adapt a traditional routing protocol to SDN, focused on multimedia transmissions. Then, the final proposal is described. Its messages, architecture and algorithms are depicted. As regards AI, chapter 5 details the module of the architecture that implements it, along with all the intelligent methods used in the routing proposal. Furthermore, the intelligent route decision algorithm is described and the final proposal is compared to the traditional routing protocol and its adaptation to SDN, showing an increment of the end quality of the transmission. Finally, some applications based on the routing proposal are described. The applications are presented to demonstrate that the proposed solution can work with heterogeneous networks.Rego Máñez, A. (2020). Intelligent multimedia flow transmission through heterogeneous networks using cognitive software defined networks [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/160483TESI

INTELIGENTNA TECHNIKA WYBORU OPTYMALIZATORA: BADANIE PORÓWNAWCZE ZMODYFIKOWANEGO MODELU DENSENET201 Z INNYMI MODELAMI GŁĘBOKIEGO UCZENIA

The rapid growth and development of AI-based applications introduce a wide range of deep and transfer learning model architectures. Selecting an optimal optimizer is still challenging to improve any classification type's performance efficiency and accuracy. This paper proposes an intelligent optimizer selection technique using a new search algorithm to overcome this difficulty. A dataset used in this work was collected and customized for controlling and monitoring roads, especially when emergency vehicles are approaching. In this regard, several deep and transfer learning models have been compared for accurate detection and classification. Furthermore, DenseNet201 layers are frizzed to choose the perfect optimizer. The main goal is to improve the performance accuracy of emergency car classification by performing the test of various optimization methods, including (Adam, Adamax, Nadam, and RMSprob). The evaluation metrics utilized for the model’s comparison with other deep learning techniques are based on classification accuracy, precision, recall, and F1-Score. Test results show that the proposed selection-based optimizer increased classification accuracy and reached 98.84%.Szybki wzrost i rozwój aplikacji opartych na sztucznej inteligencji wprowadzają szeroki zakres architektur modeli głębokiego uczenia i uczenia transferowego. Wybór optymalnego optymalizatora wciąż stanowi wyzwanie w celu poprawy wydajności i dokładności każdego rodzaju klasyfikacji. W niniejszej pracy proponowana jest inteligentna technika wyboru optymalizatora, wykorzystująca nowy algorytm wyszukiwania, aby pokonać to wyzwanie. Zbiór danych użyty w tej pracy został zebrany i dostosowany do celów kontroli i monitorowania dróg, zwłaszcza w sytuacjach, gdy zbliżają się pojazdy ratunkowe. W tym kontekście porównano kilka modeli głębokiego uczenia i uczenia transferowego w celu dokładnej detekcji i klasyfikacji. Ponadto, warstwy DenseNet201 zostały zamrożone, aby wybrać optymalizatora idealnego. Głównym celem jest poprawa dokładności klasyfikacji samochodów ratunkowych poprzez przeprowadzenie testów różnych metod optymalizacji, w tym (Adam, Adamax, Nadam i RMSprob). Metryki oceny wykorzystane do porównania modelu z innymi technikami głębokiego uczenia opierają się na dokładności klasyfikacji, precyzji, czułości i miarze F1. Wyniki testów pokazują, że zaproponowany optymalizator oparty na wyborze zwiększył dokładność klasyfikacji i osiągnął wynik na poziomie 98,84%

Adapting reinforcement learning for multimedia transmission on SDN

[EN] Multimedia transmissions require a high quantity of resources to ensure their quality. In the last years, some technologies that provide a better resource management have appeared. Software defined networks (SDNs) are presented as a solution to improve this management. Furthermore, combining SDN with artificial intelligence (AI) techniques, networks are able to provide a higher performance using the same resources. In this paper, a redefinition of reinforcement learning is proposed. This model is focused on multimedia transmission in a SDN environment. Moreover, the architecture needed and the algorithm of the reinforcement learning are described. Using the Openflow protocol, several sample actions are defined in the system. Results show that using the system users perceive an increase in the image quality three times better. Moreover, the loss rate is reduced more than half the value of losses recorded when the algorithm is not applied. Regarding bandwidth, the maximum throughput increases from 987.16 kbps to 24.73 Mbps while the average bandwidth improves from 412.42 kbps to 7.83 Mbps.Ayudas para contratos predoctorales de Formación del Profesorado Universitario FPU (Convocatoria 2015), Grant/Award Number: FPU15/06837; Programa Estatal de Investigación Científica y Técnica de Excelencia (Convocatoria 2017), Grant/Award Number: TIN2017-84802-C2-1-P; Programa Estatal De Investigación, Desarrollo e Innovación Orientada a los retos de la sociedad (Convocatoria 2016), Grant/Award Number: TEC2016-76795-C6-4-R; ERANETMED, Grant/Award Number: ERANETMED3-227 SMARTWATIRRego Mañez, A.; Sendra, S.; García-García, L.; Lloret, J. (2019). Adapting reinforcement learning for multimedia transmission on SDN. Transactions on Emerging Telecommunications Technologies. 30(9):1-15. https://doi.org/10.1002/ett.3643S11530

Communication and Control in Collaborative UAVs: Recent Advances and Future Trends

The recent progress in unmanned aerial vehicles (UAV) technology has significantly advanced UAV-based applications for military, civil, and commercial domains. Nevertheless, the challenges of establishing high-speed communication links, flexible control strategies, and developing efficient collaborative decision-making algorithms for a swarm of UAVs limit their autonomy, robustness, and reliability. Thus, a growing focus has been witnessed on collaborative communication to allow a swarm of UAVs to coordinate and communicate autonomously for the cooperative completion of tasks in a short time with improved efficiency and reliability. This work presents a comprehensive review of collaborative communication in a multi-UAV system. We thoroughly discuss the characteristics of intelligent UAVs and their communication and control requirements for autonomous collaboration and coordination. Moreover, we review various UAV collaboration tasks, summarize the applications of UAV swarm networks for dense urban environments and present the use case scenarios to highlight the current developments of UAV-based applications in various domains. Finally, we identify several exciting future research direction that needs attention for advancing the research in collaborative UAVs

Towards Autonomous Computer Networks in Support of Critical Systems

L'abstract è presente nell'allegato / the abstract is in the attachmen

Dynamic Network State Learning Model for Mobility Based WMSN Routing Protocol

The rising demand of wireless multimedia sensor networks (WMSNs) has motivated academia-industries to develop energy efficient, Quality of Service (QoS) and delay sensitive communication systems to meet major real-world demands like multimedia broadcast, security and surveillance systems, intelligent transport system, etc. Typically, energy efficiency, QoS and delay sensitive transmission are the inevitable requirements of WMSNs. Majority of the existing approaches either use physical layer or system level schemes that individually can’t assure optimal transmission decision to meet the demand. The cumulative efficiency of physical layer power control, adaptive modulation and coding and system level dynamic power management (DPM) are found significant to achieve these demands. With this motivation, in this paper a unified model is derived using enhanced reinforcement learning and stochastic optimization method. Exploiting physical as well as system level network state information, our proposed dynamic network state learning model (NSLM) applies stochastic optimization to learn network state-activity that derives an optimal DPM policy and PHY switching scheduling. NSLM applies known as well as unknown network state variables to derive transmission and PHY switching policy, where it considers DPM as constrained Markov decision process (MDP) problem. Here,the use of Hidden Markov Model and Lagrangian relaxation has made NSLM convergence swift that assures delay-sensitive, QoS enriched, and bandwidth and energy efficient transmission for WMSN under uncertain network conditions. Our proposed NSLM DPM model has outperformed traditional Q-Learning based DPM in terms of buffer cost, holding cost, overflow, energy consumption and bandwidth utilization