Reinforcement Learning Scheduler for Vehicle-to-Vehicle Communications Outside Coverage

Radio resources in vehicle-to-vehicle (V2V) communication can be scheduled either by a centralized scheduler residing in the network (e.g., a base station in case of cellular systems) or a distributed scheduler, where the resources are autonomously selected by the vehicles. The former approach yields a considerably higher resource utilization in case the network coverage is uninterrupted. However, in case of intermittent or out-of-coverage, due to not having input from centralized scheduler, vehicles need to revert to distributed scheduling. Motivated by recent advances in reinforcement learning (RL), we investigate whether a centralized learning scheduler can be taught to efficiently pre-assign the resources to vehicles for out-of-coverage V2V communication. Specifically, we use the actor-critic RL algorithm to train the centralized scheduler to provide non-interfering resources to vehicles before they enter the out-of-coverage area. Our initial results show that a RL-based scheduler can achieve performance as good as or better than the state-of-art distributed scheduler, often outperforming it. Furthermore, the learning process completes within a reasonable time (ranging from a few hundred to a few thousand epochs), thus making the RL-based scheduler a promising solution for V2V communications with intermittent network coverage.Comment: Article published in IEEE VNC 201

Deep Reinforcement Learning-Based Offloading Scheduling for Vehicular Edge Computing

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this recordVehicular edge computing (VEC) is a new computing paradigm that has great potential to enhance the capability of vehicle terminals (VT) to support resource-hungry in-vehicle applications with low latency and high energy efficiency. In this paper, we investigate an important computation offloading scheduling problem in a typical VEC scenario, where a VT traveling along an expressway intends to schedule its tasks waiting in the queue to minimize the long-term cost in terms of a trade-off between task latency and energy consumption. Due to diverse task characteristics, dynamic wireless environment, and frequent handover events caused by vehicle movements, an optimal solution should take into account both where to schedule (i.e., local computation or offloading) and when to schedule (i.e., the order and time for execution) each task. To solve such a complicated stochastic optimization problem, we model it by a carefully designed Markov decision process (MDP) and resort to deep reinforcement learning (DRL) to deal with the enormous state space. Our DRL implementation is designed based on the state-of-the-art proximal policy optimization (PPO) algorithm. A parameter-shared network architecture combined with a convolutional neural network (CNN) is utilized to approximate both policy and value function, which can effectively extract representative features. A series of adjustments to the state and reward representations are taken to further improve the training efficiency. Extensive simulation experiments and comprehensive comparisons with six known baseline algorithms and their heuristic combinations clearly demonstrate the advantages of the proposed DRL-based offloading scheduling method.European Commissio

Coordination and Self-Adaptive Communication Primitives for Low-Power Wireless Networks

The Internet of Things (IoT) is a recent trend where objects are augmented with computing and communication capabilities, often via low-power wireless radios. The Internet of Things is an enabler for a connected and more sustainable modern society: smart grids are deployed to improve energy production and consumption, wireless monitoring systems allow smart factories to detect faults early and reduce waste, while connected vehicles coordinate on the road to ensure our safety and save fuel. Many recent IoT applications have stringent requirements for their wireless communication substrate: devices must cooperate and coordinate, must perform efficiently under varying and sometimes extreme environments, while strict deadlines must be met. Current distributed coordination algorithms have high overheads and are unfit to meet the requirements of today\u27s wireless applications, while current wireless protocols are often best-effort and lack the guarantees provided by well-studied coordination solutions. Further, many communication primitives available today lack the ability to adapt to dynamic environments, and are often tuned during their design phase to reach a target performance, rather than be continuously updated at runtime to adapt to reality.In this thesis, we study the problem of efficient and low-latency consensus in the context of low-power wireless networks, where communication is unreliable and nodes can fail, and we investigate the design of a self-adaptive wireless stack, where the communication substrate is able to adapt to changes to its environment. We propose three new communication primitives: Wireless Paxos brings fault-tolerant consensus to low-power wireless networking, STARC is a middleware for safe vehicular coordination at intersections, while Dimmer builds on reinforcement learning to provide adaptivity to low-power wireless networks. We evaluate in-depth each primitive on testbed deployments and we provide an open-source implementation to enable their use and improvement by the community

Aerial base station placement in temporary-event scenarios

Die Anforderungen an den Netzdatenverkehr sind in den letzten Jahren dramatisch gestiegen, was ein großes Interesse an der Entwicklung neuartiger Lösungen zur Erhöhung der Netzkapazität in Mobilfunknetzen erzeugt hat. Besonderes Augenmerk wurde auf das Problem der Kapazitätsverbesserung bei temporären Veranstaltungen gelegt, bei denen das Umfeld im Wesentlichen dynamisch ist. Um der Dynamik der sich verändernden Umgebung gerecht zu werden und die Bodeninfrastruktur durch zusätzliche Kapazität zu unterstützen, wurde der Einsatz von Luftbasisstationen vorgeschlagen. Die Luftbasisstationen können in der Nähe des Nutzers platziert werden und aufgrund der im Vergleich zur Bodeninfrastruktur höheren Lage die Vorteile der Sichtlinienkommunikation nutzen. Dies reduziert den Pfadverlust und ermöglicht eine höhere Kanalkapazität. Das Optimierungsproblem der Maximierung der Netzkapazität durch die richtige Platzierung von Luftbasisstationen bildet einen Schwerpunkt der Arbeit. Es ist notwendig, das Optimierungsproblem rechtzeitig zu lösen, um auf Veränderungen in der dynamischen Funkumgebung zu reagieren. Die optimale Platzierung von Luftbasisstationen stellt jedoch ein NP-schweres Problem dar, wodurch die Lösung nicht trivial ist. Daher besteht ein Bedarf an schnellen und skalierbaren Optimierungsalgorithmen. Als Erstes wird ein neuartiger Hybrid-Algorithmus (Projected Clustering) vorgeschlagen, der mehrere Lösungen auf der Grundlage der schnellen entfernungsbasierten Kapazitätsapproximierung berechnet und sie auf dem genauen SINR-basierten Kapazitätsmodell bewertet. Dabei werden suboptimale Lösungen vermieden. Als Zweites wird ein neuartiges verteiltes, selbstorganisiertes Framework (AIDA) vorgeschlagen, welches nur lokales Wissen verwendet, den Netzwerkmehraufwand verringert und die Anforderungen an die Kommunikation zwischen Luftbasisstationen lockert. Bei der Formulierung des Platzierungsproblems konnte festgestellt werden, dass Unsicherheiten in Bezug auf die Modellierung der Luft-Bodensignalausbreitung bestehen. Da dieser Aspekt im Rahmen der Analyse eine wichtige Rolle spielt, erfolgte eine Validierung moderner Luft-Bodensignalausbreitungsmodelle, indem reale Messungen gesammelt und das genaueste Modell für die Simulationen ausgewählt wurden.As the traffic demands have grown dramatically in recent years, so has the interest in developing novel solutions that increase the network capacity in cellular networks. The problem of capacity improvement is even more complex when applied to a dynamic environment during a disaster or temporary event. The use of aerial base stations has received much attention in the last ten years as the solution to cope with the dynamics of the changing environment and to supplement the ground infrastructure with extra capacity. Due to higher elevations and possibility to place aerial base stations in close proximity to the user, path loss is significantly smaller in comparison to the ground infrastructure, which in turn enables high data capacity. We are studying the optimization problem of maximizing network capacity by proper placement of aerial base stations. To handle the changes in the dynamic radio environment, it is necessary to promptly solve the optimization problem. However, we show that the optimal placement of aerial base stations is the NP-hard problem and its solution is non-trivial, and thus, there is a need for fast and scalable optimization algorithms. This dissertation investigates how to solve the placement problem efficiently and to support the dynamics of temporary events. First, we propose a novel hybrid algorithm (Projected Clustering), which calculates multiple solutions based on the fast distance-based capacity approximation and evaluates them on the accurate SINR-based capacity model, avoiding sub-optimal solutions. Second, we propose a novel distributed, self-organized framework (AIDA), which conducts a decision-making process using only local knowledge, decreasing the network overhead and relaxing the requirements for communication between aerial base stations. During the formulation of the placement problem, we found that there is still considerable uncertainty with regard to air-to-ground propagation modeling. Since this aspect plays an important role in our analysis, we validated state-of-the-art air-to-ground propagation models by collecting real measurements and chose the most accurate model for the simulations

On the Design of Sidelink for Cellular V2X: A Literature Review and Outlook for Future

Connected and fully automated vehicles are expected to revolutionize our mobility in the near future on a global scale, by significantly improving road safety, traffic efficiency, and traveling experience. Enhanced vehicular applications, such as cooperative sensing and maneuvering or vehicle platooning, heavily rely on direct connectivity among vehicles, which is enabled by sidelink communications. In order to set the ground for the core contribution of this paper, we first analyze the main streams of the cellular-vehicle-to-everything (C-V2X) technology evolution within the Third Generation Partnership Project (3GPP), with focus on the sidelink air interface. Then, we provide a comprehensive survey of the related literature, which is classified and critically dissected, considering both the Long-Term Evolution-based solutions and the 5G New Radio-based latest advancements that promise substantial improvements in terms of latency and reliability. The wide literature review is used as a basis to finally identify further challenges and perspectives, which may shape the C-V2X sidelink developments in the next-generation vehicles beyond 5G

Applications

Volume 3 describes how resource-aware machine learning methods and techniques are used to successfully solve real-world problems. The book provides numerous specific application examples: in health and medicine for risk modelling, diagnosis, and treatment selection for diseases in electronics, steel production and milling for quality control during manufacturing processes in traffic, logistics for smart cities and for mobile communications

Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Unmanned Aerial Vehicle (UAV)-Enabled Wireless Communications and Networking

The emerging massive density of human-held and machine-type nodes implies larger traffic deviatiolns in the future than we are facing today. In the future, the network will be characterized by a high degree of flexibility, allowing it to adapt smoothly, autonomously, and efficiently to the quickly changing traffic demands both in time and space. This flexibility cannot be achieved when the network’s infrastructure remains static. To this end, the topic of UAVs (unmanned aerial vehicles) have enabled wireless communications, and networking has received increased attention. As mentioned above, the network must serve a massive density of nodes that can be either human-held (user devices) or machine-type nodes (sensors). If we wish to properly serve these nodes and optimize their data, a proper wireless connection is fundamental. This can be achieved by using UAV-enabled communication and networks. This Special Issue addresses the many existing issues that still exist to allow UAV-enabled wireless communications and networking to be properly rolled out

Advances in Intelligent Vehicle Control

This book is a printed edition of the Special Issue Advances in Intelligent Vehicle Control that was published in the journal Sensors. It presents a collection of eleven papers that covers a range of topics, such as the development of intelligent control algorithms for active safety systems, smart sensors, and intelligent and efficient driving. The contributions presented in these papers can serve as useful tools for researchers who are interested in new vehicle technology and in the improvement of vehicle control systems