Reinforcement Learning Scheduler for Vehicle-to-Vehicle Communications Outside Coverage

Radio resources in vehicle-to-vehicle (V2V) communication can be scheduled either by a centralized scheduler residing in the network (e.g., a base station in case of cellular systems) or a distributed scheduler, where the resources are autonomously selected by the vehicles. The former approach yields a considerably higher resource utilization in case the network coverage is uninterrupted. However, in case of intermittent or out-of-coverage, due to not having input from centralized scheduler, vehicles need to revert to distributed scheduling. Motivated by recent advances in reinforcement learning (RL), we investigate whether a centralized learning scheduler can be taught to efficiently pre-assign the resources to vehicles for out-of-coverage V2V communication. Specifically, we use the actor-critic RL algorithm to train the centralized scheduler to provide non-interfering resources to vehicles before they enter the out-of-coverage area. Our initial results show that a RL-based scheduler can achieve performance as good as or better than the state-of-art distributed scheduler, often outperforming it. Furthermore, the learning process completes within a reasonable time (ranging from a few hundred to a few thousand epochs), thus making the RL-based scheduler a promising solution for V2V communications with intermittent network coverage.Comment: Article published in IEEE VNC 201

Multi-Agent Reinforcement Learning for Joint Channel Assignment and Power Allocation in Platoon-Based C-V2X Systems

We consider the problem of joint channel assignment and power allocation in underlaid cellular vehicular-to-everything (C-V2X) systems where multiple vehicle-to-infrastructure (V2I) uplinks share the time-frequency resources with multiple vehicle-to-vehicle (V2V) platoons that enable groups of connected and autonomous vehicles to travel closely together. Due to the nature of fast channel variant in vehicular environment, traditional centralized optimization approach relying on global channel information might not be viable in C-V2X systems with large number of users. Utilizing a reinforcement learning (RL) approach, we propose a distributed resource allocation (RA) algorithm to overcome this challenge. Specifically, we model the RA problem as a multi-agent system. Based solely on the local channel information, each platoon leader, who acts as an agent, collectively interacts with each other and accordingly selects the optimal combination of sub-band and power level to transmit its signals. Toward this end, we utilize the double deep Q-learning algorithm to jointly train the agents under the objectives of simultaneously maximizing the V2I sum-rate and satisfying the packet delivery probability of each V2V link in a desired latency limitation. Simulation results show that our proposed RL-based algorithm achieves a close performance compared to that of the well-known exhaustive search algorithm.Comment: 6 pages, 4 figure

Radio resource management for V2X in cellular systems

The thesis focuses on the provision of cellular vehicle-to-everything (V2X) communications, which have attracted great interest for 5G due to the potential of improving traffic safety and enabling new services related to intelligent transportation systems. These types of services have strict requirements on reliability, access availability, and end-to-end (E2E) latency. V2X requires advanced network management techniques that must be developed based on the characteristics of the networks and traffic requirements. The integration of the Sidelink (SL), which enables the direct communication between vehicles (i.e., vehicle-to-vehicle (V2V)) without passing through the base station into cellular networks is a promising solution for enhancing the performance of V2X in cellular systems. In this thesis, we addressed some of the challenges arising from the integration of V2V communication in cellular systems and validated the potential of this technology by providing appropriate resource management solutions. Our main contributions have been in the context of radio access network slicing, mode selection, and radio resource allocation mechanisms. With regard to the first research direction that focuses on the RAN slicing management, a novel strategy based on offline Q-learning and softmax decision-making has been proposed as an enhanced solution to determine the adequate split of resources between a slice for eMBB communications and a slice for V2X. Then, starting from the outcome of the off-line Q-learning algorithm, a low-complexity heuristic strategy has been proposed to achieve further improvements in the use of resources. The proposed solution has been compared against proportional and fixed reference schemes. The extensive performance assessment have revealed the ability of the proposed algorithms to improve network performance compared to the reference schemes, especially in terms of resource utilization, throughput, latency and outage probability. Regarding the second research direction that focuses on the mode selection, two different mode selection solutions referred to as MSSB and MS-RBRS strategies have been proposed for V2V communication over a cellular network. The MSSB strategy decides when it is appropriate to use one or the other mode, i.e. sidelink or cellular, for the involved vehicles, taking into account the quality of the links between V2V users, the available resources, and the network traffic load situation. Moreover, the MS-RBRS strategy not only selects the appropriate mode of operation but also decides efficiently the amount of resources needed by V2V links in each mode and allows reusing RBs between different SL users while guaranteeing the minimum signal to interference requirements. The conducted simulations have revealed that the MS-RBRS and MSSB strategies are beneficial in terms of throughput, radio resource utilization, outage probability and latency under different offered loads comparing to the reference scheme. Last, we have focused on the resource allocation problem including jointly mode selection and radio resource scheduling. For the mode selection, a novel mode selection has been presented to decide when it is appropriate to select sidelink mode and use a distributed approach for radio resource allocation or cellular mode and use a centralized radio resource allocation. It takes into account three aspects: the quality of the links between V2V users, the available resources, and the latency. As for the radio resource allocation, the proposed approach includes a distributed radio resource allocation for sidelink mode and a centralized radio resource allocation for cellular mode. The proposed strategy supports dynamic assignments by allowing transmission over mini-slots. A simulation-based analysis has shown that the proposed strategies improved the network performance in terms of latency of V2V services, packet success rate and resource utilization under different network loads.La tesis se centra en la provisión de comunicaciones para vehículos sistemas celulares (V2X: Vehicle to Everything), que han atraído un gran interés en el contexto de 5G debido a su potencial de mejorar la seguridad del tráfico y habilitar nuevos servicios relacionados con los sistemas inteligentes de transporte. Estos tipos de servicios tienen requisitos estrictos en términos fiabilidad, disponibilidad de acceso y latencia de extremo a extremo (E2E). Para ello, V2X requiere técnicas avanzadas de gestión de red que deben desarrollarse en función de las características de las redes y los requisitos de tráfico. La integración del Sidelink (SL), que permite la comunicación directa entre vehículos (es decir, vehículo a vehículo (V2V)) sin pasar por la estación base de las redes celulares, es una solución prometedora para mejorar el rendimiento de V2X en el sistema celular. En esta tesis, abordamos algunos de los desafíos derivados de la integración de la comunicación V2V en los sistemas celulares y validamos el potencial de esta tecnología al proporcionar soluciones de gestión de recursos adecuadas. Nuestras principales contribuciones han sido en el contexto del denominado "slicing" de redes de acceso radio, la selección de modo y los mecanismos de asignación de recursos radio. Respecto a la primera dirección de investigación que se centra en la gestión del RAN slicing, se ha propuesto una estrategia novedosa basada en Q-learning y toma de decisiones softmax como una solución para determinar la división adecuada de recursos entre un slice para comunicaciones eMBB y un slice para V2X. Luego, a partir del resultado del algoritmo de Q-learning, se ha propuesto una estrategia heurística de baja complejidad para lograr mejoras adicionales en el uso de los recursos. La solución propuesta se ha comparado con esquemas de referencia proporcionales y fijos. La evaluación ha revelado la capacidad de los algoritmos propuestos para mejorar el rendimiento de la red en comparación con los esquemas de referencia, especialmente en términos de utilización de recursos, rendimiento, y latencia . Con respecto a la segunda dirección de investigación que se centra en la selección de modo, se han propuesto dos soluciones de diferentes llamadas estrategias MSSB y MS-RBRS para la comunicación V2V a través de una red celular. La estrategia MSSB decide cuándo es apropiado usar el modo SL o el modo celular, para los vehículos involucrados, teniendo en cuenta la calidad de los enlaces entre los usuarios de V2V, los recursos disponibles y la situación de carga de tráfico de la red. Además, la estrategia MS-RBRS no solo selecciona el modo de operación apropiado, sino que también decide eficientemente la cantidad de recursos que los enlaces V2V necesitan en cada modo, y permite que los RB se reutilicen entre diferentes usuarios de SL al tiempo que garantiza requisitos mínimos de señal a interferencia. Se ha presentado un análisis basado en simulación para evaluar el desempeño de las estrategias propuestas. Finalmente, nos hemos centrado en el problema conjunto de la selección de modo y la asignación de recursos de radio. Para la selección de modo, se ha presentado una nueva estrategia para decidir cuándo es apropiado seleccionar el modo SL y usar un enfoque distribuido para la asignación de recursos de radio o el modo celular y usar la asignación de recursos de radio centralizada. Tiene en cuenta tres aspectos: la calidad de los enlaces entre los usuarios de V2V, los recursos disponibles y la latencia. En términos de asignación de recursos de radio, el enfoque propuesto incluye una asignación de recursos de radio distribuida para el modo SL y una asignación de recursos de radio centralizada para el modo celular. La estrategia propuesta admite asignaciones dinámicas al permitir la transmisión a través de mini-slots. Los resultados muestran las mejoras en términos de latencia, tasa de recepción y la utilización de recursos bajo diferentes cargas de red.Postprint (published version

Internet of Vehicles and Real-Time Optimization Algorithms: Concepts for Vehicle Networking in Smart Cities

Achieving sustainable freight transport and citizens’ mobility operations in modern cities are becoming critical issues for many governments. By analyzing big data streams generated through IoT devices, city planners now have the possibility to optimize traffic and mobility patterns. IoT combined with innovative transport concepts as well as emerging mobility modes (e.g., ridesharing and carsharing) constitute a new paradigm in sustainable and optimized traffic operations in smart cities. Still, these are highly dynamic scenarios, which are also subject to a high uncertainty degree. Hence, factors such as real-time optimization and re-optimization of routes, stochastic travel times, and evolving customers’ requirements and traffic status also have to be considered. This paper discusses the main challenges associated with Internet of Vehicles (IoV) and vehicle networking scenarios, identifies the underlying optimization problems that need to be solved in real time, and proposes an approach to combine the use of IoV with parallelization approaches. To this aim, agile optimization and distributed machine learning are envisaged as the best candidate algorithms to develop efficient transport and mobility systems

셀룰러 사이드링크 성능 향상을 위한 상위계층 기법

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2020. 8. 박세웅.In typical cellular communications, User Equipments (UEs) have always had to go through a Base Station (BS) to communicate with each other, e.g., a UE transmits a packet to a BS via uplink and then the BS transmits the packet to another UE via downlink. Although the communication method can serve UEs efficiently, the communication method can cause latency problems and overload problems in BS. Thus, sidelink has been proposed to overcome these problems in 3GPP release 12. Through sidelink, UEs can communicate directly with each other. There are two representative communications using sidelink, i.e., Device-to-Device (D2D) communication and Vehicle-to-Vehicle (V2V) communication. In this dissertation, we consider three strategies to enhance the performances of D2D and V2V communications: (i) efficient feedback mechanism for D2D communications, (ii) context-aware congestion control scheme for V2V communication, and (iii) In-Device Coexistence (IDC)-aware LTE and NR sidelink resource allocation scheme. Firstly, in the related standard, there is no feedback mechanism for D2D communication because D2D communications only support broadcast-type communications. A feedback mechanism is presented for D2D communications. Through our proposed mechanism, UEs can use the feedback mechanism without the help of BS and UEs do not need additional signals to allocate feedback resources. We also propose a rate adaptation algorithm, which consider in-band emission problem, on top of the proposed feedback mechanism. We find that our rate adaptation achieves higher and stable throughput compared with the legacy scheme that complies to the standard. Secondly, we propose a context-aware congestion control scheme for LTE-V2V communication. Through LTE-V2V communication, UEs transmit Cooperative Awareness Message (CAM), which is a periodic message, and Decentralized Environmental Notification Message (DENM), which is a event-driven message and allows one-hop relay. The above two messages have different characteristics and generation rule. Thus, it is difficult and inefficient to apply the same congestion control scheme to two messages. We propose a congestion control schemes for each message. Through the proposed congestion control schemes, UEs decide whether to transmit according to their situation. Through simulation results, we show that our proposed schemes outperform comparison schemes as well as the legacy scheme. Finally, we propose a NR sidelink resource allocation scheme based on multi-agent reinforcement learning, which awares a IDC problem between LTE and NR in Intelligent Transport System (ITS) band. First, we model a realistic IDC interference based on spectrum emission mask specified at the standard. Then, we formulate the resource allocation as a multi-agent reinforcement learning with fingerprint method. Each UE achieves its local observation and rewards, and learns its policy to increase its rewards through updating Q-network. Through simulation results, we observe that the proposed resource allocation scheme further improves Packet Delivery Ratio (PDR) performances compared to the legacy scheme.전형적인 셀룰러 통신에서는, 단말들은 서로 통신하기 위해 항상 기지국을 거쳐야 한다. 예를 들면, 단말이 uplink를 통해 기지국에게 패킷을 전송한 다음 기지국은 downlink를 통해 해당 패킷을 전송해준다. 이러한 통신방식은 단말들에게 효율적으로 서비스를 제공할 수 있지만, 상황에 따라서는 지연문제와 기지국의 과부하 문제를 야기할 수 있다. 따라서 3GPP release12에서 이러한 문제점들을 극복하기 위해 sidelink가 제안되었다. 덕분에 단말들은 sidelink를 통해서 서로 직접 통신을 할 수 있게 되었다. Sidelink를 사용하는 두 가지 대표적인 통신은 D2D(Device-to-Device) 통신과 V2V(Vehicle-to-Vehicle) 통신이다. 본 논문에서는 D2D 와 V2V 통신 성능을 향상시키기 위한 세가지 전략을 고려한다. (i) D2D 통신을 위한 효율적인 피드백 메커니즘, (ii) V2V 통신을 위한 상황인식기반 혼잡제어 기법, 그리고 (iii) IDC(In-Device Coexistence) 인지 기반 sidelink 자원 할당 방식. 첫째, 관련 표준에는 D2D 통신이 브로드캐스트 유형의 통신만을 지원하기 때문에 D2D 통신에 대한 피드백 메커니즘이 없다. 우리는 이러한 한계점을 극복하고자 D2D 통신을 위한 피드백 메커니즘을 제안한다. 제안된 메커니즘을 통해, 단말은 기지국의 도움없이 피드백 메커니즘을 사용할 수 있으며 피드백 자원을 할당하기 위한 추가 신호를 필요로 하지 않는다. 우리는 또한 제안된 피드백 메커니즘위에서 동작할 수 있는 data rate 조절 기법을 제안하였다. 우리는 시뮬레이션 결과를 통하여, 제안한 data rate 조절 기법이 기존 방식보다 더 높고 안정적인 수율을 제공하는 것을 확인하였다. 둘째, LTE-V2V 통신을 위한 상황 인지 기반 혼잡 제어 기법을 제안한다. LTE-V2V 통신에서 단말들은 주기적인 메시지인 CAM(Cooperative Awareness Message) 및 비주기적 메시지이며 one-hop릴레이를 허용하는 DENM(Decentralized Environmental Notification Message)를 전송한다. 위의 두 메시지는 특성과 생성 규칙이 다르기 때문에 동일한 혼잡 제어 기법을 적용하는 것은 비효율적이다. 따라서 우리는 각 메시지에 적용할 수 있는 혼잡 제어 기법들을 제안한다. 제안된 기법들을 통해서 단말들은 그들의 상황에 따라서 전송 여부를 결정하게 된다. 시뮬레이션 결과를 통해 제안된 기법이 기존 표준 방식 뿐만 아니라 최신의 비교 기법들보다 우수한 성능을 얻는 것을 확인하였다. 마지막으로 ITS(Intelligent Transport System)대역에서 LTE와 NR사이의 IDC문제를 고려하는 NR sidelink 자원할당 기법을 제안한다. 먼저, 표준에 지정된 스펙트럼 방출 마스크를 기반으로 현실적인 IDC 간섭을 모델링한다. 그런 다음 다중 에이전트 강화학습으로 자원할당 기법을 제안한다. 각 단말들은 자신들의 주변 환경을 관측하고 관측된 환경을 기반으로 행동하여 보상을 얻고 Q-network을 자신의 보상을 증가시키도록 정책을 업데이트 및 학습한다. 우리는 시뮬레이션 결과를 통하여 제안된 자원할당 박식이 기존기법 대비하여 PDR(Packet Delivery Ratio) 성능을 향상시키는 것을 확인하였다.Introduction 1 Efficient feedback mechanism for LTE-D2D Communication 8 CoCo: Context-aware congestion control scheme for C-V2X communications 35 IDC-aware resource allocation based on multi-agents reinforcement learning 67 Concluding remarks 84 Abstract(In Korean) 96 감사의 글 99Docto

On the Design of Sidelink for Cellular V2X: A Literature Review and Outlook for Future

Connected and fully automated vehicles are expected to revolutionize our mobility in the near future on a global scale, by significantly improving road safety, traffic efficiency, and traveling experience. Enhanced vehicular applications, such as cooperative sensing and maneuvering or vehicle platooning, heavily rely on direct connectivity among vehicles, which is enabled by sidelink communications. In order to set the ground for the core contribution of this paper, we first analyze the main streams of the cellular-vehicle-to-everything (C-V2X) technology evolution within the Third Generation Partnership Project (3GPP), with focus on the sidelink air interface. Then, we provide a comprehensive survey of the related literature, which is classified and critically dissected, considering both the Long-Term Evolution-based solutions and the 5G New Radio-based latest advancements that promise substantial improvements in terms of latency and reliability. The wide literature review is used as a basis to finally identify further challenges and perspectives, which may shape the C-V2X sidelink developments in the next-generation vehicles beyond 5G