29 research outputs found

    Towards Learning Feasible Hierarchical Decision-Making Policies in Urban Autonomous Driving

    Get PDF
    Modern learning-based algorithms, powered by advanced deep structured neural nets, have multifacetedly facilitated automated driving platforms, spanning from scene characterization and perception to low-level control and state estimation schemes. Nonetheless, urban autonomous driving is regarded as a challenging application for machine learning (ML) and artificial intelligence (AI) since the learnt driving policies must handle complex multi-agent driving scenarios with indeterministic intentions of road participants. In the case of unsignalized intersections, automating the decision-making process at these safety-critical environments entails comprehending numerous layers of abstractions associated with learning robust driving behaviors to allow the vehicle to drive safely and efficiently. Based on our in-depth investigation, we discern that an efficient, yet safe, decision-making scheme for navigating real-world unsignalized intersections does not exist yet. The state-of-the-art schemes lacked practicality to handle real-life complex scenarios as they utilize Low-fidelity vehicle dynamic models which makes them incapable of simulating the real dynamic motion in real-life driving applications. In addition, the conservative behavior of autonomous vehicles, which often overreact to threats which have low likelihood, degrades the overall driving quality and jeopardizes safety. Hence, enhancing driving behavior is essential to attain agile, yet safe, traversing maneuvers in such multi-agent environments. Therefore, the main goal of conducting this PhD research is to develop high-fidelity learning-based frameworks to enhance the autonomous decision-making process at these safety-critical environments. We focus this PhD dissertation on three correlated and complementary research challenges. In our first research challenge, we conduct an in-depth and comprehensive survey on the state-of-the-art learning-based decision-making schemes with the objective of identifying the main shortcomings and potential research avenues. Based on the research directions concluded, we propose, in Problem II and Problem III, novel learning-based frameworks with the objective of enhancing safety and efficiency at different decision-making levels. In Problem II, we develop a novel sensor-independent state estimation for a safety-critical system in urban driving using deep learning techniques. A neural inference model is developed and trained via deep-learning training techniques to obtain accurate state estimates using indirect measurements of vehicle dynamic states and powertrain states. In Problem III, we propose a novel hierarchical reinforcement learning-based decision-making architecture for learning left-turn policies at four-way unsignalized intersections with feasibility guarantees. The proposed technique involves an integration of two main decision-making layers; a high-level learning-based behavioral planning layer which adopts soft actor-critic principles to learn high-level, non-conservative yet safe, driving behaviors, and a motion planning layer that uses low-level Model Predictive Control (MPC) principles to ensure feasibility of the two-dimensional left-turn maneuver. The high-level layer generates reference signals of velocity and yaw angle for the ego vehicle taking into account safety and collision avoidance with the intersection vehicles, whereas the low-level planning layer solves an optimization problem to track these reference commands considering several vehicle dynamic constraints and ride comfort

    Multi-Agent Chance-Constrained Stochastic Shortest Path with Application to Risk-Aware Intelligent Intersection

    Full text link
    In transportation networks, where traffic lights have traditionally been used for vehicle coordination, intersections act as natural bottlenecks. A formidable challenge for existing automated intersections lies in detecting and reasoning about uncertainty from the operating environment and human-driven vehicles. In this paper, we propose a risk-aware intelligent intersection system for autonomous vehicles (AVs) as well as human-driven vehicles (HVs). We cast the problem as a novel class of Multi-agent Chance-Constrained Stochastic Shortest Path (MCC-SSP) problems and devise an exact Integer Linear Programming (ILP) formulation that is scalable in the number of agents' interaction points (e.g., potential collision points at the intersection). In particular, when the number of agents within an interaction point is small, which is often the case in intersections, the ILP has a polynomial number of variables and constraints. To further improve the running time performance, we show that the collision risk computation can be performed offline. Additionally, a trajectory optimization workflow is provided to generate risk-aware trajectories for any given intersection. The proposed framework is implemented in CARLA simulator and evaluated under a fully autonomous intersection with AVs only as well as in a hybrid setup with a signalized intersection for HVs and an intelligent scheme for AVs. As verified via simulations, the featured approach improves intersection's efficiency by up to 200%200\% while also conforming to the specified tunable risk threshold

    EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control System

    Full text link
    Emergency vehicles (EMVs) play a crucial role in responding to time-critical calls such as medical emergencies and fire outbreaks in urban areas. Existing methods for EMV dispatch typically optimize routes based on historical traffic-flow data and design traffic signal pre-emption accordingly; however, we still lack a systematic methodology to address the coupling between EMV routing and traffic signal control. In this paper, we propose EMVLight, a decentralized reinforcement learning (RL) framework for joint dynamic EMV routing and traffic signal pre-emption. We adopt the multi-agent advantage actor-critic method with policy sharing and spatial discounted factor. This framework addresses the coupling between EMV navigation and traffic signal control via an innovative design of multi-class RL agents and a novel pressure-based reward function. The proposed methodology enables EMVLight to learn network-level cooperative traffic signal phasing strategies that not only reduce EMV travel time but also shortens the travel time of non-EMVs. Simulation-based experiments indicate that EMVLight enables up to a 42.6%42.6\% reduction in EMV travel time as well as an 23.5%23.5\% shorter average travel time compared with existing approaches.Comment: 19 figures, 10 tables. Manuscript extended on previous work arXiv:2109.05429, arXiv:2111.0027

    Developments in Estimation and Control for Cloud-Enabled Automotive Vehicles.

    Full text link
    Cloud computing is revolutionizing access to distributed information and computing resources that can facilitate future data and computation intensive vehicular control functions and improve vehicle driving comfort and safety. This dissertation investigates several potential Vehicle-to-Cloud-to-Vehicle (V2C2V) applications that can enhance vehicle control and enable additional functionalities by integrating onboard and cloud resources. Firstly, this thesis demonstrates that onboard vehicle sensors can be used to sense road profiles and detect anomalies. This information can be shared with other vehicles and transportation authorities within a V2C2V framework. The response of hitting a pothole is characterized by a multi-phase dynamic model which is validated by comparing simulation results with a higher-fidelity commercial modeling package. A novel framework of simultaneous road profile estimation and anomaly detection is developed by combining a jump diffusion process (JDP)-based estimator and a multi-input observer. The performance of this scheme is evaluated in an experimental vehicle. In addition, a new clustering algorithm is developed to compress anomaly information by processing anomaly report streams. Secondly, a cloud-aided semi-active suspension control problem is studied demonstrating for the first time that road profile information and noise statistics from the cloud can be used to enhance suspension control. The problem of selecting an optimal damping mode from a finite set of damping modes is considered and the best mode is selected based on performance prediction on the cloud. Finally, a cloud-aided multi-metric route planner is investigated in which safety and comfort metrics augment traditional planning metrics such as time, distance, and fuel economy. The safety metric is developed by processing a comprehensive road and crash database while the comfort metric integrates road roughness and anomalies. These metrics and a planning algorithm can be implemented on the cloud to realize the multi-metric route planning. Real-world case studies are presented. The main contribution of this part of the dissertation is in demonstrating the feasibility and benefits of enhancing the existing route planning algorithms with safety and comfort metrics.PhDAerospace EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/120710/1/zhaojli_1.pd

    New Perspectives on Modelling and Control for Next Generation Intelligent Transport Systems

    Get PDF
    This PhD thesis contains 3 major application areas all within an Intelligent Transportation System context. The first problem we discuss considers models that make beneficial use of the large amounts of data generated in the context of traffic systems. We use a Markov chain model to do this, where important data can be taken into account in an aggregate form. The Markovian model is simple and allows for fast computation, even on low end computers, while at the same time allowing meaningful insight into a variety of traffic system related issues. This allows us to both model and enable the control of aggregate, macroscopic features of traffic networks. We then discuss three application areas for this model: the modelling of congestion, emissions, and the dissipation of energy in electric vehicles. The second problem we discuss is the control of pollution emissions in eets of hybrid vehicles. We consider parallel hybrids that have two power units, an internal combustion engine and an electric motor. We propose a scheme in which we can in uence the mix of the two engines in each car based on simple broadcast signals from a central infrastructure. The infrastructure monitors pollution levels and can thus make the vehicles react to its changes. This leads to a context aware system that can be used to avoid pollution peaks, yet does not restrict drivers unnecessarily. In this context we also discuss technical constraints that have to be taken into account in the design of traffic control algorithms that are of a microscopic nature, i.e. they affect the operation of individual vehicles. We also investigate ideas on decentralised trading of emissions. The goal here is to allocate the rights to pollute fairly among the eet's vehicles. Lastly we discuss the usage of decentralised stochastic assignment strategies in traffic applications. Systems are considered in which reservation schemes can not reliably be provided or enforced and there is a signifficant delay between decisions and their effect. In particular, our approach facilitates taking into account the feedback induced into traffic systems by providing forecasts to large groups of users. This feedback can invalidate the predictions if not modelled carefully. At the same time our proposed strategies are simple rules that are easy to follow, easy to accept, and significantly improve the performance of the systems under study. We apply this approach to three application areas, the assignment of electric vehicles to charging stations, the assignment of vehicles to parking facilities, and the assignment of customers to bike sharing stations. All discussed approaches are analysed using mathematical tools and validated through extensive simulations

    Review of graph-based hazardous event detection methods for autonomous driving systems

    Get PDF
    Automated and autonomous vehicles are often required to operate in complex road environments with potential hazards that may lead to hazardous events causing injury or even death. Therefore, a reliable autonomous hazardous event detection system is a key enabler for highly autonomous vehicles (e.g., Level 4 and 5 autonomous vehicles) to operate without human supervision for significant periods of time. One promising solution to the problem is the use of graph-based methods that are powerful tools for relational reasoning. Using graphs to organise heterogeneous knowledge about the operational environment, link scene entities (e.g., road users, static objects, traffic rules) and describe how they affect each other. Due to a growing interest and opportunity presented by graph-based methods for autonomous hazardous event detection, this paper provides a comprehensive review of the state-of-the-art graph-based methods that we categorise as rule-based, probabilistic, and machine learning-driven. Additionally, we present an in-depth overview of the available datasets to facilitate hazardous event training and evaluation metrics to assess model performance. In doing so, we aim to provide a thorough overview and insight into the key research opportunities and open challenges

    Resource allocation and congestion control in vehicular ad-hoc networks through optimization algorithms and artificial intelligence

    Get PDF
    [SPA] Esta tesis doctoral se presenta bajo la modalidad de compendio de publicaciones. En los 煤ltimos a帽os la creciente demanda de la industria del transporte junto con requisitos de seguridad cada vez m谩s estrictos han promovido el r谩pido desarrollo de las comunicaciones vehiculares. Tales comunicaciones se basan en el intercambio de mensajes peri贸dicos (beacons) que contienen informaci贸n cr铆tica de los veh铆culos. Esta difusi贸n de informaci贸n da origen a lo que com煤nmente se denomina conciencia cooperativa, que permite ampliar las capacidades de numerosos sistemas de asistencia en carretera y las diferentes aplicaciones de seguridad. Ciertamente, la difusi贸n de informaci贸n entre veh铆culos es la base de la conducci贸n aut贸noma y reduce dr谩sticamente el riesgo de colisi贸n y otros eventos indeseados. Sin embargo, es importante tener en cuenta que la carga agregada de los beacons transmitidos puede congestionar r谩pidamente el canal, comprometiendo la recepci贸n de paquetes y, por lo tanto, poniendo en peligro las ventajas que ofrecen tales comunicaciones. Para garantizar la disponibilidad del canal tanto para la recepci贸n correcta de mensajes de emergencia y de las m铆nimas balizas necesarias para satisfacer los requisitos de las aplicaciones de seguridad, una determinada fracci贸n del canal debe de ser reservada. En la literatura relacionada, el control de la congesti贸n se ha abordado mediante el ajuste de varios par谩metros de transmisi贸n (tasa de mensaje, potencia y tasa de bit), pero todav铆a existen numerosos desaf铆os por abordar. Por ejemplo, aunque los par谩metros de transmisi贸n suelen ajustarse individualmente debido a la simplicidad del problema de optimizaci贸n, aqu铆 se muestran las ventajas de ajustar varios par谩metros de forma simult谩nea. En esta tesis, se propone el uso de diferentes algoritmos distribuidos que alcancen el nivel de congesti贸n deseado sin requerir infraestructura ninguna en carretera. La primera parte de esta tesis aborda la asignaci贸n de la tasa de balizamiento mediante la maximizaci贸n de la utilidad de red (NUM) y diferentes m茅tricas de riesgo como el tiempo de colisi贸n y la velocidad de la carretera de aviso. En la segunda parte, no solo se estudian diferentes combinaciones consistentes de par谩metros, sino que tambi茅n nos sumergimos en el paradigma de los algoritmos no cooperativos, en los que no se requiere informaci贸n de los veh铆culos vecinos. El problema de control de la congesti贸n es formulado como un Proceso de Decisi贸n de Markov (MDP) y resuelto mediante t茅cnicas de inteligencia artificial, m谩s concretamente, mediante aprendizaje por refuerzo (RL). Se proponen diferentes soluciones que van desde simples m茅todos tabulares, adecuados para entornos discretos, como Q-learning, hasta funciones de aproximaci贸n m谩s complejas adecuadas para espacios continuos, como SARSA basado en semi-gradiente o redes neuronales artificiales.[ENG] This doctoral dissertation has been presented in the form of thesis by publication. The ever-increasing growth of the transportation industry demands combined with new safety requirements has triggered the development of vehicular communications. These communications among vehicles are based on the exchange of periodical messages or beacons containing valuable information about vehicle state. This gives rise to the socalled cooperative awareness, which allows extending the capabilities of numerous driver assistance systems and safety applications. Disseminating information among vehicles certainly lessens the risk of collision and other undesired events. Nevertheless, the aggregated beaconing load can rapidly jam the channel, compromising packet reception, and therefore endangering the advantages offered by such communications. To guarantee the availability of the channel for emergency messages and the minimum beacons receptions that satisfy safety application requirements, a given fraction of the channel capacity should be available. This congestion control has been addressed by adjusting several transmission parameters but some challenges are still unresolved. Although these parameters are usually optimized individually because of the convexity of the optimization problem, we show the advantages of combining them. In this thesis, we propose the use of different distributed algorithms that reach the desired congestion level without explicitly requiring any costly infrastructure. The first part of this thesis addresses beaconing rate allocation. We propose several distributed solutions based on Network Utility Maximization (NUM) and different risk metrics such as time-to-collision and advisory road speed. In the second part, we not only study different combinations of well-coupled parameters but also dive into the paradigm of noncooperative algorithms, in which no information from neighboring vehicles or centralized infrastructure are required. We formulate the congestion control problem as a Markov Decision Process and solve it by means of different reinforcement learning techniques. In particular, we propose different solutions ranging from tabular methods suitable for simple and discrete environments, like Q-learning, to more complex functions approximations for continuous action-state spaces, such as Semi-gradient SARSA or artificial neural networks.Esta tesis doctoral se presenta bajo la modalidad de compendio de publicaciones. Est谩 formada por estos seis art铆culos: 1. (j1) Aznar-Poveda, J., Egea-Lopez, E., Garcia-Sanchez, A. J., and Pavon-Mari帽o, P. (2019, October). Time-to-Collision-Based Awareness and Congestion Control for Vehicular Communications. IEEE Access, 7, 154192-154208. DOI: 10.1109/ACCESS.2019.2949131. 2. (c1) Aznar-Poveda, J., Egea-Lopez, E., and Garcia-Sanchez, A. J. (2020, May). Cooperative Awareness Message Dissemination in EN 302 637-2: An Adaptation for Winding Roads. IEEE 91st Vehicular Technology Conference (VTC2020-Spring) (pp. 1-5). IEEE. DOI: 10.1109/VTC2020-Spring48590.2020.9128815. 3. (c2) Aznar-Poveda, J., Egea-Lopez, E., Garcia-Sanchez, A. J., and Garcia-Haro, J. (2020, July). Advisory Speed Estimation for an Improved V2X Communications Awareness in Winding Roads. In 2020 22nd International Conference on Transparent Optical Networks (ICTON) (pp. 1-4). IEEE. DOI: 10.1109/ICTON51198.2020.9203478 4. (j2) Aznar-Poveda, J., Garcia-Sanchez, A. J., Egea-Lopez, E., and Garcia-Haro, J. (2021, January). MDPRP: A Q-Learning Approach for the Joint Control of Beaconing Rate and Transmission Power in VANETs. IEEE Access, 9, 10166-10178. DOI: 10.1109/ACCESS.2021.3050625 5. (j3) Aznar-Poveda, J., Garcia-Sanchez, A. J., Egea-Lopez, E., and Garcia-Haro, J. (2021, August). Simultaneous Data Rate and Transmission Power Adaptation in V2V Communications: A Deep Reinforcement Learning Approach. IEEE Access 9, 122067-122081. DOI: 10.1109/ACCESS.2021.3109422 6. (j4) Aznar-Poveda, J., Garcia-Sanchez, A. J., Egea-Lopez, E., and Garcia-Haro, J. (2021, December). Approximate Reinforcement Learning to Control Beaconing Congestion in Distributed Networks. Scientific Reports, 12, 142. DOI: 10.1038/s41598-021-04123-9Escuela Internacional de Doctorado de la Universidad Polit茅cnica de CartagenaUniversidad Polit茅cnica de CartagenaPrograma de Doctorado en Tecnolog铆as de la Informaci贸n y las Comunicacione

    Proceedings, MSVSCC 2012

    Get PDF
    Proceedings of the 6th Annual Modeling, Simulation & Visualization Student Capstone Conference held on April 19, 2012 at VMASC in Suffolk, Virginia
    corecore