Human Behavior Modeling and Human Behavior-aware Control of Automated Vehicles for Trustworthy Navigation

First and foremost, I would like to thank my advisor, Professor Dawn Tilbury, for her constant guidance and encouragement. She has been extremely helpful in developing my technical, research, and personal skills and immensely supportive of my ideas and endeavors throughout graduate school. She has been an excellent mentor and has always been there in my time of need, encouraging and boosting my confidence when I needed them the most. I would like to specially thank my committee members and collaborators, Professors Lionel Robert and Jessie Yang, for their support and encouragement, right from the start of my graduate program. The multi-disciplinary nature of the research initiated by these three Professors is what first drew me towards pursuing a Ph.D. I would also like to thank my other committee members Professors Ilya Kolmanovsky and Ram Vasudevan, for providing their support and feedback that improved the dissertation. I would like to thank the Department of Mechanical Engineering, Rackham Graduate School, and the University of Michigan for giving me the opportunity to pursue the doctoral degree and providing financial support during my time at the university. In addition, I would like to thank the Toyota Research Institute and the Automotive Research Center for providing financial assistance. I really appreciate the support I received from the MAVRIC lab members. The multi-disciplinary culture and environment that the Professors have fostered in the MAVRIC lab have deeply broadened my perspectives. Specically, I would like to thank Hebert Azevedo-Sa. He is usually the first person I discuss my ideas with and has been an excellent critique. I would also like to thank Connor Esterwood, Na Du, Qiaoning Zhang, and Huajing Zhao for the numerous discussions and help with my user studies; especially Connor, who took on a variety of roles to help with my user study|from an engineer to a tailor, to even a hidden driver. Outside of the University of Michigan, I would like to thank my undergraduate advisor, Professor Madhu M., and my internship advisor at the Indian Institute of Technology-Madras, Professor Saravanan Gurunathan. They encouraged me to pursue research and provided me with the necessary opportunities. A special thanks to Sajaysurya Ganesh, a close friend, and collaborator in my early research projects, with who I discuss ideas even now. Last but not least, I would like to thank my family and friends for supporting me during the past several years. My friends at Ann Arbor made life away from home much easier; they are like my second family. A long list of people from my Master's and Ph.D. programs at the University of Michigan has played an essential role in my graduate experience. Still, I would like to especially thank Sandipp Krishnan Ravi, Subramaniam Balakrishna, Rahasudha Kannan, and Paavai Pari for all their love and support. I will fondly remember my time at the University of Michigan and in Ann Arbor because of all of the people I encountered, the friends I made, and the experiences I had. My parents, wife, and extended family have all been incredibly supportive of the pursuit of my degree, and I am eternally grateful for their love and guidance.PHDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169640/1/jskumaar_1.pd

Traffic light detection and V2I communications of an autonomous vehicle with the traffic light for an effective intersection navigation using MAVS simulation

Intersection Navigation plays a significant role in autonomous vehicle operation. This paper focuses on enhancing autonomous vehicle intersection navigation through advanced computer vision and Vehicle-to-Infrastructure (V2I) communication systems. The research unfolds in two phases. In the first phase, an approach utilizing YOLOv8s is proposed for precise traffic light detection and recognition, trained on the Small-Scale Traffic Light Dataset (S2TLD). The second phase establishes seamless connectivity between autonomous vehicles and traffic lights in a simulated Mississippi State University Autonomous Vehicle Simulation (MAVS) environment resembling a small city with multiple intersections. This V2I system enables the transmission of Signal Phase and Timing (SPaT) messages to vehicles, providing information on current traffic light phases and time until the next phase change which enables the vehicles to adjust their speed and behavior in real-time. The simulation demonstrates accurate traffic light detection, with vehicles receiving SPaT messages, showcasing the system’s effectiveness in a multi-intersection scenario

Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Abstract—Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, inter- active motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behaviour as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behaviour, from prediction of individual pedestrians’ likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behaviour, high-level psychological and social modelling of pedestrian behaviour still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behaviour, but much work is still needed to translate them into quantitative algorithms for practical AV control

Towards Learning Feasible Hierarchical Decision-Making Policies in Urban Autonomous Driving

Modern learning-based algorithms, powered by advanced deep structured neural nets, have multifacetedly facilitated automated driving platforms, spanning from scene characterization and perception to low-level control and state estimation schemes. Nonetheless, urban autonomous driving is regarded as a challenging application for machine learning (ML) and artificial intelligence (AI) since the learnt driving policies must handle complex multi-agent driving scenarios with indeterministic intentions of road participants. In the case of unsignalized intersections, automating the decision-making process at these safety-critical environments entails comprehending numerous layers of abstractions associated with learning robust driving behaviors to allow the vehicle to drive safely and efficiently. Based on our in-depth investigation, we discern that an efficient, yet safe, decision-making scheme for navigating real-world unsignalized intersections does not exist yet. The state-of-the-art schemes lacked practicality to handle real-life complex scenarios as they utilize Low-fidelity vehicle dynamic models which makes them incapable of simulating the real dynamic motion in real-life driving applications. In addition, the conservative behavior of autonomous vehicles, which often overreact to threats which have low likelihood, degrades the overall driving quality and jeopardizes safety. Hence, enhancing driving behavior is essential to attain agile, yet safe, traversing maneuvers in such multi-agent environments. Therefore, the main goal of conducting this PhD research is to develop high-fidelity learning-based frameworks to enhance the autonomous decision-making process at these safety-critical environments. We focus this PhD dissertation on three correlated and complementary research challenges. In our first research challenge, we conduct an in-depth and comprehensive survey on the state-of-the-art learning-based decision-making schemes with the objective of identifying the main shortcomings and potential research avenues. Based on the research directions concluded, we propose, in Problem II and Problem III, novel learning-based frameworks with the objective of enhancing safety and efficiency at different decision-making levels. In Problem II, we develop a novel sensor-independent state estimation for a safety-critical system in urban driving using deep learning techniques. A neural inference model is developed and trained via deep-learning training techniques to obtain accurate state estimates using indirect measurements of vehicle dynamic states and powertrain states. In Problem III, we propose a novel hierarchical reinforcement learning-based decision-making architecture for learning left-turn policies at four-way unsignalized intersections with feasibility guarantees. The proposed technique involves an integration of two main decision-making layers; a high-level learning-based behavioral planning layer which adopts soft actor-critic principles to learn high-level, non-conservative yet safe, driving behaviors, and a motion planning layer that uses low-level Model Predictive Control (MPC) principles to ensure feasibility of the two-dimensional left-turn maneuver. The high-level layer generates reference signals of velocity and yaw angle for the ego vehicle taking into account safety and collision avoidance with the intersection vehicles, whereas the low-level planning layer solves an optimization problem to track these reference commands considering several vehicle dynamic constraints and ride comfort

Predicting pedestrian road-crossing assertiveness for autonomous vehicle control

Autonomous vehicles (AVs) must interact with other road users including pedestrians. Unlike passive environments, pedestrians are active agents having their own utilities and decisions, which must be inferred and predicted by AVs in order to control interactions with them and navigation around them. In particular, when a pedestrian wishes to cross the road in front of the vehicle at an unmarked crossing, the pedestrian and AV must compete for the space, which may be considered as a game-theoretic interaction in which one agent must yield to the other. To inform AV controllers in this setting, this study collects and analyses data from real-world human road crossings to determine what features of crossing behaviours are predictive about the level of assertiveness of pedestrians and of the eventual winner of the interactions. It presents the largest and most detailed data set of its kind known to us, and new methods to analyze and predict pedestrian-vehicle interactions based upon it. Pedestrian-vehicle interactions are decomposed into sequences of independent discrete events. We use probabilistic methods - logistic regression and decision tree regression - and sequence analysis to analyze sets and sub-sequences of actions used by both pedestrians and human drivers while crossing at an intersection, to find common patterns of behaviour and to predict the winner of each interaction. We report on the particular features found to be predictive and which can thus be integrated into game-theoretic AV controllers to inform real-time interactions

Naturalistic Driver Intention and Path Prediction using Machine Learning

Autonomous vehicles are still yet to be available to the public. This is because there are a number of challenges that have not been overcome to ensure that autonomous vehicles can safely and efficiently drive on public roads. Accurate prediction of other vehicles is vital for safe driving, as interacting with other vehicles is unavoidable on public streets. This thesis explores reasons why this problem of scene understanding is still unsolved, and presents methods for driver intention and path prediction. The thesis focuses on intersections, as this is a very complex scenario in which to predict the actions of human drivers. There is very limited data available for intersection studies from the perspective of an autonomous vehicle. This thesis presents a very large dataset of over 23,000 vehicle trajectories, used to validate the algorithms presented in this thesis. This dataset was collected using a lidar based vehicle detection and tracking system onboard a vehicle. Analytics of this data is presented. To determine the intent of vehicle at an intersection, a method for manoeuvre classification through the use of recurrent neural networks is presented. This allows accurate predictions of which destination a vehicle will take at an unsignalised intersection, based on that vehicle's approach. The final contribution of this thesis presents a method for driver path prediction, based on recurrent neural networks. It produces a multi-modal prediction for the vehicle’s path with uncertainty assigned to each mode. The output modes are not hand labelled, but instead learned from the data. This results in there not being a fixed number of output modes. Whilst the application of this method is vehicle prediction, this method shows significant promise to be used in other areas of robotics