Development of Detection and Tracking Systems for Autonomous Vehicle using Machine Learning

A thesis presented to the faculty of the Elmer R. Smith College of Business and Technology at Morehead State University in partial fulfillment of the requirements for the Degree Master of Science by Tyler Ward on April 25, 2023

Development of Detection and Tracking Systems for Autonomous Vehicles using Machine Learning

A thesis presented to the faculty of the Elmer R. Smith College of Business and Technology at Morehead State University in partial fulfillment of the requirements for the degree of Master of Science by Tyler Ward April 25, 2023

Real-Time On-Site OpenGL-Based Object Speed Measuring Using Constant Sequential Image

This thesis presents a method that can detect moving objects and measure their speed of movement, using a constant rate series of sequential images, such as video recordings. It uses the industry standard non-vendor specific OpenGL ES so can be implemented on any platform with OpenGL ES support. It can run on low-end embedded system as it uses simple and basic foundations based on a few assumptions to lowering the overall implementation complexity in OpenGL ES. It also does not require any special peripheral devices, so existing infrastructure can be used with minimal modification, which will further lower the cost of this system. The sequential images are streamed from an IO device via the CPU into the GPU where a custom shader is used to detect changing pixels between frames to find potential moving objects. The GPU shader continues by measuring the pixel displacement of each object, and then maps this into a practical distance. These results are then sent back to the CPU for future processing. The algorithm was tested on two real world traffic videos (720p video at 10 FPS) and it successfully extracted the speed data of road vehicles in view on a low-end embedded system (Raspberry Pi 4)

Autonomous Pedestrian Detection in Transit Buses

This project created a proof of concept for an automated pedestrian detection and avoidance system designed for transit buses. The system detects objects up to 12 meters away, calculates the distance from the system using a solid-state LIDAR, and determines if that object is human by passive infrared. This triggers a visual and sound warning. A Xilinx Zynq-SoC utilizing programmable logic and an ARM-based processing system drive data fusion, and an external power unit makes it configurable for transit-buses

A LiDAR Based Semi-Autonomous Collision Avoidance System and the Development of a Hardware-in-the-Loop Simulator to Aid in Algorithm Development and Human Studies

In this paper, the architecture and implementation of an embedded controller for a steering based semi-autonomous collision avoidance system on a 1/10th scale model is presented. In addition, the development of a 2D hardware-in-the-loop simulator with vehicle dynamics based on the bicycle model is described. The semi-autonomous collision avoidance software is fully contained onboard a single-board computer running embedded GNU/Linux. To eliminate any wired tethers that limit the system’s abilities, the driver operates the vehicle at a user-control-station through a wireless Bluetooth interface. The user-control-station is outfitted with a game-controller that provides standard steering wheel and pedal controls along with a television monitor equipped with a wireless video receiver in order to provide a real-time driver’s perspective video feed. The hardware-in-the-loop simulator was developed in order to aid in the evaluation and further development of the semi-autonomous collision avoidance algorithms. In addition, a post analysis tool was created to numerically and visually inspect the controller’s responses. The ultimate goal of this project was to create a wireless 1/10th scale collision avoidance research platform to facilitate human studies surrounding driver assistance and active safety systems in automobiles. This thesis is a continuation of work done by numerous Cal Poly undergraduate and graduate students

Lidar-based Obstacle Detection and Recognition for Autonomous Agricultural Vehicles

Today, agricultural vehicles are available that can drive autonomously and follow exact route plans more precisely than human operators. Combined with advancements in precision agriculture, autonomous agricultural robots can reduce manual labor, improve workflow, and optimize yield. However, as of today, human operators are still required for monitoring the environment and acting upon potential obstacles in front of the vehicle. To eliminate this need, safety must be ensured by accurate and reliable obstacle detection and avoidance systems.In this thesis, lidar-based obstacle detection and recognition in agricultural environments has been investigated. A rotating multi-beam lidar generating 3D point clouds was used for point-wise classification of agricultural scenes, while multi-modal fusion with cameras and radar was used to increase performance and robustness. Two research perception platforms were presented and used for data acquisition. The proposed methods were all evaluated on recorded datasets that represented a wide range of realistic agricultural environments and included both static and dynamic obstacles.For 3D point cloud classification, two methods were proposed for handling density variations during feature extraction. One method outperformed a frequently used generic 3D feature descriptor, whereas the other method showed promising preliminary results using deep learning on 2D range images. For multi-modal fusion, four methods were proposed for combining lidar with color camera, thermal camera, and radar. Gradual improvements in classification accuracy were seen, as spatial, temporal, and multi-modal relationships were introduced in the models. Finally, occupancy grid mapping was used to fuse and map detections globally, and runtime obstacle detection was applied on mapped detections along the vehicle path, thus simulating an actual traversal.The proposed methods serve as a first step towards full autonomy for agricultural vehicles. The study has thus shown that recent advancements in autonomous driving can be transferred to the agricultural domain, when accurate distinctions are made between obstacles and processable vegetation. Future research in the domain has further been facilitated with the release of the multi-modal obstacle dataset, FieldSAFE

Mitigating Emergent Safety and Security Incidents of CPS by a Protective Shell

In today's modern world, Cyber-Physical Systems (CPS) have gained widespread prevalence, offering tremendous benefits while also increasing society's dependence on them. Given the direct interaction of CPS with the physical environment, their malfunction or compromise can pose significant risks to human life, property, and the environment. However, as the complexity of CPS rises due to heightened expectations and expanded functional requirements, ensuring their trustworthy operation solely during the development process becomes increasingly challenging. This thesis introduces and delves into the novel concept of the 'Protective Shell' – a real-time safeguard actively monitoring CPS during their operational phases. The protective shell serves as a last line of defence, designed to detect abnormal behaviour, conduct thorough analyses, and initiate countermeasures promptly, thereby mitigating unforeseen risks in real-time. The primary objective of this research is to enhance the overall safety and security of CPS by refining, partly implementing, and evaluating the innovative protective shell concept. To provide context for collaborative systems working towards higher objectives — common within CPS as system-of-systems (SoS) — the thesis introduces the 'Emergence Matrix'. This matrix categorises outcomes of such collaboration into four quadrants based on their anticipated nature and desirability. Particularly concerning are outcomes that are both unexpected and undesirable, which frequently serve as the root cause of safety accidents and security incidents in CPS scenarios. The protective shell plays a critical role in mitigating these unfavourable outcomes, as conventional vulnerability elimination procedures during the CPS design phase prove insufficient due to their inability to proactively anticipate and address these unforeseen situations. Employing the design science research methodology, the thesis is structured around its iterative cycles and the research questions imposed, offering a systematic exploration of the topic. A detailed analysis of various safety accidents and security incidents involving CPS was conducted to retrieve vulnerabilities that led to dangerous outcomes. By developing specific protective shells for each affected CPS and assessing their effectiveness during these hazardous scenarios, a generic core for the protective shell concept could be retrieved, indicating general characteristics and its overall applicability. Furthermore, the research presents a generic protective shell architecture, integrating advanced anomaly detection techniques rooted in explainable artificial intelligence (XAI) and human machine teaming. While the implementation of protective shells demonstrate substantial positive impacts in ensuring CPS safety and security, the thesis also articulates potential risks associated with their deployment that require careful consideration. In conclusion, this thesis makes a significant contribution towards the safer and more secure integration of complex CPS into daily routines, critical infrastructures and other sectors by leveraging the capabilities of the generic protective shell framework.:1 Introduction 1.1 Background and Context 1.2 Research Problem 1.3 Purpose and Objectives 1.3.1 Thesis Vision 1.3.2 Thesis Mission 1.4 Thesis Outline and Structure 2 Design Science Research Methodology 2.1 Relevance-, Rigor- and Design Cycle 2.2 Research Questions 3 Cyber-Physical Systems 3.1 Explanation 3.2 Safety- and Security-Critical Aspects 3.3 Risk 3.3.1 Quantitative Risk Assessment 3.3.2 Qualitative Risk Assessment 3.3.3 Risk Reduction Mechanisms 3.3.4 Acceptable Residual Risk 3.4 Engineering Principles 3.4.1 Safety Principles 3.4.2 Security Principles 3.5 Cyber-Physical System of Systems (CPSoS) 3.5.1 Emergence 4 Protective Shell 4.1 Explanation 4.2 System Architecture 4.3 Run-Time Monitoring 4.4 Definition 4.5 Expectations / Goals 5 Specific Protective Shells 5.1 Boeing 737 Max MCAS 5.1.1 Introduction 5.1.2 Vulnerabilities within CPS 5.1.3 Specific Protective Shell Mitigation Mechanisms 5.1.4 Protective Shell Evaluation 5.2 Therac-25 5.2.1 Introduction 5.2.2 Vulnerabilities within CPS 5.2.3 Specific Protective Shell Mitigation Mechanisms 5.2.4 Protective Shell Evaluation 5.3 Stuxnet 5.3.1 Introduction 5.3.2 Exploited Vulnerabilities 5.3.3 Specific Protective Shell Mitigation Mechanisms 5.3.4 Protective Shell Evaluation 5.4 Toyota 'Unintended Acceleration' ETCS 5.4.1 Introduction 5.4.2 Vulnerabilities within CPS 5.4.3 Specific Protective Shell Mitigation Mechanisms 5.4.4 Protective Shell Evaluation 5.5 Jeep Cherokee Hack 5.5.1 Introduction 5.5.2 Vulnerabilities within CPS 5.5.3 Specific Protective Shell Mitigation Mechanisms 5.5.4 Protective Shell Evaluation 5.6 Ukrainian Power Grid Cyber-Attack 5.6.1 Introduction 5.6.2 Vulnerabilities in the critical Infrastructure 5.6.3 Specific Protective Shell Mitigation Mechanisms 5.6.4 Protective Shell Evaluation 5.7 Airbus A400M FADEC 5.7.1 Introduction 5.7.2 Vulnerabilities within CPS 5.7.3 Specific Protective Shell Mitigation Mechanisms 5.7.4 Protective Shell Evaluation 5.8 Similarities between Specific Protective Shells 5.8.1 Mitigation Mechanisms Categories 5.8.2 Explanation 5.8.3 Conclusion 6 AI 6.1 Explainable AI (XAI) for Anomaly Detection 6.1.1 Anomaly Detection 6.1.2 Explainable Artificial Intelligence 6.2 Intrinsic Explainable ML Models 6.2.1 Linear Regression 6.2.2 Decision Trees 6.2.3 K-Nearest Neighbours 6.3 Example Use Case - Predictive Maintenance 7 Generic Protective Shell 7.1 Architecture 7.1.1 MAPE-K 7.1.2 Human Machine Teaming 7.1.3 Protective Shell Plugin Catalogue 7.1.4 Architecture and Design Principles 7.1.5 Conclusion Architecture 7.2 Implementation Details 7.3 Evaluation 7.3.1 Additional Vulnerabilities introduced by the Protective Shell 7.3.2 Summary 8 Conclusion 8.1 Summary 8.2 Research Questions Evaluation 8.3 Contribution 8.4 Future Work 8.5 Recommendatio

Autonomous real-time object detection and identification

Sensor devices are regularly used on unmanned aerial vehicles (UAVs) as reconnaissance and intelligence gathering systems and as support for front line troops on operations. This platform provides a wealth of sensor data and has limited computational power available for processing. The objective of this work is to detect and identify objects in real-time, with a low power footprint so that it can operate on a UAV. An appraisal of current computer vision methods is presented, with reference to their performance and applicability to the objectives. Experimentation with real-time methods of background subtraction and motion estimation was carried out and limitations of each method described. A new, assumption free, data driven method for object detection and identification was developed. The core ideas of the development were based on models that propose that the human vision system analyses edges of objects to detect and separate them and perceives motion separately, a function which has been modelled here by optical flow. The initial development in the temporal domain combined object and motion detection in the analysis process. This approach was found to have limitations. The second iteration used a detection component in the spatial domain that extracts texture patches based on edge contours, their profile, and internal texture structure. Motion perception was performed separately on the texture patches using optical flow. The motion and spatial location of texture patches was used to define physical objects. A clustering method is used on the rich feature set extracted by the detection method to characterise the objects. The results show that the method carries out detection and identification of both moving and static objects, in real-time, irrespective of camera motion

Performance Evaluation of Vehicular Ad Hoc Networks using simulation tools

Recent studies demonstrate that the routing protocol performances in vehicular networks can improve using dynamic information on the traffic conditions. WSNs (Wireless Sensor Networks) and VANETs (Vehicular Ad Hoc Networks) are exactly related with this statement and represent the trend of wireless networks research program in the last years. In this context, a new type of network has been developed: in fact, HSVN (Hybrid Sensor and Vehicular Network) let WSNs and VANETs cooperate through dynamic information data exchanges with the aim to improve road safety, and especially to warn the driver and the co-pilot of any event occurred in the road ahead, such as traffic jam, accidents or bad weather. The results will be immediate: less accidents means more saved lives, less traffic means a pollution decrease, and from the technological point of view, this communication protocol will open the door to attractive services, such as downloading of multimedia services or internet browsing, that means easier, safer and more comfortable trips. It is out of doubt that speaking about cars and road technology developments, the market and the interests about this field increase exponentially. Recent projects such as CVIS [1] and COMeSafety [2], focused on improving the road driving, and are the concrete demonstration that this entire context can get soon very close to reality. Owing to their peculiar characteristics, VANETs require the definition of specific networking techniques, whose feasibility and performance are usually tested by means of simulation. Starting from this point, this project will present a HSVN platform, and will also introduce and evaluate a communication protocol between VANETs and WSNs using the NCTUns 6.0 [3] simulator. We will particularly analyze the performances of 2 types of Scenarios developed during our project. Both of them are in an urban context, but we will extract different useful results analyzing the packet losses, the throughput and the end-to-end packet delay