MOZARD: Multi-Modal Localization for Autonomous Vehicles in Urban Outdoor Environments

Visually poor scenarios are one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present MOZARD, a multi-modal localization system for urban outdoor environments using vision and LiDAR. By extending our preexisting key-point based visual multi-session local localization approach with the use of semantic data, an improved localization recall can be achieved across vastly different appearance conditions. In particular we focus on the use of curbstone information because of their broad distribution and reliability within urban environments. We present thorough experimental evaluations on several driving kilometers in challenging urban outdoor environments, analyze the recall and accuracy of our localization system and demonstrate in a case study possible failure cases of each subsystem. We demonstrate that MOZARD is able to bridge scenarios where our previous work VIZARD fails, hence yielding an increased recall performance, while a similar localization accuracy of 0.2m is achieve

A New Curb Detection Method for Unmanned Ground Vehicles Using 2D Sequential Laser Data

sensor

Knowledge Discovery and Data Mining for Shared Mobility and Connected and Automated Vehicle Applications

The rapid development of shared mobility and connected and automated vehicles (CAVs) has not only brought new intelligent transportation system (ITS) challenges with the new types of mobility, but also brought a huge opportunity to accelerate the connectivity and informatization of transportation systems, particularly when we consider all the new forms of data that is becoming available. The primary challenge is how to take advantage of the enormous amount of data to discover knowledge, build effective models, and develop impactful applications. With the theoretical and experimental progress being made over the last two decades, data mining and machine learning technologies have become key approaches for parsing data, understanding information, and making informed decisions, especially as the rise of deep learning algorithms bringing new levels of performance to the analysis of large datasets. The combination of data mining and ITS can greatly benefit research and advances in shared mobility and CAVs.This dissertation focuses on knowledge discovery and data mining for shared mobility and CAV applications. When considering big data associated with shared mobility operations and CAV research, data mining techniques can be customized with transportation knowledge to initially parse the data. Then machine learning methods can be used to model the parsed data to elicit hidden knowledge. Finally, the discovered knowledge and extracted information can help in the development of effective shared mobility and CAV applications to achieve the goals of a safer, faster, and more eco-friendly transportation systems.In this dissertation, there are four main sections that are addressed. First, new methodologies are introduced for extracting lane-level road features from rough crowdsourced GPS trajectories via data mining, which is subsequently used as the fundamental information for CAV applications. The proposed method results in decimeter level accuracy, which satisfies the positioning needs for many macroscopic and microscopic shared mobility and CAV applications. Second, macroscopic ride-hailing service big data has been analyzed for demand prediction, vehicle operation, and system efficiency monitoring. The proposed deep learning algorithms increase the ride-hailing demand prediction accuracy to 80% and can help the fleet dispatching system reduce 30% of vacant travel distance. Third, microscopic automated vehicle perception data has been analyzed for a real-time computer vision system that can be used for lane change behavior detection. The proposed deep learning design combines the residual neural network image input with time serious control data and reaches 95% of lane change behavior prediction accuracy. Last but not least, new ride sharing and CAV applications have been simulated in a behavior modeling framework to analyze the impact of mobility and energy consumption, which addresses key barriers by quantifying the transportation system-wide mobility, energy and behavior impacts from new mobility technologies using real-world data

Semi-automated Generation of Road Transition Lines Using Mobile Laser Scanning Data

Recent advances in autonomous vehicles (AVs) are exponential. Prominent car manufacturers, academic institutions, and corresponding governmental departments around the world are taking active roles in the AV industry. Although the attempts to integrate AV technology into smart roads and smart cities have been in the works for more than half a century, the High Definition Road Maps (HDRMs) that assists full self-driving autonomous vehicles did not yet exist. Mobile Laser Scanning (MLS) has enormous potential in the construction of HDRMs due to its flexibility in collecting wide coverage of street scenes and 3D information on scanned targets. However, without proper and efficient execution, it is difficult to generate HDRMs from MLS point clouds. This study recognizes the research gaps and difficulties in generating transition lines (the paths that pass through a road intersection) in road intersections from MLS point clouds. The proposed method contains three modules: road surface detection, lane marking extraction, and transition line generation. Firstly, the points covering road surface are extracted using the voxel- based upward-growing and the improved region growing. Then, lane markings are extracted and identified according to the multi-thresholding and the geometric filtering. Finally, transition lines are generated through a combination of the lane node structure generation algorithm and the cubic Catmull-Rom spline algorithm. The experimental results demonstrate that transition lines can be successfully generated for both T- and cross-intersections with promising accuracy. In the validation of lane marking extraction using the manually interpreted lane marking points, the method can achieve 90.80% precision, 92.07% recall, and 91.43% F1-score, respectively. The success rate of transition line generation is 96.5%. Furthermore, the Buffer-overlay-statistics (BOS) method validates that the proposed method can generate lane centerlines and transition lines within 20 cm-level localization accuracy from MLS point clouds. In addition, a comparative study is conducted to indicate the better performance of the proposed road marking extraction method than that of three other existing methods. In conclusion, this study makes a considerable contribution to the research on generating transition lines for HDRMs, which further contributes to the research of AVs

Detecção e Rastreamento de Veículos em Movimento para Automóveis Robóticos Autônomos

Neste trabalho, foi investigado o problema de detecção e rastreamento de objetos em movimento (detection and tracking of moving objects - DATMO) para automóveis robóticos autônomos. DATMO envolve a detecção de objetos em movimento no ambiente ao redor do robô e a estimativa do estado (e.g., posição, orientação e velocidade) dos objetos ao longo do tempo. O robô precisa estimar o estado de cada objeto ao longo do tempo, de forma que possa predizer o estado destes objetos alguns segundos mais tarde para fins de mapeamento, localização e navegação. Foram estudadas várias abordagens para a solução deste problema e foi proposto um sistema de detecção e rastreamento de múltiplos veículos em movimento no ambiente ao redor do automóvel robótico autônomo usando um sensor Light Detection and Ranging (LIDAR) 3D. O sistema proposto opera em três etapas: segmentação, associação e rastreamento. A cada varredura do sensor, após a conversão dos dados do sensor em uma nuvem de pontos 3D, na etapa de segmentação os pontos associados ao plano do solo são removidos; a nuvem de pontos é segmentada em agrupamentos de pontos 3D usando a distância Euclidiana, sendo que cada agrupamento representa um objeto no ambiente ao redor do automóvel robótico; nesta etapa os agrupamentos relacionados a meio-fios são também removidos. Na etapa de associação, os objetos observados na varredura atual do sensor são associados aos mesmos objetos observados na varredura anterior usando o algoritmo do vizinho mais próximo (nearest neighbor). Finalmente, na etapa de rastreamento, o estado (posição, orientação e velocidade) dos objetos é estimado usando um filtro de partículas. Os objetos com velocidade acima de um determinado limiar são considerados veículos em movimento. O desempenho do sistema de DATMO proposto foi avaliado usando dados de um sensor LIDAR 3D, além de dados de outros sensores, coletados ao longo de uma volta pelo anel viário do campus da Universidade Federal do Espírito Santo (UFES). Os resultados experimentais mostraram que o sistema de DATMO proposto foi capaz de detectar e rastrear com bom desempenho múltiplos veículos em movimento

Rapid Inspection of Pavement Markings Using Mobile Laser Scanning Point Clouds

Intelligent Transportation System (ITS) is the combination of information technology, sensors and communications for more efficient, safer, more secure and more eco-friendly surface transport. One of the most viable forms of ITS is the driverless car, which exist mainly as prototypes. Serval automobile manufacturers (e.g. Ford, GM, BMW, Toyota, Tesla, Honda) and non-automobile companies (e.g. Apple, Google, Nokia, Baidu, Huawei) have invested in this field, and wider commercialization of the driverless car is estimated in 2025 to 2030. Currently, the key elements of the driverless car are the sensors and a prior 3D map. The sensors mounted on the vehicle are the “eyes” of the driverless car to capture the 3D data of its environment. Comparing its environment and a pre-prepared prior known 3D map, the driverless car can distinguish moving targets (e.g. vehicles, pedestrians) and permanent surface features (e.g. buildings, trees, roads, traffic signs) and take relevant actions. With a centimetre-accuracy prior map, the intractable perception problem is transformed into a solvable localization task. The most important technology for generating the prior map is Mobile Laser Scanning (MLS). MLS technology can safely and rapidly acquire highly dense and accurate georeferenced 3D point clouds with the measurement of surface reflectivity. Therefore, the 3D point clouds with intensity data not only offer the detailed 3D surface of the road but also contains pavement marking information that are embedded in the prior map for automatic navigation. Relevant researches have been focused on the pavement marking extraction from MLS data to collect, update and maintain the 3D prior maps. However, the accuracy and efficiency of automatic extraction of pavement markings can be further improved by intensity correction and window-based enhancement. Thus, this study aims at building a robust method for semi-automated information extraction of pavement markings detected from MLS point clouds. The proposed workflow consists of three components: preprocessing, extraction, and classification. In preprocessing, the 3D MLS point clouds are converted into the radiometrically corrected and enhanced 2D intensity imagery of the road surface. Then the pavement markings are automatically extracted with the intensity using a set of algorithms, including Otsu’s thresholding, neighbour-counting filtering, and region growing. Finally, the extracted pavement markings are classified with the geometric parameters using a manually defined decision tree. Case studies are conducted using the MLS datasets acquired in Kingston (Ontario, Canada) and Xiamen (Fujian, China), respectively, with significantly different road environments by two RIEGL VMX-450 systems. The results demonstrated that the proposed workflow and method can achieve 93% in completeness, 95% in correctness, and 94% in F-score respectively when using Xiamen dataset, while 84%, 93%, 89% respectively when using Kingston dataset

Automated Extraction of Road Information from Mobile Laser Scanning Data

Effective planning and management of transportation infrastructure requires adequate geospatial data. Existing geospatial data acquisition techniques based on conventional route surveys are very time consuming, labor intensive, and costly. Mobile laser scanning (MLS) technology enables a rapid collection of enormous volumes of highly dense, irregularly distributed, accurate geo-referenced point cloud data in the format of three-dimensional (3D) point clouds. Today, more and more commercial MLS systems are available for transportation applications. However, many transportation engineers have neither interest in the 3D point cloud data nor know how to transform such data into their computer-aided model (CAD) formatted geometric road information. Therefore, automated methods and software tools for rapid and accurate extraction of 2D/3D road information from the MLS data are urgently needed. This doctoral dissertation deals with the development and implementation aspects of a novel strategy for the automated extraction of road information from the MLS data. The main features of this strategy include: (1) the extraction of road surfaces from large volumes of MLS point clouds, (2) the generation of 2D geo-referenced feature (GRF) images from the road-surface data, (3) the exploration of point density and intensity of MLS data for road-marking extraction, and (4) the extension of tensor voting (TV) for curvilinear pavement crack extraction. In accordance with this strategy, a RoadModeler prototype with three computerized algorithms was developed. They are: (1) road-surface extraction, (2) road-marking extraction, and (3) pavement-crack extraction. Four main contributions of this development can be summarized as follows. Firstly, a curb-based approach to road surface extraction with assistance of the vehicle’s trajectory is proposed and implemented. The vehicle’s trajectory and the function of curbs that separate road surfaces from sidewalks are used to efficiently separate road-surface points from large volume of MLS data. The accuracy of extracted road surfaces is validated with manually selected reference points. Secondly, the extracted road enables accurate detection of road markings and cracks for transportation-related applications in road traffic safety. To further improve computational efficiency, the extracted 3D road data are converted into 2D image data, termed as a GRF image. The GRF image of the extracted road enables an automated road-marking extraction algorithm and an automated crack detection algorithm, respectively. Thirdly, the automated road-marking extraction algorithm applies a point-density-dependent, multi-thresholding segmentation to the GRF image to overcome unevenly distributed intensity caused by the scanning range, the incidence angle, and the surface characteristics of an illuminated object. The morphological operation is then implemented to deal with the presence of noise and incompleteness of the extracted road markings. Fourthly, the automated crack extraction algorithm applies an iterative tensor voting (ITV) algorithm to the GRF image for crack enhancement. The tensor voting, a perceptual organization method that is capable of extracting curvilinear structures from the noisy and corrupted background, is explored and extended into the field of crack detection. The successful development of three algorithms suggests that the RoadModeler strategy offers a solution to the automated extraction of road information from the MLS data. Recommendations are given for future research and development to be conducted to ensure that this progress goes beyond the prototype stage and towards everyday use