Building and Infrastructure Defect Detection and Visualization Using Drone and Deep Learning Technologies

This paper presents an accurate and stable method for object and defect detection and visualization on building and infrastructural facilities. This method uses drones and cameras to collect three- dimensional (3D) point clouds via photogrammetry, and uses orthographic or arbitrary views of the target objects to generate the feature images of points’ spectral, elevation, and normal features. U-Net is implemented in the pixelwise segmentation for object and defect detection using multiple feature images. This method was validated on four applications, including on-site path detection, pavement cracking detection, highway slope detection, and building facade window detection. The comparative experimental results confirmed that U-Net with multiple features has a better pixelwise segmentation performance than separately using each single feature. The developed method can implement object and defect detection with different shapes, including striped objects, thin objects, recurring and regularly shaped objects, and bulky objects, which will improve the accuracy and efficiency of inspection, assessment, and management of buildings and infrastructural facilities

Proposition d’une approche multidisciplinaire pour la maintenance prédictive des chaussées

La dégradation d’une chaussée a pour origine de multiples facteurs tels que le trafic, les conditions climatiques et ses caractéristiques structurelles. Pour maintenir la qualité des infrastructures routières et prolonger leurs durées de vie tout en réduisant les coûts de maintenance, il devient essentiel de prédire et d'anticiper ces dégradations. Dans cette optique, l’utilisation de la maintenance prédictive, basée sur des moyens de surveillance in-situ, d'analyse statistique et d'intelligence artificielle, est donc nécessaire. Cependant, sa mise en œuvre fait face à de nombreux défis tels que la gestion de grandes quantités de données collectées par diverses sources mais aussi la modélisation dans un environnement incertain. Dans ce contexte, pour améliorer la surveillance des infrastructures routières, cette étude combine trois disciplines scientifiques pour démontrer la faisabilité d’un jumeau numérique d'une section de chaussée. La méthodologie proposée s’appuie sur l'optimisation des moyens d'instrumentation des routes, à l'aide de capteurs sans fil, pour alimenter des modèles mécaniques et issus des données destinés à prédire l'endommagement de la chaussée et ainsi anticiper son état de santé. Cette approche pluridisciplinaire est mise en œuvre sur un cas d'étude : une section d’autoroute instrumentée dans la région de Bordeaux en France.The pavement deterioration can be caused by multiple factors such as traffic, weather conditions and structural characteristics. To maintain the quality of roads and extend their life while reducing maintenance costs, it is essential to predict and anticipate deterioration. The use of predictive maintenance, based on in-situ monitoring, statistical analysis and artificial intelligence, is therefore necessary. Nevertheless, its implementation must deal with several challenges such as managing large amounts of data collected from different sources or modelling in an uncertain environment. In this context, to improve road infrastructure monitoring, this study combines three scientific fields to demonstrate the feasibility of a digital twin of a pavement section. The proposed methodology is based on the optimization of road instrumentation tools, using wireless sensors, to feed mechanical and data-driven models to predict pavement damage and thus anticipate its health. This multidisciplinary approach is implemented on a case study: an instrumented highway section in the Bordeaux region in France

Scan4Façade: Automated As-Is Façade Modeling of Historic High-Rise Buildings Using Drones and AI

This paper presents an automated as-is façade modeling method for existing and historic high-rise buildings, named Scan4Façade. To begin with, a camera drone with a spiral path is employed to capture building exterior images, and photogrammetry is used to conduct three-dimensional (3D) reconstruction and create mesh models for the scanned building façades. High-resolution façade orthoimages are then generated from mesh models and pixelwise segmented by an artificial intelligence (AI) model named U-net. A combined data augmentation strategy, including random flipping, rotation, resizing, perspective transformation, and color adjustment, is proposed for model training with a limited number of labels. As a result, the U-net achieves an average pixel accuracy of 0.9696 and a mean intersection over union of 0.9063 in testing. Then, the developed twoStagesClustering algorithm, with a two-round shape clustering and a two-round coordinates clustering, is used to precisely extract façade elements’ dimensions and coordinates from façade orthoimages and pixelwise label. In testing with the Michigan Central Station (office tower), a historic high-rise building, the developed algorithm achieves an accuracy of 99.77% in window extraction. In addition, the extracted façade geometric information and element types are transformed into AutoCAD command and script files to create CAD drawings without manual interaction. Experimental results also show that the proposed Scan4Façade method can provide clear and accurate information to assist BIM feature creation in Revit. Future research recommendations are also stated in this paper

Identifying Asphalt Pavement Distress Using UAV LiDAR Point Cloud Data and Random Forest Classification

Asphalt pavement ages and incurs various distresses due to natural and human factors. Thus, it is crucial to rapidly and accurately extract different types of pavement distress to effectively monitor road health status. In this study, we explored the feasibility of pavement distress identification using low-altitude unmanned aerial vehicle light detection and ranging (UAV LiDAR) and random forest classification (RFC) for a section of an asphalt road that is located in the suburb of Shihezi City in Xinjiang Province of China. After a spectral and spatial feature analysis of pavement distress, a total of 48 multidimensional and multiscale features were extracted based on the strength of the point cloud elevations and reflection intensities. Subsequently, we extracted the pavement distresses from the multifeature dataset by utilizing the RFC method. The overall accuracy of the distress identification was 92.3%, and the kappa coefficient was 0.902. When compared with the maximum likelihood classification (MLC) and support vector machine (SVM), the RFC had a higher accuracy, which confirms its robustness and applicability to multisample and high-dimensional data classification. Furthermore, the method achieved an overall accuracy of 95.86% with a validation dataset. This result indicates the validity and stability of our method, which highway maintenance agencies can use to evaluate road health conditions and implement maintenance

Measurement and Evaluation of Roadway Geometry for Safety Analyses and Pavement Material Volume Estimation for Resurfacing and Rehabilitation Using Mobile LiDAR and Imagery-based Point Clouds

Roadway safety is a multifaceted issue affected by several variables including geometric design features of the roadway, weather conditions, sight distance issues, user behavior, and pavement surface condition. In recent years, transportation agencies have demonstrated a growing interest in utilizing Light Detecting and Ranging (LiDAR) and other remote sensing technologies to enhance data collection productivity, safety, and facilitate the development of strategies to maintain and improve existing roadway infrastructure. Studies have shown that three-dimensional (3D) point clouds acquired using mobile LiDAR systems are highly accurate, dense, and have numerous applications in transportation. Point cloud data applications include extraction of roadway geometry features, asset management, as-built documentation, and maintenance operations. Another source of highly accurate 3D data in the form of point clouds is close-range aerial photogrammetry using unmanned aerial vehicle (UAV) systems. One of the main advantages of these systems over conventional surveying methods is the ability to obtain accurate continuous data in a timely manner. Traditional surveying techniques allow for the collection of road surface data only at specified intervals. Point clouds from LiDAR and imagery-based data can be imported into modeling and design software to create a virtual representation of constructed roadways using 3D models. From a roadway safety assessment standpoint, mobile LiDAR scanning (MLS) systems and UAV close-range photogrammetry (UAV-CRP) can be used as effective methods to produce accurate digital representations of existing roadways for various safety evaluations. This research used LiDAR data collected by five vendors and UAV imagery data collected by the research team to achieve the following objectives: a) evaluate the accuracy of point clouds from MLS and UAV imagery data for collection roadway cross slopes for system-wide cross slope verification; b) evaluate the accuracy of as-built geometry features extracted from MLS and UAV imagery-based point clouds for estimating design speeds on horizontal and vertical curves of existing roadways; c) Determine whether MLS and UAV imagery-based point clouds can be used to produce accurate road surface models for material volume estimation purposes. Ground truth data collected using manual field survey measurements were used to validate the results of this research. Cross slope measurements were extracted from ten randomly selected stations along a 4-lane roadway. This resulted in a total of 42 cross slope measurements per data set including measurements from left turn lanes. The roadway is an urban parkway classified as an urban principal arterial located in Anderson, South Carolina. A comparison of measurements from point clouds and measurements from field survey data using t-test statical analysis showed that deviations between field survey data and MLS and UAV imagery-based point clouds were within the acceptable range of ±0.2% specified by SHRP2 and the South Carolina Department of Transportation (SCDOT). A surface-to-surface method was used to compute and compare material volumes between terrain models from MLS and UAV imagery-based point clouds and a terrain model from field survey data. The field survey data consisted of 424 points collected manually at sixty-nine 100-ft stations over the 1.3-mile study area. The average difference in height for all MLS data was less than 1 inch except for one of the vendors which appeared to be due to a systematic error. The average height difference for the UAV imagery-based data was approximately 1.02 inches. The relatively small errors indicated that these data sets can be used to obtain reliable material volume estimates. Lastly, MLS and UAV imagery-based point clouds were used to obtain horizontal curve radii and superelevation data to estimate design speeds on horizontal curves. Results from paired t-test statistical analyses using a 95% confidence level showed that geometry data extracted from point clouds can be used to obtain realistic estimates of design speeds on horizontal curves. Similarly, road grade and sight distance were obtained from point clouds for design speed estimation on crest and sag vertical curves. A similar approach using a paired t-test statistical analysis at a 95% confidence level showed that point clouds can be used to obtain reliable design speed information on crest and sag vertical curves. The proposed approach offers advantages over extracting information from design drawings which may provide an inaccurate representation of the as-built roadway

Pavement Defect Classification and Localization Using Hybrid Weakly Supervised and Supervised Deep Learning and GIS

Automated detection of road defects has historically been challenging for the pavement management industry. As a result, new methods have been developed over the past few years to handle this issue. Most of these methods relied on supervised machine learning techniques, such as object detection and segmentation methods, which need a large, annotated image dataset to train their models. However, annotating pavement defects is difficult and time-consuming due to their ununiformed and complex shapes. To address this challenge, a hybrid pavement defect classification and localization framework using weakly supervised and supervised deep learning methods is proposed in this thesis. This framework has two steps: (1) A robust hierarchical two-level classifier that classifies the defects in images, and (2) A method for defect localization combining weakly supervised and supervised techniques. In the localization method, first, defects are primarily localized using a weakly supervised method (i.e. Class Activation Mapping (CAM)). Next, based on the results of the first classifiers, the defects are segmented from the localized patches obtained in the previous step. The feature maps extracted from the CAM method are used to train a segmentation network once (i.e. U-Net or Mask R-CNN) to localize and segment the defects in the images. Thus, the proposed framework combines the advantages of weakly supervised and supervised methods. The supervised modules in the framework are trained once and can be used for any new data without the need to train. In other words, to use our framework on new dataset only the classifiers should be fine-tuned. Furthermore, the proposed framework introduced an innovative method designed to calculate the maximum crack width in pixels within linear segmented defect patches, derived from the localization module of the proposed framework. This method is particularly advantageous as it provides critical information that can be further employed in the calculation of the Pavement Condition Index (PCI). Additionally, the proposed method benefits from an asset management inspection system based on Geographic Information System (GIS) technology to prepare the dataset used in the training and testing. Thus, this advanced system serves a dual role within our framework. Firstly, it assists in the assembly and preparation of the dataset used in the model training process, providing a geographically organized collection of images and related data. Secondly, it plays a crucial role in the testing phase, offering a spatially accurate platform for evaluating the effectiveness of the model in real-world scenarios. A dataset from Georgia State in the USA was used in the case study. The proposed framework obtained high precision of 97%, 88%, 92% and 97% for localizing the alligator, block, longitudinal and transverse cracks, respectively. Considering all factors, such as annotation cost, and performance on the test dataset, the proposed localization method outperforms the supervised localization methods, such as instance segmentation and object detection for localizing road pavement defect