Detection and Localization of Root Damages in Underground Sewer Systems using Deep Neural Networks and Computer Vision Techniques

Indiana University-Purdue University Indianapolis (IUPUI)The maintenance of a healthy sewer infrastructure is a major challenge due to the root damages from nearby plants that grow through pipe cracks or loose joints, which may lead to serious pipe blockages and collapse. Traditional inspections based on video surveillance to identify and localize root damages within such complex sewer networks are inefficient, laborious, and error-prone. Therefore, this study aims to develop a robust and efficient approach to automatically detect root damages and localize their circumferential and longitudinal positions in CCTV inspection videos by applying deep neural networks and computer vision techniques. With twenty inspection videos collected from various resources, keyframes were extracted from each video according to the difference in a LUV color space with certain selections of local maxima. To recognize distance information from video subtitles, OCR models such as Tesseract and CRNN-CTC were implemented and led to a 90% of recognition accuracy. In addition, a pre-trained segmentation model was applied to detect root damages, but it also found many false positive predictions. By applying a well-tuned YoloV3 model on the detection of pipe joints leveraging the Convex Hull Overlap (CHO) feature, we were able to achieve a 20% improvement on the reliability and accuracy of damage identifications. Moreover, an end-to-end deep learning pipeline that involved Triangle Similarity Theorem (TST) was successfully designed to predict the longitudinal position of each identified root damage. The prediction error was less than 1.0 feet

Defect Detection and Classification in Sewer Pipeline Inspection Videos Using Deep Neural Networks

Sewer pipelines as a critical civil infrastructure become a concern for municipalities as they are getting near to the end of their service lives. Meanwhile, new environmental laws and regulations, city expansions, and budget constraints make it harder to maintain these networks. On the other hand, access and inspect sewer pipelines by human-entry based methods are problematic and risky. Current practice for sewer pipeline assessment uses various types of equipment to inspect the condition of pipelines. One of the most used technologies for sewer pipelines inspection is Closed Circuit Television (CCTV). However, application of CCTV method in extensive sewer networks involves certified operators to inspect hours of videos, which is time-consuming, labor-intensive, and error prone. The main objective of this research is to develop a framework for automated defect detection and classification in sewer CCTV inspection videos using computer vision techniques and deep neural networks. This study presents innovative algorithms to deal with the complexity of feature extraction and pattern recognition in sewer inspection videos due to lighting conditions, illumination variations, and unknown patterns of various sewer defects. Therefore, this research includes two main sub-models to first identify and localize anomalies in sewer inspection videos, and in the next phase, detect and classify the defects among the recognized anomalous frames. In the first phase, an innovative approach is proposed for identifying the frames with potential anomalies and localizing them in the pipe segment which is being inspected. The normal and anomalous frames are classified utilizing a one-class support vector machine (OC-SVM). The proposed approach employs 3D Scale Invariant Feature Transform (SIFT) to extract spatio-temporal features and capture scene dynamic statistics in sewer CCTV videos. The OC-SVM is trained by the frame-features which are considered normal, and the outliers to this model are considered abnormal frames. In the next step, the identified anomalous frames are located by recognizing the present text information in them using an end-to-end text recognition approach. The proposed localization approach is performed in two steps, first the text regions are detected using maximally stable extremal regions (MSER) algorithm, then the text characters are recognized using a convolutional neural network (CNN). The performance of the proposed model is tested using videos from real-world sewer inspection reports, where the accuracies of 95% and 86% were achieved for anomaly detection and frame localization, respectively. Identifying the anomalous frames and excluding the normal frames from further analysis could reduce the time and cost of detection. It also ensures the accuracy and quality of assessment by reducing the number of neglected anomalous frames caused by operator error. In the second phase, a defect detection framework is proposed to provide defect detection and classification among the identified anomalous frames. First, a deep Convolutional Neural Network (CNN) which is pre-trained using transfer learning, is used as a feature extractor. In the next step, the remaining convolutional layers of the constructed model are trained by the provided dataset from various types of sewer defects to detect and classify defects in the anomalous frames. The proposed methodology was validated by referencing the ground truth data of a dataset including four defects, and the mAP of 81.3% was achieved. It is expected that the developed model can help sewer inspectors in much faster and more accurate pipeline inspection. The whole framework would decrease the condition assessment time and increase the accuracy of sewer assessment reports

The role of deep learning in urban water management: A critical review

This is the final version. Available on open access from Elsevier via the DOI in this recordDeep learning techniques and algorithms are emerging as a disruptive technology with the potential to transform global economies, environments and societies. They have been applied to planning and management problems of urban water systems in general, however, there is lack of a systematic review of the current state of deep learning applications and an examination of potential directions where deep learning can contribute to solving urban water challenges. Here we provide such a review, covering water demand forecasting, leakage and contamination detection, sewer defect assessment, wastewater system state prediction, asset monitoring and urban flooding. We find that the application of deep learning techniques is still at an early stage as most studies used benchmark networks, synthetic data, laboratory or pilot systems to test the performance of deep learning methods with no practical adoption reported. Leakage detection is perhaps at the forefront of receiving practical implementation into day-to-day operation and management of urban water systems, compared with other problems reviewed. Five research challenges, i.e., data privacy, algorithmic development, explainability and trustworthiness, multi-agent systems and digital twins, are identified as key areas to advance the application and implementation of deep learning in urban water management. Future research and application of deep learning systems are expected to drive urban water systems towards high intelligence and autonomy. We hope this review will inspire research and development that can harness the power of deep learning to help achieve sustainable water management and digitalise the water sector across the world.Royal SocietyAlan Turing InstituteNational Natural Science Foundation of Chin

Artificial Intelligence in Civil Infrastructure Health Monitoring—historical Perspectives, Current Trends, and Future Visions

Over the past 2 decades, the use of artificial intelligence (AI) has exponentially increased toward complete automation of structural inspection and assessment tasks. This trend will continue to rise in image processing as unmanned aerial systems (UAS) and the internet of things (IoT) markets are expected to expand at a compound annual growth rate of 57.5% and 26%, respectively, from 2021 to 2028. This paper aims to catalog the milestone development work, summarize the current research trends, and envision a few future research directions in the innovative application of AI in civil infrastructure health monitoring. A blow-by-blow account of the major technology progression in this research field is provided in a chronological order. Detailed applications, key contributions, and performance measures of each milestone publication are presented. Representative technologies are detailed to demonstrate current research trends. A road map for future research is outlined to address contemporary issues such as explainable and physics-informed AI. This paper will provide readers with a lucid memoir of the historical progress, a good sense of the current trends, and a clear vision for future research

Simultaneous localization and mapping for inspection robots in water and sewer pipe networks: a review

At the present time, water and sewer pipe networks are predominantly inspected manually. In the near future, smart cities will perform intelligent autonomous monitoring of buried pipe networks, using teams of small robots. These robots, equipped with all necessary computational facilities and sensors (optical, acoustic, inertial, thermal, pressure and others) will be able to inspect pipes whilst navigating, selflocalising and communicating information about the pipe condition and faults such as leaks or blockages to human operators for monitoring and decision support. The predominantly manual inspection of pipe networks will be replaced with teams of autonomous inspection robots that can operate for long periods of time over a large spatial scale. Reliable autonomous navigation and reporting of faults at this scale requires effective localization and mapping, which is the estimation of the robot’s position and its surrounding environment. This survey presents an overview of state-of-the-art works on robot simultaneous localization and mapping (SLAM) with a focus on water and sewer pipe networks. It considers various aspects of the SLAM problem in pipes, from the motivation, to the water industry requirements, modern SLAM methods, map-types and sensors suited to pipes. Future challenges such as robustness for long term robot operation in pipes are discussed, including how making use of prior knowledge, e.g. geographic information systems (GIS) can be used to build map estimates, and improve the multi-robot SLAM in the pipe environmen

CONCRETE CRACK EVALUATION FOR CIVIL INFRASTRUCTURE USING COMPUTER VISION AND DEEP LEARNING

Department of Urban and Environmental Engineering (Urban Infrastructure Engineering)Surface cracks of civil infrastructure are one of the important indicators for structural durability and integrity. Concrete cracks are typically investigated by manual visual observation on the surface, which is intrinsically subjective because it highly depends on the experience of inspectors. Furthermore, manual visual inspection is time-consuming, expensive, and often unsafe when inaccessible structural components need to be assessed. Computer vision-based approach is recognized as a promising alternative that can automatically extract crack information from images captured by the digital camera. As texts and cracks are similar in terms of consisting distinguishable lines and curves, image binarization developed for text detection can be appropriate for crack identification purposes. However, although image binarization is useful to separate cracks and backgrounds, the crack assessment is difficult to standardize owing to the high dependence of binarization parameters determined by users. Another critical challenge in digital image processing for crack detection is to automatically distinguish cracks from an image containing actual cracks and crack-like noise patterns (e.g., stains, holes, dark shadows, and lumps), which are often seen on the surface of concrete structures. In addition, a tailored camera system and the corresponding strategy are necessary to effectively address the practical issues in terms of the skewed angle and the process of the sequential crack images for efficient measurement. This research develops a computer vision-based approach in conjunction with deep learning for accurate crack evaluation of for civil infrastructure. The main contribution of the proposed approach can be summarized as follows: (1) a deep learning-based approach for crack detection, (2) a hybrid image processing for crack quantification, and (3) camera systems for the practical issues on civil infrastructure in terms of a skewed angle problem and an efficient measurement with the sequential crack images. The proposed research allows accurate crack evaluation to provide a proper maintenance strategy for civil infrastructure in practice.clos

Point Cloud-based Deep Learning and UAV Path Planning for Surface Defect Detection of Concrete Bridges

Over the past decades, several bridges have collapsed, causing many losses due to the lack of proper monitoring and inspection. Although several new techniques have been developed to detect bridge defects, annual visual inspection remains the main approach. Visual inspection, using naked eyes, is time-consuming and subjective because of human errors. Light Detection and Ranging (LiDAR) scanning is a new technology to collect 3D point clouds. The main strength of point clouds over 2D images is collecting the third dimension of the scanned objects. Deep Learning (DL)-based methods have attracted the researchers’ attention for concrete surface defect detection. However, no point cloud-based DL method is currently available for semantic segmentation of bridge surface defects without converting the raw point cloud dataset into other representations, which results in increasing the size of the dataset and leads to some challenges regarding storage capacity, cost, and training time. Some promising point cloud-based semantic segmentation methods (i.e., PointNet and PointNet++) have been applied in segmenting bridge components (i.e., slabs, piers), but not for segmenting surface defects (i.e., cracks, spalls). Moreover, most of the current point cloud-based concrete surface defect detection methods focus on only one type of defects. On the other hand, in DL, a dataset plays a key role in terms of variety, diversity, accuracy, and size. The lack of publicly available point cloud datasets for bridge surface defects is one of the reasons of the lack of studies in the area of point cloud-based methods. Furthermore, compared with terrestrial LiDAR scanning, LiDAR-equipped Unmanned Aerial Vehicle (UAV) is capable of scanning the inaccessible surfaces of the bridges at a closer distance with higher safety. Although the UAV flying path can be controlled using remote controllers, automating and optimizing UAV path planning is preferable for being able to trace a collision-free path with minimum flight time. To increase the efficiency and accuracy of this approach, it is crucial to scan all parts of the bridge with a near perpendicular view. However, in the case of obstacle existence (e.g., bridge piers), achieving full coverage with near perpendicular view may not be possible. To provide more accurate results, using overlapping views is recommended. However, this method could result in increasing the inspection cost and time. Therefore, overlapping views should be considered only for surface areas where defects are expected. Addressing the above issues, this research aims to: (1) create a publicly available point cloud dataset for concrete bridge surface defect semantic segmentation, (2) develop a point cloud-based semantic segmentation DL method to detect different types of concrete surface defects, and (3) propose a novel near-optimal path planning method for LiDAR-equipped UAV with respect to the minimum path length and maximum coverage considering the potential locations of defects. On this premise, a point cloud-based DL method for semantic segmentation of concrete bridge surface defects (i.e., cracks and spalls), called SNEPointNet++, is developed. To have a network with high-performance, SNEPointNet++ focuses on two main characteristics related to surface defects (i.e., normal vector and depth) and takes into account the issues related to the point cloud dataset (i.e., small size and imbalanced dataset). Sensitivity analysis is applied to capture the best combination of hyperparameters and investigate their effects on network performance. The dataset, which was collected from four concrete bridges, was annotated, augmented, and classified into three classes: cracks, spalls, and no defect. This dataset is made available for other researchers. The model was trained and evaluated using 60% and 20% of the dataset, respectively. Testing on the remaining part of the dataset resulted in 93% recall (69% IoU) and 92% recall (82.5% IoU) for cracks and spalls, respectively. Moreover, the results show that the spalls of the segments deeper than 7 cm (severe spalls) can be detected with 99% recall. On the other hand, this research proposes a 3D path planning method for using a UAV equipped with a LiDAR for bridge inspection to have efficient data collection. The method integrates a Genetic Algorithm (GA) and A* algorithm to solve the Traveling Salesman Problem (TSP), considering the potential locations of bridge surface defects such as cracks. The objective is to minimize the time of flight while achieving maximum visibility. The method provides the potential locations of surface defects to efficiently achieve perpendicular and overlapping views for sampling the viewpoints. Calculating the visibility with respect to the level of criticality leads to giving the priority to covering the areas with higher risk levels. Applying the proposed method on a 3-span bridge in Alberta, the results reveal that considering overlapping views based on the level of criticality of the zones and perpendicular views for all viewpoints leads to accurate and time-efficient data collection