Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark

Perhaps surprisingly sewerage infrastructure is one of the most costly infrastructures in modern society. Sewer pipes are manually inspected to determine whether the pipes are defective. However, this process is limited by the number of qualified inspectors and the time it takes to inspect a pipe. Automatization of this process is therefore of high interest. So far, the success of computer vision approaches for sewer defect classification has been limited when compared to the success in other fields mainly due to the lack of public datasets. To this end, in this work we present a large novel and publicly available multi-label classification dataset for image-based sewer defect classification called Sewer-ML. The Sewer-ML dataset consists of 1.3 million images annotated by professional sewer inspectors from three different utility companies across nine years. Together with the dataset, we also present a benchmark algorithm and a novel metric for assessing performance. The benchmark algorithm is a result of evaluating 12 state-of-the-art algorithms, six from the sewer defect classification domain and six from the multi-label classification domain, and combining the best performing algorithms. The novel metric is a class-importance weighted F2 score, $\text{F}2_{\text{CIW}}$, reflecting the economic impact of each class, used together with the normal pipe F1 score, $\text{F}1_{\text{Normal}}$. The benchmark algorithm achieves an $\text{F}2_{\text{CIW}}$ score of 55.11% and $\text{F}1_{\text{Normal}}$ score of 90.94%, leaving ample room for improvement on the Sewer-ML dataset. The code, models, and dataset are available at the project page https://vap.aau.dk/sewer-ml/Comment: CVPR 2021. Project webpage: https://vap.aau.dk/sewer-ml

Automatic Analysis of Sewer Pipes Based on Unrolled Monocular Fisheye Images

The task of detecting and classifying damages in sewer pipes offers an important application area for computer vision algorithms. This paper describes a system, which is capable of accomplishing this task solely based on low quality and severely compressed fisheye images from a pipe inspection robot. Relying on robust image features, we estimate camera poses, model the image lighting, and exploit this information to generate high quality cylindrical unwraps of the pipes' surfaces.Based on the generated images, we apply semantic labeling based on deep convolutional neural networks to detect and classify defects as well as structural elements.Comment: Published in: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV

A Systematic Review of Convolutional Neural Network-Based Structural Condition Assessment Techniques

With recent advances in non-contact sensing technology such as cameras, unmanned aerial and ground vehicles, the structural health monitoring (SHM) community has witnessed a prominent growth in deep learning-based condition assessment techniques of structural systems. These deep learning methods rely primarily on convolutional neural networks (CNNs). The CNN networks are trained using a large number of datasets for various types of damage and anomaly detection and post-disaster reconnaissance. The trained networks are then utilized to analyze newer data to detect the type and severity of the damage, enhancing the capabilities of non-contact sensors in developing autonomous SHM systems. In recent years, a broad range of CNN architectures has been developed by researchers to accommodate the extent of lighting and weather conditions, the quality of images, the amount of background and foreground noise, and multiclass damage in the structures. This paper presents a detailed literature review of existing CNN-based techniques in the context of infrastructure monitoring and maintenance. The review is categorized into multiple classes depending on the specific application and development of CNNs applied to data obtained from a wide range of structures. The challenges and limitations of the existing literature are discussed in detail at the end, followed by a brief conclusion on potential future research directions of CNN in structural condition assessment

Defect Detection and Classification in Sewer Pipeline Inspection Videos Using Deep Neural Networks

Sewer pipelines as a critical civil infrastructure become a concern for municipalities as they are getting near to the end of their service lives. Meanwhile, new environmental laws and regulations, city expansions, and budget constraints make it harder to maintain these networks. On the other hand, access and inspect sewer pipelines by human-entry based methods are problematic and risky. Current practice for sewer pipeline assessment uses various types of equipment to inspect the condition of pipelines. One of the most used technologies for sewer pipelines inspection is Closed Circuit Television (CCTV). However, application of CCTV method in extensive sewer networks involves certified operators to inspect hours of videos, which is time-consuming, labor-intensive, and error prone. The main objective of this research is to develop a framework for automated defect detection and classification in sewer CCTV inspection videos using computer vision techniques and deep neural networks. This study presents innovative algorithms to deal with the complexity of feature extraction and pattern recognition in sewer inspection videos due to lighting conditions, illumination variations, and unknown patterns of various sewer defects. Therefore, this research includes two main sub-models to first identify and localize anomalies in sewer inspection videos, and in the next phase, detect and classify the defects among the recognized anomalous frames. In the first phase, an innovative approach is proposed for identifying the frames with potential anomalies and localizing them in the pipe segment which is being inspected. The normal and anomalous frames are classified utilizing a one-class support vector machine (OC-SVM). The proposed approach employs 3D Scale Invariant Feature Transform (SIFT) to extract spatio-temporal features and capture scene dynamic statistics in sewer CCTV videos. The OC-SVM is trained by the frame-features which are considered normal, and the outliers to this model are considered abnormal frames. In the next step, the identified anomalous frames are located by recognizing the present text information in them using an end-to-end text recognition approach. The proposed localization approach is performed in two steps, first the text regions are detected using maximally stable extremal regions (MSER) algorithm, then the text characters are recognized using a convolutional neural network (CNN). The performance of the proposed model is tested using videos from real-world sewer inspection reports, where the accuracies of 95% and 86% were achieved for anomaly detection and frame localization, respectively. Identifying the anomalous frames and excluding the normal frames from further analysis could reduce the time and cost of detection. It also ensures the accuracy and quality of assessment by reducing the number of neglected anomalous frames caused by operator error. In the second phase, a defect detection framework is proposed to provide defect detection and classification among the identified anomalous frames. First, a deep Convolutional Neural Network (CNN) which is pre-trained using transfer learning, is used as a feature extractor. In the next step, the remaining convolutional layers of the constructed model are trained by the provided dataset from various types of sewer defects to detect and classify defects in the anomalous frames. The proposed methodology was validated by referencing the ground truth data of a dataset including four defects, and the mAP of 81.3% was achieved. It is expected that the developed model can help sewer inspectors in much faster and more accurate pipeline inspection. The whole framework would decrease the condition assessment time and increase the accuracy of sewer assessment reports

Artificial Intelligence in Civil Infrastructure Health Monitoring—historical Perspectives, Current Trends, and Future Visions

Over the past 2 decades, the use of artificial intelligence (AI) has exponentially increased toward complete automation of structural inspection and assessment tasks. This trend will continue to rise in image processing as unmanned aerial systems (UAS) and the internet of things (IoT) markets are expected to expand at a compound annual growth rate of 57.5% and 26%, respectively, from 2021 to 2028. This paper aims to catalog the milestone development work, summarize the current research trends, and envision a few future research directions in the innovative application of AI in civil infrastructure health monitoring. A blow-by-blow account of the major technology progression in this research field is provided in a chronological order. Detailed applications, key contributions, and performance measures of each milestone publication are presented. Representative technologies are detailed to demonstrate current research trends. A road map for future research is outlined to address contemporary issues such as explainable and physics-informed AI. This paper will provide readers with a lucid memoir of the historical progress, a good sense of the current trends, and a clear vision for future research

Automated analysis of sewer CCTV surveys

Sewers across the globe must be regularly inspected to ensure their smooth running and effective maintenance. Furthermore, surveys are often performed reactively, often diagnosing suspected faults within a network. Almost all surveys in the UK and abroad are performed using closed circuit television (CCTV) cameras where pipes are too small for manual inspection. As such vast quantities of footage are recorded by surveying teams on a daily basis. This footage is currently analysed manually, requiring a trained engineer to watch through its entirety, annotating potential faults. This thesis examines methods of improving this labelling process, implementing various machine learning and image processing techniques to automate this procedure. The thesis presents two distinct methodologies: the first for the detection of faults, using only raw CCTV footage, whilst the second identifies the type of a detected fault according to the Manual of Sewer Condition Classification. The fault detection methodology identifies the presence of a fault within a CCTV image. The methodology calculates a GIST feature descriptor for each video frame, before utilising a Random Forest classifier, to predict the presence of a fault. The basic methodology was further refined with the inclusion of smoothing, to eliminate isolated inconstancies, and stacking to intuitively combine the results of multiple machine learning classifiers. The final methodology achieved a detection accuracy of 86% on unseen real-life data from the UK. The fault classification methodology identifies the fault type in images, where faults have been previously detected using the above technique. The tool again calculates a frame’s GIST descriptor before applying multiple Random Forest classifiers in a ‘1 vs all’ architecture to predict the type of a given fault. This architecture allowed for comparative classifications and later enabled the identification of multiple faults within a single frame. The methodology achieved a peak accuracy of 74% when classifying faults well represented by the dataset (at least 100 examples). Furthermore, when including multi-label functionality, the tool achieved an accuracy of 67% across all fault types. Both methods have been developed to be holistic and practical, utilising only industry standard CCTV footage and generalising well across all types of sewer system (size, shape and material). Furthermore, as both methodologies rely on the same feature descriptor, they integrate well to form a methodology that could be applied in real time. As such the thesis also explores the practical implications of creating a detection support tool capable of integration with current working practices. In combination both methodologies and their additions present a unique contribution to the field of automated sewer surveying. Achieving competitive accuracies with a streamlined methodology, the technology shows promise for future application in industry, greatly increasing the speed, accuracy and consistency of CCTV sewer surveys

Data Analytics for Automated Near Real Time Detection of Blockages in Smart Wastewater Systems

Blockage events account for a substantial portion of the reported failures in the wastewater network, causing flooding, loss of service, environmental pollution and significant clean-up costs. Increasing telemetry in Combined Sewer Overflows (CSOs) provides the opportunity for near real-time data-driven modelling of the sewer network. The research work presented in this thesis describes the development and testing of a novel system, designed for the automatic detection of blockages and other unusual events in near real-time. The methodology utilises an Evolutionary Artificial Neural Network (EANN) model for short term CSO level predictions and Statistical Process Control (SPC) techniques to analyse unusual CSO level behaviour. The system is designed to mimic the work of a trained, experience human technician in determining if a blockage event has occurred. The detection system has been applied to real blockage events from a UK wastewater network. The results obtained illustrate that the methodology can identify different types of blockage events in a reliable and timely manner, and with a low number of false alarms. In addition, a model has been developed for the prediction of water levels in a CSO chamber and the generation of alerts for upcoming spill events. The model consists of a bi-model committee evolutionary artificial neural network (CEANN), composed of two EANN models optimised for wet and dry weather, respectively. The models are combined using a non-linear weighted averaging approach to overcome bias arising from imbalanced data. Both methodologies are designed to be generic and self-learning, thus they can be applied to any CSO location, without requiring input from a human operator. It is envisioned that the technology will allow utilities to respond proactively to developing blockages events, thus reducing potential harm to the sewer network and the surrounding environment

Condition Classification in Underground Pipes Based on Acoustical Characteristics. Acoustical characteristics are used to classify the structural and operational conditions in underground pipes with advanced signal classification methods

This thesis is concerned with the development and study of a pattern recognition system for siphon and sewer condition/defect analysis based on acoustic characteristics. Pattern recognition has been studied and used widely in many fields including: identification and authentication; medical diagnosis and musical modelling. Audio based classification and research has been mainly focusing on speech recognition and music retrieval, but few applications have attempted to use acoustic characteristics for underground pipe condition classification. Traditional CCTV inspection methods are relatively expensive and subjective so remote techniques have been developed to overcome this concern and increase the inspection efficiency. The acoustic environment provides a rich source of information about the internal conditions of a pipe. This thesis reports on a classification system based on measuring the direct and reflected acoustic signals and describing the energy spectrum for each condition/pipe defect. A K-nearest neighbour classifier (KNN) and Support vector machines (SVMs) classifier have been adopted to train the classification system to identify sediment and pipe surface defects by comparing the measured acoustic signals with a database containing a range of typical conditions. Laboratory generated data and field collected data were used to train the proposed system and evaluate its ability. The overall accuracy of the system recognizing blockage and structural aspects in each of the series of experiments varies between 70% and 95%