Few-shot learning for image-based bridge damage detection

Autonomous bridge visual inspection is a real-world challenge due to various materials, surface coatings, and changing light and weather conditions. Traditional supervised learning relies on massive annotated data to establish a robust model, which requires a time-consuming data acquisition process. This work proposes a few-shot learning (FSL) approach based on improved ProtoNet for damage detection with just a few labeled examples. Feature embedding is achieved through cross-domain transfer learning from ImageNet instead of episodic training. The ProtoNet is improved with embedding normalization to enhance transduction performance based on Euclidean distance and a linear classifier for classification. The approach is explored on a public dataset through different ablation experiments and achieves over 94% mean accuracy for 2-way 5-shot classification via the pre-trained GoogleNet after fine-tuning. Moreover, the proposed fine-tuning methods based on a fully connected layer (FCN) and Hadamard product are demonstrated with better performance than the previous method. Finally, the approach is validated using real bridge inspection images, demonstrating its capability of fast implementation for practical damage inspection with weakly supervised information

Learning-based Image Scale Estimation for Quantitative Visual Inspection of Civil Structures

The number of assets of civil infrastructure (e.g., bridges or roads) have been increasing to meet the demands of growing populations around the world. However, they degrade over time due to environmental factors and must be maintained and monitored to ensure the safety of its users. The increasing number of infrastructure assets which deteriorate over time is fast outpacing the rate at which they are inspected and rehabilitated. Currently, the main mode of structure condition assessment is visual inspection, where human inspectors manually identify, classify, track, and measure, as needed, deterioration over time to make assessments of a structure’s overall condition. However, the current process is highly time consuming, expensive, and subject to the inspector’s judgement and expertise, which could lead to inconsistent assessments of a given structure when surveyed by several different inspectors over a period of time. As a result, there is a clear need for the current inspection process to be improved in terms of efficiency and consistency. Developments in computer vision algorithms, vision sensors, sensing platforms, and high-performance computing have shown promise in improving the current inspection processes to enable consistent and rapid structural assessments. Recent work often involves rapid collection and/or analysis of imagery captured from personnel or mobile data collection platforms (e.g., smart phones, unmanned aerial or ground vehicles) to detect and classify visual features (e.g., structural components or deterioration). These works often involve the use of advanced image processing or computer vision algorithms such as convolutional neural networks to detect and/or classify regions of interest. However, a major shortfall of vision-based inspection is the inability to deduce physical measurements (e.g., mm or cm) from the collected images. The lack of an image scale (e.g., pixel/mm) on 2D images does not permit quantitative inspection. To address this challenge, a learning-based scale estimation technique is proposed. The underlying assumption is that the surface texture of structures, captured in images, contains enough information to estimate scale for each corresponding image (e.g., pixel/mm). This permits the training of a regression model to establish the relationship between surface textures in images and their scales. A convolutional neural network was trained to extract scale-related features from textures captured in images. The trained model is used to estimate scales for all images captured from surfaces of a structure with similar textures in subsequent inspections. The capability of the proposed technique was demonstrated using data collected from surface textures of three different structures. An average scale estimation error, from images of each structure, is less than 15%, which is acceptable in typical visual inspection settings. The source code and data are available from a data repository (GitHub)

Monitoring Fatigue Cracks in Steel Bridges using Advanced Structural Health Monitoring Technologies

Fatigue cracks that develop in steel highway bridges under repetitive traffic loads are one of the major mechanisms that degrades structural integrity. If bridges are not appropriately inspected and maintained, fatigue cracks can eventually lead to catastrophic failures, in particular for fracture-critical bridges. Despite various levels of success of crack monitoring methods over the past decades in the fields of structural health monitoring (SHM) and non-destructive evaluation (NDE), monitoring fatigue cracks in steel bridges is still challenging due to the complex structural joint layout and unpredictable crack propagation paths. In this dissertation, advanced SHM technologies are proposed for detecting and monitoring fatigue cracks in steel bridges. These technologies are categorized as: 1) a large-area strain sensing technology based on the soft elastomeric capacitor (SEC) sensor; and 2) non-contact vision-based fatigue crack detection approaches. In SEC-based fatigue crack sensing, the research focuses are placed on numerical prediction of the SEC’s response under fatigue cracking and experimental validations of sensing algorithms for monitoring fatigue cracks over long-term. In vision-based fatigue crack detection approaches, two novel sensing methodologies are established through feature tracking and image overlapping, respectively. Laboratory test results verified that the proposed approaches can robustly identify the true fatigue crack from many non-crack edges. Overall, the proposed advanced SHM technologies show great promise for fatigue crack damage detection of steel bridges in laboratory configurations, hence form the basis for long-term fatigue sensing solutions in field applications

Computer Vision Sensing Systems for Structural Health Monitoring in Challenging Field Conditions

Computer vision sensing techniques enable easy-to-install and remote non-contact monitoring of structures and have great potentials in field applications. This study will develop/implement novel computer vision techniques for two sensing systems for monitoring different aspects of infrastructures in challenging field conditions. The dissertation is therefore composed of two parts: robust measurement of global multi-point structural displacements, and accurate and robust monitoring of local surface displacements/strains. Computer vision based displacement measurement has become popular in the recent decade. The first part presents InnoVision, a vision sensing system developed to address a number of challenging problems associated with applying vision sensors to the measurement of multi-point structural displacement in field conditions that are rarely comprehensively studied in the literature. The challenging problems include tracking low-contrast natural targets on the structural surface, insufficient resolution for long distance measurement, inevitable camera vibration, and image distortion due to heat haze in hot weather. Several techniques are developed in InnoVision to tackle these challenges. Laboratory and field tests are conducted to evaluate the performance of these techniques. In the second part, another vision sensing system SurfaceVision is developed for accurate and robust monitoring two-dimensional (2D) structural surface displacements/strains. Important structures, such as nuclear power plants, need the continuous inspection of surface conditions. As an alternative to the human inspection, conventional digital-image-correlation (DIC) based methods have been applied to surfaces painted with speckle patterns in a controlled environment. However, it is highly challenging for DIC methods to accurately measure displacement on natural concrete surfaces in outdoor conditions with changing illumination and weather conditions. Additionally, common surface displacement measurement is based on segmenting the surface image into small subsets and tracking each subset individually through template matching, the surface displacement thus obtained has obvious discontinuity and low spatial resolution. Therefore, for applicability in the outdoor environment, SurfaceVision is proposed for accurate and robust monitoring of surface displacements/strains. Advanced computer vision techniques are developed/implemented to enable surface displacement measurement with high continuity, spatial resolution, accuracy, and robustness. An intuitive strain calculation method is also developed for converting surface displacements into surface strains. A numerical simulation is formulated based on four-point bending tests to validate the accuracy and robustness of SurfaceVision in surface displacements. Four-point bending experiments using reinforced concrete specimens are conducted to demonstrate the performance of SurfaceVision under different cases of optical noises and its effectiveness in predicting crack formations

Investigation of Computer Vision Concepts and Methods for Structural Health Monitoring and Identification Applications

This study presents a comprehensive investigation of methods and technologies for developing a computer vision-based framework for Structural Health Monitoring (SHM) and Structural Identification (St-Id) for civil infrastructure systems, with particular emphasis on various types of bridges. SHM is implemented on various structures over the last two decades, yet, there are some issues such as considerable cost, field implementation time and excessive labor needs for the instrumentation of sensors, cable wiring work and possible interruptions during implementation. These issues make it only viable when major investments for SHM are warranted for decision making. For other cases, there needs to be a practical and effective solution, which computer-vision based framework can be a viable alternative. Computer vision based SHM has been explored over the last decade. Unlike most of the vision-based structural identification studies and practices, which focus either on structural input (vehicle location) estimation or on structural output (structural displacement and strain responses) estimation, the proposed framework combines the vision-based structural input and the structural output from non-contact sensors to overcome the limitations given above. First, this study develops a series of computer vision-based displacement measurement methods for structural response (structural output) monitoring which can be applied to different infrastructures such as grandstands, stadiums, towers, footbridges, small/medium span concrete bridges, railway bridges, and long span bridges, and under different loading cases such as human crowd, pedestrians, wind, vehicle, etc. Structural behavior, modal properties, load carrying capacities, structural serviceability and performance are investigated using vision-based methods and validated by comparing with conventional SHM approaches. In this study, some of the most famous landmark structures such as long span bridges are utilized as case studies. This study also investigated the serviceability status of structures by using computer vision-based methods. Subsequently, issues and considerations for computer vision-based measurement in field application are discussed and recommendations are provided for better results. This study also proposes a robust vision-based method for displacement measurement using spatio-temporal context learning and Taylor approximation to overcome the difficulties of vision-based monitoring under adverse environmental factors such as fog and illumination change. In addition, it is shown that the external load distribution on structures (structural input) can be estimated by using visual tracking, and afterward load rating of a bridge can be determined by using the load distribution factors extracted from computer vision-based methods. By combining the structural input and output results, the unit influence line (UIL) of structures are extracted during daily traffic just using cameras from which the external loads can be estimated by using just cameras and extracted UIL. Finally, the condition assessment at global structural level can be achieved using the structural input and output, both obtained from computer vision approaches, would give a normalized response irrespective of the type and/or load configurations of the vehicles or human loads