Human-Robot Collaboration for Effective Bridge Inspection in the Artificial Intelligence Era

Advancements in sensor, Artificial Intelligence (AI), and robotic technologies have formed a foundation to enable a transformation from traditional engineering systems to complex adaptive systems. This paradigm shift will bring exciting changes to civil infrastructure systems and their builders, operators and managers. Funded by the INSPIRE University Transportation Center (UTC), Dr. Qin’s group investigated the holism of an AI-robot-inspector system for bridge inspection. Dr. Qin will discuss the need for close collaboration among the constituent components of the AI-robot-inspector system. In the workplace of bridge inspection using drones, the mobile robotic inspection platform rapidly collected big inspection video data that need to be processed prior to element-level inspections. She will illustrate how human intelligence and artificial intelligence can collaborate in creating an AI model both efficiently and effectively. Obtaining a large amount of expert-annotated data for model training is less desirable, if not unrealistic, in bridge inspection. This INSPIRE project addressed this annotation challenge by developing a semi-supervised self-learning (S3T) algorithm that utilizes a small amount of time and guidance from inspectors to help the model achieve an excellent performance. The project evaluated the improvement in job efficacy produced by the developed AI model. This presentation will conclude by introducing some of the on-going work to achieve the desired adaptability of AI models to new or revised tasks in bridge inspection as the National Bridge Inventory includes over 600,000 bridges of various types in material, shape, and age

Rapid post-disaster infrastructure damage characterisation enabled by remote sensing and deep learning technologies -- a tiered approach

Critical infrastructure, such as transport networks and bridges, are systematically targeted during wars and suffer damage during extensive natural disasters because it is vital for enabling connectivity and transportation of people and goods, and hence, underpins national and international economic growth. Mass destruction of transport assets, in conjunction with minimal or no accessibility in the wake of natural and anthropogenic disasters, prevents us from delivering rapid recovery and adaptation. As a result, systemic operability is drastically reduced, leading to low levels of resilience. Thus, there is a need for rapid assessment of its condition to allow for informed decision-making for restoration prioritisation. A solution to this challenge is to use technology that enables stand-off observations. Nevertheless, no methods exist for automated characterisation of damage at multiple scales, i.e. regional (e.g., network), asset (e.g., bridges), and structural (e.g., road pavement) scales. We propose a methodology based on an integrated, multi-scale tiered approach to fill this capability gap. In doing so, we demonstrate how automated damage characterisation can be enabled by fit-for-purpose digital technologies. Next, the methodology is applied and validated to a case study in Ukraine that includes 17 bridges, damaged by human targeted interventions. From regional to component scale, we deploy technology to integrate assessments using Sentinel-1 SAR images, crowdsourced information, and high-resolution images for deep learning to facilitate automatic damage detection and characterisation. For the first time, the interferometric coherence difference and semantic segmentation of images were deployed in a tiered multi-scale approach to improve the reliability of damage characterisations at different scales

Automatic Scaffolding Productivity Measurement through Deep Learning

This study developed a method to automatically measure scaffolding productivity by extracting and analysing semantic information from onsite vision data

CONCRETE CRACK EVALUATION FOR CIVIL INFRASTRUCTURE USING COMPUTER VISION AND DEEP LEARNING

Department of Urban and Environmental Engineering (Urban Infrastructure Engineering)Surface cracks of civil infrastructure are one of the important indicators for structural durability and integrity. Concrete cracks are typically investigated by manual visual observation on the surface, which is intrinsically subjective because it highly depends on the experience of inspectors. Furthermore, manual visual inspection is time-consuming, expensive, and often unsafe when inaccessible structural components need to be assessed. Computer vision-based approach is recognized as a promising alternative that can automatically extract crack information from images captured by the digital camera. As texts and cracks are similar in terms of consisting distinguishable lines and curves, image binarization developed for text detection can be appropriate for crack identification purposes. However, although image binarization is useful to separate cracks and backgrounds, the crack assessment is difficult to standardize owing to the high dependence of binarization parameters determined by users. Another critical challenge in digital image processing for crack detection is to automatically distinguish cracks from an image containing actual cracks and crack-like noise patterns (e.g., stains, holes, dark shadows, and lumps), which are often seen on the surface of concrete structures. In addition, a tailored camera system and the corresponding strategy are necessary to effectively address the practical issues in terms of the skewed angle and the process of the sequential crack images for efficient measurement. This research develops a computer vision-based approach in conjunction with deep learning for accurate crack evaluation of for civil infrastructure. The main contribution of the proposed approach can be summarized as follows: (1) a deep learning-based approach for crack detection, (2) a hybrid image processing for crack quantification, and (3) camera systems for the practical issues on civil infrastructure in terms of a skewed angle problem and an efficient measurement with the sequential crack images. The proposed research allows accurate crack evaluation to provide a proper maintenance strategy for civil infrastructure in practice.clos

Learning-based Image Scale Estimation for Quantitative Visual Inspection of Civil Structures

The number of assets of civil infrastructure (e.g., bridges or roads) have been increasing to meet the demands of growing populations around the world. However, they degrade over time due to environmental factors and must be maintained and monitored to ensure the safety of its users. The increasing number of infrastructure assets which deteriorate over time is fast outpacing the rate at which they are inspected and rehabilitated. Currently, the main mode of structure condition assessment is visual inspection, where human inspectors manually identify, classify, track, and measure, as needed, deterioration over time to make assessments of a structure’s overall condition. However, the current process is highly time consuming, expensive, and subject to the inspector’s judgement and expertise, which could lead to inconsistent assessments of a given structure when surveyed by several different inspectors over a period of time. As a result, there is a clear need for the current inspection process to be improved in terms of efficiency and consistency. Developments in computer vision algorithms, vision sensors, sensing platforms, and high-performance computing have shown promise in improving the current inspection processes to enable consistent and rapid structural assessments. Recent work often involves rapid collection and/or analysis of imagery captured from personnel or mobile data collection platforms (e.g., smart phones, unmanned aerial or ground vehicles) to detect and classify visual features (e.g., structural components or deterioration). These works often involve the use of advanced image processing or computer vision algorithms such as convolutional neural networks to detect and/or classify regions of interest. However, a major shortfall of vision-based inspection is the inability to deduce physical measurements (e.g., mm or cm) from the collected images. The lack of an image scale (e.g., pixel/mm) on 2D images does not permit quantitative inspection. To address this challenge, a learning-based scale estimation technique is proposed. The underlying assumption is that the surface texture of structures, captured in images, contains enough information to estimate scale for each corresponding image (e.g., pixel/mm). This permits the training of a regression model to establish the relationship between surface textures in images and their scales. A convolutional neural network was trained to extract scale-related features from textures captured in images. The trained model is used to estimate scales for all images captured from surfaces of a structure with similar textures in subsequent inspections. The capability of the proposed technique was demonstrated using data collected from surface textures of three different structures. An average scale estimation error, from images of each structure, is less than 15%, which is acceptable in typical visual inspection settings. The source code and data are available from a data repository (GitHub)

Post-earthquake Serviceability Assessment of RC Bridge Columns Using Computer Vision

Modern seismic design codes ensure a large displacement capacity and prevent total collapse for bridges. However, this performance objective is usually attained at the cost of damage to target ductile members. For reinforced concrete (RC) bridges, the columns are usually the main source of ductility during an earthquake in which concrete cover, core, and reinforcement may damage, and the column may experience a large permanent lateral deformation. A significant number of the US bridges will experience large earthquakes in the next 50 years that may result in the bridge closure due to excessive damage. A quick assessment of bridges immediately after severe events is needed to maximize serviceability and access to the affected sites, and to minimize casualties and costs. The main goal of this project was to accelerate post-earthquake RC bridge column assessment using \u201ccomputer vision\u201d. When sending trained personnel to the affect sites is limited or will take time, local personnel equipped with an assessment software (on various platforms such as mobile applications, cloud-based tools, or built-in with drones) can be deployed to evaluate the bridge condition. The project in this phase was focused on the damage assessment of modern RC bridge columns after earthquakes. Substandard columns, other bridge components, and other hazards were not included