Integration of Small Unmanned Aircraft Systems and Deep Learning for Efficient Airfield Pavement Crack Detection and Assessment

Airfield pavement inspection and maintenance are critical aspects of aviation infrastructure, representing a substantial portion of life-cycle costs. Longitudinal, transverse, and diagonal (LTD) cracks; corner breaks; shattered slabs in Portland cement concrete (PCC) pavement; and longitudinal and transverse (L&T) cracks of asphalt concrete (AC) pavement consist of most of the airfield pavement distresses. Traditional airfield pavement inspection methods are manual, time-consuming, laborious, and reliant on the inspector’s experience, leading to increased expenses and safety risks. This research explores the potential to automatically identify those distresses in red-green-blue images using four variants of deep learning (DL) model YOLOv8, ranging from nano to large. YOLOv8 is a widely used off-the-shelf DL object detection model that allows rapid training and easy execution. A DL training dataset of 5,273 small uncrewed aircraft systems (sUAS) collected images was developed. The transfer learning technique was used, and the dataset passed through each model 100 times for adequate training. The model exhibits mean average precision values exceeding 0.65, with varying processing times. Such accuracy showed that crack-related distress detection using DL models could enhance airfield pavement inspection efficiency.This is a manuscript of a proceeding published as Sourav, Md Abdullah All, Halil Ceylan, Sunghwan Kim, and Matthew Brynick. "Integration of Small Unmanned Aircraft Systems and Deep Learning for Efficient Airfield Pavement Crack Detection and Assessment." In International Conference on Transportation and Development 2024, pp. 884-893. 2024. doi:https://doi.org/10.1061/9780784485514.078. Copyright 2024 The Authors. "This material may be downloaded for personal use only. Any other use requires prior permission of the American Society of Civil Engineers

Small Unmanned Aircraft System for Pavement Inspection: Task 4\u2014Execute the Field Demonstration Plan and Analyze the Collected Data

The primary objectives of this research project are to develop recommended processes and procedures for using small unmanned/uncrewed aircraft system (sUAS) to complement current methods of airport Pavement Management Program (PMP) inspections and to evaluate various types of sUAS platforms and sensors that will lead to recommended minimum specifications required for consistently safe, reliable, and effective sUAS-assisted airport PMP inspections. Under Task 4, the research team developed and executed field demonstrated plans to safely deploy several sUAS at six airports in Michigan, Illinois, Iowa, and New Jersey from December 2020 to August 2021. Red, green, and blue (RGB) optical orthophotos, digital elevation models (DEMs), hillshades derived from DEMs, and thermal orthophotos collected using several sUAS at different altitudes were analyzed for their usefulness in airfield distress detection. Based on the data analyses and results, RGB orthophotos of 1.5 mm/pixel and DEMs of 6 mm/pixel resolution, or higher, are highly recommended for airfield pavement distress detection and rating

Automatic visual sensemaking approach for the behavioral understanding of beef cattle

Computer vision has been extensively used for livestock monitoring in recent years. Most research in this domain focuses on the performance evaluation of algorithms used to analyze visual sensing data collected from specially designed environments. There is a scope for finding the best combination of hardware placement to collect data, evaluate algorithm performance, and develop a set of recommendations for livestock behavioral data analysis. Thus, this dissertation seeks to find the appropriate automatic visual sensemaking approach for achieving a behavioral understanding of steer in a confined feeding operation. The camera configuration needed for observing steer, the algorithm required for behavioral data analysis, and the impact of the pen environment on behavioral analysis were investigated to answer the question. Data collection with a sensor or camera is the first part of the complete workflow. This study sought to determine the optimal camera placement locations in a confined steer feeding operation. Measurements of cattle pens were used to create a 3D farm model using Blender 3D computer graphic software. A method was developed to calculate the camera coverage in a 3D farm environment, and a genetic algorithm-based model was designed to find optimal placements of a multi-camera and multi-pen setup. The algorithm's objective was to maximize the multi-camera coverage while minimizing the budget. Two different optimization methods involving multiple cameras and pen combinations were used. The results demonstrated the applicability of the genetic algorithm in achieving maximum coverage and thereby enhancing the quality of the livestock visual-sensing data. The algorithm also provided the top 25 solutions for each camera and pen combination with a maximum coverage difference of less than 3.5% between them. It offers other alternative options when one solution is not practical for any reason, for example, the unavailability of a power outlet. The detection and tracking of individual steers are essential steps for a computer vision-based steer monitoring system. An attempt was made to detect, track, and classify steer’s behaviors using state-of-the-art deep learning object detection methods. We trained Faster RCNN ResNet-50, Faster RCNN ResNet-101, Faster RCNN Inception V2, SSD MobileNet V1, YOLO V3, and YOLO V5 with our custom dataset containing 3305 images of the pen to identify and localize steers. A simple online tracker was used to track the steer and a location-based function was used to initiate re-track when tracking is lost. The current behaviors of the cattle were classified based on spatial location, travel path, and body shape detected by the object detection algorithm. All Faster RCNN variants, YOLO V3, and YOLO V5 models achieved more than 99% F1 score in detection. The result also showed that eating, standing, and laying behaviors are correctly classified in all models with around 90% accuracy. However, misclassification between standing and laying behaviors was observed in several cases due to similarities in steer body image between standing and laying and lack of visual details. The promising performance of combining the Faster RCNN and YOLO-based model with the tracking algorithm demonstrates the feasibility of deep learning-based object detection and tracking. The behavior detection model also provided time spent on each behavior by each steer in given pen videos. In addition, the time-series behavioral data analysis method to understand the current behaviors were also summarized. Contributions of this dissertation include (a) it is the first focused study on the role of hardware placement and behavior detection algorithm integration in livestock monitoring, (b) a novel automated visual system for behavior monitoring in a confined beef cattle operation, and (c) a set of recommendations provided for livestock behavior monitoring with visual sensing and sensemaking

Heavy rainfall and moisture susceptibility of pavement foundation: A case study coupling finite element method and MnROAD moisture monitoring data

The Midwest region, including Minnesota, has been experiencing increased heavy precipitation events due to climate change, and the Minnesota Department of Transportation (MnDOT) is currently investigating the effect of climate change on pavement foundation and other transportation assets. As part of this effort, a study was conducted to investigate the impact of heavy rainfall on pavement foundation performance by focusing on moisture dynamics and resilient modulus changes in the pavement base layer. This study is aimed at understanding the adverse effects of heavy rainfall on moisture fluxes within pavement foundation and corresponding stiffness of the base aggregate layer. A two-step approach was adopted for predicting changes in saturation when estimating corresponding resilient modulus values using the resilient modulus prediction equation employed in AASHTOWare Pavement Mechanistic-Empirical (ME) Design. PLAXIS 3D, a finite-element analysis tool, was utilized to simulate the movement of moisture within the pavement layers under varying heavy rainfall scenarios. By incorporating predicted saturation from PLAXIS 3D simulations into the Pavement ME equation, corresponding resilient modulus values were estimated for the base layer. To ensure its accuracy and reliability, the model was validated using field sensor data from the MnROAD facility. Multiple linear regression models were developed to provide a means for estimating resilient modulus changes due to heavy rainfall. This study highlights the importance of considering moisture effects in pavement design and maintenance in regions prone to heavy rainfall events, and findings can be used by transportation agencies as part of their transportation/geotechnical asset management programs.This is a manuscript of the article Published as Jibon, Md, Md Abdullah All Sourav, Masrur Mahedi, Sunghwan Kim, Halil Ceylan, and Raul Velasquez. "Heavy rainfall and moisture susceptibility of pavement foundation: A case study coupling finite element method and MnROAD moisture monitoring data." Transportation Geotechnics (2024): 101312. doi: https://doi.org/10.1016/j.trgeo.2024.101312. Posted with Permission. CC BY-NC-ND

Use of Digital Elevation Model for Detecting Airfield Pavement Distress

A digital elevation model (DEM) is a topographic representation of a bare surface, usually referring to the Earth’s surface. DEMs have been used for flood assessment, landside hazard detection, soil properties’ quantification, road and highway planning, etc. Recently, DEMs generated using small unmanned aircraft systems (sUAS) have shown potential in transportation infrastructure inspection. In this study, we evaluated the performance of sUAS-collected DEM data to detect airfield pavement distresses. Natural-color red-green-blue (RGB) data were collected from both Portland cement concrete (PCC) and asphalt concrete (AC) pavements at three airports with UAS. The data were processed to generate an RGB optical orthophoto using close-range photogrammetry, and one product of that process was a DEM representing the airfield pavement in three-dimensional space. The resolution of the generated DEMs varied from 2.8 mm/pix to 84 mm/pix based on sUAS sensors, flying altitude, and parameters selected for RGB data processing. The DEMs were studied for their usefulness in visually identifying 14 PCC pavement distresses and nine AC pavement distresses. Analysis showed that DEM with 6 mm/pix resolution or better can be useful for detecting one or more severity levels of 13 PCC pavement distresses and 6 AC pavement distresses. High-resolution DEMs were also useful in identifying distresses related to elevation differences, such as shoving, depression, and faulting. This study illustrates the potential of DEM data in airfield pavement distress detection.This is a manuscript of a proceeding published as All Sourav, Md Abdullah, Halil Ceylan, Sunghwan Kim, Colin Brooks, David Peshkin, Richard Dobson, and Matthew Brynick. "Use of Digital Elevation Model for Detecting Airfield Pavement Distress." In Airfield and Highway Pavements 2023, pp. 254-265. https://ascelibrary.org/doi/abs/10.1061/9780784484906.024. Posted with Permission

Small Uncrewed Aircraft Systems-Based Orthophoto and Digital Elevation Model Creation and Accuracy Evaluation for Airfield Portland Cement Concrete Pavement Distress Detection and Rating

Early detection and repair of airfield pavement distresses are essential aspects of preserving airport pavements for the full spans of their design periods and maintaining their structural integrity, ride quality, and safety. In recent years, various studies have been conducted for pavement distress detection and mapping using alternate data collection methodologies, including small uncrewed aerial systems (sUAS). This study evaluates the usefulness of red-green-blue (RGB) orthophoto and digital elevation models (DEMs) generated from sUAS data in detecting various airfield Portland cement concrete (PCC) pavement distress. Fourteen airfield PCC pavement distresses with different severities were studied using RGB orthophotos with resolutions ranging from 0.8 mm/pixel to 21 mm/pixel and DEM resolutions ranging from 3.0 mm/pixel to 84 mm/pixel. The analysis showed that the RGB optical orthophotos and DEM were useful for detecting 13 out of 14 PCC pavement distresses and confirmation of the location of 11 out of 14 distresses, respectively.This article is published as All Sourav, Md Abdullah, Halil Ceylan, Sunghwan Kim, Colin Brooks, David Peshkin, Richard Dobson, Matthew Brynick, and Mike DiPilato. "Small Uncrewed Aircraft Systems-Based Orthophoto and Digital Elevation Model Creation and Accuracy Evaluation for Airfield Portland Cement Concrete Pavement Distress Detection and Rating." In International Conference on Transportation and Development 2022, pp. 168-180. 2022. DOI: 10.1061/9780784484371.016. Works produced by employees of the U.S. Government as part of their official duties are not copyrighted within the U.S. The content of this document is not copyrighted

Evaluation of Small Uncrewed Aircraft Systems Data in Airfield Pavement Crack Detection and Rating

Current practice for airport Pavement Management Program (PMP) inspection relies on visual surveys and manual interpretation of reports and sketches prepared by inspectors in the field to quantify pavement conditions using the Pavement Condition Index method set forth in ASTM D5340. In recent years, several attempts have been made, both by the industry and by airport operators, to use small Uncrewed (Unpersonned/Unmanned) Aircraft Systems (sUAS), or “drones,” to conduct various types of imaging and inspection of airport pavements. As part of a comprehensive study on the use of such sUAS to evaluate airfield pavement conditions, the objectives of this study were to assess the performance of various sUAS platforms and sensors in detecting and rating a subset of crack-based pavement distresses and to evaluate the use of a combination of different sUAS datasets to complement current methods used to support airport PMP. Two airports in Michigan were selected for sUAS data collection, and five sUAS platforms equipped with eight different sensors were flown at these airports at different altitudes to collect red, green, and blue (RGB) optical and thermal data at different resolutions. RGB orthophotos, digital elevation models, and thermal images were visually analyzed to study their usefulness in detecting and rating longitudinal and transverse cracks in flexible/asphalt pavements and longitudinal, transverse, and diagonal cracks, corner breaks, and durability cracks in rigid/concrete pavements. This study demonstrated the capability of using sUAS data in detecting and rating multiple crack-related distresses in both flexible and rigid airfield pavement systems.This article is published as Sourav, Md Abdullah All, Masrur Mahedi, Halil Ceylan, Sunghwan Kim, Colin Brooks, David Peshkin, Richard Dobson, and Matthew Brynick. "Evaluation of Small Uncrewed Aircraft Systems Data in Airfield Pavement Crack Detection and Rating." Transportation Research Record 2677, no. 1 (2023): 653-668. DOI: 10.1177/03611981221101030. Works produced by employees of the U.S. Government as part of their official duties are not copyrighted within the U.S. The content of this document is not copyrighted

Practical Lessons Learned from Planning, Collecting, Processing, and Analyzing Small Unmanned Aircraft System Data for Airfield Pavement Inspection

692M15-20-T-00039A small unmanned aircraft system (sUAS) or drone has proven to be a valuable tool for civil infrastructure inspection, highway inspection, unpaved road inspection, bridge inspection, construction work progress monitoring, and other applications. Additionally, several proof-of-concept studies showed that sUAS could be helpful for airfield pavement distress detection. This report documents a comprehensive study that evaluated the usefulness of sUAS-collected data in detection and rating both asphalt concrete and Portland cement concrete pavement distresses. Multiple sUASs were deployed at different altitudes to collect data from six airports in Michigan, Illinois, Iowa, and New Jersey. The collected data were then processed to create red, green, and blue (RGB) orthophotos, Digital Elevation Models (DEMs), hillshades from DEMs, and stereo-thermal orthophotos to assess what resolution and methods were best to detect and rate each type of airfield pavement distress. Based on the subsequent analysis, 1.5-mm/pix resolution for RGB orthophotos and a 6.0-mm/pix resolution for DEMs are recommended to detect and rate airfield pavement distresses. During data collection and processing, recommended standard processes were established, such as the use of high-quality ground control points (GCPs) for data acquisition, a maximum distance of 100 m between two GCPs, and a minimum three-person data collection team. Additionally, it is recommended to ensure the usability of flight-control software prior to data collection and the scheduling of sUAS deployment during low wind speed and precipitation free weather conditions. With these recommended practices in place, sUAS data can be collected effectively, safely, and efficiently. In addition, using the recommended data resolutions, most severity levels of airfield asphalt and Portland concrete cement distresses could be identified and rated

Small Unmanned Aircraft System for Pavement Inspection

Airport pavements require routine maintenance, upgrading, and rehabilitation to reach or exceed their design period. While pavement distresses caused by environmental conditions cannot be prevented, early and routine maintenance work can minimize the deterioration. The current practice for airport pavement inspections relies on time-consuming visual surveys and manual interpretation of reports and sketches prepared by inspectors in the field. Recently, the use of small Unmanned Aircraft Systems (sUAS) has attracted attention as an option for performing cost-effective and efficient pavement inspections. In this study, the research team deployed several sUAS at different altitudes at six airports in Michigan, Illinois, Iowa, and New Jersey from December 2020 to November 2021. Red, green, and blue (RGB) optical orthophotos, Digital Elevation Models (DEMs), hillshades from DEMs, and thermal orthophotos collected using several sUAS at different altitudes were analyzed for their usefulness in airfield distress detection. The results showed that RGB optical data could detect as many as 13 Portland cement concrete (PCC) pavement distresses out of 14 available in this study and 6 out of 9 asphalt concrete (AC) pavement distresses available on the airports. Similarly, DEMs were found to be useful for confirming the location of distresses with elevation change, such as faulting in PCC pavement and shoving in AC pavement. In addition, thermal orthophotos showed potential to detect crack-based distresses. Based on the data analysis, RGB orthophoto resolution of 1.5 millimeters per pixel (mm/pix), DEM resolution of 6 mm/pix, and thermal orthophoto resolution of 30 mm/pix or higher were recommended for airfield pavement distress detection and rating. This research also concluded that sUAS-based PCI inspection not only detects and rates a number of airfield pavement distresses, but also provides PCI values close to the foot-on-ground traditional PCI inspection value. Recommendations on sUAS data collection plan development, safe and efficient data collection, data processing, data analysis, and the process of incorporating sUAS-based PCI inspection to complement traditional PCI inspections are discussed in detail