Traffic Surveillance and Automated Data Extraction from Aerial Video Using Computer Vision, Artificial Intelligence, and Probabilistic Approaches

In transportation engineering, sufficient, reliable, and diverse traffic data is necessary for effective planning, operations, research, and professional practice. Using aerial imagery to achieve traffic surveillance and collect traffic data is one of the feasible ways that is facilitated by the advances of technologies in many related areas. A great deal of aerial imagery datasets are currently available and more datasets are collected every day for various applications. It will be beneficial to make full and efficient use of the attribute rich imagery as a resource for valid and useful traffic data for many applications in transportation research and practice. In this dissertation, a traffic surveillance system that can collect valid and useful traffic data using quality-limited aerial imagery datasets with diverse characteristics is developed. Two novel approaches, which can achieve robust and accurate performance, are proposed and implemented for this system. The first one is a computer vision-based approach, which uses convolutional neural network (CNN) to detect vehicles in aerial imagery and uses features to track those detections. This approach is capable of detecting and tracking vehicles in the aerial imagery datasets with a very limited quality. Experimental results indicate the performance of this approach is very promising and it can achieve accurate measurements for macroscopic traffic data and is also potential for reliable microscopic traffic data. The second approach is a multiple hypothesis tracking (MHT) approach with innovative kinematics and appearance models (KAM). The implemented MHT module is designed to cooperate with the CNN module in order to extend and improve the vehicle tracking system. Experiments are designed based on a meticulously established synthetic vehicle detection datasets, originally induced scale-agonistic property of MHT, and comprehensively identified metrics for performance evaluation. The experimental results not only indicate that the performance of this approach can be very promising, but also provide solutions for some long-standing problems and reveal the impacts of frame rate, detection noise, and traffic configurations as well as the effects of vehicle appearance information on the performance. The experimental results of both approaches prove the feasibility of traffic surveillance and data collection by detecting and tracking vehicles in aerial video, and indicate the direction of further research as well as solutions to achieve satisfactory performance with existing aerial imagery datasets that have very limited quality and frame rates. This traffic surveillance system has the potential to be transformational in how large area traffic data is collected in the future. Such a system will be capable of achieving wide area traffic surveillance and extracting valid and useful traffic data from wide area aerial video captured with a single platfor

Image similarity metrics suitable for infrared video stabilization during active wildfire monitoring: a comparative analysis

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration.Peer ReviewedPostprint (published version

Image similarity metrics suitable for infrared video stabilization during active wildfire monitoring : a comparative analysis

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration. These results provide a sound basis for image alignment measurement and open the door to further developments in image registration, motion estimation and video stabilization for aerial monitoring of active wildland fires

Multisource data integration to investigate one century of evolution for the Agnone landslide (Molise, southern Italy)

Landslides are one of the most relevant geohazards worldwide, causing direct and indirect costs and fatalities. Italy is one of the countries most affected by mass movements, and the Molise region, southern Italy, is known to be susceptible to erosional processes and landslides. In January 2003, a landslide in the municipality of Agnone, in the Colle Lapponi-Piano Ovetta (CL-PO) territory, occurred causing substantial damage to both structures and civil infrastructure. To investigate the evolution of the landslide-affected catchment over approximately one century, different data were taken into account: (i) literature information at the beginning of the twentieth century; (ii) historical sets of aerial optical photographs to analyse the geomorphological evolution from 1945 to 2003; (iii) SAR (Synthetic Aperture Radar) data fromthe ERS1/2, ENVISATand COSMO-SkyMed satellites tomonitor the displacement from 1992 to 2015; (iv) traditional measurements carried out through geological and geomorphological surveys, inclinometers and GPS campaigns to characterize the geological setting of the area; and (v) recent optical photographs of the catchment area to determine the enlargement of the landslide. Using the structure from motion technique, a 3D reconstruction of each set of historical aerial photographs was made to investigate the geomorphological evolution and to trace the boundary of the mass movements. As a result, the combination of multitemporal and multitechnique analysis of the evolution of the CL-PO landslide enabled an assessment of the landslide expansion, which resulted in a maximum length of up to approximately 1500 m. A complete investigation of the past and present deformational sequences of the area was performed to potentially plan further mitigation and prevention strategies to avoid possible reactivations

Mapping ground instability in areas of geotechnical infrastructure using satellite InSAR and Small UAV Surveying: a case study in Northern Ireland

Satellite Interferometric Synthetic Aperture Radar (InSAR), geological data and Small Unmanned Aerial Vehicle (SUAV) surveying was used to enhance our understanding of ground movement at five areas of interest in Northern Ireland. In total 68 ERS-1/2 images 1992–2000 were processed with the Small Baseline Subset (SBAS) InSAR technique to derive the baseline ground instability scenario of key areas of interest for five stakeholders: TransportNI, Northern Ireland Railways, Department for the Economy, Arup, and Belfast City Council. These stakeholders require monitoring of ground deformation across either their geotechnical infrastructure (i.e., embankments, cuttings, engineered fills and earth retaining structures) or assessment of subsidence risk as a result of abandoned mine workings, using the most efficient, cost-effective methods, with a view to minimising and managing risk to their businesses. The InSAR results provided an overview of the extent and magnitude of ground deformation for a 3000 km2 region, including the key sites of the disused salt mines in Carrickfergus, the Belfast–Bangor railway line, Throne Bend and Ligoniel Park in Belfast, Straidkilly and Garron Point along the Antrim Coast Road, plus other urbanised areas in and around Belfast. Tailored SUAV campaigns with a X8 airframe and generation of very high resolution ortho-photographs and a 3D surface model via the Structure from Motion (SfM) approach at Maiden Mount salt mine collapse in Carrickfergus in 2016 and 2017 also demonstrate the benefits of very high resolution surveying technologies to detect localised deformation and indicators of ground instabilit

Post-eruption deformation processes measured using ALOS-1 and UAVSAR InSAR at Pacaya Volcano, Guatemala

Pacaya volcano is a persistently active basaltic cone complex located in the Central American Volcanic Arc in Guatemala. In May of 2010, violent Volcanic Explosivity Index-3 (VEI-3) eruptions caused significant topographic changes to the edifice, including a linear collapse feature 600 m long originating from the summit, the dispersion of ~20 cm of tephra and ash on the cone, the emplacement of a 5.4 km long lava flow, and ~3 m of co-eruptive movement of the southwest flank. For this study, Interferometric Synthetic Aperture Radar (InSAR) images (interferograms) processed from both spaceborne Advanced Land Observing Satellite-1 (ALOS-1) and aerial Uninhabited Aerial Vehicle Synthetic Aperture Radar (UAVSAR) data acquired between 31 May 2010 and 10 April 2014 were used to measure post-eruptive deformation events. Interferograms suggest three distinct deformation processes after the May 2010 eruptions, including: (1) subsidence of the area involved in the co-eruptive slope movement; (2) localized deformation near the summit; and (3) emplacement and subsequent subsidence of about a 5.4 km lava flow. The detection of several different geophysical signals emphasizes the utility of measuring volcanic deformation using remote sensing techniques with broad spatial coverage. Additionally, the high spatial resolution of UAVSAR has proven to be an excellent compliment to satellite data, particularly for constraining motion components. Measuring the rapid initiation and cessation of flank instability, followed by stabilization and subsequent influence on eruptive features, provides a rare glimpse into volcanic slope stability processes. Observing these and other deformation events contributes both to hazard assessment at Pacaya and to the study of the stability of stratovolcanoes

Spatial Pyramid Context-Aware Moving Object Detection and Tracking for Full Motion Video and Wide Aerial Motion Imagery

A robust and fast automatic moving object detection and tracking system is essential to characterize target object and extract spatial and temporal information for different functionalities including video surveillance systems, urban traffic monitoring and navigation, robotic. In this dissertation, I present a collaborative Spatial Pyramid Context-aware moving object detection and Tracking system. The proposed visual tracker is composed of one master tracker that usually relies on visual object features and two auxiliary trackers based on object temporal motion information that will be called dynamically to assist master tracker. SPCT utilizes image spatial context at different level to make the video tracking system resistant to occlusion, background noise and improve target localization accuracy and robustness. We chose a pre-selected seven-channel complementary features including RGB color, intensity and spatial pyramid of HoG to encode object color, shape and spatial layout information. We exploit integral histogram as building block to meet the demands of real-time performance. A novel fast algorithm is presented to accurately evaluate spatially weighted local histograms in constant time complexity using an extension of the integral histogram method. Different techniques are explored to efficiently compute integral histogram on GPU architecture and applied for fast spatio-temporal median computations and 3D face reconstruction texturing. We proposed a multi-component framework based on semantic fusion of motion information with projected building footprint map to significantly reduce the false alarm rate in urban scenes with many tall structures. The experiments on extensive VOTC2016 benchmark dataset and aerial video confirm that combining complementary tracking cues in an intelligent fusion framework enables persistent tracking for Full Motion Video and Wide Aerial Motion Imagery.Comment: PhD Dissertation (162 pages