A System for the Generation of Synthetic Wide Area Aerial Surveillance Imagery

The development, benchmarking and validation of aerial Persistent Surveillance (PS) algorithms requires access to specialist Wide Area Aerial Surveillance (WAAS) datasets. Such datasets are difficult to obtain and are often extremely large both in spatial resolution and temporal duration. This paper outlines an approach to the simulation of complex urban environments and demonstrates the viability of using this approach for the generation of simulated sensor data, corresponding to the use of wide area imaging systems for surveillance and reconnaissance applications. This provides a cost-effective method to generate datasets for vehicle tracking algorithms and anomaly detection methods. The system fuses the Simulation of Urban Mobility (SUMO) traffic simulator with a MATLAB controller and an image generator to create scenes containing uninterrupted door-to-door journeys across large areas of the urban environment. This ‘pattern-of-life’ approach provides three-dimensional visual information with natural movement and traffic flows. This can then be used to provide simulated sensor measurements (e.g. visual band and infrared video imagery) and automatic access to ground-truth data for the evaluation of multi-target tracking systems

ClusterNet: Detecting Small Objects in Large Scenes by Exploiting Spatio-Temporal Information

Object detection in wide area motion imagery (WAMI) has drawn the attention of the computer vision research community for a number of years. WAMI proposes a number of unique challenges including extremely small object sizes, both sparse and densely-packed objects, and extremely large search spaces (large video frames). Nearly all state-of-the-art methods in WAMI object detection report that appearance-based classifiers fail in this challenging data and instead rely almost entirely on motion information in the form of background subtraction or frame-differencing. In this work, we experimentally verify the failure of appearance-based classifiers in WAMI, such as Faster R-CNN and a heatmap-based fully convolutional neural network (CNN), and propose a novel two-stage spatio-temporal CNN which effectively and efficiently combines both appearance and motion information to significantly surpass the state-of-the-art in WAMI object detection. To reduce the large search space, the first stage (ClusterNet) takes in a set of extremely large video frames, combines the motion and appearance information within the convolutional architecture, and proposes regions of objects of interest (ROOBI). These ROOBI can contain from one to clusters of several hundred objects due to the large video frame size and varying object density in WAMI. The second stage (FoveaNet) then estimates the centroid location of all objects in that given ROOBI simultaneously via heatmap estimation. The proposed method exceeds state-of-the-art results on the WPAFB 2009 dataset by 5-16% for moving objects and nearly 50% for stopped objects, as well as being the first proposed method in wide area motion imagery to detect completely stationary objects.Comment: Main paper is 8 pages. Supplemental section contains a walk-through of our method (using a qualitative example) and qualitative results for WPAFB 2009 datase

Aerial Vehicle Tracking by Adaptive Fusion of Hyperspectral Likelihood Maps

Hyperspectral cameras can provide unique spectral signatures for consistently distinguishing materials that can be used to solve surveillance tasks. In this paper, we propose a novel real-time hyperspectral likelihood maps-aided tracking method (HLT) inspired by an adaptive hyperspectral sensor. A moving object tracking system generally consists of registration, object detection, and tracking modules. We focus on the target detection part and remove the necessity to build any offline classifiers and tune a large amount of hyperparameters, instead learning a generative target model in an online manner for hyperspectral channels ranging from visible to infrared wavelengths. The key idea is that, our adaptive fusion method can combine likelihood maps from multiple bands of hyperspectral imagery into one single more distinctive representation increasing the margin between mean value of foreground and background pixels in the fused map. Experimental results show that the HLT not only outperforms all established fusion methods but is on par with the current state-of-the-art hyperspectral target tracking frameworks.Comment: Accepted at the International Conference on Computer Vision and Pattern Recognition Workshops, 201

Traffic Surveillance and Automated Data Extraction from Aerial Video Using Computer Vision, Artificial Intelligence, and Probabilistic Approaches

In transportation engineering, sufficient, reliable, and diverse traffic data is necessary for effective planning, operations, research, and professional practice. Using aerial imagery to achieve traffic surveillance and collect traffic data is one of the feasible ways that is facilitated by the advances of technologies in many related areas. A great deal of aerial imagery datasets are currently available and more datasets are collected every day for various applications. It will be beneficial to make full and efficient use of the attribute rich imagery as a resource for valid and useful traffic data for many applications in transportation research and practice. In this dissertation, a traffic surveillance system that can collect valid and useful traffic data using quality-limited aerial imagery datasets with diverse characteristics is developed. Two novel approaches, which can achieve robust and accurate performance, are proposed and implemented for this system. The first one is a computer vision-based approach, which uses convolutional neural network (CNN) to detect vehicles in aerial imagery and uses features to track those detections. This approach is capable of detecting and tracking vehicles in the aerial imagery datasets with a very limited quality. Experimental results indicate the performance of this approach is very promising and it can achieve accurate measurements for macroscopic traffic data and is also potential for reliable microscopic traffic data. The second approach is a multiple hypothesis tracking (MHT) approach with innovative kinematics and appearance models (KAM). The implemented MHT module is designed to cooperate with the CNN module in order to extend and improve the vehicle tracking system. Experiments are designed based on a meticulously established synthetic vehicle detection datasets, originally induced scale-agonistic property of MHT, and comprehensively identified metrics for performance evaluation. The experimental results not only indicate that the performance of this approach can be very promising, but also provide solutions for some long-standing problems and reveal the impacts of frame rate, detection noise, and traffic configurations as well as the effects of vehicle appearance information on the performance. The experimental results of both approaches prove the feasibility of traffic surveillance and data collection by detecting and tracking vehicles in aerial video, and indicate the direction of further research as well as solutions to achieve satisfactory performance with existing aerial imagery datasets that have very limited quality and frame rates. This traffic surveillance system has the potential to be transformational in how large area traffic data is collected in the future. Such a system will be capable of achieving wide area traffic surveillance and extracting valid and useful traffic data from wide area aerial video captured with a single platfor

Is Wide-Area Persistent Surveillance by State and Local Governments Constitutional?

This dissertation addresses the following question: “Can wide-area persistent surveillance (WAPS) developed by the United States military and employed abroad as a tool in the Global War on Terror be employed domestically as a law enforcement tool without violating the US Constitution’s Fourth Amendment?” The most likely and controversial application of WAPS by state and local governments is for law enforcement. Aircraft will loiter over a city persistently taking high-definition photographs to capture locations of unidentified persons with the intent to identify persons and areas of interest for criminal investigations. Based on the Flyover Cases, aerial surveillance has few constitutional limitations which WAPS can be consistent. The key challenge in determining the constitutionality of WAPS depends on the Court’s interpretation of the Fourth Amendment concerning emerging technologies. Legal scholars have suggested various forms of the Mosaic Theory, which was introduced in two concurring opinions in Jones v. United States . The Supreme Court has been reticent to engage new technology’s constitutionality. WAPS is among the less intrusive tools when compared to other emerging technologies like digital information or facial recognition. This research argues why the Courts should view Personal Identifying Information (PII) as the line of reasonable expectations of privacy for WAPS and other emerging technologies. Aerial surveillance by nature, collects passive information, new data is not being created by photographing the happenings in public spaces from an aerial platform. In Carpenter v. United States, the Court ruled that warrantless surveillance of cell site location information (CSLI) for more than seven days was an unreasonable search. However, the court repeatedly referred to CSLI as “unique,” whereas “conventional surveillance and tools, such as security cameras,” are not. WAPS should not be limited by the Constitution for the operational duration, time of day/night, camera resolution, location of collection, altitude, or any other variable at the collection stage of the operations. The analysis and exploitation of WAPS data encounters constitutional limits necessary to protect individuals’ PII absent probable cause standards