Mitigating the Effects of Boom Occlusion on Automated Aerial Refueling through Shadow Volumes

In flight refueling of Unmanned Aerial Vehicles (UAVs) is critical to the United States Air Force (USAF). However, the large communication latency between a ground-based operator and his/her remote UAV makes docking with a refueling tanker unsafe. This latency may be mitigated by leveraging a tanker-centric stereo vision system. The vision system observes and computes an approaching receiver\u27s relative position and orientation offering a low-latency, high frequency docking solution. Unfortunately, the boom -- an articulated refueling arm responsible for physically pumping fuel into the receiver -- occludes large portions of the receiver especially as the receiver approaches and docks with the tanker. The vision system must be able to compensate for the boom\u27s occlusion of the receiver aircraft. We present a novel algorithm for mitigating the negative effects of boom occlusion in stereo-based aerial environments. Our algorithm dynamically compensates for occluded receiver geometry by transforming the occluded areas into shadow volumes. These shadow volumes are then used to cull hidden geometry that is traditionally consumed, in error, by the vision processing and point registration pipeline. Our algorithm improves computer-vision pose estimates by an average of 74% over a naive approach without shadow volume culling

Maximizing Accuracy through Stereo Vision Camera Positioning for Automated Aerial Refueling

Aerial refueling is a key component of the U.S. Air Force strategic arsenal. When two aircraft interact in an aerial refueling operation, the accuracy of relative navigation estimates are critical for the safety, accuracy and success of the mission. Automated Aerial Refueling (AAR) looks to improve the refueling process by creating a more effective system and allowing for Unmanned Aerial Vehicle(s) (UAV) support. This paper considers a cooperative aerial refueling scenario where stereo cameras are used on the tanker to direct a \boom (a large, long structure through which the fuel will ow) into a port on the receiver aircraft. The analysis focuses on the effects of camera positioning with the rear-facing stereo vision system. In particular, the research seeks the optimal system design for the camera system to achieve the most accurate navigational estimates. The testing process consists of utilizing a simulation engine and recreating real world flights based on previously collected Global Positioning System (GPS) data. Using the pose estimation results and the ground truth information, the system computes the error between the incoming aircraft\u27s position in the virtual world and its calculated location based on the stereo matching algorithm. The testing process includes both un-obscured scenarios and cases where the boom causes significant occlusions in the camera images. The results define the improvements in position and orientation estimation of camera positioning from the consolidated simulation data. Conclusions drawn from this research will propose and help provide recommendations for future Air Force acquisition and development of aerial refueling systems

Stereo Vision: A Comparison of Synthetic Imagery vs. Real World Imagery for the Automated Aerial Refueling Problem

Missions using unmanned aerial vehicles have increased in the past decade. Currently, there is no way to refuel these aircraft. Accomplishing automated aerial refueling can be made possible using the stereo vision system on a tanker. Real world experiments for the automated aerial refueling problem are expensive and time consuming. Currently, simulations performed in a virtual world have shown promising results using computer vision. It is possible to use the virtual world as a substitute environment for the real world. This research compares the performance of stereo vision algorithms on synthetic and real world imagery

Infrared and Electro-Optical Stereo Vision for Automated Aerial Refueling

Currently, Unmanned Aerial Vehicles are unsafe to refuel in-flight due to the communication latency between the UAVs ground operator and the UAV. Providing UAVs with an in-flight refueling capability would improve their functionality by extending their flight duration and increasing their flight payload. Our solution to this problem is Automated Aerial Refueling (AAR) using stereo vision from stereo electro-optical and infrared cameras on a refueling tanker. To simulate a refueling scenario, we use ground vehicles to simulate a pseudo tanker and pseudo receiver UAV. Imagery of the receiver is collected by the cameras on the tanker and processed by a stereo block matching algorithm to calculate a position and orientation estimate of the receiver. GPS and IMU truth data is then used to validate these results

Towards Automated Aerial Refueling: Real Time Position Estimation with Stereo Vision

Aerial refueling is essential to the United States Air Force (USAF) core mission of rapid global mobility. However, in-flight refueling is not available to remotely piloted aircraft (RPA) or unmanned aerial systems (UAS). As reliance on drones for intelligence, surveillance, and reconnaissance (ISR) and other USAF core missions grows, the ability to automate aerial refueling for such systems becomes increasingly critical. New refueling platforms include sensors that could be used to estimate the relative position of an approaching aircraft. Relative position estimation is a key component to solving the automated aerial refueling (AAR) problem. Analysis of data from a one-seventh scale, real world refueling scenario demonstrates that the relative position of an approaching aircraft can be estimated at rates between 10 Hz and 30 Hz using stereo vision. Linear regression models on position estimate accuracies predict results reported by other research in the simulation domain, suggesting that real world accuracies are comparable to simulation domain accuracies reported by others. Further, by seeding the position estimation algorithm with previous position estimates, subsequent errors in position estimation are reduce

Automated Aerial Refueling Position Estimation Using a Scanning LiDAR

This research examines the application of using a scanning Light Detection and Ranging(LiDAR) to perform Automated Aerial Refueling(AAR). Specifically this thesis presents two algorithms to determine the relative position between the tanker and receiver aircraft. These two algorithms require a model of the tanker aircraft and the relative attitude between the aircraft. The first algorithm fits the measurements to the model of the aircraft using a modified Iterative Closest Point (ICP) algorithm. The second algorithm uses the model to predict LiDAR scans and compare them to actual measurements while perturbing the estimated location of the tanker. Each algorithm was tested with simulated LiDAR data before real data became available from test flights. The data collected from this test ight was used to determine the accuracy of the two algorithms with real LiDAR data. After correcting for modeling errors the accuracy of each algorithm is about a Mean Radial Spherical Error of 40cm

Determining Virtual Practicality from Physical Stereo Vision Images and GPS

Current research efforts for Automated Aerial Refueling (AAR) at The Air Force Institute of Technology (AFIT) utilize Stereo Computer Vision to compute a relative pose between a tanker and receiver aircraft. Due to costs, time, and availability, it can be onerous to test these algorithms using actual Air Force (AF) aircraft. Our solution to this problem consists of using a 3D Graphics Engine to simulate AAR endeavors. However, the question then arises, “Does the virtual world accurately represent the physical world?” This can be explored by comparing a set of truth data to a similar set of virtual data. First, a set of truth data is collected using physical aircraft. Next, using the same flight path as that of the truth data, a set of virtual data is collected. Finally, a comparison of the physical and virtual data can provide information regarding how well the virtual world accurately represents the physical world, and if so, to within what margin of error? The results show that the virtual world roughly approximates the physical world but performs 2-6x better on average

FPGA Accelerated Discrete-SURF for Real-Time Homography Estimation

This paper describes our hardware accelerated, FPGA implementation of SURF, named Discrete SURF, to support real-time homography estimation for close range aerial navigation. The SURF algorithm provides feature matches between a model and a scene which can be used to find the transformation between the camera and the model. Previous implementations of SURF have partially employed FPGAs to accelerate the feature detection stage of upright only image comparisons. We extend the work of previous implementations by providing an FPGA implementation that allows rotation during image comparisons in order to facilitate aerial navigation. We also expand beyond feature detection as the complete Discrete SURF algorithm is run on the FPGA, rather than piped into processors. This not only minimizes overhead and increases the parallelization of the algorithm, but also allows the algorithm to be easily ported to different FPGAs. Furthermore, the Discrete SURF module is a logic-only implementation that does not rely on external hardware which therefore decreases the overall size, weight and power of the device while also allowing for easy FPGA to ASIC conversion. We evaluate the Discrete SURF algorithm in terms of performance against the original SURF and upright SURF algorithms implemented in OpenCV. Finally, we show how Discrete SURF is more compatible with an aerial navigation scenario than previous works, since rotation invariance must be considered in addition to scale

Air Force Institute of Technology Research Report 2016

This Research Report presents the FY16 research statistics and contributions of the Graduate School of Engineering and Management (EN) at AFIT. AFIT research interests and faculty expertise cover a broad spectrum of technical areas related to USAF needs, as reflected by the range of topics addressed in the faculty and student publications listed in this report. In most cases, the research work reported herein is directly sponsored by one or more USAF or DOD agencies. AFIT welcomes the opportunity to conduct research on additional topics of interest to the USAF, DOD, and other federal organizations when adequate manpower and financial resources are available and/or provided by a sponsor. In addition, AFIT provides research collaboration and technology transfer benefits to the public through Cooperative Research and Development Agreements (CRADAs)

System-of-Systems Considerations in the Notional Development of a Metropolitan Aerial Transportation System

There are substantial future challenges related to sustaining and improving efficient, cost-effective, and environmentally friendly transportation options for urban regions. Over the past several decades there has been a worldwide trend towards increasing urbanization of society. Accompanying this urbanization are increasing surface transportation infrastructure costs and, despite public infrastructure investments, increasing surface transportation "gridlock." In addition to this global urbanization trend, there has been a substantial increase in concern regarding energy sustainability, fossil fuel emissions, and the potential implications of global climate change. A recently completed study investigated the feasibility of an aviation solution for future urban transportation (refs. 1, 2). Such an aerial transportation system could ideally address some of the above noted concerns related to urbanization, transportation gridlock, and fossil fuel emissions (ref. 3). A metro/regional aerial transportation system could also provide enhanced transportation flexibility to accommodate extraordinary events such as surface (rail/road) transportation network disruptions and emergency/disaster relief responses