Autonomous search and tracking of objects using model predictive control of unmanned aerial vehicle and gimbal: Hardware-in-the-loop simulation of payload and avionics

This paper describes the design of model predictive control (MPC) for an unmanned aerial vehicle (UAV) used to track objects of interest identified by a real-time camera vision (CV) module in a search and track (SAT) autonomous system. A fully functional UAV payload is introduced, which includes an infra-red (IR) camera installed in a two-axis gimbal system. Hardware-in-loop (HIL) simulations are performed to test the MPC's performance in the SAT system, where the gimbal attitude and the UAV's flight trajectory are optimized to place the object to be tracked in the center of the IR camera's image.(c) 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works

Infrared Object Detection & Tracking in UAVs

The present thesis describes the design and implementation of a small, light weight and power efficient payload system for the use in unmanned aerial vehicles (UAVs). The primary application of the payload system is that of performing real-time object detection and tracking based on an infrared camera. The implemented object detection algorithm utilizes pre-trained classifiers to perform detection, and the implemented object tracking algorithm is based on an estimate-and-measure tracking approach. The estimator used is a standard Kalman filter, which assumes a linear motion model for the tracked objects. A global nearest neighbor approach was used to match the measurements to the tracked objects. Two types of classifiers were trained. The first classifier was trained with a support vector machine in combination with the histogram of oriented gradients feature representation. The second classifier was created by constructing a boosted cascade of classifiers trained using the AdaBoost algorithm by means of a set of Haar-like features. Furthermore, experiments are presented which demonstrate that the system is able to consistently track humans in many simulated real-time scenarios. However, it was found that in the presence of abrupt and relatively large object displacements, the linear motion model was not sufficiently accurate to keep track of the object. This implies that the tracking algorithm can benefit from the implementation of a non-linear estimator. Finally, the payload was found to be able to perform simulated real-time tracking, while at the same time performing several additional tasks. This includes sending important data to a control station located on the ground, and also simulating control over the UAV

A UAV Ice Tracking Framework for Autonomous Sea Ice Management

This paper describes an unmanned aerial vehicle (UAV) ice tracking framework for use in sea ice management applications. The framework is intended to be used in an ice management scenario where the UAV should detect and track the movement of icebergs and ice floes in an Arctic environment, and seeks to enable the UAV to do so autonomously. This is achieved by using an occupancy grid map algorithm and a locations of interest generator coupled with a Model Predictive Control (MPC) UAV path planner. The main contribution of this paper is interfacing the occupancy grid map algorithm with a machine vision object detection module in order to enable the UAV to generate an occupancy grid map of a pre-defined search area in real-time using on-board processing of UAV sensor data. Further, the paper presents a locations of interest generator module which generates locations that the UAV should investigate based on the generated occupancy grid map. These locations of interest are then used by an MPC path planner in order to make the UAV autonomously investigate and track ice features at said locations. Furthermore, the paper verifies the use of the developed ice tracking framework for autonomously detecting and tracking ice features based on thermal images captured with a UAV, as well as verifying the usefulness and role of UAVs in ice management scenarios by conducting two flight experiments

Colored-Noise Tracking of Floating Objects using UAVs with Thermal Cameras

Tracking of floating objects using a fixed-wing UAV equipped with a thermal camera requires precise knowledge about the position and attitude of the UAV. Errors in the navigation estimates reduce the accuracy of the tracking system. Navigation errors are usually correlated in time and can propagate colored noise into the tracking filter. This work analyzes two approaches that seek to mitigate colored noise and they are compared experimentally with a third approach which assumes that the noise in the tracking system is purely white. Two independent flight experiments have been carried out where a small marine vessel was used as target. Thermal images of the target were captured and the position and velocity of the target have been estimated in an Earth-fixed coordinate system only using the images. The results show that objects can be tracked with an accuracy of a few meters when measurements are available, and that the estimates do not drift significantly in periods without measurements. Moreover, the results demonstrate that colored noise need to be accounted for in the measurement model to estimate the covariance precisely and maintain filter consistency, which is critical in multi-target tracking

A UAV Ice Tracking Framework for Autonomous Sea Ice Management

This paper describes an unmanned aerial vehicle (UAV) ice tracking framework for use in sea ice management applications. The framework is intended to be used in an ice management scenario where the UAV should detect and track the movement of icebergs and ice floes in an Arctic environment, and seeks to enable the UAV to do so autonomously. This is achieved by using an occupancy grid map algorithm and a locations of interest generator coupled with a Model Predictive Control (MPC) UAV path planner. The main contribution of this paper is interfacing the occupancy grid map algorithm with a machine vision object detection module in order to enable the UAV to generate an occupancy grid map of a pre-defined search area in real-time using on-board processing of UAV sensor data. Further, the paper presents a locations of interest generator module which generates locations that the UAV should investigate based on the generated occupancy grid map. These locations of interest are then used by an MPC path planner in order to make the UAV autonomously investigate and track ice features at said locations. Furthermore, the paper verifies the use of the developed ice tracking framework for autonomously detecting and tracking ice features based on thermal images captured with a UAV, as well as verifying the usefulness and role of UAVs in ice management scenarios by conducting two flight experiments

Automatic detection, classification and tracking of objects in the ocean surface from UAVs using a thermal camera

The use of unmanned aerial vehicles (UAVs) that can operate autonomously in dynamic and dangerous operational environments are becoming increasingly common. In such operations, object detection, classification and tracking can often be one of the main goals. In recent years there has been an increased focus on embedded hardware that is both small and powerful, making UAV on-board data processing more viable. Being able to process the video feed on-board the UAV calls for fast and robust real-time algorithms for object identification and tracking. This paper discusses the development and implementation of a machine vision system for a low-cost fixed-wing UAV with a total flying weight of less than 4kg. The machine vision system incorporates the use of a thermal imaging camera and on-board processing power to perform real-time object detection, classification and tracking of objects in the ocean surface. The system is tested on thermal video data from a test flight, and is found to be able to detect 99;6% of objects of interest located in the ocean surface. Of the detected objects, only 5% were false positives. Furthermore, it classifies 93; 3% of the object types it is trained to classify correctly. The classifier is highly agile, allowing the user to quickly define which object characteristics that should be considered during classification, and what types of objects to classify. Finally, the system is found to successfully track 85% of the object types it is actively searching for in a real-time simulation test

Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera

This paper presents four vision-based tracking system architectures for marine surface objects using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera mounted in a pan/tilt gimbal. The tracking systems estimate the position and velocity of an object in the North-East (NE) plane, and differ in how the measurement models are defined. The first tracking system measures the position and velocity of the target with georeferencing and optical flow. The states are estimated in a Kalman filter. A Kalman filter is also utilized in the second architecture, but only the georeferenced position is used as a measurement. A bearing-only measurement model is the basis for the third tracking system, and because the measurement model is nonlinear, an extended Kalman filter is used for state estimation. The fourth tracking system extends the bearing-only tracking system to let navigation uncertainty in the UAV position affect the target estimates in a Schmidt-Kalman filter. All tracking architectures are evaluated on data gathered at a flight experiment near the Azores islands outside of Portugal. The results show that various marine vessels can be tracked quite accurately

Object Detection, Recognition and Tracking from UAVs using a Thermal Camera

In this paper a multiple object detection, recognition, and tracking system for unmanned aerial vehicles (UAVs) has been studied. The system can be implemented on any UAVs platform, with the main requirement being that the UAV has a suitable onboard computational unit and a camera. It is intended to be used in a maritime object tracking system framework for UAVs, which enables a UAV to perform multiobject tracking and situational awareness of the sea surface, in real time, during a UAV operation. Using machine vision to automatically detect objects in the camera's image stream combined with the UAV's navigation data, the onboard computer is able to georeference each object detection to measure the location of the detected objects in a local North‐East (NE) coordinate frame. A tracking algorithm which uses a Kalman filter and a constant velocity motion model utilizes an object's position measurements, automatically found using the object detection algorithm, to track and estimate an object's position and velocity. Furthermore, a global‐nearest‐neighbor algorithm is applied for data association. This is achieved using a measure of distance that is based not only on the physical distance between an object's estimated position and the measured position, but also how similar the objects appear in the camera image. Four field tests were conducted at sea to verify the object detection and tracking system. One of the flight tests was a two‐object tracking scenario, which is also used in three scenarios with an additional two simulated objects. The tracking results demonstrate the effectiveness of using visual recognition for data association to avoid interchanging the two estimated object trajectories. Furthermore, real‐time computations performed on the gathered data show that the system is able to automatically detect and track the position and velocity of a boat. Given that the system had at least 100 georeferenced measurements of the boat's position, the position was estimated and tracked with an accuracy of 5–15 m from 400 m altitude while the boat was in the camera's field of view (FOV). The estimated speed and course would also converge to the object's true trajectories (measured by Global Positioning System, GPS) for the tested scenarios. This enables the system to track boats while they are outside the FOV of the camera for extended periods of time, with tracking results showing a drift in the boat's position estimate down to 1–5 m/min outside of the FOV of the camera

Autonomous ballistic airdrop of objects from a small fixed-wing unmanned aerial vehicle

Autonomous airdrop is a useful basic operation for a fixed-wing unmanned aerial system. Being able to deliver an object to a known target position extends operational range without risking human lives, but is still limited to known delivery locations. If the fixed-wing unmanned aerial vehicle delivering the object could also recognize its target, the system would take one step further in the direction of autonomy. This paper presents a closed-loop autonomous delivery system that uses machine vision to identify a target marked with a distinct colour, calculates the geographical coordinates of the target location and plans a path to a release point, where it delivers the object. Experimental results present a visual target estimator with a mean error distance of 3.4 m and objects delivered with a mean error distance of 5.5 m

Real-time Georeferencing of Thermal Images using Small Fixed-Wing UAVs in Maritime Environments

This article considers real-time georeferencing using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera. A flexible system for direct georeferencing is proposed without the need for ground reference points. Moreover, as the system is tailored for highly maneuverable and agile fixed-wing UAVs, no restrictions on the motion are assumed. The system is designed with a solution for accurate time synchronization between sensors. This feature enables tracking of objects with low uncertainty. Sensors for navigation, permitting estimation of the UAV pose with a nonlinear observer, are employed in addition to a thermal camera. The estimated UAV pose is utilized in georeferencing to acquire Earth-fixed coordinates of objects. The main examples studied in this research are georeferencing of a static object and of a moving marine vessel. To obtain the desired accuracy, thermal camera calibration and compensation of mounting misalignment errors are discussed. The entire system is validated in two independent field experiments with a thorough analysis of the results. Georeferencing of a static object is conducted with centimeter accuracy when the average position of all measurements is used. The position of a moving marine vessel is obtained with mean accuracy of two meters