Object Detection and Tracking Based on Optical Flow in Unmanned Aerial Vehicles

In recent years the Unmanned Aerial Vehicle (UAV) community has discovered the enormous amount of information that can be extracted from a video camera. It can be used for collision avoidance, navigation, velocity estimation, terrain mapping, object detection and object tracking in addition to many other applications. This thesis looks into several different ways a video camera in the payload of a fixed-wing UAV can be utilized. The first part investigates a way to calculate the body-fixed velocity of the UAV with the images captured by the camera. It can be achieved with optical flow and measurements of the roll angle, pitch angle and the altitude of the UAV. The calculated body-fixed velocity can be used as a measurement in the navigation system. This thesis looks into a navigation system based on a nonlinear observer that estimates attitude, position, velocity and gyro bias. In order to estimate these states the nonlinear observer utilizes measurements from an Inertial Measurement Unit (IMU), a Global Navigation Satellite System (GNSS) receiver and a video camera. The second part looks into moving object detection and tracking. An algorithm for detection of moving objects has been developed. It utilizes the navigation states of the UAV and optical flow in order to extract the moving objects from the images. Furthermore a tracking system based on the moving object detection algorithm has been proposed. The tracking system consists of the moving object detection algorithm, a classifier that describes each object and a discrete Kalman filter that estimates the motion of the object. The objects are tracked in the image plane and the estimates are transformed to the North-East-Down (NED) coordinate system. A fixed-wing UAV experiment has been carried out to gather images captured from an airborne UAV and collect data from an IMU and a GPS receiver. The navigation system has been evaluated offline by computer simulations based on the data collected at the experiment. The tracking system and the moving object detection algorithm have also been evaluated in computer simulations. Moving objects have been inserted into the images because moving objects did not exist in the experiment. Promising and accurate results were shown for both the navigation system and the proposed tracking system

Detection and Trcking of Floating Objects using UAVs with Optical Sensors

This thesis focuses on detection and tracking of floating objects using fixed-wing unmanned aerial vehicles (UAVs) equipped with a monocular thermal camera. UAVs with optical sensors are useful in a vast number of applications such as search and rescue, inspection, target tracking and surveillance. The usefulness of UAVs in remote sensing applications is going to increase in the future so further research on these topics is needed. The main motivation behind this work was to identify and develop suitable realtime algorithms for detection and tracking of objects located on the sea surface. Moreover, real-time performance on small embedded computers was desired and used as a guideline. Much work has been carried out to study detection and tracking on stationary platforms, but less work has been conducted to find methods that fit the operating envelope of fixed-wing UAVs. This thesis aims to provide new insight into detection and tracking, and handle issues that arise from rapid camera motion. Another objective was to provide new insight through analysis of experimental data. The majority of the results presented in this thesis are based on experimental data that have been collected in several independent field experiments. A huge effort was made to plan and conduct these experiments, both with regard to payload development and mission design. It is hard to simulate detection and tracking in a realistic manner because some of the most prominent issues are difficult to recreate in simulations. Therefore, a goal was to collect much experimental data and process the data using different methods. Obviously, this has influenced the direction of this work because issues revealed along the way have been prioritized and investigated later in the research period. Tracking and georeferencing of floating objects have been the main tasks studied in this thesis. Consequently, some of the methods are tailored for these applications. Nevertheless, the methods are also applicable for complementary missions with changes to smaller parts of the system. The experimental analysis has focused both on empirical and theoretical aspects. Improving the accuracy of existing systems has been desirable, but concepts such as optimality and consistency of tracking filters have also been central. Target tracking is usually divided into three subproblems, namely detection, data or measurement association, and filtering (state estimation). Detection concerns identification of objects within a sensor scan. Object detection in images involves identifying pixels that belong to objects. It is traditionally solved with image processing techniques and more recently with machine learning. The filtering part of target tracking deals with estimation of target states, typically the position and velocity. The filtering part utilizes image detections to improve and correct the predicted states. Measurement association is needed when multiple targets are present in the surveillance region or when clutter and false detections are expected. New detections are related to existing tracks through data association. This thesis looks into every part of the tracking system, but focuses particularly on filtering and state estimation. The first part of this thesis consists of three chapters which give a fundamental introduction to remote sensing applications, UAVs, optical sensors and target tracking. Research objectives and goals are stated together with description of related and existing work. Chapter 4 to Chapter 7 investigate detection in thermal images, georeferencing of thermal images, and target tracking. Navigation uncertainty influences georeferencing and target tracking in a negative way so one chapter is devoted to tracking in the presence of navigation uncertainty. These four chapters consist of several methods in addition to numerous case studies that investigate the effectiveness of these methods. The thesis also touches upon navigation for UAVs because errors in the UAV pose is a bottleneck with respect to the accuracy and performance of the tracking system. The final part of the dissertation concludes the results and discusses future possibilities

Sensor Combinations in Heterogeneous Multi-sensor Fusion for Maritime Target Tracking

Safe navigation for autonomous surface vehicles requires a robust and reliable tracking system that maintains and estimates position and velocity of other vessels. This paper demonstrates a measurement level sensor fusion system for tracking in a maritime environment using lidar, radar, electrooptical and infrared cameras. The backbone of the system is a multi-sensor version of the Joint Integrated Probabilistic Data Association (JIPDA) with both existence and visibility probabilities. Using reference targets equipped with GPS receivers, the performance of different sensors and sensor combinations are evaluated for autonomous surface vehicles (ASVs), Several interesting observations are made, among them that passive sensors can help resolve merged measurements issues in radar tracking, and that the choice between radar and lidar may boil down to a trade-off between fast track initiation and large numbers of false tracks

Colored-Noise Tracking of Floating Objects using UAVs with Thermal Cameras

Tracking of floating objects using a fixed-wing UAV equipped with a thermal camera requires precise knowledge about the position and attitude of the UAV. Errors in the navigation estimates reduce the accuracy of the tracking system. Navigation errors are usually correlated in time and can propagate colored noise into the tracking filter. This work analyzes two approaches that seek to mitigate colored noise and they are compared experimentally with a third approach which assumes that the noise in the tracking system is purely white. Two independent flight experiments have been carried out where a small marine vessel was used as target. Thermal images of the target were captured and the position and velocity of the target have been estimated in an Earth-fixed coordinate system only using the images. The results show that objects can be tracked with an accuracy of a few meters when measurements are available, and that the estimates do not drift significantly in periods without measurements. Moreover, the results demonstrate that colored noise need to be accounted for in the measurement model to estimate the covariance precisely and maintain filter consistency, which is critical in multi-target tracking

Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera

This paper presents four vision-based tracking system architectures for marine surface objects using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera mounted in a pan/tilt gimbal. The tracking systems estimate the position and velocity of an object in the North-East (NE) plane, and differ in how the measurement models are defined. The first tracking system measures the position and velocity of the target with georeferencing and optical flow. The states are estimated in a Kalman filter. A Kalman filter is also utilized in the second architecture, but only the georeferenced position is used as a measurement. A bearing-only measurement model is the basis for the third tracking system, and because the measurement model is nonlinear, an extended Kalman filter is used for state estimation. The fourth tracking system extends the bearing-only tracking system to let navigation uncertainty in the UAV position affect the target estimates in a Schmidt-Kalman filter. All tracking architectures are evaluated on data gathered at a flight experiment near the Azores islands outside of Portugal. The results show that various marine vessels can be tracked quite accurately

Object Detection, Recognition and Tracking from UAVs using a Thermal Camera

In this paper a multiple object detection, recognition, and tracking system for unmanned aerial vehicles (UAVs) has been studied. The system can be implemented on any UAVs platform, with the main requirement being that the UAV has a suitable onboard computational unit and a camera. It is intended to be used in a maritime object tracking system framework for UAVs, which enables a UAV to perform multiobject tracking and situational awareness of the sea surface, in real time, during a UAV operation. Using machine vision to automatically detect objects in the camera's image stream combined with the UAV's navigation data, the onboard computer is able to georeference each object detection to measure the location of the detected objects in a local North‐East (NE) coordinate frame. A tracking algorithm which uses a Kalman filter and a constant velocity motion model utilizes an object's position measurements, automatically found using the object detection algorithm, to track and estimate an object's position and velocity. Furthermore, a global‐nearest‐neighbor algorithm is applied for data association. This is achieved using a measure of distance that is based not only on the physical distance between an object's estimated position and the measured position, but also how similar the objects appear in the camera image. Four field tests were conducted at sea to verify the object detection and tracking system. One of the flight tests was a two‐object tracking scenario, which is also used in three scenarios with an additional two simulated objects. The tracking results demonstrate the effectiveness of using visual recognition for data association to avoid interchanging the two estimated object trajectories. Furthermore, real‐time computations performed on the gathered data show that the system is able to automatically detect and track the position and velocity of a boat. Given that the system had at least 100 georeferenced measurements of the boat's position, the position was estimated and tracked with an accuracy of 5–15 m from 400 m altitude while the boat was in the camera's field of view (FOV). The estimated speed and course would also converge to the object's true trajectories (measured by Global Positioning System, GPS) for the tested scenarios. This enables the system to track boats while they are outside the FOV of the camera for extended periods of time, with tracking results showing a drift in the boat's position estimate down to 1–5 m/min outside of the FOV of the camera

Autonomous ballistic airdrop of objects from a small fixed-wing unmanned aerial vehicle

Autonomous airdrop is a useful basic operation for a fixed-wing unmanned aerial system. Being able to deliver an object to a known target position extends operational range without risking human lives, but is still limited to known delivery locations. If the fixed-wing unmanned aerial vehicle delivering the object could also recognize its target, the system would take one step further in the direction of autonomy. This paper presents a closed-loop autonomous delivery system that uses machine vision to identify a target marked with a distinct colour, calculates the geographical coordinates of the target location and plans a path to a release point, where it delivers the object. Experimental results present a visual target estimator with a mean error distance of 3.4 m and objects delivered with a mean error distance of 5.5 m

Real-time Georeferencing of Thermal Images using Small Fixed-Wing UAVs in Maritime Environments

This article considers real-time georeferencing using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera. A flexible system for direct georeferencing is proposed without the need for ground reference points. Moreover, as the system is tailored for highly maneuverable and agile fixed-wing UAVs, no restrictions on the motion are assumed. The system is designed with a solution for accurate time synchronization between sensors. This feature enables tracking of objects with low uncertainty. Sensors for navigation, permitting estimation of the UAV pose with a nonlinear observer, are employed in addition to a thermal camera. The estimated UAV pose is utilized in georeferencing to acquire Earth-fixed coordinates of objects. The main examples studied in this research are georeferencing of a static object and of a moving marine vessel. To obtain the desired accuracy, thermal camera calibration and compensation of mounting misalignment errors are discussed. The entire system is validated in two independent field experiments with a thorough analysis of the results. Georeferencing of a static object is conducted with centimeter accuracy when the average position of all measurements is used. The position of a moving marine vessel is obtained with mean accuracy of two meters