Towards an autonomous flying cameraman

This PhD research project focuses on the development of a robotic flying camera man. Within the Cametron project we aim at making the process of filming an event completely automatic. A human camera camera crew will be replaced by automated static PTZ cameras and UAV-drones sporting a camera and microphone. The movie director will be replaced by a virtual director so that an event, for example a music performance or a sports game, can be captured in real time with minimal human intervention. This project focuses on the virtual camera man part of Cametron, in which an unmanned aerial vehicle (UAV), sporting a PTZ camera and microphone, is robotized such that it totally autonomously can gather video of an event, based on high abstraction level instructions from the virtual director. These instructions are no more detailed than framing a certain (set of) actor(s) in a certain cinematographic shot composition (long shot, shoulder shot, mid shot, close-up, extreme close-up, etc.). The main challenges that have to be conquered in this project are real-time actor detection and tracking, topological localization, image-based visual servoing and motion planning. Indeed, first task is the detection of persons in the UAV's camera. Because of the very broad definition of ‘actor’ in this project (e.g. lecturer, rock star, cyclist), this task requires fast general person detectors and trackers to be developed to keep the right object centered in the image while objects and/or cams move. Tracking trough time can be done by the well-known “tracking-by-detection”-framework, but for this specific application extreme conditions apply, such as unconstrained varying viewpoints and the very limited amount of on-board computing power. Therefore real time person and action tracking algorithms should be implemented on specific hardware that fits on the units. Knowing its viewpoint is crucial for each camera unit. Yet, for the present application, it is not necessary to know the exact metric location. The quality of the source data matters more than exact localization. In order to record data according to the orders of the director, a unit needs to know its approximate, relative position w.r.t. other units and the action. Therefore, a qualitative, topological layout of the camera positions should suffice. A topological model of the (dynamic) environment will be developed in which each flying camera can localize itself. The last challenge in the project is visual servoing. We also want the drones to fly very smooth so that the captured images aren't blurry or shaking, therefor a good control algorithm will be designed for the drones. Moreover, because the UAVs have 6 degrees of freedom, and the PTZ unit mounted on the adds another 3 DOFs, this control system is certainly not trivial. Fortunately, because in this application, the image produced by the on-board camera is most important, rather than the exact position of the UAV, we can use the image-based visual servoing (IBVS) paradigm. The first goal we want to reach is capturing a lesson or presentation with static PTZ cameras that automatically pan, tilt and zoom to follow the speaker. Afterward, we will expand this to track and capture a single person with a single camera mounted on a UAV-drone and the final goal is to capture an event with several flying units collaborating together.Hulens D., ''Towards an autonomous flying cameraman'', Proefschrift voorgedragen tot het behalen van het doctoraat in de industriële ingenieurswetenschappen, KU Leuven, March 2018, Leuven, Belgium.status: publishe

Autonomous flying cameraman with embedded person detection and tracking while applying cinematographic rules

Unmanned Aerial Vehicles (UAVs) enable nu- merous applications such as search and rescue operations, structural inspection of buildings, crop growth analysis in agriculture, performing 3D reconstruction and so on. For such applications, currently the UAV is steered manually. However, in this paper we aim to record semi-professional video footage (e.g. concerts, sport events) using fully autonomous UAVs. Evidently, this is challenging since we need to detect and track the actor on-board a UAV in real-time, while automatically – and smoothly – controlling the UAV based on these detections. For this, all four DOF (Degrees of freedom) are controlled in separate simultaneous control loops by our vision-based algorithms. Furthermore cinematographic rules need to be taken into account (e.g. the rule of thirds) which position the actor at the visually optimal location in the frame. We extensively validated our algorithms: each control loop and the overall final system is thoroughly evaluated with respect to both accuracy and control speed. We show that our system is able to efficiently control the UAV such that professional recordings are obtained.Hulens D., Goedemé T., ''Autonomous flying cameraman with embedded person detection and tracking while applying cinematographic rules'', Proceedings 14th conference on computer and robot vision - CRV 2017, 8 pp., May 17-19, 2017, Edmonton, Alberta, Canada.status: publishe

Real-time vision-based UAV navigation in fruit orchards

Unmanned Aerial Vehicles (UAV) enable numerous agricultural applications such as terrain mapping, monitor crop growth, detecting areas with diseases and so on. For these applications a UAV flies above the terrain and has a global view of the plants. When the individual fruits or plants have to be examined, an oblique view is better, e.g. via an inspection-camera mounted on expensive all-terrain wheeled robots that drive through the orchard. However, in this paper we aim to autonomously navigate through the orchard with a low-cost UAV and cheap sensors (e.g. a webcam). Evidently, this is challenging since every orchard or even every corridor looks different. For this we developed a vision-based system that detects the center and end of the corridor to autonomously navigate the UAV towards the end of the orchard without colliding with the trees. Furthermore extensive experiments were performed to prove that our algorithm is able to navigate through the orchard with high accuracy and in real-time, even on embedded hardware. A connection with a ground station is thus unnecessary which makes the UAV fully autonomous.Hulens D., Vandersteegen M., Goedemé T., ''Real-time vision-based UAV navigation in fruit orchards'', Proceedings 12th international conference on computer vision theory and applications - VISAPP 2017, 6 pp., February 27 - March 1, 2017, Porto, Portugal.status: publishe

How to choose the best embedded processing platform for on-board UAV image processing ?

For a variety of tasks, complex image processing algorithms are a necessity to make UAVs more autonomous. Often, the processing of images of the on-board camera is performed on a ground station, which severely limits the operating range of the UAV. Often, offline processing is used since it is difficult to find a suitable hardware platform to run a specific vision algorithm on-board the UAV. First of all, it is very hard to find a good trade-off between speed, power consumption and weight of a specific hardware platform and secondly, due to the variety of hardware platforms, it is difficult to find a suitable hardware platform and to estimate the speed the user’s algorithm will run on that hardware platform. In this paper we tackle those problems by presenting a framework that automatically determines the most-suited hardware platform for each arbitrary complex vision algorithm. Additionally, our framework estimates the speed, power consumption and flight time of this algorithm for a variety of hardware platforms on a specific UAV. We demonstrate this methodology on two real-life cases and give an overview of the present top processing CPU-based platforms for on-board UAV image processing.Hulens D., Goedemé T., Verbeke J., ''How to choose the best embedded processing platform for on-board UAV image processing ?'', Proceedings 10th international conference on computer vision theory and applications - VISAPP 2015, 10 pp., March 11-14, 2015, Berlin, Germany.status: publishe

Autonomous lecture recording with a PTZ camera while complying with cinematographic rules

Nowadays, many lectures and presentations are recorded and broadcasted for teleteaching applications. When no human camera crew is present, the most obvious choice is for static cameras. In order to enhance the viewing experience, more advanced systems automatically track and steer the camera towards the lecturer. In this paper we propose an even more advanced system that tracks the lecturer while taking cinematographic rules into account. On top of that, the lecturer can be filmed in different types of shots. Our system is able to detect and track the position of the lecturer, even with non-static backgrounds and in difficult illumination. We developed an action axis determination system, needed to apply cinematographic rules and to steer the Pan-Tilt-Zoom (PTZ) camera towards the lecturer.Hulens D., Rumes T., Goedemé T., ''Autonomous lecture recording with a PTZ camera while complying with cinematographic rules'', Proceedings CRV 2014 - the conference on computer and robot vision, pp. 371-377, May 7-9, 2014, Montréal, Québec, Canada.status: publishe

Choosing the Best Embedded Processing Platform for On-Board UAV Image Processing

Nowadays, complex image processing algorithms are a necessity to make UAVs more autonomous. Currently, the processing of images of the on-board camera is often performed on a ground station, thus severely limiting the operating range. On-board processing has numerous advantages, however determining a good trade-off between speed, power consumption and weight of a specific hardware platform for on-board processing is hard. Many hardware platforms exist, and finding the most suited one for a specific vision algorithm is difficult. We present a framework that automatically determines the most-suited hardware platform given an arbitrary complex vision algorithm. Our framework estimates the speed, power consumption and flight time of this algorithm for multiple hardware platforms on a specific UAV. We demonstrate this methodology on two real-life cases and give an overview of the present top performing CPU-based platforms for on-board UAV image processing.Hulens D., Verbeke J., Goedemé T., ''Choosing the best embedded processing platform for on-board UAV image processing'', Computer vision, imaging and computer graphics theory and applications, 10th international joint conference, VISIGRAPP 2015, revised selected papers - communications in computer and information science series, vol. 598, pp. 455-472, 2016, Springer (10th international conference on computer vision theory and applications - VISAPP 2015, March 11-14, 2015, Berlin, Germany).status: publishe

Fast and Accurate Face Orientation Measurement in Low-resolution Images on Embedded Hardware

In numerous applications it is important to collect information about the gaze orientation or head-angle of a person. Examples are measuring the alertness of a car driver to see if he is still awake, or the attentiveness of people crossing a street to see if they noticed the cars driving by. In our own application we want to apply cinematographic rules (e.g. the rule of thirds where a face should be positioned left or right in the frame depending on the gaze direction) on images taken from an Unmanned Aerial Vehicle (UAV). For this an accurate estimation of the angle of the head is needed. These applications should run on embedded hardware so that they can be easily attached to e.g. a car or a UAV. This implies that the head angle detection algorithm should run in real-time on minimal hardware. Therefore we developed an approach that runs in real-time on embedded hardware while achieving excellent accuracy. We demonstrated these approaches on both a publicly available face dataset and our own dataset recorded from a UAV.status: publishe

Autonomous Visual Navigation for a Flower Pollination Drone

In this paper, we present the development of a visual navigation capability for a small drone enabling it to autonomously approach flowers. This is a very important step towards the development of a fully autonomous flower pollinating nanodrone. The drone we developed is totally autonomous and relies for its navigation on a small on-board color camera, complemented with one simple ToF distance sensor, to detect and approach the flower. The proposed solution uses a DJI Tello drone carrying a Maix Bit processing board capable of running all deep-learning-based image processing and navigation algorithms on-board. We developed a two-stage visual servoing algorithm that first uses a highly optimized object detection CNN to localize the flowers and fly towards it. The second phase, approaching the flower, is implemented by a direct visual steering CNN. This enables the drone to detect any flower in the neighborhood, steer the drone towards the flower and make the drone’s pollinating rod touch the flower. We trained all deep learning models based on an artificial dataset with a mix of images of real flowers, artificial (synthetic) flowers and virtually rendered flowers. Our experiments demonstrate that the approach is technically feasible. The drone is able to detect, approach and touch the flowers totally autonomously. Our 10 cm sized prototype is trained on sunflowers, but the methodology presented in this paper can be retrained for any flower type

Deep Diamond Re-ID.

Re-identification neural networks are widely used in numerous applications such as crowd control, crime investi- gations, safety systems and even in most smartphones to unlock the phone with a picture of the owner. These techniques are mostly used to re-identify faces or persons but in this paper we investigate the possibility to adapt these to also re-identify similar looking objects such as diamonds. Since polished diamonds are very similar to the naked eye, it is difficult to distinguish one diamond from another. We have indications that diamonds are sometimes switched by trained switchers with fake or less expensive stones, while they pretend to inspect the stone. A solution to this is diamond fingerprinting. We therefore propose a technique to generate a unique ID for each stone, which allows to re-identify the diamond solely based on an image of the gem. Since each diamond is assigned a unique ID it is even possible to keep track of the diamonds over time. This allows the seller to verify his stones before and after trading while switchers don’t stand a chance. For this task we trained and adapted a classification network optimized for both speed and accuracy.status: submitte

The Design and Construction of a High Endurance Hexacopter suited for Narrow Corridors

We have conceived a novel compound multicopter (helicopter type utilizing multiple different size propellers for separate lift and attitude control) configuration specifically for flight through narrow corridors. Its design combines the contradictory requirements of limited width, high agility and long endurance while carrying a significant payload. This configuration can be scaled for both indoor and outdoor applications. The development is part of a doctoral research in which an autonomous unmanned rotary helicopter is designed, constructed and flight tested for inspecting fruit orchards and vineyards while flying in between the tree rows in outdoor conditions such as wind and gusts. The compound hexacopter configuration combines two large lift propellers, with a constant rotational velocity, with four small control propellers commanded by an autopilot. The autopilot is configured as a quadcopter commanding only the control propellers as only these change the attitude and overall thrust of the hexacopter. The benefit of using large lift propellers is their lower disk loading (thrust divided by disk area) which results in a higher Figure of Merit and lower power consumption compared to the smaller control propellers, while the latter are better suited for outdoor (windy) conditions due to their fast reaction time in spooling up and down. Compared to a standard quadcopter with the same width, payload and battery capacity, the endurance of the compound hexacopter is potentially up to 60% higher. As a concept validator, a small-scale prototype has been designed, constructed and successfully flight tested.status: publishe