12 research outputs found
AlphaPilot: Autonomous Drone Racing
This paper presents a novel system for autonomous, vision-based drone racing
combining learned data abstraction, nonlinear filtering, and time-optimal
trajectory planning. The system has successfully been deployed at the first
autonomous drone racing world championship: the 2019 AlphaPilot Challenge.
Contrary to traditional drone racing systems, which only detect the next gate,
our approach makes use of any visible gate and takes advantage of multiple,
simultaneous gate detections to compensate for drift in the state estimate and
build a global map of the gates. The global map and drift-compensated state
estimate allow the drone to navigate through the race course even when the
gates are not immediately visible and further enable to plan a near
time-optimal path through the race course in real time based on approximate
drone dynamics. The proposed system has been demonstrated to successfully guide
the drone through tight race courses reaching speeds up to 8m/s and ranked
second at the 2019 AlphaPilot Challenge.Comment: Accepted at Robotics: Science and Systems 2020, associated video at
https://youtu.be/DGjwm5PZQT
RGB-D-E: Event Camera Calibration for Fast 6-DOF Object Tracking
Augmented reality devices require multiple sensors to perform various tasks
such as localization and tracking. Currently, popular cameras are mostly
frame-based (e.g. RGB and Depth) which impose a high data bandwidth and power
usage. With the necessity for low power and more responsive augmented reality
systems, using solely frame-based sensors imposes limits to the various
algorithms that needs high frequency data from the environement. As such,
event-based sensors have become increasingly popular due to their low power,
bandwidth and latency, as well as their very high frequency data acquisition
capabilities. In this paper, we propose, for the first time, to use an
event-based camera to increase the speed of 3D object tracking in 6 degrees of
freedom. This application requires handling very high object speed to convey
compelling AR experiences. To this end, we propose a new system which combines
a recent RGB-D sensor (Kinect Azure) with an event camera (DAVIS346). We
develop a deep learning approach, which combines an existing RGB-D network
along with a novel event-based network in a cascade fashion, and demonstrate
that our approach significantly improves the robustness of a state-of-the-art
frame-based 6-DOF object tracker using our RGB-D-E pipeline.Comment: 9 pages, 9 figure
Human-Piloted Drone Racing: Visual Processing and Control
Humans race drones faster than algorithms, despite being limited to a fixed
camera angle, body rate control, and response latencies in the order of
hundreds of milliseconds. A better understanding of the ability of human pilots
of selecting appropriate motor commands from highly dynamic visual information
may provide key insights for solving current challenges in vision-based
autonomous navigation. This paper investigates the relationship between human
eye movements, control behavior, and flight performance in a drone racing task.
We collected a multimodal dataset from 21 experienced drone pilots using a
highly realistic drone racing simulator, also used to recruit professional
pilots. Our results show task-specific improvements in drone racing performance
over time. In particular, we found that eye gaze tracks future waypoints (i.e.,
gates), with first fixations occurring on average 1.5 seconds and 16 meters
before reaching the gate. Moreover, human pilots consistently looked at the
inside of the future flight path for lateral (i.e., left and right turns) and
vertical maneuvers (i.e., ascending and descending). Finally, we found a strong
correlation between pilots eye movements and the commanded direction of
quadrotor flight, with an average visual-motor response latency of 220 ms.
These results highlight the importance of coordinated eye movements in
human-piloted drone racing. We make our dataset publicly available.Comment: 8 pages, 6 figure
Event-based Vision: A Survey
Event cameras are bio-inspired sensors that differ from conventional frame
cameras: Instead of capturing images at a fixed rate, they asynchronously
measure per-pixel brightness changes, and output a stream of events that encode
the time, location and sign of the brightness changes. Event cameras offer
attractive properties compared to traditional cameras: high temporal resolution
(in the order of microseconds), very high dynamic range (140 dB vs. 60 dB), low
power consumption, and high pixel bandwidth (on the order of kHz) resulting in
reduced motion blur. Hence, event cameras have a large potential for robotics
and computer vision in challenging scenarios for traditional cameras, such as
low-latency, high speed, and high dynamic range. However, novel methods are
required to process the unconventional output of these sensors in order to
unlock their potential. This paper provides a comprehensive overview of the
emerging field of event-based vision, with a focus on the applications and the
algorithms developed to unlock the outstanding properties of event cameras. We
present event cameras from their working principle, the actual sensors that are
available and the tasks that they have been used for, from low-level vision
(feature detection and tracking, optic flow, etc.) to high-level vision
(reconstruction, segmentation, recognition). We also discuss the techniques
developed to process events, including learning-based techniques, as well as
specialized processors for these novel sensors, such as spiking neural
networks. Additionally, we highlight the challenges that remain to be tackled
and the opportunities that lie ahead in the search for a more efficient,
bio-inspired way for machines to perceive and interact with the world