Vision-based Learning for Drones: A Survey

Drones as advanced cyber-physical systems are undergoing a transformative shift with the advent of vision-based learning, a field that is rapidly gaining prominence due to its profound impact on drone autonomy and functionality. Different from existing task-specific surveys, this review offers a comprehensive overview of vision-based learning in drones, emphasizing its pivotal role in enhancing their operational capabilities under various scenarios. We start by elucidating the fundamental principles of vision-based learning, highlighting how it significantly improves drones' visual perception and decision-making processes. We then categorize vision-based control methods into indirect, semi-direct, and end-to-end approaches from the perception-control perspective. We further explore various applications of vision-based drones with learning capabilities, ranging from single-agent systems to more complex multi-agent and heterogeneous system scenarios, and underscore the challenges and innovations characterizing each area. Finally, we explore open questions and potential solutions, paving the way for ongoing research and development in this dynamic and rapidly evolving field. With growing large language models (LLMs) and embodied intelligence, vision-based learning for drones provides a promising but challenging road towards artificial general intelligence (AGI) in 3D physical world

Adaptive Tracking and Perching for Quadrotor in Dynamic Scenarios

Perching on the moving platforms is a promising solution to enhance the endurance and operational range of quadrotors, which could benefit the efficiency of a variety of air-ground cooperative tasks. To ensure robust perching, tracking with a steady relative state and reliable perception is a prerequisite. This paper presents an adaptive dynamic tracking and perching scheme for autonomous quadrotors to achieve tight integration with moving platforms. For reliable perception of dynamic targets, we introduce elastic visibility-aware planning to actively avoid occlusion and target loss. Additionally, we propose a flexible terminal adjustment method that adapts the changes in flight duration and the coupled terminal states, ensuring full-state synchronization with the time-varying perching surface at various angles. A relaxation strategy is developed by optimizing the tangential relative speed to address the dynamics and safety violations brought by hard boundary conditions. Moreover, we take SE(3) motion planning into account to ensure no collision between the quadrotor and the platform until the contact moment. Furthermore, we propose an efficient spatiotemporal trajectory optimization framework considering full state dynamics for tracking and perching. The proposed method is extensively tested through benchmark comparisons and ablation studies. To facilitate the application of academic research to industry and to validate the efficiency of our scheme under strictly limited computational resources, we deploy our system on a commercial drone (DJI-MAVIC3) with a full-size sport-utility vehicle (SUV). We conduct extensive real-world experiments, where the drone successfully tracks and perches at 30~km/h (8.3~m/s) on the top of the SUV, and at 3.5~m/s with 60{\deg} inclined into the trunk of the SUV

Vision and Learning for Deliberative Monocular Cluttered Flight

Cameras provide a rich source of information while being passive, cheap and lightweight for small and medium Unmanned Aerial Vehicles (UAVs). In this work we present the first implementation of receding horizon control, which is widely used in ground vehicles, with monocular vision as the only sensing mode for autonomous UAV flight in dense clutter. We make it feasible on UAVs via a number of contributions: novel coupling of perception and control via relevant and diverse, multiple interpretations of the scene around the robot, leveraging recent advances in machine learning to showcase anytime budgeted cost-sensitive feature selection, and fast non-linear regression for monocular depth prediction. We empirically demonstrate the efficacy of our novel pipeline via real world experiments of more than 2 kms through dense trees with a quadrotor built from off-the-shelf parts. Moreover our pipeline is designed to combine information from other modalities like stereo and lidar as well if available

Adaptive Airborne Separation to Enable UAM Autonomy in Mixed Airspace

The excitement and promise generated by Urban Air Mobility (UAM) concepts have inspired both new entrants and large aerospace companies throughout the world to invest hundreds of millions in research and development of air vehicles, both piloted and unpiloted, to fulfill these dreams. The management and separation of all these new aircraft have received much less attention, however, and even though NASAs lead is advancing some promising concepts for Unmanned Aircraft Systems (UAS) Traffic Management (UTM), most operations today are limited to line of sight with the vehicle, airspace reservation and geofencing of individual flights. Various schemes have been proposed to control this new traffic, some modeled after conventional air traffic control and some proposing fully automatic management, either from a ground-based entity or carried out on board among the vehicles themselves. Previous work has examined vehicle-based traffic management in the very low altitude airspace within a metroplex called UTM airspace in which piloted traffic is rare. A management scheme was proposed in that work that takes advantage of the homogeneous nature of the traffic operating in UTM airspace. This paper expands that concept to include a traffic management plan usable at all altitudes desired for electric Vertical Takeoff and Landing urban and short-distance, inter-city transportation. The interactions with piloted aircraft operating under both visual and instrument flight rules are analyzed, and the role of Air Traffic Control services in the postulated mixed traffic environment is covered. Separation values that adapt to each type of traffic encounter are proposed, and the relationship between required airborne surveillance range and closure speed is given. Finally, realistic scenarios are presented illustrating how this concept can reliably handle the density and traffic mix that fully implemented and successful UAM operations would entail