Real-time on-board obstacle avoidance for UAVs based on embedded stereo vision

In order to improve usability and safety, modern unmanned aerial vehicles (UAVs) are equipped with sensors to monitor the environment, such as laser-scanners and cameras. One important aspect in this monitoring process is to detect obstacles in the flight path in order to avoid collisions. Since a large number of consumer UAVs suffer from tight weight and power constraints, our work focuses on obstacle avoidance based on a lightweight stereo camera setup. We use disparity maps, which are computed from the camera images, to locate obstacles and to automatically steer the UAV around them. For disparity map computation we optimize the well-known semi-global matching (SGM) approach for the deployment on an embedded FPGA. The disparity maps are then converted into simpler representations, the so called U-/V-Maps, which are used for obstacle detection. Obstacle avoidance is based on a reactive approach which finds the shortest path around the obstacles as soon as they have a critical distance to the UAV. One of the fundamental goals of our work was the reduction of development costs by closing the gap between application development and hardware optimization. Hence, we aimed at using high-level synthesis (HLS) for porting our algorithms, which are written in C/C++, to the embedded FPGA. We evaluated our implementation of the disparity estimation on the KITTI Stereo 2015 benchmark. The integrity of the overall realtime reactive obstacle avoidance algorithm has been evaluated by using Hardware-in-the-Loop testing in conjunction with two flight simulators.Comment: Accepted in the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Scienc

A smart wheelchair system using a combination of stereoscopic and spherical vision cameras

University of Technology, Sydney. Faculty of Engineering and Information Technology.Reports have shown growing numbers of people who fall into the categories of the elderly or those living with some form of disability. Physical and functional impairments are broad-ranging across these groups and the causes are numerous, including strokes, spinal cord injury, spina bifida, multiple sclerosis, muscular dystrophy, and various degenerative disorders. Rehabilitation technologies are a solution to many of these impairments and aim to improve the quality of life for the people who require them. Smart wheelchair developments, in particular, have the purpose of assisting those with mobility disabilities. Providing independence in mobility can have many significant benefits to the users in their daily lives, including improved physical, cognitive, confidence, communication, and social skills. Unfortunately for many, particularly those with tetraplegia (partial or total loss of functionality through illness or injury to all four limbs and torso), there is a serious lack of options available for adequately and safely controlling mobility devices such as wheelchairs. There are few options for hands-free controlling wheelchairs, and furthermore, there are no accessible options for intelligent assistance from the wheelchair to make hands-free control easy and safe. This is a serious issue since the limited hands-free control options available can be difficult to use, resulting in many accidents. There are also new control technology devices emerging in research, such as brain-computer interfaces (BCIs), which could potentially provide an adequate means of control for many people who cannot use currently commercial options, but require intelligent assistance from the wheelchair to make use of such a system safe. In this thesis, the design and development of a new smart wheelchair, named TIM, is introduced to address these issues. The TIM smart wheelchair was created with the intention of providing intelligent assistance during navigation for any hands-free control technology, both currently commercial and new devices produced in research. This aims to vastly improve the options available to the people who are in need of such smart wheelchair developments. A method of utilising stereoscopic cameras for adaptive, real-time vision mapping is presented in this thesis, as cameras are increasingly becoming a more accessible and inexpensive form of artificial sensor. The mapping process in this method involves acquisition from the left and right stereo pair of cameras, which then undergo a range of image pre-processing techniques before being stereo-processed, which includes matching and correlation algorithms, to produce a disparity image. This disparity image contains depth information about the scene, which is then converted into a 3-dimensional (3D) point map, placing all mapped pixels of the environment, and features within, into a 3D plane. Unnecessary information, such as the floor and everything above the maximum height of the TIM smart wheelchair, is removed and the remaining data extracted into a 2-dimensional (2D) bird’s eye view environment map. This mapping representation assists the wheelchair in the later steps of making intelligent navigational decisions based on the relative placement of objects in the environment. Wheel encoders on the drive wheels are also acquired during operation, and odometry change calculations are performed to facilitate the ability of the system to ‘remember’ mapped object points that have passed outside the vision range. This is performed frequently to construct a real-time environment map, and to remember the placement of objects that have moved out of the range of vision, in order to further avoid collisions. This is particularly useful for static environments and creating maps of the static object placements within. A wheel parameter correction process was also employed to increase the accuracy of this mapping process and successfully reduce the errors associated with drive wheel specifications, which in turn can affect the mapping process based on the wheel encoder information. Correction of these parameters helped optimise the ‘memory mapping’ process and reduce skewing and accumulative errors in the maps. A process for intelligent stereo processing parameter selection was designed, as the quality of disparity images, and hence the quality of environmental-mapping, is heavily dependent on the stereo processing parameters, which may work well when set for one environment but produce problems in another. The differences that affected performance between environments were found to mostly be the lighting conditions which resulted from the varying types of environments. As such, this proposed method involves classifying environmental categories in real-time, based on image data available, and adapting the parameters accordingly. The environment types were separated into four categories to account for most encountered environmental situations, being 1) ‘General Illumination Contrast’, 2) ‘Extreme Illumination Contrast’, 3) ‘Consistent Dark’, and 4) ‘Consistent Bright’. The proposed method successfully allowed classification in real-time of the environment categories and adaptation of the stereo processing parameters in accordance, producing a system that can change its settings ‘on the fly’ to suit the environment the wheelchair is navigating through. Limited vision and trouble with dynamic objects were found to be downfalls with the stereoscopic vision, so to address these, methods of utilising a spherical vision camera system were introduced for obstacle detection over a wide vision range. Spherical vision is an extension of monoscopic cameras, producing 360º of panoramic vision. A strategy to utilise these panoramic images is presented, in which the images are separated into segments and ‘Traffic Light’ zones. The segments display different areas of the image representing the allocated areas around the wheelchair. The ‘Traffic Light’ zones within the segments are separated into three categories: 1) Red, meaning an obstruction is present around the wheelchair, 2) Yellow, indicating to take caution as an object is nearby, and 3) Green, meaning there are no objects close to the wheelchair in this segment. Image processing techniques have been assembled as a pre-processing strategy, and neural networks are used for intelligent classification of the segmented images into the zone categories. This method provides a wider range of vision than the stereoscopic cameras alone, and also takes into account the issue of detecting dynamic obstacles, such as people moving around. A unique combination of the stereoscopic cameras and the spherical vision cameras is then introduced. This combination and system configuration is biologically inspired by the equine vision system. Horses inherently have a large vision range, which includes a wide monocular vision range around and a binocular vision overlap ahead of the horse. In accordance with this effective vision system, the camera configuration on the TIM smart wheelchair was modelled similarly and advanced software integration strategies then followed. A method for advanced real-time obstacle avoidance is presented, which utilises algorithms in research, such as Vector Field Histogram (VFH) and Vector Polar Histogram (VPH) methods, and adapts them for use with the specified camera configuration. Further improvement upon the algorithms for this application provides safer obstacle avoidance during navigation in unknown environments, with an added emphasis on making automated navigational decisions towards areas with more available free space. Speed and manner of obstacle avoidance is dependent upon the placement spread of objects in an environment and how close they are to the wheelchair during navigation. Finally, the combination and integration of the automated guidance and obstacle avoidance capabilities of the wheelchair with hands-free control technologies are introduced. The aim of the TIM smart wheelchair system was to effectively provide safe navigation with automated obstacle avoidance in a manner that ultimately executes the user’s intentions for travel. As such, a head-movement control (HMC) device and a brain-computer interface (BCI) device are both separately integrated with the TIM smart wheelchair, providing a display of two new options for hands-free control. Experimental studies were conducted using these two control devices separately, to assess the performance of the TIM smart wheelchair, as well as its ability to carry out user’s navigational intentions safely and effectively. Eight able-bodied participants trialled the system, including four male and four female, with ages ranging from 21 to 56 years old. All of these able-bodied participants have not previously had any experience operating a wheelchair. In addition, two male tetraplegic (C-6 to C-7) participants also completed the experimental study, aged 20 and 33. Both tetraplegic participants are wheelchair users, so these experiments were of great importance. The same tasks applied to all, and included navigating obstacle courses with the use of the head-movement control system and the brain-computer interface for control. Experiment runs were conducted for each control system with automated navigational guidance assistance from TIM, and repeated for some capable participants without the assistance from TIM. This process was also conducted in two types of obstacle courses, being 1) a ‘Static Course’ and 2) a ‘Dynamic Course’, requiring different types of challenges in obstacle avoidance. This provided results to assess the performance and safety of the TIM smart wheelchair in a range of environments and situations. Evaluation of the results displayed the feasibility and effectiveness of the developed TIM smart wheelchair system. This system, once equipped with a unique camera configuration and reliable obstacle avoidance strategies, was able to successfully allow users to control the wheelchair with research-produced hands-free interface devices and effectively navigate safely through challenging environments. The TIM smart wheelchair system is able to adapt to people with various types and levels of physical impairment, and ultimately provide ease-of-use as well as safety during navigation

J-MOD2^{2}: Joint Monocular Obstacle Detection and Depth Estimation

In this work, we propose an end-to-end deep architecture that jointly learns to detect obstacles and estimate their depth for MAV flight applications. Most of the existing approaches either rely on Visual SLAM systems or on depth estimation models to build 3D maps and detect obstacles. However, for the task of avoiding obstacles this level of complexity is not required. Recent works have proposed multi task architectures to both perform scene understanding and depth estimation. We follow their track and propose a specific architecture to jointly estimate depth and obstacles, without the need to compute a global map, but maintaining compatibility with a global SLAM system if needed. The network architecture is devised to exploit the joint information of the obstacle detection task, that produces more reliable bounding boxes, with the depth estimation one, increasing the robustness of both to scenario changes. We call this architecture J-MOD$^{2}$. We test the effectiveness of our approach with experiments on sequences with different appearance and focal lengths and compare it to SotA multi task methods that jointly perform semantic segmentation and depth estimation. In addition, we show the integration in a full system using a set of simulated navigation experiments where a MAV explores an unknown scenario and plans safe trajectories by using our detection model

Depth Image Processing for Obstacle Avoidance of an Autonomous VTOL UAV

We describe a new approach for stereo-based obstacle avoidance. This method analyzes the images of a stereo camera in realtime and searches for a safe target point that can be reached without collision. The obstacle avoidance system is used by our unmanned helicopter ARTIS (Autonomous Rotorcraft Testbed for Intelligent Systems) and its simulation environment. It is optimized for this UAV, but not limited to aircraft systems

Pushbroom Stereo for High-Speed Navigation in Cluttered Environments

We present a novel stereo vision algorithm that is capable of obstacle detection on a mobile-CPU processor at 120 frames per second. Our system performs a subset of standard block-matching stereo processing, searching only for obstacles at a single depth. By using an onboard IMU and state-estimator, we can recover the position of obstacles at all other depths, building and updating a full depth-map at framerate. Here, we describe both the algorithm and our implementation on a high-speed, small UAV, flying at over 20 MPH (9 m/s) close to obstacles. The system requires no external sensing or computation and is, to the best of our knowledge, the first high-framerate stereo detection system running onboard a small UAV

Safe Local Exploration for Replanning in Cluttered Unknown Environments for Micro-Aerial Vehicles

In order to enable Micro-Aerial Vehicles (MAVs) to assist in complex, unknown, unstructured environments, they must be able to navigate with guaranteed safety, even when faced with a cluttered environment they have no prior knowledge of. While trajectory optimization-based local planners have been shown to perform well in these cases, prior work either does not address how to deal with local minima in the optimization problem, or solves it by using an optimistic global planner. We present a conservative trajectory optimization-based local planner, coupled with a local exploration strategy that selects intermediate goals. We perform extensive simulations to show that this system performs better than the standard approach of using an optimistic global planner, and also outperforms doing a single exploration step when the local planner is stuck. The method is validated through experiments in a variety of highly cluttered environments including a dense forest. These experiments show the complete system running in real time fully onboard an MAV, mapping and replanning at 4 Hz.Comment: Accepted to ICRA 2018 and RA-L 201