Plants Detection, Localization and Discrimination using 3D Machine Vision for Robotic Intra-row Weed Control

Weed management is vitally important in crop production systems. However, conventional herbicide-based weed control can lead to negative environmental impacts. Manual weed control is laborious and impractical for large scale production. Robotic weeding offers a possibility of controlling weeds precisely, particularly for weeds growing close to or within crop rows. The fusion of two-dimensional textural images and three-dimensional spatial images to recognize and localize crop plants at different growth stages were investigated. Images of different crop plants at different growth stages with weeds were acquired. Feature extraction algorithms were developed, and different features were extracted and used to train plant and background classifiers, which also addressed the problems of canopy occlusion and leaf damage. Then, the efficacy and accuracy of the proposed methods in classification were demonstrated by experiments. Currently, the algorithms were only developed and tested for broccoli and lettuce. For broccoli plants, the crop plants detection true positive rate was 93.1%, and the false discover rate was 1.1%, with the average crop-plant-localization error of 15.9 mm. For lettuce plants, the crop plants detection true positive rate was 92.3%, and the false discover rate was 4.0%, with the average crop-plant-localization error of 8.5 mm. The results have shown that 3D imaging based plant recognition algorithms are effective and reliable for crop/weed differentiation

Image Segmentation from RGBD Images by 3D Point Cloud Attributes and High-Level Features

In this paper, an approach is developed for segmenting an image into major surfaces and potential objects using RGBD images and 3D point cloud data retrieved from a Kinect sensor. In the proposed segmentation algorithm, depth and RGB data are mapped together. Color, texture, XYZ world coordinates, and normal-, surface-, and graph-based segmentation index features are then generated for each pixel point. These attributes are used to cluster similar points together and segment the image. The inclusion of new depth-related features provided improved segmentation performance over RGB-only algorithms by resolving illumination and occlusion problems that cannot be handled using graph-based segmentation algorithms, as well as accurately identifying pixels associated with the main structure components of rooms (walls, ceilings, floors). Since each segment is a potential object or structure, the output of this algorithm is intended to be used for object recognition. The algorithm has been tested on commercial building images and results show the usability of the algorithm in real time applications

Plant Localization and Discrimination using 2D+3D Computer Vision for Robotic Intra-row Weed Control

Weed management is vitally important in crop production systems. However, conventional herbicide based weed control can lead to negative environmental impacts. Manual weed control is laborious and impractical for large scale production. Robotic weed control offers a possibility of controlling weeds precisely, particularly for weeds growing near or within crop rows. A computer vision system was developed based on Kinect V2 sensor, using the fusion of two-dimensional textural data and three-dimensional spatial data to recognize and localized crop plants different growth stages. Images were acquired of different plant species such as broccoli, lettuce and corn at different growth stages. A database system was developed to organize these images. Several feature extraction algorithms were developed which addressed the problems of canopy occlusion and damaged leaves. With our proposed algorithms, different features were extracted and used to train plant and background classifiers. Finally, the efficiency and accuracy of the proposed classification methods were demonstrated and validated by experiments

Robot Mapping and Navigation in Real-World Environments

Robots can perform various tasks, such as mapping hazardous sites, taking part in search-and-rescue scenarios, or delivering goods and people. Robots operating in the real world face many challenges on the way to the completion of their mission. Essential capabilities required for the operation of such robots are mapping, localization and navigation. Solving all of these tasks robustly presents a substantial difficulty as these components are usually interconnected, i.e., a robot that starts without any knowledge about the environment must simultaneously build a map, localize itself in it, analyze the surroundings and plan a path to efficiently explore an unknown environment. In addition to the interconnections between these tasks, they highly depend on the sensors used by the robot and on the type of the environment in which the robot operates. For example, an RGB camera can be used in an outdoor scene for computing visual odometry, or to detect dynamic objects but becomes less useful in an environment that does not have enough light for cameras to operate. The software that controls the behavior of the robot must seamlessly process all the data coming from different sensors. This often leads to systems that are tailored to a particular robot and a particular set of sensors. In this thesis, we challenge this concept by developing and implementing methods for a typical robot navigation pipeline that can work with different types of the sensors seamlessly both, in indoor and outdoor environments. With the emergence of new range-sensing RGBD and LiDAR sensors, there is an opportunity to build a single system that can operate robustly both in indoor and outdoor environments equally well and, thus, extends the application areas of mobile robots. The techniques presented in this thesis aim to be used with both RGBD and LiDAR sensors without adaptations for individual sensor models by using range image representation and aim to provide methods for navigation and scene interpretation in both static and dynamic environments. For a static world, we present a number of approaches that address the core components of a typical robot navigation pipeline. At the core of building a consistent map of the environment using a mobile robot lies point cloud matching. To this end, we present a method for photometric point cloud matching that treats RGBD and LiDAR sensors in a uniform fashion and is able to accurately register point clouds at the frame rate of the sensor. This method serves as a building block for the further mapping pipeline. In addition to the matching algorithm, we present a method for traversability analysis of the currently observed terrain in order to guide an autonomous robot to the safe parts of the surrounding environment. A source of danger when navigating difficult to access sites is the fact that the robot may fail in building a correct map of the environment. This dramatically impacts the ability of an autonomous robot to navigate towards its goal in a robust way, thus, it is important for the robot to be able to detect these situations and to find its way home not relying on any kind of map. To address this challenge, we present a method for analyzing the quality of the map that the robot has built to date, and safely returning the robot to the starting point in case the map is found to be in an inconsistent state. The scenes in dynamic environments are vastly different from the ones experienced in static ones. In a dynamic setting, objects can be moving, thus making static traversability estimates not enough. With the approaches developed in this thesis, we aim at identifying distinct objects and tracking them to aid navigation and scene understanding. We target these challenges by providing a method for clustering a scene taken with a LiDAR scanner and a measure that can be used to determine if two clustered objects are similar that can aid the tracking performance. All methods presented in this thesis are capable of supporting real-time robot operation, rely on RGBD or LiDAR sensors and have been tested on real robots in real-world environments and on real-world datasets. All approaches have been published in peer-reviewed conference papers and journal articles. In addition to that, most of the presented contributions have been released publicly as open source software

Automated Semantic Content Extraction from Images

In this study, an automatic semantic segmentation and object recognition methodology is implemented which bridges the semantic gap between low level features of image content and high level conceptual meaning. Semantically understanding an image is essential in modeling autonomous robots, targeting customers in marketing or reverse engineering of building information modeling in the construction industry. To achieve an understanding of a room from a single image we proposed a new object recognition framework which has four major components: segmentation, scene detection, conceptual cueing and object recognition. The new segmentation methodology developed in this research extends Felzenswalb\u27s cost function to include new surface index and depth features as well as color, texture and normal features to overcome issues of occlusion and shadowing commonly found in images. Adding depth allows capturing new features for object recognition stage to achieve high accuracy compared to the current state of the art. The goal was to develop an approach to capture and label perceptually important regions which often reflect global representation and understanding of the image. We developed a system by using contextual and common sense information for improving object recognition and scene detection, and fused the information from scene and objects to reduce the level of uncertainty. This study in addition to improving segmentation, scene detection and object recognition, can be used in applications that require physical parsing of the image into objects, surfaces and their relations. The applications include robotics, social networking, intelligence and anti-terrorism efforts, criminal investigations and security, marketing, and building information modeling in the construction industry. In this dissertation a structural framework (ontology) is developed that generates text descriptions based on understanding of objects, structures and the attributes of an image