Constraint-based automated reconstruction of grape bunches from 3D range data for high-throughput phenotyping

With increasing global population, the resources for agriculture required to feed the growing number of people are becoming scarce. Estimates expect that by 2050, 60 % more food will be necessary. Nowadays, 70 % of fresh water is used by agriculture and experts see no potential for new land to use for crop plants. This means that existing land has to be used efficiently in a sustainable way. To support this, plant breeders aim at the improvement of yield, quality, disease-resistance, and other important characteristics of the crops. Reports show that grapevine cultivation uses more than three times of the amount of fungicides than the cultivation of fruit trees or vegetables. This is caused by grapevine being prone to various fungal diseases and pests that quickly spread over fields. A loose grape bunch architecture is one of the most important physical barriers that make the establishment of a fungal infection less likely. The grape bunch architecture is mostly defined by the inner stem skeleton. The phenotyping of grape bunches refers to the measurement of the phenotypes, i.e., the observable traits of a plant, like the diameter of berries or the lengths of stems. Because of their perishable nature, grape bunches have to be processed in a relatively short time. On the other hand, genetic analyses require data from a large number of them. Manual phenotyping is error-prone and highly labor- and time-intensive, yielding the need for automated, high-throughput methods. The objective of this thesis is to develop a completely automated pipeline that gets as input a 3D pointcloud showing a grape bunch and computes a 3D reconstruction of the complete grape bunch, including the inner stem skeleton. The result is a 3D estimation of the grape bunch that represents not only dimensions (e.g. berry diameters) or statistics (e.g. the number of berries), but the geometry and topology as well. All architectural (i.e., geometrical and topological) traits can be derived from this complete 3D reconstruction. We aim at high-throughput phenotyping by automatizing all steps and removing any requirement for interaction with the user, while still providing an interface for a detailed visualization and possible adjustments of the parameters. There are several challenges to this task: ripe grape bunches are subject to a high amount of self-occlusion, rendering a direct reconstruction of the stem skeleton impossible. The stem skeleton structure is complex, thus, the manual creation of training data is hard. We aim at a cross-cultivation approach and there is high variability between cultivars and even between grape bunches of the same cultivar. Thus, we cannot rely on statistical distributions for single plant organ dimensions. We employ geometrical and topological constraints to meet the challenge of cross-cultivar optimization and foster efficient sampling of infinitely large hypotheses spaces, resulting in Pearson correlation coefficients between 0.7 and 0.9 for established traits traditionally used by breeders. The active working time is reduced by a factor of 12. We evaluate the pipeline for the application on scans taken in a lab environment and in the field

Vineyard yield estimation using image analysis – a review

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoYield estimation is one of the main goals of the wine industry, this because with an accurate yield estimation it is possible to have a significant reduction in production costs and a better management of the wine industry. Traditional methods for yield estimation are laborious and time consuming, for these reasons in the last years we are witnessing to the development of new methodologies, most of which are based on image analysis. Thanks to the continuous updating and improvement of the computer vision techniques and of the robotic platforms, image analysis applied to the yield estimation is becoming more and more efficient. In fact the results shown by the different studies are very satisfying, at least as regards the estimation of what is possible to see, while are under development several procedures which have the objective to estimate what is not possible to see, due to bunch occlusion by leaves and by others clusters. I this work the different methodologies and the different approaches used for yield estimation are described, including both traditional methods and new approaches based on image analysis, in order to present the advantages and disadvantages of each of themN/

Symmetry-based 3D shape completion for fruit localisation for harvesting robots

Fruit localisation is a crucial step in developing a robotic fruit-harvesting system. This paper aims to improve the localisation accuracy of fruits in 3D space. In the machine vision system of a harvesting robot, in a single view the visible area of a target is often incomplete and therefore, cannot be directly used to accurately determine the target location. A 3D shape completion method is proposed that can be used on the partially visible images of strawberries obtained from a single view. This method proposed a given number of symmetric plane candidates based on the assumption that the targets are symmetrical, which is normally true for fruits such as such apples, citrus fruits and strawberries. Corresponding rating rules were proposed to select the optimal symmetry to be used for the shape completion. The algorithm was then tested on reconstructed point clouds and implemented on a strawberry harvester equipped with a Red Green Blue-Depth (RGB-D) camera. The evaluation on reconstructed strawberry data showed that the intersection over union (IoU) and centre deviation between the results obtained by this method and ground truth were 0.77 and 6.9 mm, respectively, whilst those of the unprocessed partial data were 0.56 and 14.1 mm. The evaluation results of the strawberry data captured with the RGB-D camera showed that the IoU and centre deviation between the results obtained by this method and ground truth were 0.61 and 5.7 mm, respectively, whilst those of the unprocessed partial data were 0.47 and 8.9 mm

Development of a new non-invasive vineyard yield estimation method based on image analysis

Doutoramento em Engenharia Agronómica / Instituto Superior de Agronomia. Universidade de LisboaPredicting vineyard yield with accuracy can provide several advantages to the whole vine and wine industry. Today this is majorly done using manual and sometimes destructive methods, based on bunch samples. Yield estimation using computer vision and image analysis can potentially perform this task extensively, automatically, and non-invasively. In the present work this approach is explored in three main steps: image collection, occluded fruit estimation and image traits conversion to mass. On the first step, grapevine images were collected in field conditions along some of the main grapevine phenological stages. Visible yield components were identified in the image and compared to ground truth. When analyzing inflorescences and bunches, more than 50% were occluded by leaves or other plant organs, on three cultivars. No significant differences were observed on bunch visibility after fruit set. Visible bunch projected area explained an average of 49% of vine yield variation, between veraison and harvest. On the second step, vine images were collected, in field conditions, with different levels of defoliation intensity at bunch zone. A regression model was computed combining canopy porosity and visible bunch area, obtained via image analysis, which explained 70-84% of bunch exposure variation. This approach allowed for an estimation of the occluded fraction of bunches with average errors below |10|%. No significant differences were found between the model’s output at veraison and harvest. On the last step, the conversion of bunch image traits into mass was explored in laboratory and field conditions. In both cases, cultivar differences related to bunch architecture were found to affect weight estimation. A combination of derived variables which included visible bunch area, estimated total bunch area, visible bunch perimeter, visible berry number and bunch compactness was used to estimate yield on undisturbed grapevines. The final model achieved a R2 = 0.86 between actual and estimated yield (n = 213). If performed automatically, the final approach suggested in this work has the potential to provide a non-invasive method that can be performed accurately across whole vineyards.N/

Exploiting Spatio-Temporal Coherence for Video Object Detection in Robotics

This paper proposes a method to enhance video object detection for indoor environments in robotics. Concretely, it exploits knowledge about the camera motion between frames to propagate previously detected objects to successive frames. The proposal is rooted in the concepts of planar homography to propose regions of interest where to find objects, and recursive Bayesian filtering to integrate observations over time. The proposal is evaluated on six virtual, indoor environments, accounting for the detection of nine object classes over a total of ∼ 7k frames. Results show that our proposal improves the recall and the F1-score by a factor of 1.41 and 1.27, respectively, as well as it achieves a significant reduction of the object categorization entropy (58.8%) when compared to a two-stage video object detection method used as baseline, at the cost of small time overheads (120 ms) and precision loss (0.92).</p

Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

High throughput analysis of leaf chlorophyll content in sorghum using RGB, hyperspectral, and fluorescence imaging and sensor fusion

Background: Leaf chlorophyll content plays an important role in indicating plant stresses and nutrient status. Traditional approaches for the quantification of chlorophyll content mainly include acetone ethanol extraction, spectrophotometry and high-performance liquid chromatography. Such destructive methods based on laboratory procedures are time consuming, expensive, and not suitable for high-throughput analysis. High throughput imaging techniques are now widely used for non-destructive analysis of plant phenotypic traits. In this study three imaging modules (RGB, hyperspectral, and fluorescence imaging) were, separately and in combination, used to estimate chlorophyll content of sorghum plants in a greenhouse environment. Color features, spectral indices, and chlorophyll fluorescence intensity were extracted from these three types of images, and multiple linear regression models and PLSR (partial least squares regression) models were built to predict leaf chlorophyll content (measured by a handheld leaf chlorophyll meter) from the image features. Results: The models with a single color feature from RGB images predicted chlorophyll content with R2 ranging from 0.67 to 0.88. The models using the three spectral indices extracted from hyperspectral images (Ration Vegetation Index, Normalized Difference Vegetation Index, and Modified Chlorophyll Absorption Ratio Index) predicted chlorophyll content with R2 ranging from 0.77 to 0.78. The model using the fluorescence intensity extracted from fluorescence images predicted chlorophyll content with R2 of 0.79. The PLSR model that involved all the image features extracted from the three different imaging modules exhibited the best performance for predicting chlorophyll content, with R2 of 0.90. It was also found that inclusion of SLW (Specific Leaf Weight) into the image-based models further improved the chlorophyll prediction accuracy. Conclusion: All three imaging modules (RGB, hyperspectral, and fluorescence) tested in our study alone could estimate chlorophyll content of sorghum plants reasonably well. Fusing image features from different imaging modules with PLSR modeling significantly improved the predictive performance. Image-based phenotyping could provide a rapid and non-destructive approach for estimating chlorophyll content in sorghum

Clarification of Water Stress in Apple Seedlings Using HSI Texture with Machine Learning Technique

Apples are known for their nutrition and economic value. Accurate and rapid diagnosis of water status in apple seedlings on an individual rootstock basis is a prerequisite for precision water management. This study presents a rapid and non-destructive approach for estimating water content in apple seedlings at leaf levels. A PIKA L system collects hyperspectral images(400-1000nm) of apple leaves. To the author's knowledge, no prior work was conducted using the spectral-texture approach in plant water stress. Our research extracts spatial information, gray-level co-occurrence matrix (GLCM), from feature wavelength images of hypercubes. Machine learning methods are applied to these spatial feature matrixs to identify apple leaves under different water stresses. In addition, differences in spectral responses were analysed using machine learning techniques for sorting apple seedlings with varying water treatments (dry, normal, and overwatering). Also, we measure chlorophyll to determine the relationship between hyperspectral characteristics and physiological changes. The achievements of the research indicate that the fusion of texture and hyperspectral imaging coupled with machine learning techniques is promising and presents a powerful potential to determine the water stress in the leaves of apple seedlings

Highlighting Water Stress in Apple Seedlings Using HSI Texture with Machine Learning Technique

Apples are known for their nutrition and economic value. Accurate and rapid diagnosis of water status in apple seedlings on an individual rootstock basis is a prerequisite for precision water management. This study presents a rapid and non-destructive approach for estimating water content in apple seedlings at leaf levels. A PIKA L system collects hyperspectral images (400-1000nm) of apple leaves. Our research extracts spatial information, gray-level co-occurrence matrix (GLCM), from feature wavelength images of hypercubes. Machine learning methods are applied to these spatial feature matrixs to identify apple leaves under different water stresses. In addition, differences in spectral responses were analysed using machine learning techniques for sorting apple seedlings with varying water treatments (dry, normal, and overwatering). Also, we measure chlorophyll to determine the relationship between hyperspectral characteristics and physiological changes. The achievements of the research indicate that the fusion of texture and hyperspectral imaging coupled with machine learning techniques is promising and presents a powerful potential to determine the water stress in the leaves of apple seedlings