Deep neural networks for grape bunch segmentation in natural images from a consumer-grade camera

AbstractPrecision agriculture relies on the availability of accurate knowledge of crop phenotypic traits at the sub-field level. While visual inspection by human experts has been traditionally adopted for phenotyping estimations, sensors mounted on field vehicles are becoming valuable tools to increase accuracy on a narrower scale and reduce execution time and labor costs, as well. In this respect, automated processing of sensor data for accurate and reliable fruit detection and characterization is a major research challenge, especially when data consist of low-quality natural images. This paper investigates the use of deep learning frameworks for automated segmentation of grape bunches in color images from a consumer-grade RGB-D camera, placed on-board an agricultural vehicle. A comparative study, based on the estimation of two image segmentation metrics, i.e. the segmentation accuracy and the well-known Intersection over Union (IoU), is presented to estimate the performance of four pre-trained network architectures, namely the AlexNet, the GoogLeNet, the VGG16, and the VGG19. Furthermore, a novel strategy aimed at improving the segmentation of bunch pixels is proposed. It is based on an optimal threshold selection of the bunch probability maps, as an alternative to the conventional minimization of cross-entropy loss of mutually exclusive classes. Results obtained in field tests show that the proposed strategy improves the mean segmentation accuracy of the four deep neural networks in a range between 2.10 and 8.04%. Besides, the comparative study of the four networks demonstrates that the best performance is achieved by the VGG19, which reaches a mean segmentation accuracy on the bunch class of 80.58%, with IoU values for the bunch class of 45.64%

Three-dimensional quantitative characterization of grapes morphology and possible relation with grey mould susceptibility

Grey mould is one of the most important diseases of grapevine in the Mediterranean regions caused by the fungi Botrytis cinerea. Many factors are responsible for this disease among them, the morphology of grapes plays a crucial role in grey mould infection. The grapes with highly compact berries are the most susceptible to infection. The common methods applied to evaluate the compactness of grapes cannot apply to grapevine bunches from the same variety. Therefore, novel methods are used to detect compactness by image processing analyses such as photogrammetry for 3D model reconstruction. This study proposes an alternative analysis of bunch morphology and compaction assessment based on virtual 3D models. Seventeen Pinot Gris clones and six Pinot Noir clones were manually collected at harvest time, and the grey mould severity evaluation was carried out in the field. All the grapes were photographed at different angulations, and the 3D model reconstruction was performed by the photogrammetry technique. Several measures and indexes were extracted from each bunch. Principal component analysis (PCA) and two multiple linear regression models (MLR) were applied to identify the descriptors of the clones most related to grey mould infection. The first model assessed the correlation between the grey mould severity and the descriptors from the 2D analysis, while the second model analyzed both descriptors from the 2D and 3D analysis. The 3D MLR presented higher performances than the 2D MLR. The R-square value (R2) and the root mean square error (RMSE) were compared between models. For Pinot Gris, the R2 rose from 0.656 to 0.838, moving from the 2D to the 3D MLR, while the RMSE decreased from 1.713 to 1.175. In Pinot Noir, the 2D model did not provide sufficient robustness, while the proposed MLR estimated R2 with 0.936 value and RMSE with 0.29 value. Additional studies were performed by analyzing the data with graphs and statistics. Consequently, the most significant traits include the estimated empty volume, the width of the grape, weight, volume, shape, and the ratio between surface and height.Grey mould is one of the most important diseases of grapevine in the Mediterranean regions caused by the fungi Botrytis cinerea. Many factors are responsible for this disease among them, the morphology of grapes plays a crucial role in grey mould infection. The grapes with highly compact berries are the most susceptible to infection. The common methods applied to evaluate the compactness of grapes cannot apply to grapevine bunches from the same variety. Therefore, novel methods are used to detect compactness by image processing analyses such as photogrammetry for 3D model reconstruction. This study proposes an alternative analysis of bunch morphology and compaction assessment based on virtual 3D models. Seventeen Pinot Gris clones and six Pinot Noir clones were manually collected at harvest time, and the grey mould severity evaluation was carried out in the field. All the grapes were photographed at different angulations, and the 3D model reconstruction was performed by the photogrammetry technique. Several measures and indexes were extracted from each bunch. Principal component analysis (PCA) and two multiple linear regression models (MLR) were applied to identify the descriptors of the clones most related to grey mould infection. The first model assessed the correlation between the grey mould severity and the descriptors from the 2D analysis, while the second model analyzed both descriptors from the 2D and 3D analysis. The 3D MLR presented higher performances than the 2D MLR. The R-square value (R2) and the root mean square error (RMSE) were compared between models. For Pinot Gris, the R2 rose from 0.656 to 0.838, moving from the 2D to the 3D MLR, while the RMSE decreased from 1.713 to 1.175. In Pinot Noir, the 2D model did not provide sufficient robustness, while the proposed MLR estimated R2 with 0.936 value and RMSE with 0.29 value. Additional studies were performed by analyzing the data with graphs and statistics. Consequently, the most significant traits include the estimated empty volume, the width of the grape, weight, volume, shape, and the ratio between surface and height

Assessing Berry Number for Grapevine Yield Estimation by Image Analysis: Case Study with the Red Variety “Syrah”

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoThe yield estimation provides information that help growers to make decisions in order to optimize crop growth and to organize the harvest operations in field and in the cellar. In most vineyard estates yield is forecasted using manual methods. However, image analysis methods, which are less invasive low cost and more representative are now being developed. The main objective of this work was to estimate yield through data obtained in the frame of Vinbot project during the 2019 season. In this thesis, images of the grapevine variety Syrah taken in the laboratory and in the vineyards of the “Instituto Superior de Agronomia” in Lisbon were analyzed. In the laboratory the images were taken manually with an RGB camera, while in the field vines were imaged either manually and by the Vinbot robot. From these images, the number of visible berries were counted with MATLAB. From the laboratory values, the relationships between the number of visible berries and actual bunch weight and berry number were studied. From the data obtained in the field, it was analyzed the visibility of the berries at different levels of defoliation and the relationship between the area of visible bunches and the visible berries. Berry-by-berry occlusion showed a value of 6.4% at pea-size, 14.5% at veraison and 25% at maturation. In addition, high and significant determination coefficient were obtained between actual yield and visible berries. The comparison of estimated yield, obtained using the regression models with actual yield, showed an underestimation at all the three phonological stages. This low accuracy of the developed models show that the use of algorithms based on visible berry number on the images to estimate yield still needs further researchN/

Identifying veraison process of colored wine grapes in field conditions combining deep learning and image analysis

Acknowledgments This work was supported by the National Key R&D Program Project of China (Grant No. 2019YFD1002500) and Guangxi Key R&D Program Project (Grant No. Gui Ke AB21076001) The authors would like to thank the anonymous reviewers for their helpful comments and suggestions.Peer reviewedPostprin

Assessing berry number for grapevine yield estimation by image analysis: case study with the white variety “Encruzado”

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoNowadays, yield estimation represents one of the most important topics in viticulture. It can lead to a better vineyard management and to a better organization of harvesting operations in the vineyard and in the cellar. In recent years, image analysis has become an important tool to improve yield forecast, with the advantages of saving time and being non-invasive. This research aims to estimate the yield of the white cultivar ‘Encruzado’ using visible berry number counted in the images aquired at veraison and near harvest, using a manual RGB camera and the robot VINBOT. Images were collected in laboratory and in the field at the experimental vineyard of the Instituto Superior de Agronomia (ISA) in Lisbon. In the field images the number of visible berries per canopy meter was higher at maturation than at veraison, respectively 72.6 and 66.3. Regarding the percentage of visible berries, 30.2% where visible at veraison and 24.1% at maturation. Concerning percentage of berries occluded by other berries it was observed 28.7% at veraison and 24.3% at maturation. Regression analysis showed that the number of berries in the image explained a very high proportion of bunch weight variability, R2=0.64 at veraison and 0.91 at maturation. Regression analysis also showed that the canopy porosity explained a very high proportion of visible berries variability, R2=0.81 at veraison and 0.88 at maturation. The obtained regression models underestimated the yield with an higher error at veraison than at maturation. This underestimation indicates that the use of visible berry number on the images to estimate yield still needs further research to improve the algorithms accuracyN/

A survey of image processing techniques for agriculture

Computer technologies have been shown to improve agricultural productivity in a number of ways. One technique which is emerging as a useful tool is image processing. This paper presents a short survey on using image processing techniques to assist researchers and farmers to improve agricultural practices. Image processing has been used to assist with precision agriculture practices, weed and herbicide technologies, monitoring plant growth and plant nutrition management. This paper highlights the future potential for image processing for different agricultural industry contexts

Development of a new non-invasive vineyard yield estimation method based on image analysis

Doutoramento em Engenharia Agronómica / Instituto Superior de Agronomia. Universidade de LisboaPredicting vineyard yield with accuracy can provide several advantages to the whole vine and wine industry. Today this is majorly done using manual and sometimes destructive methods, based on bunch samples. Yield estimation using computer vision and image analysis can potentially perform this task extensively, automatically, and non-invasively. In the present work this approach is explored in three main steps: image collection, occluded fruit estimation and image traits conversion to mass. On the first step, grapevine images were collected in field conditions along some of the main grapevine phenological stages. Visible yield components were identified in the image and compared to ground truth. When analyzing inflorescences and bunches, more than 50% were occluded by leaves or other plant organs, on three cultivars. No significant differences were observed on bunch visibility after fruit set. Visible bunch projected area explained an average of 49% of vine yield variation, between veraison and harvest. On the second step, vine images were collected, in field conditions, with different levels of defoliation intensity at bunch zone. A regression model was computed combining canopy porosity and visible bunch area, obtained via image analysis, which explained 70-84% of bunch exposure variation. This approach allowed for an estimation of the occluded fraction of bunches with average errors below |10|%. No significant differences were found between the model’s output at veraison and harvest. On the last step, the conversion of bunch image traits into mass was explored in laboratory and field conditions. In both cases, cultivar differences related to bunch architecture were found to affect weight estimation. A combination of derived variables which included visible bunch area, estimated total bunch area, visible bunch perimeter, visible berry number and bunch compactness was used to estimate yield on undisturbed grapevines. The final model achieved a R2 = 0.86 between actual and estimated yield (n = 213). If performed automatically, the final approach suggested in this work has the potential to provide a non-invasive method that can be performed accurately across whole vineyards.N/