Applications of Image Processing in Viticulture: A Review

The production of high quality grapes for wine making is challenging. Significant progress has been made in the automated prediction of harvest yields from images but the analysis of images to predict the quality of the harvest has yet to be fully addressed. The quality of wine produced depends in part on the quality of the grapes harvested and therefore on the presence of disease in the vineyard. There is potential for automated early detection of disease in grape crops through the development of accurate techniques for image processing. This paper presents a review of current research and highlights some of the key challenges for geo-computation (image processing, computer vision and data mining techniques) to inform the management of vineyards and highlights the key challenges for in-field image capture and analysis. An exploration of potential applications for the knowledge generated by imaging techniques is then presented. This discussion is driven by the current interest in the effect of rapid and dramatic climate change on the production of wine and focuses on how this information might be utilized to inform the design and validation of accurate predictive models

High-throughput phenotyping of yield parameters for modern grapevine breeding

Weinbau wird auf 1% der deutschen Agrarfläche betrieben. Auf dieser vergleichsweise kleinen Anbaufläche wird jedoch ein Drittel aller in der deutschen Landwirtschaft verwendeten Fungizide appliziert, was auf die Einführung von Schaderregern im 19. Jahrhundert zurück zu führen ist. Für einen nachhaltigen Anbau ist eine Reduktion des Pflanzenschutzmittelaufwands dringend notwendig. Dieses Ziel kann durch die Züchtung und den Anbau neuer, pilzwiderstandsfähiger Rebsorten erreicht werden. Die Rebenzüchtung als solche ist sehr zeitaufwendig, da die Entwicklung neuer Rebsorten 20 bis 25 Jahre dauert. Der Einsatz der markergestützten Selektion (MAS) erhöht die Effizienz der Selektion in der Rebenzüchtung fortwährend. Eine weitere Effizienzsteigerung ist mit der andauernden Verbesserung der Hochdurchsatz Genotypisierung zu erwarten. Im Vergleich zu den Methoden der Genotypisierung ist die Qualität, Objektivität und Präzision der traditionellen Phänotypisierungsmethoden begrenzt. Die Effizienz in der Rebenzüchtung soll mit der Entwicklung von Hochdurchsatz Methoden zur Phänotypisierung durch sensorgestützte Selektion weiter gesteigert werden. Hierfür sind bisher vielfältige Sensortechniken auf dem Markt verfügbar. Das Spektrum erstreckt sich von RGB-Kameras über Multispektral-, Hyperspektral-, Wärmebild- und Fluoreszenz- Kameras bis hin zu 3D-Techniken und Laserscananwendungen. Die Phänotypisierung von Pflanzen kann unter kontrollierten Bedingungen in Klimakammern oder Gewächshäusern beziehungsweise im Freiland stattfinden. Die Möglichkeit einer standardisierten Datenaufnahme nimmt jedoch kontinuierlich ab. Bei der Rebe als Dauerkultur erfolgt die Aufnahme äußerer Merkmale, mit Ausnahme junger Sämlinge, deshalb auch überwiegend im Freiland. Variierende Lichtverhältnisse, Ähnlichkeit von Vorder- und Hintergrund sowie Verdeckung des Merkmals stellen aus methodischer Sicht die wichtigsten Herausforderungen in der sensorgestützen Merkmalserfassung dar. Bis heute erfolgt die Aufnahme phänotypischer Merkmale im Feld durch visuelle Abschätzung. Hierbei werden die BBCH Skala oder die OIV Deskriptoren verwendet. Limitierende Faktoren dieser Methoden sind Zeit, Kosten und die Subjektivität bei der Datenerhebung. Innerhalb des Züchtungsprogramms kann daher nur ein reduziertes Set an Genotypen für ausgewählte Merkmale evaluiert werden. Die Automatisierung, Präzisierung und Objektivierung phänotypischer Daten soll dazu führen, dass (1) der bestehende Engpass an phänotypischen Methoden verringert, (2) die Effizienz der Rebenzüchtung gesteigert, und (3) die Grundlage zukünftiger genetischer Studien verbessert wird, sowie (4) eine Optimierung des weinbaulichen Managements stattfindet. Stabile und über die Jahre gleichbleibende Erträge sind für eine Produktion qualitativ hochwertiger Weine notwendig und spielen daher eine Schlüsselrolle in der Rebenzüchtung. Der Fokus dieser Studie liegt daher auf Ertragsmerkmalen wie der Beerengröße, Anzahl der Beeren pro Traube und Menge der Trauben pro Weinstock. Die verwandten Merkmale Traubenarchitektur und das Verhältnis von generativem und vegetativem Wachstum wurden zusätzlich bearbeitet. Die Beurteilung von Ertragsmerkmalen auf Einzelstockniveau ist aufgrund der genotypischen Varianz und der Vielfältigkeit des betrachteten Merkmals komplex und zeitintensiv. Als erster Schritt in Richtung Hochdurchsatz (HT) Phänotypisierung von Ertragsmerkmalen wurden zwei voll automatische Bildinterpretationsverfahren für die Anwendung im Labor entwickelt. Das Cluster Analysis Tool (CAT) ermöglicht die bildgestützte Erfassung der Traubenlänge, -breite und -kompaktheit, sowie der Beerengröße. Informationen über Anzahl, Größe (Länge, Breite) und das Volumen der einzelnen Beeren liefert das Berry Analysis Tool (BAT). Beide Programme ermöglichen eine gleichzeitige Erhebung mehrerer, präziser phänotypischer Merkmale und sind dabei schnell, benutzerfreundlich und kostengünstig. Die Möglichkeit, den Vorder- und Hintergrund in einem Freilandbild zu unterscheiden, ist besonders in einem frühen Entwicklungsstadium der Rebe aufgrund der fehlenden Laubwand schwierig. Eine Möglichkeit, die beiden Ebenen in der Bildanalyse zu trennen, ist daher unerlässlich. Es wurde eine berührungsfreie, schnelle sowie objektive Methode zur Bestimmung des Winterschnittholzgewichts, welches das vegetative Wachstum der Rebe beschreibt, entwickelt. In einem innovativen Ansatz wurde unter Kombination von Tiefenkarten und Bildsegmentierung die sichtbare Winterholzfläche im Bild bestimmt. Im Zuge dieser Arbeit wurde die erste HT Phänotypisierungspipeline für die Rebenzüchtung aufgebaut. Sie umfasst die automatisierte Bildaufnahme im Freiland unter Einsatz des PHENObots, das Datenmanagement mit Datenanalyse sowie die Interpretation des erhaltenen phänotypischen Datensatzes. Die Basis des PHENObots ist ein automatisiert gesteuertes Raupenfahrzeug. Des Weiteren umfasst er ein Multi-Kamera- System, ein RTK-GPS-System und einen Computer zur Datenspeicherung. Eine eigens entwickelte Software verbindet die Bilddaten mit der Standortreferenz. Diese Referenz wird anschließend für das Datenmanagement in einer Datenbank verwendet. Um die Funktionalität der Phänotypisierungspipeline zu demonstrieren, wurden die Merkmale Beerengröße und -farbe im Rebsortiment des Geilweilerhofes unter Verwendung des Berries In Vineyard (BIVcolor) Programms erfasst. Im Durschnitt werden 20 Sekunden pro Weinstock für die Bildaufnahme im Feld benötigt, gefolgt von der Extraktion der Merkmale mittels automatischer, objektiver und präziser Bildauswertung. Im Zuge dieses Versuches konnten mit dem PHENObot 2700 Weinstöcke in 12 Stunden erfasst werden, gefolgt von einer automatischen Bestimmung der Merkmale Beerengröße und -farbe aus den Bildern. Damit konnte die grundsätzliche Machbarkeit bewiesen werden. Diese Pilotpipeline bietet nun die Möglichkeit zur Entwicklung weiterer innovativer Programme zur Erhebung neuer Merkmale sowie die Integration zusätzlicher Sensoren auf dem PHENObot.Grapevine is grown on about 1% of the German agricultural area requiring one third of all fungicides sprayed due to pathogens being introduced within the 19th century. In spite of this requirement for viticulture a reduction is necessary to improve sustainability. This objective can be achieved by growing fungus resistant grapevine cultivars. The development of new cultivars, however, is very time-consuming, taking 20 to 25 years. In recent years the breeding process could be increased considerably by using marker assisted selection (MAS). Further improvements of MAS applications in grapevine breeding will come along with developing of faster and more cost efficient high-throughput (HT) genotyping methods.Complementary to genotyping techniques the quality, objectivity and precision of current phenotyping methods is limited and HT phenotyping methods need to be developed to further increase the efficiency of grapevine breeding through sensor assisted selection. Many different types of sensors technologies are available ranging from visible light sensors (Red Green Blue (RGB) cameras), multispectral, hyperspectral, thermal, and fluorescence cameras to three dimensional (3D) camera and laser scan approaches. Phenotyping can either be done under controlled environments (growth chamber, greenhouse) or can take place in the field, with a decreasing level of standardization. Except for young seedlings, grapevine as a perennial plant needs ultimately to be screened in the field. From a methodological point of view a variety of challenges need to be considered like the variable light conditions, the similarity of fore- and background, and in the canopy hidden traits.The assessment of phenotypic data in grapevine breeding is traditionally done directly in the field by visual estimations. In general the BBCH scale is used to acquire and classify the stages of annual plant development or OIV descriptors are applied to assess the phenotypes into classes. Phenotyping is strongly limited by time, costs and the subjectivity of records. Therefore, only a comparably small set of genotypes is evaluated for certain traits within the breeding process. Due to that limitation, automation, precision and objectivity of phenotypic data evaluation is crucial in order to (1) reduce the existing phenotyping bottleneck, (2) increase the efficiency of grapevine breeding, (3) assist further genetic studies and (4) ensure improved vineyard management. In this theses emphasis was put on the following aspects: Balanced and stable yields are important to ensure a high quality wine production playing a key role in grapevine breeding. Therefore, the main focus of this study is on phenotyping different parameters of yield such as berry size, number of berries per cluster, and number of clusters per vine. Additionally, related traits like cluster architecture and vine balance (relation between vegetative and generative growth) were considered. Quantifying yield parameters on a single vine level is challenging. Complex shapes and slight variations between genotypes make it difficult and very time-consuming.As a first step towards HT phenotyping of yield parameters two fully automatic image interpretation tools have been developed for an application under controlled laboratory conditions to assess individual yield parameters. Using the Cluster Analysis Tool (CAT) four important phenotypic traits can be detected in one image: Cluster length, cluster width, berry size and cluster compactness. The utilization of the Berry Analysis Tool (BAT) provides information on number, size (length and width), and volume of grapevine berries. Both tools offer a fast, user-friendly and cheap procedure to provide several precise phenotypic features of berries and clusters at once with dimensional units in a shorter period of time compared to manual measurements.The similarity of fore- and background in an image captured under field conditions is especially difficult and crucial for image analysis at an early grapevine developmental stage due to the missing canopy. To detect the dormant pruning wood weight, partly determining vine balance, a fast and non-invasive tool for objective data acquisition in the field was developed. In an innovative approach it combines depth map calculation and image segmentation to subtract the background of the vine obtaining the pruning area visible in the image. For the implementation of HT field phenotyping in grapevine breeding a phenotyping pipeline has been set up. It ranges from the automated image acquisition directly in the field using the PHENObot, to data management, data analysis and the interpretation of obtained phenotypic data for grapevine breeding aims. The PHENObot consists of an automated guided tracked vehicle system, a calibrated multi camera system, a Real-Time-Kinematic GPS system and a computer for image data handling. Particularly developed software was applied in order to acquire geo referenced images directly in the vineyard. The geo-reference is afterwards used for the post-processing data management in a database. As phenotypic traits to be analysed within the phenotyping pipeline the detection of berries and the determination of the berry size and colour were considered. The highthroughput phenotyping pipeline was tested in the grapevine repository at Geilweilerhof to extract the characteristics of berry size and berry colour using the Berries In Vineyards (BIVcolor) tool. Image data acquisition took about 20 seconds per vine, which afterwards was followed by the automatic image analysis to extract objective and precise phenotypic data. In was possible to capture images of 2700 vines within 12 hours using the PHENObot and subsequently automatic analysis of the images and extracting berry size and berry colour. With this analysis proof of principle was demonstrated. The pilot pipeline providesthe basis for further development of additional evaluation modules as well as the integration of other sensors

Assessing berry number for grapevine yield estimation by image analysis: case study with the white variety “Encruzado”

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoNowadays, yield estimation represents one of the most important topics in viticulture. It can lead to a better vineyard management and to a better organization of harvesting operations in the vineyard and in the cellar. In recent years, image analysis has become an important tool to improve yield forecast, with the advantages of saving time and being non-invasive. This research aims to estimate the yield of the white cultivar ‘Encruzado’ using visible berry number counted in the images aquired at veraison and near harvest, using a manual RGB camera and the robot VINBOT. Images were collected in laboratory and in the field at the experimental vineyard of the Instituto Superior de Agronomia (ISA) in Lisbon. In the field images the number of visible berries per canopy meter was higher at maturation than at veraison, respectively 72.6 and 66.3. Regarding the percentage of visible berries, 30.2% where visible at veraison and 24.1% at maturation. Concerning percentage of berries occluded by other berries it was observed 28.7% at veraison and 24.3% at maturation. Regression analysis showed that the number of berries in the image explained a very high proportion of bunch weight variability, R2=0.64 at veraison and 0.91 at maturation. Regression analysis also showed that the canopy porosity explained a very high proportion of visible berries variability, R2=0.81 at veraison and 0.88 at maturation. The obtained regression models underestimated the yield with an higher error at veraison than at maturation. This underestimation indicates that the use of visible berry number on the images to estimate yield still needs further research to improve the algorithms accuracyN/

Assessing berry number for grapevine yield estimation by image analysis: case study with the white variety “Arinto”

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoYield estimation in recent years is identified as one of more important topics in viticulture because it can lead to more efficiently managed vineyards producing wines of highly quality. Recently, to improve the efficiency of yield estimation, image analysis is becoming an important tool to collect detailed information from the vines regarding the yield. New technologies were developed for yield estimation using a new ground platform, such as VINBOT, using image analysis. This work was done in a vineyard of the “Instituto Superior de Agronomia”, with the aim to estimate the final yield, during the growing cycle 2019 of the variety “Arinto”, using images collected in three different modality: laboratory condition (1), field condition (2) and VINBOT robot. In the every condition, the images were captured with the RGB-D camera. For (1) and (2) the photos were acquired manually through the use of a digital camera placed on a tripod but in the (3) the RGB-D camera was fixed on the VINBOT robot. In this work, the correlation of yield components between field data and images data was evaluated. In addition, throught MATLAB, it was evaluate the number of visible berries in the images and the percentage of visible berries not occluded by leaves and by other berries. Througt the laboratory results was calculate a growth factor of bunches on the periods pea-size and veraison. On the VINBOT analysis the efficacy to estimate the total yield from the number of berries was higher at maturation with a 10% error ratio. The relationship between canopy porosity and exposed berries showed for all the stages high and significant R2 indicating that we can use it to estimate berries occlusion through image analysis. This accuracy makes the proposed methodology ideal for early yield prediction as a very helpful tool for the grape and wine industryN/

Yield components detection and image-based indicators for non-invasive grapevine yield prediction at different phenological phases

Forecasting vineyard yield with accuracy is one of the most important trends of research in viticulture today. Conventional methods for yield forecasting are manual, require a lot of labour and resources and are often destructive. Recently, image-analysis approaches have been explored to address this issue. Many of these approaches encompass cameras deployed on ground platforms that collect images in proximal range, on-the-go. As the platform moves, yield components and other image-based indicators are detected and counted to perform yield estimations. However, in most situations, when image acquisition is done in non-disturbed canopies, a high fraction of yield components is occluded. The present work’s goal is twofold. Firstly, to evaluate yield components’ visibility in natural conditions throughout the grapevine’s phenological stages. Secondly, to explore single bunch images taken in lab conditions to obtain the best visible bunch attributes to use as yield indicators. In three vineyard plots of red (Syrah) and white varieties (Arinto and Encruzado), several canopy 1 m segments were imaged using the robotic platform Vinbot. Images were collected from winter bud stage until harvest and yield components were counted in the images as well as in the field. At pea-sized berries, veraison and full maturation stages, a bunch sample was collected and brought to lab conditions for detailed assessments at a bunch scale. At early stages, all varieties showed good visibility of spurs and shoots, however, the number of shoots was only highly and significantly correlated with the yield for the variety Syrah. Inflorescence and bunch occlusion reached high percentages, above 50 %. In lab conditions, among the several bunch attributes studied, bunch volume and bunch projected area showed the highest correlation coefficients with yield. In field conditions, using non-defoliated vines, the bunch projected area of visible bunches presented high and significant correlation coefficients with yield, regardless of the fruit’s occlusion. Our results show that counting yield components with image analysis in non-defoliated vines may be insufficient for accurate yield estimation. On the other hand, using bunch projected area as a predictor can be the best option to achieve that goal, even with high levels of occlusioninfo:eu-repo/semantics/publishedVersio

Vineyard yield estimation using image analysis – a review

Mestrado em Engenharia de Viticultura e Enologia (Double degree) / Instituto Superior de Agronomia. Universidade de Lisboa / Faculdade de Ciências. Universidade do PortoYield estimation is one of the main goals of the wine industry, this because with an accurate yield estimation it is possible to have a significant reduction in production costs and a better management of the wine industry. Traditional methods for yield estimation are laborious and time consuming, for these reasons in the last years we are witnessing to the development of new methodologies, most of which are based on image analysis. Thanks to the continuous updating and improvement of the computer vision techniques and of the robotic platforms, image analysis applied to the yield estimation is becoming more and more efficient. In fact the results shown by the different studies are very satisfying, at least as regards the estimation of what is possible to see, while are under development several procedures which have the objective to estimate what is not possible to see, due to bunch occlusion by leaves and by others clusters. I this work the different methodologies and the different approaches used for yield estimation are described, including both traditional methods and new approaches based on image analysis, in order to present the advantages and disadvantages of each of themN/

Development of a new non-invasive vineyard yield estimation method based on image analysis

Doutoramento em Engenharia Agronómica / Instituto Superior de Agronomia. Universidade de LisboaPredicting vineyard yield with accuracy can provide several advantages to the whole vine and wine industry. Today this is majorly done using manual and sometimes destructive methods, based on bunch samples. Yield estimation using computer vision and image analysis can potentially perform this task extensively, automatically, and non-invasively. In the present work this approach is explored in three main steps: image collection, occluded fruit estimation and image traits conversion to mass. On the first step, grapevine images were collected in field conditions along some of the main grapevine phenological stages. Visible yield components were identified in the image and compared to ground truth. When analyzing inflorescences and bunches, more than 50% were occluded by leaves or other plant organs, on three cultivars. No significant differences were observed on bunch visibility after fruit set. Visible bunch projected area explained an average of 49% of vine yield variation, between veraison and harvest. On the second step, vine images were collected, in field conditions, with different levels of defoliation intensity at bunch zone. A regression model was computed combining canopy porosity and visible bunch area, obtained via image analysis, which explained 70-84% of bunch exposure variation. This approach allowed for an estimation of the occluded fraction of bunches with average errors below |10|%. No significant differences were found between the model’s output at veraison and harvest. On the last step, the conversion of bunch image traits into mass was explored in laboratory and field conditions. In both cases, cultivar differences related to bunch architecture were found to affect weight estimation. A combination of derived variables which included visible bunch area, estimated total bunch area, visible bunch perimeter, visible berry number and bunch compactness was used to estimate yield on undisturbed grapevines. The final model achieved a R2 = 0.86 between actual and estimated yield (n = 213). If performed automatically, the final approach suggested in this work has the potential to provide a non-invasive method that can be performed accurately across whole vineyards.N/

A machine-learning approach for automatic grape-bunch detection based on opponent colors

This paper presents a novel and automatic artificial-intelligence (AI) method for grape-bunch detection from RGB images. It mainly consists of a cascade of support vector machine (SVM)-based classifiers that rely on visual contrast-based features that, in turn, are defined according to grape bunch color visual perception. Due to some principles of opponent color theory and proper visual contrast measures, a precise estimate of grape bunches is achieved. Extensive experimental results show that the proposed method is able to accurately segment grapes even in uncontrolled acquisition conditions and with limited computational load. Finally, such an approach requires a very small number of training samples, making it appropriate for onsite and real-time applications that are implementable on smart devices, usable and even set up by winemakers

PHENOquad: A new multi sensor platform for field phenotyping and screening of yield relevant characteristics within grapevine breeding research

Balanced and stable yield is a major trait in grapevine breeding and breeding research. Grapevine yield hereby is a complex quantitative trait, as it is influenced by multiple plant parameters, like berry size, number of berries per bunch, number of bunches per shoot, management, and environmental factors. In the current breeding process, the complexity of this trait has shown that a classification according to descriptive factors for marker development is only possible to a limited extent. Precise field phenotyping of yield-related traits is the basic prerequisite to be able to measure such quantitative traits. This, however, is the major bottleneck due to labor, time and constrains of plant material in the breeding process. For this reason, one of our main goals with the newly developed phenotyping platform PHENOquad with its multisensor system PHENOboxx is to improve phenotyping efficiency of grapevine yield to overcome the phenotyping bottleneck. The newly developed embedded vision system PHENOboxx is mounted on an "all-terrain vehicle (ATV)". This allows a fast data acquisition on a large number of individual vines. In order to evaluate the yield potential of breeding material in comparison to established grapevine cultivars, various yield-related parameters of the vines are quantified directly in the field with high spatial and temporal resolution. As key parameters for yield-related phenotyping, the number of shoots, bunches, berries and the weight of dormant pruning wood was identified. The image data acquired are annotated to train the artificial intelligence (AI). Within the process, the image analysis results are compared to annotated ground truth data and correlated with the field reference data. We expect to increase the precision, target specificity and throughput of screening grapevine material without reducing its accuracy over time by using the PHENOquad. In addition, a weighting of yield-relevant parameters would be possible. This opens up new possibilities for efficient plant evaluation in the scope of grapevine breeding. Also new application possibilities for precision viticulture are conceivable