The Use of Agricultural Robots in Orchard Management

Book chapter that summarizes recent research on agricultural robotics in orchard management, including Robotic pruning, Robotic thinning, Robotic spraying, Robotic harvesting, Robotic fruit transportation, and future trends.Comment: 22 page

Local Motion Planner for Autonomous Navigation in Vineyards with a RGB-D Camera-Based Algorithm and Deep Learning Synergy

With the advent of agriculture 3.0 and 4.0, researchers are increasingly focusing on the development of innovative smart farming and precision agriculture technologies by introducing automation and robotics into the agricultural processes. Autonomous agricultural field machines have been gaining significant attention from farmers and industries to reduce costs, human workload, and required resources. Nevertheless, achieving sufficient autonomous navigation capabilities requires the simultaneous cooperation of different processes; localization, mapping, and path planning are just some of the steps that aim at providing to the machine the right set of skills to operate in semi-structured and unstructured environments. In this context, this study presents a low-cost local motion planner for autonomous navigation in vineyards based only on an RGB-D camera, low range hardware, and a dual layer control algorithm. The first algorithm exploits the disparity map and its depth representation to generate a proportional control for the robotic platform. Concurrently, a second back-up algorithm, based on representations learning and resilient to illumination variations, can take control of the machine in case of a momentaneous failure of the first block. Moreover, due to the double nature of the system, after initial training of the deep learning model with an initial dataset, the strict synergy between the two algorithms opens the possibility of exploiting new automatically labeled data, coming from the field, to extend the existing model knowledge. The machine learning algorithm has been trained and tested, using transfer learning, with acquired images during different field surveys in the North region of Italy and then optimized for on-device inference with model pruning and quantization. Finally, the overall system has been validated with a customized robot platform in the relevant environment

Characterization of vineyard's canopy through fuzzy clustering and svm over color images

In this work we propose an image acquisition and processing methodology (framework) developed for performance in-field grapes and leaves detection and quantification, based on a six step methodology: 1) image segmentation through Fuzzy C-Means with Gustafson Kessel (FCM-GK) clustering; 2) obtaining of FCM-GK outputs (centroids) for acting as seeding for K-Means clustering; 3) Identification of the clusters generated by K-Means using a Support Vector Machine (SVM) classifier. 4) Performance of morphological operations over the grapes and leaves clusters in order to fill holes and to eliminate small pixels clusters; 5)Creation of a mosaic image by Scale-Invariant Feature Transform (SIFT) in order to avoid overlapping between images; 6) Calculation of the areas of leaves and grapes and finding of the centroids in the grape bunches. Image data are collected using a colour camera fixed to a mobile platform. This platform was developed to give a stabilized surface to guarantee that the images were acquired parallel to de vineyard rows. In this way, the platform avoids the distortion of the images that lead to poor estimation of the areas. Our preliminary results are promissory, although they still have shown that it is necessary to implement a camera stabilization system to avoid undesired camera movements, and also a parallel processing procedure in order to speed up the mosaicking process

Review of Variable-Rate Sprayer Applications Based on Real- Time Sensor Technologies

Precision variable rate spray is one of the research hotspots in the field of modern agriculture spraying applications. Variable rate spraying of the canopy allows growers to apply adjusted volume rate of pesticides to the target, based on canopy size, and to apply plant protection products in an economical and environmentally sound manner. In the field of pesticide application, knowledge of the geometrical characteristics of plantations will guarantee a better adjustment of the dosage of the agrochemicals applied. This technology is integrated with intelligent real-time sensors, which have a high potential for agricultural precision spray applications. This book chapter presents the foundations and applications in agriculture of the primary systems used for real-time spray target detection of the geometrical characterization of tree plantations. Systems based on infrared, ultrasonic, light detection and ranging (LIDAR), and stereo vision sensors were discussed, respectively, on their performances to detect spray targets. Among them, laser scanners and stereo vision systems are probably the most promising and complementary techniques for achieving three-dimensional (3D) pictures and maps of plants and canopies. The advantages of data fusion applied in real-time target detection and its accuracy in density estimation of the plants were stressed

High-throughput phenotyping of yield parameters for modern grapevine breeding

Weinbau wird auf 1% der deutschen Agrarfläche betrieben. Auf dieser vergleichsweise kleinen Anbaufläche wird jedoch ein Drittel aller in der deutschen Landwirtschaft verwendeten Fungizide appliziert, was auf die Einführung von Schaderregern im 19. Jahrhundert zurück zu führen ist. Für einen nachhaltigen Anbau ist eine Reduktion des Pflanzenschutzmittelaufwands dringend notwendig. Dieses Ziel kann durch die Züchtung und den Anbau neuer, pilzwiderstandsfähiger Rebsorten erreicht werden. Die Rebenzüchtung als solche ist sehr zeitaufwendig, da die Entwicklung neuer Rebsorten 20 bis 25 Jahre dauert. Der Einsatz der markergestützten Selektion (MAS) erhöht die Effizienz der Selektion in der Rebenzüchtung fortwährend. Eine weitere Effizienzsteigerung ist mit der andauernden Verbesserung der Hochdurchsatz Genotypisierung zu erwarten. Im Vergleich zu den Methoden der Genotypisierung ist die Qualität, Objektivität und Präzision der traditionellen Phänotypisierungsmethoden begrenzt. Die Effizienz in der Rebenzüchtung soll mit der Entwicklung von Hochdurchsatz Methoden zur Phänotypisierung durch sensorgestützte Selektion weiter gesteigert werden. Hierfür sind bisher vielfältige Sensortechniken auf dem Markt verfügbar. Das Spektrum erstreckt sich von RGB-Kameras über Multispektral-, Hyperspektral-, Wärmebild- und Fluoreszenz- Kameras bis hin zu 3D-Techniken und Laserscananwendungen. Die Phänotypisierung von Pflanzen kann unter kontrollierten Bedingungen in Klimakammern oder Gewächshäusern beziehungsweise im Freiland stattfinden. Die Möglichkeit einer standardisierten Datenaufnahme nimmt jedoch kontinuierlich ab. Bei der Rebe als Dauerkultur erfolgt die Aufnahme äußerer Merkmale, mit Ausnahme junger Sämlinge, deshalb auch überwiegend im Freiland. Variierende Lichtverhältnisse, Ähnlichkeit von Vorder- und Hintergrund sowie Verdeckung des Merkmals stellen aus methodischer Sicht die wichtigsten Herausforderungen in der sensorgestützen Merkmalserfassung dar. Bis heute erfolgt die Aufnahme phänotypischer Merkmale im Feld durch visuelle Abschätzung. Hierbei werden die BBCH Skala oder die OIV Deskriptoren verwendet. Limitierende Faktoren dieser Methoden sind Zeit, Kosten und die Subjektivität bei der Datenerhebung. Innerhalb des Züchtungsprogramms kann daher nur ein reduziertes Set an Genotypen für ausgewählte Merkmale evaluiert werden. Die Automatisierung, Präzisierung und Objektivierung phänotypischer Daten soll dazu führen, dass (1) der bestehende Engpass an phänotypischen Methoden verringert, (2) die Effizienz der Rebenzüchtung gesteigert, und (3) die Grundlage zukünftiger genetischer Studien verbessert wird, sowie (4) eine Optimierung des weinbaulichen Managements stattfindet. Stabile und über die Jahre gleichbleibende Erträge sind für eine Produktion qualitativ hochwertiger Weine notwendig und spielen daher eine Schlüsselrolle in der Rebenzüchtung. Der Fokus dieser Studie liegt daher auf Ertragsmerkmalen wie der Beerengröße, Anzahl der Beeren pro Traube und Menge der Trauben pro Weinstock. Die verwandten Merkmale Traubenarchitektur und das Verhältnis von generativem und vegetativem Wachstum wurden zusätzlich bearbeitet. Die Beurteilung von Ertragsmerkmalen auf Einzelstockniveau ist aufgrund der genotypischen Varianz und der Vielfältigkeit des betrachteten Merkmals komplex und zeitintensiv. Als erster Schritt in Richtung Hochdurchsatz (HT) Phänotypisierung von Ertragsmerkmalen wurden zwei voll automatische Bildinterpretationsverfahren für die Anwendung im Labor entwickelt. Das Cluster Analysis Tool (CAT) ermöglicht die bildgestützte Erfassung der Traubenlänge, -breite und -kompaktheit, sowie der Beerengröße. Informationen über Anzahl, Größe (Länge, Breite) und das Volumen der einzelnen Beeren liefert das Berry Analysis Tool (BAT). Beide Programme ermöglichen eine gleichzeitige Erhebung mehrerer, präziser phänotypischer Merkmale und sind dabei schnell, benutzerfreundlich und kostengünstig. Die Möglichkeit, den Vorder- und Hintergrund in einem Freilandbild zu unterscheiden, ist besonders in einem frühen Entwicklungsstadium der Rebe aufgrund der fehlenden Laubwand schwierig. Eine Möglichkeit, die beiden Ebenen in der Bildanalyse zu trennen, ist daher unerlässlich. Es wurde eine berührungsfreie, schnelle sowie objektive Methode zur Bestimmung des Winterschnittholzgewichts, welches das vegetative Wachstum der Rebe beschreibt, entwickelt. In einem innovativen Ansatz wurde unter Kombination von Tiefenkarten und Bildsegmentierung die sichtbare Winterholzfläche im Bild bestimmt. Im Zuge dieser Arbeit wurde die erste HT Phänotypisierungspipeline für die Rebenzüchtung aufgebaut. Sie umfasst die automatisierte Bildaufnahme im Freiland unter Einsatz des PHENObots, das Datenmanagement mit Datenanalyse sowie die Interpretation des erhaltenen phänotypischen Datensatzes. Die Basis des PHENObots ist ein automatisiert gesteuertes Raupenfahrzeug. Des Weiteren umfasst er ein Multi-Kamera- System, ein RTK-GPS-System und einen Computer zur Datenspeicherung. Eine eigens entwickelte Software verbindet die Bilddaten mit der Standortreferenz. Diese Referenz wird anschließend für das Datenmanagement in einer Datenbank verwendet. Um die Funktionalität der Phänotypisierungspipeline zu demonstrieren, wurden die Merkmale Beerengröße und -farbe im Rebsortiment des Geilweilerhofes unter Verwendung des Berries In Vineyard (BIVcolor) Programms erfasst. Im Durschnitt werden 20 Sekunden pro Weinstock für die Bildaufnahme im Feld benötigt, gefolgt von der Extraktion der Merkmale mittels automatischer, objektiver und präziser Bildauswertung. Im Zuge dieses Versuches konnten mit dem PHENObot 2700 Weinstöcke in 12 Stunden erfasst werden, gefolgt von einer automatischen Bestimmung der Merkmale Beerengröße und -farbe aus den Bildern. Damit konnte die grundsätzliche Machbarkeit bewiesen werden. Diese Pilotpipeline bietet nun die Möglichkeit zur Entwicklung weiterer innovativer Programme zur Erhebung neuer Merkmale sowie die Integration zusätzlicher Sensoren auf dem PHENObot.Grapevine is grown on about 1% of the German agricultural area requiring one third of all fungicides sprayed due to pathogens being introduced within the 19th century. In spite of this requirement for viticulture a reduction is necessary to improve sustainability. This objective can be achieved by growing fungus resistant grapevine cultivars. The development of new cultivars, however, is very time-consuming, taking 20 to 25 years. In recent years the breeding process could be increased considerably by using marker assisted selection (MAS). Further improvements of MAS applications in grapevine breeding will come along with developing of faster and more cost efficient high-throughput (HT) genotyping methods.Complementary to genotyping techniques the quality, objectivity and precision of current phenotyping methods is limited and HT phenotyping methods need to be developed to further increase the efficiency of grapevine breeding through sensor assisted selection. Many different types of sensors technologies are available ranging from visible light sensors (Red Green Blue (RGB) cameras), multispectral, hyperspectral, thermal, and fluorescence cameras to three dimensional (3D) camera and laser scan approaches. Phenotyping can either be done under controlled environments (growth chamber, greenhouse) or can take place in the field, with a decreasing level of standardization. Except for young seedlings, grapevine as a perennial plant needs ultimately to be screened in the field. From a methodological point of view a variety of challenges need to be considered like the variable light conditions, the similarity of fore- and background, and in the canopy hidden traits.The assessment of phenotypic data in grapevine breeding is traditionally done directly in the field by visual estimations. In general the BBCH scale is used to acquire and classify the stages of annual plant development or OIV descriptors are applied to assess the phenotypes into classes. Phenotyping is strongly limited by time, costs and the subjectivity of records. Therefore, only a comparably small set of genotypes is evaluated for certain traits within the breeding process. Due to that limitation, automation, precision and objectivity of phenotypic data evaluation is crucial in order to (1) reduce the existing phenotyping bottleneck, (2) increase the efficiency of grapevine breeding, (3) assist further genetic studies and (4) ensure improved vineyard management. In this theses emphasis was put on the following aspects: Balanced and stable yields are important to ensure a high quality wine production playing a key role in grapevine breeding. Therefore, the main focus of this study is on phenotyping different parameters of yield such as berry size, number of berries per cluster, and number of clusters per vine. Additionally, related traits like cluster architecture and vine balance (relation between vegetative and generative growth) were considered. Quantifying yield parameters on a single vine level is challenging. Complex shapes and slight variations between genotypes make it difficult and very time-consuming.As a first step towards HT phenotyping of yield parameters two fully automatic image interpretation tools have been developed for an application under controlled laboratory conditions to assess individual yield parameters. Using the Cluster Analysis Tool (CAT) four important phenotypic traits can be detected in one image: Cluster length, cluster width, berry size and cluster compactness. The utilization of the Berry Analysis Tool (BAT) provides information on number, size (length and width), and volume of grapevine berries. Both tools offer a fast, user-friendly and cheap procedure to provide several precise phenotypic features of berries and clusters at once with dimensional units in a shorter period of time compared to manual measurements.The similarity of fore- and background in an image captured under field conditions is especially difficult and crucial for image analysis at an early grapevine developmental stage due to the missing canopy. To detect the dormant pruning wood weight, partly determining vine balance, a fast and non-invasive tool for objective data acquisition in the field was developed. In an innovative approach it combines depth map calculation and image segmentation to subtract the background of the vine obtaining the pruning area visible in the image. For the implementation of HT field phenotyping in grapevine breeding a phenotyping pipeline has been set up. It ranges from the automated image acquisition directly in the field using the PHENObot, to data management, data analysis and the interpretation of obtained phenotypic data for grapevine breeding aims. The PHENObot consists of an automated guided tracked vehicle system, a calibrated multi camera system, a Real-Time-Kinematic GPS system and a computer for image data handling. Particularly developed software was applied in order to acquire geo referenced images directly in the vineyard. The geo-reference is afterwards used for the post-processing data management in a database. As phenotypic traits to be analysed within the phenotyping pipeline the detection of berries and the determination of the berry size and colour were considered. The highthroughput phenotyping pipeline was tested in the grapevine repository at Geilweilerhof to extract the characteristics of berry size and berry colour using the Berries In Vineyards (BIVcolor) tool. Image data acquisition took about 20 seconds per vine, which afterwards was followed by the automatic image analysis to extract objective and precise phenotypic data. In was possible to capture images of 2700 vines within 12 hours using the PHENObot and subsequently automatic analysis of the images and extracting berry size and berry colour. With this analysis proof of principle was demonstrated. The pilot pipeline providesthe basis for further development of additional evaluation modules as well as the integration of other sensors

Grapevine Yield and Leaf Area Estimation Using Supervised Classification Methodology on RGB Images Taken under Field Conditions

The aim of this research was to implement a methodology through the generation of a supervised classifier based on the Mahalanobis distance to characterize the grapevine canopy and assess leaf area and yield using RGB images. The method automatically processes sets of images, and calculates the areas (number of pixels) corresponding to seven different classes (Grapes, Wood, Background, and four classes of Leaf, of increasing leaf age). Each one is initialized by the user, who selects a set of representative pixels for every class in order to induce the clustering around them. The proposed methodology was evaluated with 70 grapevine (V. vinifera L. cv. Tempranillo) images, acquired in a commercial vineyard located in La Rioja (Spain), after several defoliation and de-fruiting events on 10 vines, with a conventional RGB camera and no artificial illumination. The segmentation results showed a performance of 92% for leaves and 98% for clusters, and allowed to assess the grapevine’s leaf area and yield with R2 values of 0.81 (p < 0.001) and 0.73 (p = 0.002), respectively. This methodology, which operates with a simple image acquisition setup and guarantees the right number and kind of pixel classes, has shown to be suitable and robust enough to provide valuable information for vineyard management

A Comparison of Fuzzy Clustering Algorithms Applied to Feature Extraction on Vineyard

Image segmentation is a process by which an image is partitioned into regions with similar features. Many approaches have been proposed for color image segmentation, but Fuzzy C-Means has been widely used, because it has a good performance in a large class of images. However, it is not adequate for noisy images and it also takes more time for execution as compared to other method as K-means. For this reason, several methods have been proposed to improve these weaknesses. Method like Possibilistic C-Means, Fuzzy Possibilistic C-Means, Robust Fuzzy Possibilistic C-Means and Fuzzy C-Means with Gustafson-Kessel algorithm. In this paper we perform a comparison of these clustering algorithms applied to feature extraction on vineyard images. Segmented images are evaluated using several quality parameters such as the rate of correctly classied area and runtim