Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

Multispecies Fruit Flower Detection Using a Refined Semantic Segmentation Network

In fruit production, critical crop management decisions are guided by bloom intensity, i.e., the number of flowers present in an orchard. Despite its importance, bloom intensity is still typically estimated by means of human visual inspection. Existing automated computer vision systems for flower identification are based on hand-engineered techniques that work only under specific conditions and with limited performance. This letter proposes an automated technique for flower identification that is robust to uncontrolled environments and applicable to different flower species. Our method relies on an end-to-end residual convolutional neural network (CNN) that represents the state-of-the-art in semantic segmentation. To enhance its sensitivity to flowers, we fine-tune this network using a single dataset of apple flower images. Since CNNs tend to produce coarse segmentations, we employ a refinement method to better distinguish between individual flower instances. Without any preprocessing or dataset-specific training, experimental results on images of apple, peach, and pear flowers, acquired under different conditions demonstrate the robustness and broad applicability of our method

KLASIFIKASI JENIS KOPI INDONESIA MENGGUNAKAN DEEP LEARNING

Indonesia adalah salah satu negara produsen dan eksportir kopi terbesar di dunia. Perkembangan bisnis kopi mengalami kemajuan yang cukup pesat, mulai dari tingkat para petani, pemasok, café kopi, hingga ke konsumen biasa. Disamping meningkatnya kemajuan industri kopi di indonesia masih terdapat banyak masalah yang menimbulkan kerugian secara material dan rasa ketidakpuasan baik bagi pelaku usaha maupun para pecinta kopi. Masalah yang muncul diakibatkan karena industry ini masih banyak dijalankan dengan menggunakan sistem kepercayaan antar pihak yang berkaitan. Sulit untuk system sederhana membedakan anatara satu varian kopi dengan varian lainnya. Diperlukannya sebuah system berbasis teknologi informasi yang dapat membantu mengenali dan memastikan secara langsung bahwa kopi yang dibutuhkan dan dinikmati sudah sesuai dengan apa yang diinginkan. Sistem informasi yang akan dibangun dapat mengklasifikasi jenis kopi berdasarkan gambar. Pengenalan pola citra tersebut menggunakan Deep Learning. Melatih algoritma Deep Learning untuk mendeteksi jenis kopi secara akurat membutuhkan jumlah gambar yang banyak untuk data pelatihan. Metode pengenalan menggunakan Convolutional Neural Network yang dapat digunakan untuk mengenali objek pada sebuah gambar dan sering digunakan untuk klasifikasi data berupa image. Metode CNN saat ini trend digunakan untuk masalah klasifikasi gambar dikarenakan tingkat akurasinya yang sangat tinggi. CNN akan mengklasifikasi pada setiap gambar yang disiapkan sebagai data latih untuk pengenalan. Data dikumpulkan dengan cara pengambilan gambar biji kopi menggunakan kamera. Kumpulan data ini berisi 4 jenis kopi asal Indonesia (Garut, Gayo, Kerinci, Temanggung) dengan jumlah 617 gambar biji kopi. Setelah dilakukan proses pengujian, system dapat mengenali objek dengan tingkat akurasi sebesar 70,68%

Ananas comosus crown image thresholding and crop counting using a colour space transformation scheme

The implementation of unmanned aerial vehicle (UAV) technology having image processing capabilities provides an alternative way to observe pineapple crowns captured from aerial images. In the majority of pineapple plantations, an agricultural officer will physically count the crop yield prior to harvesting the Ananas Comosus, also known as pineapple. This process is particularly evident in large plantation areas to accurately identify pineapple numbers. To alleviate this issue, given it is both time-consuming and arduous, automating the process using image processing is suggested. In this study, the possibilities and comparisons between two techniques associated with an image thresholding scheme known as HSV and L*A*B* colour space schemes were implemented. This was followed by determining the threshold by applying an automatic counting (AC) method to count the crop yield. The results of the study found that by applying colour thresholding for segmentation, it improved the low contrast image due to different heights and illumination levels on the acquired colour image. The images that were acquired using a UAV revealed that the best distance for capturing the images was at the height of three (3) metres above ground level. The results also confirm that the HSV colour space provides a more efficient approach with an average error increment of 47.6% when compared to the L*A*B*colour space

Peaches Detection Using a Deep Learning Technique - A Contribution to Yield Estimation, Resources Management, and Circular Economy

Fruit detection is crucial for yield estimation and fruit picking system performance. Many state-of-the-art methods for fruit detection use convolutional neural networks (CNNs). This paper presents the results for peach detection by applying a faster R-CNN framework in images captured from an outdoor orchard. Although this method has been used in other studies to detect fruits, there is no research on peaches. Since the fruit colors, sizes, shapes, tree branches, fruit bunches, and distributions in trees are particular, the development of a fruit detection procedure is specific. The results show great potential in using this method to detect this type of fruit. A detection accuracy of 0.90 using the metric average precision (AP) was achieved for fruit detection. Precision agriculture applications, such as deep neural networks (DNNs), as proposed in this paper, can help to mitigate climate change, due to horticultural activities by accurate product prediction, leading to improved resource management (e.g., irrigation water, nutrients, herbicides, pesticides), and helping to reduce food loss and waste via improved agricultural activity scheduling.The authors are thankful to Fundação para a Ciência e Tecnologia (FCT) and R&D Unit “Center for Mechanical and Aerospace Science and Technologies” (C-MAST), under project UIDB/00151/2020, for the opportunity and the financial support to carry on this project. The contributions of Hugo Proença and Pedro Inácio in this work were supported by FCT/MEC through FEDER—PT2020 Partnership Agreement under Project UIDB//50008/2021.info:eu-repo/semantics/publishedVersio

Detection of Grape Clusters in Images using Convolutional Neural Network

Convolutional Neural Networks and Deep Learning have revolutionized every field since their inception. Agriculture has also been reaping the fruits of developments in mentioned fields. Technology is being revolutionized to increase yield, save water wastage, take care of diseased weeds, and also increase the profit of farmers. Grapes are among the highest profit-yielding and important fruit related to the juice industry. Pakistan being an agricultural country, can widely benefit by cultivating and improving grapes per hectare yield. The biggest challenge in harvesting grapes to date is to detect their cluster successfully; many approaches tend to answer this problem by harvest and sort technique where the foreign objects are separated later from grapes after harvesting them using an automatic harvester. Currently available systems are trained on data that is from developed or grape-producing countries, thus showing data biases when used at any new location thus it gives rise to a need of creating a dataset from scratch to verify the results of research. Grape is available in different sizes, colors, seed sizes, and shapes which makes its detection, through simple Computer vision, even more challenging. This research addresses this issue by bringing the solution to this problem by using CNN and Neural Networks using the newly created dataset from local farms as the other research and the methods used don’t address issues faced locally by the farmers. YOLO has been selected to be trained on the locally collected dataset of grapes

Preliminary results of peach detection in images applying convolutional neuronal network

The fruit detection part is very important for a good performance in a yield estimation system. This paper presents the preliminary results using the object detection Faster R-CNN method in the peaches images. The aim is evaluate the method performance in the detection of peach RGB images. Images acquired in an orchard were used. Although this method of object detection has been applied in other studies to detect fruits, according to the literature, it has not been used to detect peaches. The results, although preliminary, show a great potential of using the method to detect peach.Este trabalho de investigação é financiado pelo projeto PrunusBot - Sistema robótico aéreo autónomo de pulverização controlada e previsão de produção frutícola, Operação n.º PDR2020-101-031358 (líder), Consórcio n.º 340, Iniciativa n.º 140, promovido pelo PDR2020 e co-financiado pelo FEADER e União Europeia no âmbito do Programa Portugal 2020.info:eu-repo/semantics/publishedVersio

AKFruitYield: Modular benchmarking and video analysis software for Azure Kinect cameras for fruit size and fruit yield estimation in apple orchards

AKFruitYield is a modular software that allows orchard data from RGB-D Azure Kinect cameras to be processed for fruit size and fruit yield estimation. Specifically, two modules have been developed: i) AK_SW_BENCHMARKER that makes it possible to apply different sizing algorithms and allometric yield prediction models to manually labelled color and depth tree images; and ii) AK_VIDEO_ANALYSER that analyses videos on which to automatically detect apples, estimate their size and predict yield at the plot or per hectare scale using the appropriate algorithms. Both modules have easy-to-use graphical interfaces and provide reports that can subsequently be used by other analysis tools.This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646) and by the Spanish Ministry of Science and Innovation/AEI/10.13039/501100011033/ERDF (grant RTI2018–094222-B-I00 [PAgFRUIT project] and PID2021–126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda's pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. The authors would also like to thank the Institut de Recerca i Tecnologia Agroalimentàries (IRTA) for allowing the use of their experimental fields, and in particular Dr. Luís Asín and Dr. Jaume Lordán who have contributed to the success of this work.info:eu-repo/semantics/publishedVersio

Classificació automàtica de fruites utilitzant tècniques d'aprenentatge profund

The productivity of the agri-food sector experiences continuous and growing challenges that make the use of innovative technologies to maintain and even improve their competitiveness a priority. One way to achieve this goal is the development of flexible and portable systems capable of obtaining 2D/3D measurements and classifying objects based on color and depth images taken from multiple sensors. In this project, deep learning methods for fruit detection and classification will be explored.És crucial disposar de sistemes de detecció d'objectes precisos i fiables per a desenvolupar feines d'alt nivell en agricultura com serien fer un mapatge del camp o robotitzar les collites. Aquest document utilitza una Faster-RCNN -que consisteix xarxa de detecció d'objectes de l'estat de l'art- orientada a la detecció de fruites que en aquest treball només seran pomes. La xarxa serà introduïda i explicada. Es fa un anàlisis d'obtenció dels paràmetres d'entrenament i diversos experiments orientats a maximitzar la finesa (accuracy en anglès) del model que es vol obtenir. La xarxa neuronal estarà consistirà en una part preentrenada una part completament per entrenar. Aquest estudi no ha aconseguit equiparar els resultats de treballs anteriors (F1 score > 0.9) però tampoc es pot dir que hagi obtingut mals resultats, com seria un F1-score de 0.85