Toward a New Approach in Fruit Recognition using Hybrid RGBD Features and Fruit Hierarchy Property

We present hierarchical multi-feature classification (HMC) system for multiclass fruit recognition problem. Our approach to HMC exploits the advantages of combining multimodal features  and  the  fruit  hierarchy  property.  In  the construction of hybrid features, we take the advantage of using color feature in the fruit recognition problem and combine it with 3D shape feature of depth channel of RGBD (Red, Green, Blue, Depth) images. Meanwhile, given a set of fruit species and variety, with a preexisting hierarchy among them, we consider the problem of assigning images to one of these fruit variety from the point of view of a hierarchy. We report on computational experiment using this approach. We show that the use of hierarchy structure along with hybrid RGBD features can improve the classification performance

Preliminary study

The concept of Precision Agriculture is usually associated with the usage of high-end technology equipment (hardware or software) to evaluate or monitor the conditions of a determined portion of land, adjusting afterwards the production factors, like seeds, fertilizers, pesticides, growing regulators, water, according to differential detected characteristics. This paper describes an algorithm developed to analyze and process images to recognize fruits, particularly peaches, and calculate it dimensions, like volume and weight. The recognition of peaches on their natural conditions on trees depends on several spatial- and time-variable parameters and requires complex segmentation algorithms. The proposed algorithm applies image segmentation for extraction of characteristics such as color and shape. These characteristics were used to train a classification method through a Support Vector Machine (SVM) to improve the recognition rate of fruits. The algorithm is designed to acquire images with a high-resolution camera installed in a drone that will fly between the tree lines. The production prediction of 29.3 tons per hectare was obtained based on volume and relation weight/volume calculated for the recognized peaches. An overall precision of 72% was achieved for the prediction rate of peaches in orchards (808 trees/ha). This is the first study regarding the application of these concepts under orchard trees aiming the production prediction along the fruit maturation. Other useful future applications are foreseen in orchard trees, related not only to production prediction, for this type of algorithm.Project "PrunusBOT – Sistema robótico aéreo autónomo de pulverização controlada e previsão de produção frutícola", n.º PDR2020-101-031358, funded by Rural Development Program of the Portuguese Government - Programa de Desenvolvimento Rural (PDR 2020), Portugal 2020.info:eu-repo/semantics/publishedVersio

BLOB Analysis for Fruit Recognition and Detection

Robot application in agriculture can ease the farming process, especially as the harvesting robot for seasonal fruit that is available in a short time. The addition of "eye" as the image sensor is an important feature for a harvesting robot. Thanks to the increment of technology, the camera is getting smaller with better performance, and lower prices. The cheap sensors and components make the creation of cheap and effective robot possible. Image processing is necessary for object detection, and open source software is available now for this purpose. This paper proposes BLOB analysis for object detection of 5 fruits with different shapes and colors. The simulation results show that the proposed method is effective for object detection regardless the shapes, colors, and noises

Inverse kinematic analysis of 4 DOF pick and place arm robot manipulator using fuzzy logic controller

The arm robot manipulator is suitable for substituting humans working in tomato plantation to ensure tomatoes are handled efficiently. The best design for this robot is four links with robust flexibility in x, y, and z-coordinates axis. Inverse kinematics and fuzzy logic controller (FLC) application are for precise and smooth motion. Inverse kinematics designs the most efficient position and motion of the arm robot by adjusting mechanical parameters. The FLC utilizes data input from the sensors to set the right position and motion of the end-effector. The predicted parameters are compared with experimental results to show the effectiveness of the proposed design and method. The position errors (in x, y, and z-axis) are 0.1%, 0.1%, and 0.04%. The rotation errors of each robot links (θ1, θ2, and θ3) are 0%, 0.7% and 0.3%. The FLC provides the suitable angle of the servo motor (θ4) responsible in gripper motion, and the experimental results correspond to FLC’s rules-based as the input to the gripper motion system. This setup is essential to avoid excessive force or miss-placed position that can damage tomatoes. The arm robot manipulator discussed in this study is a pick and place robot to move the harvested tomatoes to a packing system

Wheat ear counting in-field conditions: high throughput and low-cost approach using RGB images

Background The number of ears per unit ground area (ear density) is one of the main agronomic yield components in determining grain yield in wheat. A fast evaluation of this attribute may contribute to monitoring the efficiency of crop management practices, to an early prediction of grain yield or as a phenotyping trait in breeding programs. Currently the number of ears is counted manually, which is time consuming. Moreover, there is no single standardized protocol for counting the ears. An automatic ear-counting algorithm is proposed to estimate ear density under field conditions based on zenithal color digital images taken from above the crop in natural light conditions. Field trials were carried out at two sites in Spain during the 2014/2015 crop season on a set of 24 varieties of durum wheat with two growing conditions per site. The algorithm for counting uses three steps: (1) a Laplacian frequency filter chosen to remove low and high frequency elements appearing in an image, (2) a Median filter to reduce high noise still present around the ears and (3) segmentation using Find Maxima to segment local peaks and determine the ear count within the image. Results The results demonstrate high success rate (higher than 90%) between the algorithm counts and the manual (image-based) ear counts, and precision, with a low standard deviation (around 5%). The relationships between algorithm ear counts and grain yield was also significant and greater than the correlation with manual (field-based) ear counts. In this approach, results demonstrate that automatic ear counting performed on data captured around anthesis correlated better with grain yield than with images captured at later stages when the low performance of ear counting at late grain filling stages was associated with the loss of contrast between canopy and ears. Conclusions Developing robust, low-cost and efficient field methods to assess wheat ear density, as a major agronomic component of yield, is highly relevant for phenotyping efforts towards increases in grain yield. Although the phenological stage of measurements is important, the robust image analysis algorithm presented here appears to be amenable from aerial or other automated platforms

Apple Flower Detection Using Deep Convolutional Networks

To optimize fruit production, a portion of the flowers and fruitlets of apple trees must be removed early in the growing season. The proportion to be removed is determined by the bloom intensity, i.e., the number of flowers present in the orchard. Several automated computer vision systems have been proposed to estimate bloom intensity, but their overall performance is still far from satisfactory even in relatively controlled environments. With the goal of devising a technique for flower identification which is robust to clutter and to changes in illumination, this paper presents a method in which a pre-trained convolutional neural network is fine-tuned to become specially sensitive to flowers. Experimental results on a challenging dataset demonstrate that our method significantly outperforms three approaches that represent the state of the art in flower detection, with recall and precision rates higher than 90%. Moreover, a performance assessment on three additional datasets previously unseen by the network, which consist of different flower species and were acquired under different conditions, reveals that the proposed method highly surpasses baseline approaches in terms of generalization capability

Machine Vision-Based Crop-Load Estimation Using YOLOv8

Labor shortages in fruit crop production have prompted the development of mechanized and automated machines as alternatives to labor-intensive orchard operations such as harvesting, pruning, and thinning. Agricultural robots capable of identifying tree canopy parts and estimating geometric and topological parameters, such as branch diameter, length, and angles, can optimize crop yields through automated pruning and thinning platforms. In this study, we proposed a machine vision system to estimate canopy parameters in apple orchards and determine an optimal number of fruit for individual branches, providing a foundation for robotic pruning, flower thinning, and fruitlet thinning to achieve desired yield and quality.Using color and depth information from an RGB-D sensor (Microsoft Azure Kinect DK), a YOLOv8-based instance segmentation technique was developed to identify trunks and branches of apple trees during the dormant season. Principal Component Analysis was applied to estimate branch diameter (used to calculate limb cross-sectional area, or LCSA) and orientation. The estimated branch diameter was utilized to calculate LCSA, which served as an input for crop-load estimation, with larger LCSA values indicating a higher potential fruit-bearing capacity.RMSE for branch diameter estimation was 2.08 mm, and for crop-load estimation, 3.95. Based on commercial apple orchard management practices, the target crop-load (number of fruit) for each segmented branch was estimated with a mean absolute error (MAE) of 2.99 (ground truth crop-load was 6 apples per LCSA). This study demonstrated a promising workflow with high performance in identifying trunks and branches of apple trees in dynamic commercial orchard environments and integrating farm management practices into automated decision-making

PI‑Plat: a high‑resolution image‑based 3D reconstruction method to estimate growth dynamics of rice inflorescence traits

Background: Recent advances in image-based plant phenotyping have improved our capability to study vegetative stage growth dynamics. However, more complex agronomic traits such as inflorescence architecture (IA), which predominantly contributes to grain crop yield are more challenging to quantify and hence are relatively less explored. Previous efforts to estimate inflorescence-related traits using image-based phenotyping have been limited to destructive end-point measurements. Development of non-destructive inflorescence phenotyping platforms could accelerate the discovery of the phenotypic variation with respect to inflorescence dynamics and mapping of the underlying genes regulating critical yield components. Results: The major objective of this study is to evaluate post-fertilization development and growth dynamics of inflorescence at high spatial and temporal resolution in rice. For this, we developed the Panicle Imaging Platform (PI-Plat) to comprehend multi-dimensional features of IA in a non-destructive manner. We used 11 rice genotypes to capture multi-view images of primary panicle on weekly basis after the fertilization. These images were used to reconstruct a 3D point cloud of the panicle, which enabled us to extract digital traits such as voxel count and color intensity. We found that the voxel count of developing panicles is positively correlated with seed number and weight at maturity. The voxel count from developing panicles projected overall volumes that increased during the grain filling phase, wherein quantification of color intensity estimated the rate of panicle maturation. Our 3D based phenotyping solution showed superior performance compared to conventional 2D based approaches. Conclusions: For harnessing the potential of the existing genetic resources, we need a comprehensive understanding of the genotype-to-phenotype relationship. Relatively low-cost sequencing platforms have facilitated high-throughput genotyping, while phenotyping, especially for complex traits, has posed major challenges for crop improvement. PI-Plat offers a low cost and high-resolution platform to phenotype inflorescence-related traits using 3D reconstruction-based approach. Further, the non-destructive nature of the platform facilitates analyses of the same panicle at multiple developmental time points, which can be utilized to explore the genetic variation for dynamic inflorescence traits in cereals

RS-Net: robust segmentation of green overlapped apples

Fruit detection and segmentation will be essential for future agronomic management, with applications in yield estimation, growth monitoring, intelligent picking, disease detection and etc. In order to more accurately and efficiently realize the recognition and segmentation of apples in natural orchards, a robust segmentation net framework specially developed for fruit production is proposed. This model was improved for the more challenging problem which segments the overlapped apples from the monochromatic background regardless of various corruptions. The method extends Mask R-CNN by embedding an attention mechanism for focusing more on the informative pixels but also suppressing the noise caused by adverse factors (occlusions, overlaps, etc.), which could be more suitable and robust for operating in complex natural environment. Specifically, the Gaussian non-local attention mechanism is transplanted into Mask R-CNN for refining the semantic features generated continuously by residual network and feature pyramid network, then the model forward processing based on the balanced feature levels and finally segments the regions where the apples are located. Experimental results verify the hypothesis of current work and show that the proposed method outperforms other start-of-the-art detection and segmentation models, the AP box and AP mask metric values have reached 85.6% and 86.2% in a reasonable run time, respectively, which can meet the precision and robustness of vision system in agronomic managemen