Automated systems based on machine vision for inspecting citrus fruits from the field to postharvest - A review

[EN] Computer vision systems are becoming a scientific but also a commercial tool for food quality assessment. In the field, these systems can be used to predict yield, as well as for robotic harvesting or the early detection of potentially dangerous diseases. In postharvest handling, it is mostly used for the automated inspection of the external quality of the fruits and for sorting them into commercial categories at very high speed. More recently, the use of hyperspectral imaging is allowing not only the detection of defects in the skin of the fruits but also their association to certain diseases of particular importance. In the research works that use this technology, wavelengths that play a significant role in detecting some of these dangerous diseases are found, leading to the development of multispectral imaging systems that can be used in industry. This article reviews recent works that use colour and non-standard computer vision systems for the automated inspection of citrus. It explains the different technologies available to acquire the images and their use for the non-destructive inspection of internal and external features of these fruits. Particular attention is paid to inspection for the early detection of some dangerous diseases like citrus canker, black spot, decay or citrus Huanglongbing.This work was supported by the Instituto Nacional de Investigacion y Tecnologia Agraria y Alimentaria (INIA) through projects RTA2012-00062-C04-01 and RTA2012-00062-C04-03 with the support of European FEDER funds. The authors would like to thank and acknowledge the contributions that were made by all the students, postdocs, technicians and visiting scholars in the Precision Agriculture Laboratory at the University of Florida and the Computer Vision Laboratory at the Agricultural Engineering Centre of IVIA.Cubero García, S.; Lee, WS.; Aleixos Borrás, MN.; Albert Gil, FE.; Blasco Ivars, J. (2016). Automated systems based on machine vision for inspecting citrus fruits from the field to postharvest - A review. Food and Bioprocess Technology. 9(10):1623-1639. https://doi.org/10.1007/s11947-016-1767-1S16231639910Adebayo, S. E., Hashim, N., Abdan, K., & Hanafi, M. (2016). Application and potential of backscattering imaging techniques in agricultural and food processing—a review. Journal of Food Engineering, 169, 155–164.Aleixos, N., Blasco, J., Navarrón, F., & Moltó, E. (2002). Multispectral inspection of citrus in real time using machine vision and digital signal processors. Computers and Electronics in Agriculture, 33(2), 121–137.Annamalai, P., & Lee, W. S. (2003). Citrus yield mapping system using machine vision. ASAE Paper No. 031002. St. Joseph: ASAE.Annamalai, P., & Lee, W. S. (2004). Identification of green citrus fruits using spectral characteristics. ASAE Paper No. FL04–1001. St. Joseph: ASAE.Balasundaram, D., Burks, T. F., Bulanon, D. M., Schubert, T., & Lee, W. S. (2009). Spectral reflectance characteristics of citrus canker and other peel conditions of grapefruit. Postharvest Biology and Technology, 51, 220–226.Bansal, R., Lee, W. S., & Satish, S. (2013). Green citrus detection using fast Fourier transform (FFT) leakage. Precision Agriculture, 14(1), 59–70.Barreiro, P., Zheng, C., Sun, D.-W., Hernández-Sánchez, N., Pérez-Sánchez, J. M., & Ruiz-Cabello, J. (2008). Non-destructive seed detection in mandarins: comparison of automatic threshold methods in FLASH and COMSPIRA MRIs. Postharvest Biology and Technology, 47, 189–198.Basavaprasad, B., & Ravi, M. (2014). A comparative study on classification of image segmentation methods with a focus on graph based techniques. International Journal of Research in Engineering and Technology, 3, 310–315.Birth, G. S. (1976). How light interacts with foods. In: Gafney J.Jr.(Ed.), Quality detection in foods (pp. 6–11). St. Joseph: ASAE.Blanc, P.G.R., Blasco, J., Moltó, E., Gómez-Sanchis, J., & Cubero, S. (2010) System for the automatic selective separation of rotten citrus fruits. Patent number EP2133157 A1 CN101678405A, EP2133157A4, EP2133157B1, US20100121484Blasco, J., Aleixos, N., & Moltó, E. (2007a). Computer vision detection of peel defects in citrus by means of a region oriented segmentation algorithm. Journal of Food Engineering, 81(3), 535–543.Blasco, J., Aleixos, N., Gómez, J., & Moltó, E. (2007b). Citrus sorting by identification of the most common defects using multispectral computer vision. Journal of Food Engineering, 83(3), 384–393.Blasco, J., Aleixos, N., Gómez-Sanchis, J., & Moltó, E. (2009). Recognition and classification of external skin damages in citrus fruits using multispectral data and morphological features. Biosystems Engineering, 103(2), 137–145.Blasco, J., Cubero, S., & Moltó, E. (2016). Quality evaluation of citrus fruits. In D.-W. Sun (Ed.), Computer vision technology for food quality evaluation (2nd ed.). San Diego: Academic Press.Bulanon, D. M., Burks, T. F., & Alchanatis, V. (2009). Image fusion of visible and thermal images for fruit detection. Biosystems Engineering, 103, 12–22.Bulanon, D.M., Burks, T.F., Kim, D.G., & Ritenour, M.A. (2013). Citrus black spot detection using hyperspectral image analysis. Agricultural Engineering International: CIGR Journal, 15,(3)171.Burks, T. F., Villegas, F., Hannan, M. W., & Flood, S. (2003). Engineering and horticultural aspects of robotic fruit harvesting: opportunities and constraints. HortTechnology, 15(1), 79–87.Campbell, B. L., Nelson, R. G., Ebel, R. C., Dozier, W. A., Adrian, J. L., & Hockema, B. R. (2004). Fruit quality characteristics that affect consumer preferences for Satsuma mandarins. Hortscience, 39(7), 1664–1669.Chinchuluun, R., Lee, W. S., & Ehsani, R. (2009). Machine vision system for determining citrus count and size on a canopy shake and catch harvester. Applied Engineering in Agriculture, 25(4), 451–458.Choi, D., Lee, W. S., Ehsani, R., & Roka, F. M. (2015). A machine vision system for quantification of citrus fruit dropped on the ground under the canopy. Transactions of the ASABE, 58(4), 933–946.Codex Alimentarius, (2011). Codex standard for oranges. Available at: http://www.codexalimentarius.org/download/standards/10372/CXS_245e.pdf . Accessed March 2016Cubero, S., Aleixos, N., Albert, A., Torregrosa, A., Ortiz, C., García-Navarrete, O., & Blasco, J. (2014a). Optimised computer vision system for automatic pre-grading of citrus fruit in the field using a mobile platform. Precision Agriculture, 15(1), 80–94.Cubero, S., Aleixos, N., Moltó, E., Gómez-Sanchis, J., & Blasco, J. (2011). Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food and Bioprocess Technology, 4(4), 487–504.Cubero, S., Diago, M. P., Blasco, J., Tardáguila, J., Millán, B., & Aleixos, N. (2014b). A new method for pedicel/peduncle detection and size assessment of grapevine berries and other fruits by image analysis. Biosystems Engineering, 117, 62–72.Dong, C.-W., Ye, Y., Zhang, J.-Q., Zhu, H.-K., & Liu, F. (2014). Detection of thrips defect on green-peel citrus using hyperspectral imaging technology combining PCA and B-Spline lighting correction method. Journal of Integrative Agriculture, 13(10), 2229–2235.FAOSTAT (2012). URL: http://faostat.fao.org http://www.fao.org/fileadmin/templates/est/COMM_MARKETS_MONITORING/Citrus/Documents/CITRUS_BULLETIN_2012.pdf . Accessed March 2016.Farrell, T. J., Patterson, M. S., & Wilson, B. (1992). A diffusion-theory model of spatially resolved steady-state diffuse reflectance for the noninvasive determination of tissue optical-properties in vivo. Medical Physics, 19, 879–888.Flood, S. J., Burks, T. F., & Teixeira, A. A. (2006). Physical properties of oranges in response to applied gripping forces for robotic harvesting. Transactions of ASAE, 49(2), 341–346.Gaffney, J. J. (1973). Reflectance properties of citrus fruit. Transactions of ASAE, 16(2), 310–314.Garcia-Ruiz, F., Sankaran, S., Maja, J. M., Lee, W. S., Rasmussen, J., & Ehsani, R. (2013). Comparison of two aerial imaging platforms for identification of Huanglongbing infected citrus trees. Computers and Electronics in Agriculture, 91, 106–115.Gómez, J., Blasco, J., Moltó, E., & Camps-Valls, G. (2007). Hyperspectral detection of citrus damage with a Mahalanobis kernel classifier. Electronics Letters, 43(20), 1082–1084.Gómez-Sanchis, J., Blasco, J., Soria-Olivas, E., Lorente, D., Escandell-Montero, P., Martínez-Martínez, J. M., Martínez-Sober, M., & Aleixos, N. (2013). Hyperspectral LCTF-based system for classification of decay in mandarins caused by Penicillium digitatum and Penicillium italicum using the most relevant bands and non-linear classifiers. Postharvest Biology and Technology, 82, 76–86.Gómez-Sanchis, J., Gómez-Chova, L., Aleixos, N., Camps-Valls, G., Montesinos-Herrero, C., Moltó, E., & Blasco, J. (2008). Hyperspectral system for early detection of rottenness caused by Penicillium digitatum in mandarins. Journal of Food Engineering, 89(1), 80–86.Gómez-Sanchis, J., Lorente, D., Soria-Olivas, E., Aleixos, N., Cubero, S., & Blasco, J. (2014). Development of a hyperspectral computer vision system based on two liquid crystal tuneable filters for fruit inspection. Application to detect citrus fruits decay. Food and Bioprocess Technology, 7, 1047–1056.Gómez-Sanchis, J., Martín-Guerrero, J. D., Soria-Olivas, E., Martínez-Sober, M., Magdalena-Benedito, R., & Blasco, J. (2012). Detecting rottenness caused by Penicillium in citrus fruits using machine learning techniques. Expert Systems with Applications, 39(1), 780–785.Gong, A., Yu, J., He, Y., & Qiu, Z. (2013). Citrus yield estimation based on images processed by an android mobile phone. Biosystems Engineering, 115, 162–170.Gottwald, T. R., Graham, J. H., & Schubert, T. S. (2002). Citrus canker: the pathogen and its impact. Plant Health Progress. doi: 10.1094/PHP-2002-0812-01-RV.Hannan, M., Burks, T. F., & Bulanon, D.M. (2009). A machine vision algorithm for orange fruit detection. The CIGR Ejournal. Manuscript 1281. Vol XI. December 2009.Harrell, R. C., Adsit, P. D., & Slaughter, D. C. (1985). Real-time vision-servoing of a robotic tree-fruit harvester. ASAE Paper No (pp. 85–3550). St. Joseph: ASAE.Hernández-Sánchez, N., Barreiro, P., & Ruiz-Cabello, J. (2006). On-line identification of seeds in mandarins with magnetic resonance imaging. Biosystems Engineering, 95, 529–536.Holmes, G. J., & Eckert, J. W. (1999). Sensitivity of Penicillium digitatum and P. italicum to postharvest citrus fungicides in California. Phytopathology, 89(9), 716–721.Iqbal, S. M., Gopal, A., Sankaranarayanan, P. E., & Nair, A. B. (2016). Classification of selected citrus fruits based on color using machine vision system. International Journal of Food Properties, 19, 272–288.Jackson, J. E. (1991). A user’s guide to principal components. New York: Wiley.Jafari, A., Fazayeli, A., & Zarezadeh, M. R. (2014). Estimation of orange skin thickness based on visual texture coarseness. Biosystems Engineering, 117, 73–82.Jiménez-Cuesta, M. J., Cuquerella, J., & Martínez-Jávega, J. M. (1981). Determination of a color index for citrus fruit degreening. In Proceedings of the International Society of Citriculture, 2, 750–753.Kim, D. G., Burks, T. F., Qin, J., & Bulanon, D. M. (2009). Classification of grapefruit peel diseases using color texture feature analysis. International Journal of Agricultural and Biological Engineering, 2, 41–50.Kim, D. G., Burks, T. F., Ritenour, M. A., & Qin, J. (2014). Citrus black spot detection using hyperspectral imaging. International Journal of Agricultural and Biological Engineering, 7, 20–27.Kohno, Y., Kondo, N., Iida, M., Kurita, M., Shiigi, T., Ogawa, Y., Kaichi, T., & Okamoto, S. (2011). Development of a mobile grading machine for citrus fruit. Engineering in Agriculture, Environment and Food, 4, 7–11.Kondo, N., Kuramoto, M., Shimizu, H., Ogawa, Y., Kurita, M., Nishizu, T., Chong, V. K., & Yamamoto, K. (2009). Identification of fluorescent substance in mandarin orange skin for machine vision system to detect rotten citrus fruits. Engineering in Agriculture, Environment and Food, 2, 54–59.Kurita, M., Kondo, N., Shimizu, H., Ling, P. P., Falzea, P. D., Shiigi, T., Ninomiya, K., Nishizu, T., & Yamamoto, K. (2009). A double image acquisition system with visible and UV LEDs for citrus fruit. Journal of Robotics and Mechatronics, 21, 533–540.Kurtulmus, F., Lee, W. S., & Vardar, A. (2011). Green citrus detection using eigenfruit, color and circular Gabor texture features under natural outdoor conditions. Computers and Electronics in Agriculture, 78(2), 140–149.Ladaniya, M. S. (2010). Citrus fruit: biology, technology and evaluation. San Diego: Academic Press.Li, H., Lee, W. S., & Wang, K. (2016). Immature green citrus fruit detection and counting based on fast normalized cross correlation (FNCC) using natural outdoor colour images. Precision Agriculture. doi: 10.1007/s11119-016-9443-z.Li, H., Lee, W. S., Wang, K., Ehsani, R., & Yang, C. (2014). Extended spectral angle mapping (ESAM) for citrus greening disease detection using airborne hyperspectral imaging. Precision Agriculture, 15, 162–183.Li, J., Rao, X., & Ying, Y. (2011). Detection of common defects on oranges using hyperspectral reflectance imaging. Computers and Electronics in Agriculture, 78, 38–48.Li, J., Rao, X., & Ying, Y. (2012a). Development of algorithms for detecting citrus canker based on hyperspectral reflectance imaging. Journal of the Science of Food and Agriculture, 92, 125–134.Li, J., Rao, X., Wang, F., Wu, W., & Ying, Y. (2013). Automatic detection of common surface defects on oranges using combined lighting transform and image ratio methods. Postharvest Biology and Technology, 82, 59–69.Li, J., Rao, X., Ying, Y., & Wang, D. (2010). Detection of navel oranges canker based on hyperspectral imaging technology. Nongye Gongcheng Xuebao/Transactions of the Chinese Society of Agricultural Engineering, 26, 222–228.Li, X., Lee, W. S., Li, M., Ehsani, R., Mishra, A., Yang, C., & Mangan, R. (2012b). Spectral difference analysis and airborne imaging classification for citrus greening infected trees. Computers and Electronics in Agriculture, 83, 32–46.Li, X., Lee, W. S., Li, M., Ehsani, R., Mishra, A. R., Yang, C., & Mangan, R. L. (2015). Feasibility study on Huanglongbing (citrus greening) detection based on WorldView-2 satellite imagery. Biosystems Engineering, 132, 28–38.Lopes, L. B., VanDeWall, H., Li, H. T., Venugopal, V., Li, H. K., Naydin, S., Hosmer, J., Levendusky, M., Zheng, H., Bentley, M. V., Levin, R., & Hass, M. A. (2010). Topical delivery of lycopene using microemulsions: enhanced skin penetration and tissue antioxidant activity. Journal of Pharmaceutical Sciences, 99, 1346–1357.López, J. J., Cobos, M., & Aguilera, E. (2011). Computer-based detection and classification of flaws in citrus fruits. Neural Computing and Applications, 20, 975–981.López-García, F., Andreu, G., Blasco, J., Aleixos, N., & Valiente, J. M. (2010). Automatic detection of skin defects in citrus fruits using a multivariate image analysis approach. Computers and Electronics in Agriculture, 71, 189–197.Lorente, D., Aleixos, N., Gómez-Sanchis, J., Cubero, S., & Blasco, J. (2013a). Selection of optimal wavelength features for decay detection in citrus fruit using the ROC curve and neural networks. Food and Bioprocess Technology, 6(2), 530–541.Lorente, D., Aleixos, N., Gómez-Sanchis, J., Cubero, S., García-Navarrete, O. L., & Blasco, J. (2012). Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment. Food and Bioprocess Technology, 5(4), 1121–1142.Lorente, D., Blasco, J., Serrano, A. J., Soria-Olivas, E., Aleixos, N., & Gómez-Sanchis, J. (2013b). Comparison of ROC feature selection method for the detection of decay in citrus fruit using hyperspectral images. Food and Bioprocess Technology, 6(12), 3613–3619.Lorente, D., Zude, M., Regen, C., Palou, L., Gómez-Sanchis, J., & Blasco, J. (2013c). Early decay detection in citrus fruit using laser-light backscattering imaging. Postharvest Biology and Technology, 86, 424–430.Lorente, D., Zude, M., Idler, C., Gómez-Sanchis, J., & Blasco, J. (2015). Laser-light backscattering imaging for early decay detection in citrus fruit using both a statistical and a physical model. Journal of Food Engineering, 154, 76–85.Maf Industries. (2016). VIOTEC brochure. http://mafindustries.com/wp-content/uploads/2015/02/viotec3.pdf . Accessed March 2016.Magwaza, L. S., Opara, U. L., Nieuwoudt, H., Cronje, P. J. R., Saeys, W., & Nicolaï, B. (2012). NIR spectroscopy applications for internal and external quality analysis of citrus fruit—a review. Food and Bioprocess Technology, 5(2), 425–444.Mehta, S. S., & Burks, T. F. (2014). Vision-based control of robotic manipulator for citrus harvesting. Computers and Electronics in Agriculture, 102, 146–158.Moltó, E., Blasco, J., & Gómez-Sanchis, J. (2010). Analysis of hyperspectral images of citrus fruits. In D.-W. Sun (Ed.), Hyperspectral imaging for food quality analysis and control (pp. 321–348). California: Academic Press.Moltó, E., Plá, F., & Juste, F. (1992). Vision systems for the location of citrus fruit in a tree canopy. Journal of Agricultural Engineering Research, 52, 101–110.Momin, A., Kondo, N., Kuramoto, M., Ogawa, Y., Yamamoto, K., & Shiigi, T. (2012). Investigation of excitation wavelength for fluorescence emission of citrus peels based on UV-VIS spectra. Engineering in Agriculture, Environment and Food, 5, 126–132.Momin, A., Kondo, N., Ogawa, Y., Ido, K., & Ninomiya, K. (2013b). Patterns of fluorescence associated with citrus peel defects. Engineering in Agriculture, Environment and Food, 6, 54–60.Momin, A., Kuramoto, M., Kondo, N., Ido, K., Ogawa, Y., Shiigi, T., & Ahmad, U. (2013a). Identification of UV-fluorescence components for detecting peel defects of lemon and yuzu using machine vision. Engineering in Agriculture, Environment and Food, 6, 165–171.Morgan, S. P., & Stockford, I. M. (2003). Surface-reflection elimination in polarization imaging of superficial tissue. Optics Letters, 28, 114–116.Niphadkar, N. P., Burks, T. F., Qin, J., & Ritenour, M. (2013b). Edge effect compensation for citrus canker lesion detection due to light source variation—a hyperspectral imaging application. Agricultural Engineering International: CIGR Journal, 15, 314–327.Niphadkar, N. P., Burks, T. F., Qin, J. W., & Ritenour, M. A. (2013a). Estimation of citrus canker lesion size using hyperspectral reflectance imaging. International Journal of Agricultural and Biological Engineering, 6, 41–51.Obenland, D., Margosan, D., Smilanick, J. L., & Mackey, B. (2010). Ultraviolet fluorescence to identify navel oranges with poor peel quality and decay. HortTechnology, 20, 991–995.Ogawa, Y., Abdul, M. M., Kuramoto, M., Kohno, Y., Shiigi, T., Yamamoto, K., & Kondo, K. (2011). Rotten part detection on citrus fruit surfaces by use of fluorescent images. The Review of Laser Engineering, 394, 255–261.Okamoto, H., & Lee, W. S. (2009). Green citrus detection using hyperspectral imaging. Computers and Electronics in Agriculture, 66(2), 201–208.Omid, M., Khojastehnazhand, M., & Tabatabaeefar, A. (2010). Estimating volume and mass of citrus fruits by image processing technique. Journal of Food Engineering, 100, 315–321.Ottavian, M., Barolo, M., & García-Muñoz, S. (2013). Maintenance of machine vision systems for product quality assessment. Part I. Addressing changes in lighting conditions. Industrial & Engineering Chemistry Research, 52, 12309–12318.Ottavian, M., Barolo, M., & García-Muñoz, S. (2014). Maintenance of machine vision systems for product quality assessment. Part II. Addressing camera replacement. Industrial & Engineering Chemistry Research, 53, 1529–1536.Palou, L. (2014). Penicillium digitatum, Penicillium italicum (green mold, blue mold). In S. Bautista-Baños (Ed.), Postharvest decay. Control strategies. London: Elsevier.Palou, L., Smilanick, J. L., Montesinos-Herrero, C., Valencia-Chamorro, S., & Pérez-Gago, M. B. (2011). Novel approaches for postharvest preservation of fresh citrus fruits. In Slaker (Ed.), Citrus fruits: properties, consumption and nutrition. New York: Nova Science Publishers, Inc..Pongnumkul, S., Chaovalit, P., & Surasvadi, N. (2015). Applications of smartphone-based sensors in agriculture: a systematic review of research. Journal of Sensors, Open Access Article ID 195308.Pourreza, A., Lee, W. S., Ehsani, R., Schueller, J. K., & Raveh, E. (2015a). An optimum method for real-time in-field detection of Huanglongbing disease using a vision sensor. Computers and Electronics in Agriculture, 110, 221–232.Pourreza, A., Lee, W. S., Etxeberria, E., & Banerjee, A. (2015b). An evaluation of a vision based sensor performance in Huanglongbing disease identification. Biosystems Engineering, 130, 13–22.Qin, J., Burks, T. F., Kim, M. S., Chao, K., & Ritenour, M. A. (2008). Citrus canker detection using hyperspectral reflectance imaging and PCA-based image classification method. Sensing and Instrumentation for Food Quality and Safety, 2(3), 168–177.Qin, J., Burks, T. F., Ritenour, M. A., & Gordon Bonn, W. (2009). Detection of citrus canker using hyperspectral reflectance imaging with spectral information divergence. Journal of Food Engineering, 93, 183–191.Qin, J., Burks, T. F., Zhao, X., Niphadkar, N., & Ritenour, M. A. (2011). Multispectral detection of citrus canker using hyperspectral band selection. Transactions of the ASABE, 54, 2331–2341.Qin, J., Burks, T. F., Zhao, X., Niphadkar, N., & Ritenour, M. A. (2012). Development of a two-band spectral imaging system for real-time citrus canker detection. Journal of Food Engineering, 108, 87–93.Sengupta, S., & Lee, W. S. (2014). Identification and determination of the number of immature green citrus fruit under different ambient light conditions.

Fruit sizing using AI: A review of methods and challenges

Fruit size at harvest is an economically important variable for high-quality table fruit production in orchards and vineyards. In addition, knowing the number and size of the fruit on the tree is essential in the framework of precise production, harvest, and postharvest management. A prerequisite for analysis of fruit in a real-world environment is the detection and segmentation from background signal. In the last five years, deep learning convolutional neural network have become the standard method for automatic fruit detection, achieving F1-scores higher than 90 %, as well as real-time processing speeds. At the same time, different methods have been developed for, mainly, fruit size and, more rarely, fruit maturity estimation from 2D images and 3D point clouds. These sizing methods are focused on a few species like grape, apple, citrus, and mango, resulting in mean absolute error values of less than 4 mm in apple fruit. This review provides an overview of the most recent methodologies developed for in-field fruit detection/counting and sizing as well as few upcoming examples of maturity estimation. Challenges, such as sensor fusion, highly varying lighting conditions, occlusions in the canopy, shortage of public fruit datasets, and opportunities for research transfer, are discussed.This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646 and 2021 LLAV 00088) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / FEDER (grants RTI2018-094222-B-I00 [PAgFRUIT project] and PID2021-126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU.info:eu-repo/semantics/publishedVersio

Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities

Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection.This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation

CitDet: A Benchmark Dataset for Citrus Fruit Detection

In this letter, we present a new dataset to advance the state of the art in detecting citrus fruit and accurately estimate yield on trees affected by the Huanglongbing (HLB) disease in orchard environments via imaging. Despite the fact that significant progress has been made in solving the fruit detection problem, the lack of publicly available datasets has complicated direct comparison of results. For instance, citrus detection has long been of interest in the agricultural research community, yet there is an absence of work, particularly involving public datasets of citrus affected by HLB. To address this issue, we enhance state-of-the-art object detection methods for use in typical orchard settings. Concretely, we provide high-resolution images of citrus trees located in an area known to be highly affected by HLB, along with high-quality bounding box annotations of citrus fruit. Fruit on both the trees and the ground are labeled to allow for identification of fruit location, which contributes to advancements in yield estimation and potential measure of HLB impact via fruit drop. The dataset consists of over 32,000 bounding box annotations for fruit instances contained in 579 high-resolution images. In summary, our contributions are the following: (i) we introduce a novel dataset along with baseline performance benchmarks on multiple contemporary object detection algorithms, (ii) we show the ability to accurately capture fruit location on tree or on ground, and finally (ii) we present a correlation of our results with yield estimations.Comment: Submitted to IEEE Robotics and Automation Letters (RA-L

31 - IMPROVING SHELF-LIFE OF FRUITS USING THERMOGRAPHY

Novel technologies have always been an indispensable part of the scientific enterprise and a catalyst for new discoveries. The invisible radiation patterns of objects are converted into visible images called thermograms or thermal images. Thermal images can be utilized to estimate the ripeness of some fruits which do not change their color from yellow to green when they are ripe. Thermal imaging techniques are very helpful since color and fluorescent analytical approaches cannot be applied to these fruits. In this work, we show the different ripeness levels of avocado using thermal images non-destructively, in two-dimension. The work is based on the fact that fruits have different specific heat capacities at different temperatures, thus making their thermal images clear indicators of ripeness

Symptoms Based Image Predictive Analysis for Citrus Orchards Using Machine Learning Techniques: A Review

In Agriculture, orchards are the deciding factor in the country’s economy. There are many orchards, and citrus and sugarcane will cover 60 percent of them. These citrus orchards satisfy the necessity of citrus fruits and citrus products, and these citrus fruits contain more vitamin C. The citrus orchards have had some problems generating good yields and quality products. Pathogenic diseases, pests, and water shortages are the three main problems that plants face. Farmers can find these problems early on with the support of machine learning and deep learning, which may also change how they feel about technology.  By doing this in agriculture, the farmers can cut off the major issues of yield and quality losses. This review gives enormous methods for identifying and classifying plant pathogens, pests, and water stresses using image-based work. In this review, the researchers present detailed information about citrus pathogens, pests, and water deficits. Methods and techniques that are currently available will be used to validate the problem. These will include pre-processing for intensification, segmentation, feature extraction, and selection processes, machine learning-based classifiers, and deep learning models. In this work, researchers thoroughly examine and outline the various research opportunities in the field. This review provides a comprehensive analysis of citrus plants and orchards; Researchers used a systematic review to ensure comprehensive coverage of this topic

A comprehensive review of fruit and vegetable classification techniques

Recent advancements in computer vision have enabled wide-ranging applications in every field of life. One such application area is fresh produce classification, but the classification of fruit and vegetable has proven to be a complex problem and needs to be further developed. Fruit and vegetable classification presents significant challenges due to interclass similarities and irregular intraclass characteristics. Selection of appropriate data acquisition sensors and feature representation approach is also crucial due to the huge diversity of the field. Fruit and vegetable classification methods have been developed for quality assessment and robotic harvesting but the current state-of-the-art has been developed for limited classes and small datasets. The problem is of a multi-dimensional nature and offers significantly hyperdimensional features, which is one of the major challenges with current machine learning approaches. Substantial research has been conducted for the design and analysis of classifiers for hyperdimensional features which require significant computational power to optimise with such features. In recent years numerous machine learning techniques for example, Support Vector Machine (SVM), K-Nearest Neighbour (KNN), Decision Trees, Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN) have been exploited with many different feature description methods for fruit and vegetable classification in many real-life applications. This paper presents a critical comparison of different state-of-the-art computer vision methods proposed by researchers for classifying fruit and vegetable

Multispectral Image Analysis of Remotely Sensed Crops

The range in topography, biodiversity, and agricultural technology has led to the emergence of precision agriculture. Precision agriculture is a farming management concept based on monitoring, measuring, and responding to crop variability. Computer vision, image analysis, and image processing are gaining considerable traction. For this paper, image analysis involves recognizing individual objects and providing insights from vegetation indices. The data acquired was remote-sensed multispectral images from blueberry, maguey, and pineapple. After computing vegetation indices, histograms were analyzed to choose thresholds. The masking of vegetation indices with threshold allowed the removal of areas with shadows and soil. The four leading vegetation indices used were the Normalized Difference Vegetation Index (NDVI), the Normalized Difference Red Edge (NDRE), the Simple Ratio, the Red Edge Chlorophyll Index, and the Visible Atmospherically Resistant Index (SAVI). This research reviews literature for acquiring, preprocessing, and analyzing remote-sensed multispectral images in precision agriculture. It compiles the theoretical framework for analyzing multispectral data. Also, it describes and implements radiometric calibration and image alignment using the custom code from the MicaSense repository. As a result, it was possible to segment the blueberry, tequila agave, and pineapple plants from the background regardless of the noisy images. Non-plant pixels were excluded and shown as transparent by masking areas with shadows and low NDVI pixels, which sometimes removed plant pixels. The NDVI and NDRE helped identify crop pixels. On the other hand, it was possible to identify the pineapple fruits from the agave plantation using the SAVI vegetation index and the thresholding method. Finally, the work identifies the problems associated with an incorrect data acquisition methodology and provides suggestions.ITESO, A. C

Automatic Assessment of Seed Germination Percentage

This research was designed to investigate an automatic seed germination rate for the top of paper germination method. Chili and guinea were adopted to be used in the experiment with a 4-time repetition and 2 sets of the germination group (4-separated plates with 50 seeds per plate, 2 sets per seed type, totally 400 seeds of chili and 400 seeds of quinea). Two detection methods were proposed binary thresholding and maximum likelihood; based on color analysis. An uncontrolled environment image taking was the way to collect image data. The results were compared to a hand-labeling groundtruth. Both methods achieved accuracy rate higher than 93% which was promising to implement this system. The binary thresholding was a lightweight method suitable for a very limited resource software environment system. The maximum likelihood was more complex. The method had more potential than the binary thresholding, it was flexible to the light condition, returned few false alarms per image (less than 3 false alarms per image). Maximum likelihood could be adopted to implement in a proper environment which still could be in a mobile device