Looking behind occlusions: A study on amodal segmentation for robust on-tree apple fruit size estimation

The detection and sizing of fruits with computer vision methods is of interest because it provides relevant information to improve the management of orchard farming. However, the presence of partially occluded fruits limits the performance of existing methods, making reliable fruit sizing a challenging task. While previous fruit segmentation works limit segmentation to the visible region of fruits (known as modal segmentation), in this work we propose an amodal segmentation algorithm to predict the complete shape, which includes its visible and occluded regions. To do so, an end-to-end convolutional neural network (CNN) for simultaneous modal and amodal instance segmentation was implemented. The predicted amodal masks were used to estimate the fruit diameters in pixels. Modal masks were used to identify the visible region and measure the distance between the apples and the camera using the depth image. Finally, the fruit diameters in millimetres (mm) were computed by applying the pinhole camera model. The method was developed with a Fuji apple dataset consisting of 3925 RGB-D images acquired at different growth stages with a total of 15,335 annotated apples, and was subsequently tested in a case study to measure the diameter of Elstar apples at different growth stages. Fruit detection results showed an F1-score of 0.86 and the fruit diameter results reported a mean absolute error (MAE) of 4.5 mm and R2 = 0.80 irrespective of fruit visibility. Besides the diameter estimation, modal and amodal masks were used to automatically determine the percentage of visibility of measured apples. This feature was used as a confidence value, improving the diameter estimation to MAE = 2.93 mm and R2 = 0.91 when limiting the size estimation to fruits detected with a visibility higher than 60%. The main advantages of the present methodology are its robustness for measuring partially occluded fruits and the capability to determine the visibility percentage. The main limitation is that depth images were generated by means of photogrammetry methods, which limits the efficiency of data acquisition. To overcome this limitation, future works should consider the use of commercial RGB-D sensors. The code and the dataset used to evaluate the method have been made publicly available at https://github.com/GRAP-UdL-AT/Amodal_Fruit_SizingThis work was partly funded by the Departament de Recerca i Universitats de la Generalitat de Catalunya (grant 2021 LLAV 00088), the Spanish Ministry of Science, Innovation and Universities (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project] and PID2020-117142GB-I00 [DeeLight project] by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union). The work of Jordi Gené Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU. We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) for their support during data acquisition, and Pieter van Dalfsen and Dirk de Hoog from Wageningen University & Research for additional data collection used in the case study.info:eu-repo/semantics/publishedVersio

Assessing automatic data processing algorithms for RGB-D cameras to predict fruit size and weight in apples

Data acquired using an RGB-D Azure Kinect DK camera were used to assess different automatic algorithms to estimate the size, and predict the weight of non-occluded and occluded apples. The programming of the algorithms included: (i) the extraction of images of regions of interest (ROI) using manual delimitation of bounding boxes or binary masks; (ii) estimating the lengths of the major and minor geometric axes for the purpose of apple sizing; and (iii) predicting the final weight by allometric modelling. In addition to the use of bounding boxes, the algorithms also allowed other post-mask settings (circles, ellipses and rotated rectangles) to be implemented, and different depth options (distance between the RGB-D camera and the fruits detected) for subsequent sizing through the application of the thin lens theory. Both linear and nonlinear allometric models demonstrated the ability to predict apple weight with a high degree of accuracy (R2 greater than 0.942 and RMSE < 16 g). With respect to non-occluded apples, the best weight predictions were achieved using a linear allometric model including both the major and minor axes of the apples as predictors. The mean absolute percentage error (MAPE) ranged from 5.1% to 5.7% with respective RMSE of 11.09 g and 13.02 g, depending to whether circles, ellipses, or bounding boxes were used to adjust fruit shape. The results were therefore promising and open up the possibility of implementing reliable in-field apple measurements in real time. Importantly, final weight prediction error and intermediate size estimation errors (from sizing algorithms) interact but in a way that is not easily quantifiable when weight allometric models with implicit prediction error are used. In addition, allometric models should be reviewed when applied to other apple cultivars, fruit development stages or even for different fruit growth conditions depending on canopy management.This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017, SGR 646 and 2021 LLAV 00088), by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / ERDF (grants RTI2018-094222-B-I00 [PAgFRUIT project], PID2021-126648OB-I00 [PAgPROTECT project]) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / European Union NextGeneration / PRTR (grantTED2021-131871B-I00 [DIGIFRUIT project]). We would also like to thank the Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and the European Social Fund (ESF) for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU.info:eu-repo/semantics/publishedVersio

Fruit sizing using AI: A review of methods and challenges

Fruit size at harvest is an economically important variable for high-quality table fruit production in orchards and vineyards. In addition, knowing the number and size of the fruit on the tree is essential in the framework of precise production, harvest, and postharvest management. A prerequisite for analysis of fruit in a real-world environment is the detection and segmentation from background signal. In the last five years, deep learning convolutional neural network have become the standard method for automatic fruit detection, achieving F1-scores higher than 90 %, as well as real-time processing speeds. At the same time, different methods have been developed for, mainly, fruit size and, more rarely, fruit maturity estimation from 2D images and 3D point clouds. These sizing methods are focused on a few species like grape, apple, citrus, and mango, resulting in mean absolute error values of less than 4 mm in apple fruit. This review provides an overview of the most recent methodologies developed for in-field fruit detection/counting and sizing as well as few upcoming examples of maturity estimation. Challenges, such as sensor fusion, highly varying lighting conditions, occlusions in the canopy, shortage of public fruit datasets, and opportunities for research transfer, are discussed.This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646 and 2021 LLAV 00088) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / FEDER (grants RTI2018-094222-B-I00 [PAgFRUIT project] and PID2021-126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU.info:eu-repo/semantics/publishedVersio

Sperm quality, semen production, and fertility in young Norwegian Red bulls

Ved bruk av genomisk seleksjon i storfeavlen blir eliteokser selektert basert på deres estimerte genomiske avlsverdier i stedet for ved avkomsgransking. Oksene er derfor yngre når de blir tatt i bruk i sædproduksjon enn tidligere. Hovedmålet med denne avhandlingen var å identifisere nye indikatorer for når sædproduksjonen er i gang hos unge Norsk Rødt Fe okser, og som kan måles i løpet av testperioden og gi informasjon om oksenes potensielle fremtidige sædproduksjon, aksept for semin-stasjonen samt fruktbarhet i felt. I Artikkel 1 ble flowcytometri og Computer-Aided Sperm Analysis brukt til å analysere ulike spermiekvalitetsparametere i ejakulater fra 65 okser i alderen 9-13 måneder. Sædprøver ble utsatt for stresstester og kryokonservering. Oksene ble klassifisert i tre grupper med ulik respons på spermie-stresstester. Ved å benytte spermie-stresstester, kryokonservering og morfologianalyse tidlig i testperioden, kan en få verdifull innsikt i når oksene er tilstrekkelig utviklet for sædproduksjon. Med denne tilnærmingen vil en kunne ta i bruk yngre okser i sæduttak og -produksjon, og dermed bidra til redusert generasjonsintervall og økt genetisk framgang. I Artikkel 2 ble det fokusert på å undersøke potensialet til insulin-like factor 3 som en biomarkør for å predikere når sædproduksjonen starter hos unge Norsk Rødt Fe okser. Det ble tatt blodprøver og samtidig utført målinger av skrotumomkrets på 142 okser på fire tidspunkt mellom 2 og 12 måneders alder. Studien hadde som mål å belyse sammenhenger mellom nivået av insulin-like factor 3, skrotumomkrets og ulike sædparametere. Det ble funnet en positiv korrelasjon mellom insulin-like factor 3 og skrotumomkretsen, men det ble ikke funnet signifikante sammenhenger mellom skrotumomkretsen og sædparametere. På grunn av betydelige individuelle variasjoner i den undersøkte norske okse-populasjonen, er insulin-like factor 3 foreløpig ikke en egnet biomarkør til å kunne predikere når sædproduksjonen starter hos denne rasen. I Artikkel 3 presenteres en automatisert metode for å måle skrotumomkretsen hos Norsk Rødt Fe okser ved hjelp av 3D-bilder og konvolusjonelle nevrale nettverk. 3D-bilder ble tatt samtidig som manuelle målinger av skrotumomkretsen ble utført på oksene, noe som ble gjentatt ved ulike aldere. Studien sammenlignet de manuelle og automatiserte målingene oppnådd ved semantisk segmentering. Det ble vist at de automatiserte målingene av skrotumomkretsen ga tilsvarende resultater som de manuelle målingene. Gjennomsnittlig prediksjonsfeil varierte med oksenes alder og kvaliteten på 3D-bildene. Denne nye målemetoden har potensiale til å kunne implementeres i breeding soundness evaluation ved testings- og seminstasjoner, og kan gi en rask og effektiv vurdering av skrotumomkretsen.Abstract. With the application of genomic selection in dairy cattle breeding, the choice of elite sires is based on their estimated genomic breeding values instead of progeny testing. Consequently, bulls are introduced into semen production at a younger age than previously. The main aim of this thesis was to identify novel early indicators of sperm production onset and maturity status of young Norwegian Red bulls during their performance test period, to provide insight into their potential future semen production, acceptance for the AI station, and field fertility. In Paper 1, flow cytometry and computer-aided sperm analysis were used to analyse various sperm quality parameters in ejaculates collected from 65 bulls aged 9-13 months. Semen samples were subjected to stress tests and cryopreservation. The bulls were classified into three clusters with different responses to sperm stress tests. By incorporating sperm stress tests, cryopreservation, and early morphology analysis, valuable insights into the maturity of bulls for sperm production could be gained. This approach would allow for the integration of younger bulls into semen collection, facilitating reduced generation interval and increased genetic gain. The focus in Paper 2 is on investigating the potential of insulin-like factor 3 as a biomarker for predicting the onset of sperm production in young Norwegian Red bulls. Blood samples and scrotal circumference measurements were collected from 142 bulls at four time-points between 2 and 12 months of age. The aim of the study was to determine the relationship between insulin-like factor 3, scrotal circumference, and semen characteristics. While a positive correlation was found between insulin-like factor 3 and scrotal circumference, no significant correlations were observed between scrotal circumference and semen characteristics. Due to the substantial interindividual variability in the Norwegian Red bull population, insulin-like factor 3 is currently not a reliable biomarker for predicting the onset of sperm production in this breed. In Paper 3 an automated method for measuring scrotal circumference of Norwegian Red bulls using 3D images and convolutional neural networks is presented. 3D images were captured, and manual scrotal circumference measurements made of bulls at different ages. The study compared the manual and automated measurements obtained through semantic segmentation. The results showed that the automated scrotal circumference measurements were similar to manual measurements. Mean prediction error varied depending on bull age and image quality. This novel measurement method has the potential to be implemented in bull breeding soundness evaluations at performance test stations and semen collection centers, providing a fast and efficient approach for assessing scrotal circumference.publishedVersio