Prediction of strawberry yield based on receptacle detection and Bayesian inference

The receptacle of strawberry is a more direct part than the flower for predicting yield as they eventually become fruits. Thus, we tried to predict the yield by combining an AI technique for receptacle detection in images and statistical analysis on the relationship between the number of receptacles detected and the strawberry yield over a period of time. Five major cultivars were cultivated to consider the cultivar characteristics and environmental factors for two years were collected to consider the climate difference. Faster R–CNN based object detector was used to estimate the number of receptacles per strawberry plant in given two-dimensional images, which achieved a mAP of 0.6587 for our dataset. However, not all receptacles appear on the two-dimensional images, and Bayesian analysis was used to model the uncertainty associated with the number of receptacles missed by the AI. After estimating the probability of fruiting per receptacle, prediction models for the total strawberry yield at the end of harvest season were evaluated. Even though the detection accuracy was not perfect, the results indicated that counting the receptacles by object detection and estimating the probability of fruiting per receptacle by Bayesian modeling are more useful for predicting the total yield per plant than knowing its cumulative yield during the first month

L*a*b*fruits : a rapid and robust outdoor fruit detection system combining bio-inspired features with one-stage deep learning networks

Automation of agricultural processes requires systems that can accurately detect and classify produce in real industrial environments that include variation in fruit appearance due to illumination, occlusion, seasons, weather conditions, etc. In this paper we combine a visual processing approach inspired by colour-opponent theory in humans with recent advancements in one-stage deep learning networks to accurately, rapidly and robustly detect ripe soft fruits (strawberries) in real industrial settings and using standard (RGB) camera input. The resultant system was tested on an existent data-set captured in controlled conditions as well our new real-world data-set captured on a real strawberry farm over two months. We utilise F1 score, the harmonic mean of precision and recall, to show our system matches the state-of-the-art detection accuracy ( F1 : 0.793 vs. 0.799) in controlled conditions; has greater generalisation and robustness to variation of spatial parameters (camera viewpoint) in the real-world data-set ( F1 : 0.744); and at a fraction of the computational cost allowing classification at almost 30fps. We propose that the L*a*b*Fruits system addresses some of the most pressing limitations of current fruit detection systems and is well-suited to application in areas such as yield forecasting and harvesting. Beyond the target application in agriculture this work also provides a proof-of-principle whereby increased performance is achieved through analysis of the domain data, capturing features at the input level rather than simply increasing model complexity

합성곱 신경망과 완전 연결 계층의 앙상블 모델을 이용한 정밀한 파프리카 과실 발달 단계 예측

학위논문 (석사) -- 서울대학교 대학원 : 농업생명과학대학 농림생물자원학부, 2021. 2. 손정익.Accurate detection of individual fruits and prediction of their development stages enable growers to efficiently allocate labor and manage strategically. However, the prediction of the fruit development stage is challenging, especially in sweet peppers, because the fruit harvest is discrete and its immature stage is indistinguishable. An ensemble model of convolutional and fully connected neural networks was developed to detect sweet pepper (Capsicum annuum L.) fruits in images and predict their development stages. The plants were grown in four rows in a greenhouse, and images were collected in each row. Plant growth and environmental data were collected every minute and month, respectively. For predicting the fruit stage, an ensemble of convolutional neural network (CNN) and multilayer perceptron (MLP) models were used. The fruit development stage was classified into immature, breaking, and mature stages with a CNN using images. Moreover, the immature stage was internally divided into four stages with an MLP. The plant growth and environmental data and the information from the CNN output were used for the MLP input. That is, a total of six stages were classified using the CNN–MLP ensemble model. The ensemble model showed good agreement in predicting fruit development stages. The average accuracy of the six stages was F1 score = 0.77 and IoU = 0.86. The CNN-only model could classify the mature and breaking stages well, but the immature stages were not distinguished, while the MLP-only model could hardly classify the fruit stage except the immature stages. The most influential factors in classification were the data obtained from CNN and the plant growth and environment data, which contributed to the improvement of model accuracy. The ensemble models can help in appropriate labor allocation and strategic management by detecting individual fruits in images and predicting precise fruit development stages.온실에서는 고부가가치에 열매를 맺는 작물을 효율적으로 관리하는 것이 중요하다. 개별 과실을 감지하고 그것의 발달 단계를 예측함으로써 재배자가 노동력을 적재적소에 할당하고, 전략적인 관리를 할 수 있다. 그러나 파프리카의 과실 발달 단계를 예측하는 것은 과실 수확량이 불연속적이고, 미성숙 단계에서 과실 간 나타나는 외부적인 특징 차이를 구별하기 어렵기 때문에 쉽지 않다. 이 연구의 목적은 합성곱 신경망과 완전 연결 계층의 앙상블 모델을 이용하여 이미지에서 파프리카 과실을 찾아내고 과실 발달 단계를 예측하는 것이다. 실험용 온실에서 파프리카 (Capsicum annuum L.)를 4줄로 재배하였고, 각 줄에 양면에서 식물 이미지를 수집 하였다. 2020년 4월 6일부터 6월 24일까지 환경 데이터는 분 마다, 식물 생장 데이터는 월 마다 수집되었다. 과실 발달 단계는 이미지에서 합성곱 신경망을 이용하여 미성숙, 변화 중, 성숙 3 단계로 구분하였고, 미성숙 단계는 완전 연결 계층을 이용하여 다시 세부적으로 4 단계로 구분 하였다. 환경, 식물 생장 데이터 및 합성곱 신경망의 출력 정보가 완전 연결 계층에 입력으로 사용되었다. 즉, 총 6 개의 과실 발달 단계가 앙상블 모델을 이용하여 분류되었다. 앙상블 모델은 과실 발달 단계를 예측하는 데 좋은 성능을 보였다. 총 6 단계의 과실 발달 단계 분류에 평균 정확도는 F1 점수 = 0.77, IoU = 0.86이다. 합성곱 신경망만을 이용한 모델은 성숙 단계와 변화 중 단계를 잘 분류 할 수 있었지만 미성숙 단계를 구별하지 못하였다. 완전 연결 계층만을 이용한 모델은 미성숙 단계를 제외하고 과실 단계를 거의 분류 할 수 없었다. 과실 발달 단계의 분류에 가장 큰 영향을 미치는 요인은 합성곱 신경망의 출력 정보였고, 환경 및 식물 생장 데이터는 모델 정확도 향상에 기여했다. 본 연구 결과는 추후 다양한 환경에 이미지에서 개별 과실을 찾아내고, 정확한 과실 발달 단계를 예측함으로써 적절한 노동력 할당 및 전략적 관리에 도움이 될 수 있을 것으로 사료된다.ABSTRACT i CONTENTS iii LIST OF TABLES iv LIST OF FIGURES v INTRODUCTION 1 LITERATURE REVIEW 4 MATERIALS AND METHODS 9 RESULTS 24 DISCUSSION 34 CONCLUSION 39 LITERATURE CITED 40 ABSTRACT IN KOREAN 47Maste

Real-time Strawberry Detection Based on Improved YOLOv5s Architecture for Robotic Harvesting in open-field environment

This study proposed a YOLOv5-based custom object detection model to detect strawberries in an outdoor environment. The original architecture of the YOLOv5s was modified by replacing the C3 module with the C2f module in the backbone network, which provided a better feature gradient flow. Secondly, the Spatial Pyramid Pooling Fast in the final layer of the backbone network of YOLOv5s was combined with Cross Stage Partial Net to improve the generalization ability over the strawberry dataset in this study. The proposed architecture was named YOLOv5s-Straw. The RGB images dataset of the strawberry canopy with three maturity classes (immature, nearly mature, and mature) was collected in open-field environment and augmented through a series of operations including brightness reduction, brightness increase, and noise adding. To verify the superiority of the proposed method for strawberry detection in open-field environment, four competitive detection models (YOLOv3-tiny, YOLOv5s, YOLOv5s-C2f, and YOLOv8s) were trained, and tested under the same computational environment and compared with YOLOv5s-Straw. The results showed that the highest mean average precision of 80.3% was achieved using the proposed architecture whereas the same was achieved with YOLOv3-tiny, YOLOv5s, YOLOv5s-C2f, and YOLOv8s were 73.4%, 77.8%, 79.8%, 79.3%, respectively. Specifically, the average precision of YOLOv5s-Straw was 82.1% in the immature class, 73.5% in the nearly mature class, and 86.6% in the mature class, which were 2.3% and 3.7%, respectively, higher than that of the latest YOLOv8s. The model included 8.6*10^6 network parameters with an inference speed of 18ms per image while the inference speed of YOLOv8s had a slower inference speed of 21.0ms and heavy parameters of 11.1*10^6, which indicates that the proposed model is fast enough for real time strawberry detection and localization for the robotic picking.Comment: 20 pages; 15 figure

Yield sensing technologies for perennial and annual horticultural crops: a review

Yield maps provide a detailed account of crop production and potential revenue of a farm. This level of details enables a range of possibilities from improving input management, conducting on-farm experimentation, or generating profitability map, thus creating value for farmers. While this technology is widely available for field crops such as maize, soybean and grain, few yield sensing systems exist for horticultural crops such as berries, field vegetable or orchards. Nevertheless, a wide range of techniques and technologies have been investigated as potential means of sensing crop yield for horticultural crops. This paper reviews yield monitoring approaches that can be divided into proximal, either direct or indirect, and remote measurement principles. It reviews remote sensing as a way to estimate and forecast yield prior to harvest. For each approach, basic principles are explained as well as examples of application in horticultural crops and success rate. The different approaches provide whether a deterministic (direct measurement of weight for instance) or an empirical (capacitance measurements correlated to weight for instance) result, which may impact transferability. The discussion also covers the level of precision required for different tasks and the trend and future perspectives. This review demonstrated the need for more commercial solutions to map yield of horticultural crops. It also showed that several approaches have demonstrated high success rate and that combining technologies may be the best way to provide enough accuracy and robustness for future commercial systems

Deep neural networks with transfer learning for forest variable estimation using sentinel-2 imagery in boreal forest

Estimation of forest structural variables is essential to provide relevant insights for public and private stakeholders in forestry and environmental sectors. Airborne light detection and ranging (LiDAR) enables accurate forest inventory, but it is expensive for large area analyses. Continuously increasing volume of open Earth Observation (EO) imagery from high-resolution (|BIAS%| = 0.8%). We found 3×3 pixels to be the optimal size for the sampling window, and two to three hidden layer DNNs to produce the best results with relatively small improvement to single hidden layer networks. Including CHM features with S2 data and additional features led to reduced relative RMSE (RMSE% = 28.6–30.7%) but increased the absolute value of relative bias (|BIAS%| = 0.9–4.0%). Transfer learning was found to be beneficial mainly with training data sets containing less than 250 field plots. The performance differences of DNN and random forest models were marginal. Our results contribute to improved structural variable estimation performance in boreal forests with the proposed image sampling and input feature concept

High-throughput phenotyping for breeding targets - Current status and future directions of strawberry trait automation

Automated image-based phenotyping has become widely accepted in crop phenotyping, particularly in cereal crops, yet few traits used by breeders in the strawberry industry have been automated. Early phenotypic assessment remains largely qualitative in this area since the manual phenotyping process is laborious and domain experts are constrained by time. Precision agriculture, facilitated by robotic technologies, is increasing in the strawberry industry, and the development of quantitative automated phenotyping methods is essential to ensure that breeding programs remain economically competitive. In this review, we investigate the external morphological traits relevant to the breeding of strawberries that have been automated and assess the potential for automation of traits that are still evaluated manually, highlighting challenges and limitations of the approaches used, particularly when applying high-throughput strawberry phenotyping in real-world environmental conditions

A synthetic wheat l-system to accurately detect and visualise wheat head anomalies

Greater knowledge of wheat crop phenology and growth and improvements in measurement are beneficial to wheat agronomy and productivity. This is constrained by a lack of public plant datasets. Collecting plant data is expensive and time consuming and methods to augment this with synthetic data could address this issue. This paper describes a cost-effective and accurate Synthetic Wheat dataset which has been created by a novel L-system, based on technological advances in cameras and deep learning. The dataset images have been automatically created, categorised, masked and labelled, and used to successfully train a synthetic neural network. This network has been shown to accurately recognise wheat in pasture images taken from the Global Wheat dataset, which provides for the ongoing interest in the phenotyping of wheat characteristics around the world. The proven Mask R-CNN and Detectron2 frameworks have been used, and the created network is based on the public COCO format. The research question is “How can L-system knowledge be used to create an accurate synthetic wheat dataset and to make cost-effective wheat crop measurements?”

Remote Sensing Object Detection Meets Deep Learning: A Meta-review of Challenges and Advances

Remote sensing object detection (RSOD), one of the most fundamental and challenging tasks in the remote sensing field, has received longstanding attention. In recent years, deep learning techniques have demonstrated robust feature representation capabilities and led to a big leap in the development of RSOD techniques. In this era of rapid technical evolution, this review aims to present a comprehensive review of the recent achievements in deep learning based RSOD methods. More than 300 papers are covered in this review. We identify five main challenges in RSOD, including multi-scale object detection, rotated object detection, weak object detection, tiny object detection, and object detection with limited supervision, and systematically review the corresponding methods developed in a hierarchical division manner. We also review the widely used benchmark datasets and evaluation metrics within the field of RSOD, as well as the application scenarios for RSOD. Future research directions are provided for further promoting the research in RSOD.Comment: Accepted with IEEE Geoscience and Remote Sensing Magazine. More than 300 papers relevant to the RSOD filed were reviewed in this surve

Fruit sizing using AI: A review of methods and challenges

Fruit size at harvest is an economically important variable for high-quality table fruit production in orchards and vineyards. In addition, knowing the number and size of the fruit on the tree is essential in the framework of precise production, harvest, and postharvest management. A prerequisite for analysis of fruit in a real-world environment is the detection and segmentation from background signal. In the last five years, deep learning convolutional neural network have become the standard method for automatic fruit detection, achieving F1-scores higher than 90 %, as well as real-time processing speeds. At the same time, different methods have been developed for, mainly, fruit size and, more rarely, fruit maturity estimation from 2D images and 3D point clouds. These sizing methods are focused on a few species like grape, apple, citrus, and mango, resulting in mean absolute error values of less than 4 mm in apple fruit. This review provides an overview of the most recent methodologies developed for in-field fruit detection/counting and sizing as well as few upcoming examples of maturity estimation. Challenges, such as sensor fusion, highly varying lighting conditions, occlusions in the canopy, shortage of public fruit datasets, and opportunities for research transfer, are discussed.This work was partly funded by the Department of Research and Universities of the Generalitat de Catalunya (grants 2017 SGR 646 and 2021 LLAV 00088) and by the Spanish Ministry of Science and Innovation / AEI/10.13039/501100011033 / FEDER (grants RTI2018-094222-B-I00 [PAgFRUIT project] and PID2021-126648OB-I00 [PAgPROTECT project]). The Secretariat of Universities and Research of the Department of Business and Knowledge of the Generalitat de Catalunya and European Social Fund (ESF) are also thanked for financing Juan Carlos Miranda’s pre-doctoral fellowship (2020 FI_B 00586). The work of Jordi Gené-Mola was supported by the Spanish Ministry of Universities through a Margarita Salas postdoctoral grant funded by the European Union - NextGenerationEU.info:eu-repo/semantics/publishedVersio