Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities

Fruit detection and localization will be essential for future agronomic management of fruit crops, with applications in yield prediction, yield mapping and automated harvesting. RGB-D cameras are promising sensors for fruit detection given that they provide geometrical information with color data. Some of these sensors work on the principle of time-of-flight (ToF) and, besides color and depth, provide the backscatter signal intensity. However, this radiometric capability has not been exploited for fruit detection applications. This work presents the KFuji RGB-DS database, composed of 967 multi-modal images containing a total of 12,839 Fuji apples. Compilation of the database allowed a study of the usefulness of fusing RGB-D and radiometric information obtained with Kinect v2 for fruit detection. To do so, the signal intensity was range corrected to overcome signal attenuation, obtaining an image that was proportional to the reflectance of the scene. A registration between RGB, depth and intensity images was then carried out. The Faster R-CNN model was adapted for use with five-channel input images: color (RGB), depth (D) and range-corrected intensity signal (S). Results show an improvement of 4.46% in F1-score when adding depth and range-corrected intensity channels, obtaining an F1-score of 0.898 and an AP of 94.8% when all channels are used. From our experimental results, it can be concluded that the radiometric capabilities of ToF sensors give valuable information for fruit detection.This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya, the Spanish Ministry of Economy and Competitiveness and the European Regional Development Fund (ERDF) under Grants 2017SGR 646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry of Education is thanked for Mr. J. Gené’s predoctoral fellowships (FPU15/03355). We would also like to thank Nufri and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Adria Carbó for his assistance in Faster R-CNN implementation

Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

Making sense of light: the use of optical spectroscopy techniques in plant sciences and agriculture

As a result of the development of non-invasive optical spectroscopy, the number of prospective technologies of plant monitoring is growing. Being implemented in devices with different functions and hardware, these technologies are increasingly using the most advanced data processing algorithms, including machine learning and more available computing power each time. Optical spectroscopy is widely used to evaluate plant tissues, diagnose crops, and study the response of plants to biotic and abiotic stress. Spectral methods can also assist in remote and non-invasive assessment of the physiology of photosynthetic biofilms and the impact of plant species on biodiversity and ecosystem stability. The emergence of high-throughput technologies for plant phenotyping and the accompanying need for methods for rapid and non-contact assessment of plant productivity has generated renewed interest in the application of optical spectroscopy in fundamental plant sciences and agriculture. In this perspective paper, starting with a brief overview of the scientific and technological backgrounds of optical spectroscopy and current mainstream techniques and applications, we foresee the future development of this family of optical spectroscopic methodologies.info:eu-repo/semantics/publishedVersio

Ensuring Agricultural Sustainability through Remote Sensing in the Era of Agriculture 5.0

This work was supported by the projects: "VIRTUOUS" funded by the European Union's Horizon 2020 Project H2020-MSCA-RISE-2019. Ref. 872181, "SUSTAINABLE" funded by the European Union's Horizon 2020 Project H2020-MSCA-RISE-2020. Ref. 101007702 and the "Project of Excellence" from Junta de Andalucia 2020. Ref. P18-H0-4700. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.Timely and reliable information about crop management, production, and yield is considered of great utility by stakeholders (e.g., national and international authorities, farmers, commercial units, etc.) to ensure food safety and security. By 2050, according to Food and Agriculture Organization (FAO) estimates, around 70% more production of agricultural products will be needed to fulfil the demands of the world population. Likewise, to meet the Sustainable Development Goals (SDGs), especially the second goal of “zero hunger”, potential technologies like remote sensing (RS) need to be efficiently integrated into agriculture. The application of RS is indispensable today for a highly productive and sustainable agriculture. Therefore, the present study draws a general overview of RS technology with a special focus on the principal platforms of this technology, i.e., satellites and remotely piloted aircrafts (RPAs), and the sensors used, in relation to the 5th industrial revolution. Nevertheless, since 1957, RS technology has found applications, through the use of satellite imagery, in agriculture, which was later enriched by the incorporation of remotely piloted aircrafts (RPAs), which is further pushing the boundaries of proficiency through the upgrading of sensors capable of higher spectral, spatial, and temporal resolutions. More prominently, wireless sensor technologies (WST) have streamlined real time information acquisition and programming for respective measures. Improved algorithms and sensors can, not only add significant value to crop data acquisition, but can also devise simulations on yield, harvesting and irrigation periods, metrological data, etc., by making use of cloud computing. The RS technology generates huge sets of data that necessitate the incorporation of artificial intelligence (AI) and big data to extract useful products, thereby augmenting the adeptness and efficiency of agriculture to ensure its sustainability. These technologies have made the orientation of current research towards the estimation of plant physiological traits rather than the structural parameters possible. Futuristic approaches for benefiting from these cutting-edge technologies are discussed in this study. This study can be helpful for researchers, academics, and young students aspiring to play a role in the achievement of sustainable agriculture.European Commission 101007702 872181Junta de Andalucia P18-H0-470

Computer vision reading on stickers and direct part marking on horticultural products : challenges and possible solutions

Traceability of products from production to the consumer has led to a technological advancement in product identification. There has been development from the use of traditional one-dimensional barcodes (EAN-13, Code 128, etc.) to 2D (two-dimensional) barcodes such as QR (Quick Response) and Data Matrix codes. Over the last two decades there has been an increased use of Radio Frequency Identification (RFID) and Direct Part Marking (DPM) using lasers for product identification in agriculture. However, in agriculture there are still considerable challenges to adopting barcodes, RFID and DPM technologies, unlike in industry where these technologies have been very successful. This study was divided into three main objectives. Firstly, determination of the effect of speed, dirt, moisture and bar width on barcode detection was carried out both in the laboratory and a flower producing company, Brandkamp GmbH. This study developed algorithms for automation and detection of Code 128 barcodes under rough production conditions. Secondly, investigations were carried out on the effect of low laser marking energy on barcode size, print growth, colour and contrast on decoding 2D Data Matrix codes printed directly on apples. Three different apple varieties (Golden Delicious, Kanzi and Red Jonaprince) were marked with various levels of energy and different barcode sizes. Image processing using Halcon 11.0.1 (MvTec) was used to evaluate the markings on the apples. Finally, the third objective was to evaluate both algorithms for 1D and 2D barcodes. According to the results, increasing the speed and angle of inclination of the barcode decreased barcode recognition. Also, increasing the dirt on the surface of the barcode resulted in decreasing the successful detection of those barcodes. However, there was 100% detection of the Code 128 barcode at the company’s production speed (0.15 m/s) with the proposed algorithm. Overall, the results from the company showed that the image-based system has a future prospect for automation in horticultural production systems. It overcomes the problem of using laser barcode readers. The results for apples showed that laser energy, barcode size, print growth, type of product, contrast between the markings and the colour of the products, the inertia of the laser system and the days of storage all singularly or in combination with each other influence the readability of laser Data Matrix codes and implementation on apples. There was poor detection of the Data Matrix code on Kanzi and Red Jonaprince due to the poor contrast between the markings on their skins. The proposed algorithm is currently working successfully on Golden Delicious with 100% detection for 10 days using energy 0.108 J mm-2 and a barcode size of 10 × 10 mm2. This shows that there is a future prospect of not only marking barcodes on apples but also on other agricultural products for real time production

A novel CNN gap layer for growth prediction of palm tree plantlings

Monitoring palm tree seedlings and plantlings presents a formidable challenge because of the microscopic size of these organisms and the absence of distinguishing morphological characteristics. There is a demand for technical approaches that can provide restoration specialists with palm tree seedling monitoring systems that are high-resolution, quick, and environmentally friendly. It is possible that counting plantlings and identifying them down to the genus level will be an extremely time-consuming and challenging task. It has been demonstrated that convolutional neural networks, or CNNs, are effective in many aspects of image recognition; however, the performance of CNNs differs depending on the application. The performance of the existing CNN-based models for monitoring and predicting plantlings growth could be further improved. To achieve this, a novel Gap Layer modified CNN architecture (GL-CNN) has been proposed with an IoT effective monitoring system and UAV technology. The UAV is employed for capturing plantlings images and the IoT model is utilized for obtaining the ground truth information of the plantlings health. The proposed model is trained to predict the successful and poor seedling growth for a given set of palm tree plantling images. The proposed GL-CNN architecture is novel in terms of defined convolution layers and the gap layer designed for output classification. There are two 64×3 conv layers, two 128×3 conv layers, two 256×3 conv layers and one 512×3 conv layer for processing of input image. The output obtained from the gap layer is modulated using the ReLU classifier for determining the seedling classification. To evaluate the proposed system, a new dataset of palm tree plantlings was collected in real time using UAV technology. This dataset consists of images of palm tree plantlings. The evaluation results showed that the proposed GL-CNN model performed better than the existing CNN architectures with an average accuracy of 95.96%

Proceedings of the European Conference on Agricultural Engineering AgEng2021

This proceedings book results from the AgEng2021 Agricultural Engineering Conference under auspices of the European Society of Agricultural Engineers, held in an online format based on the University of Évora, Portugal, from 4 to 8 July 2021. This book contains the full papers of a selection of abstracts that were the base for the oral presentations and posters presented at the conference. Presentations were distributed in eleven thematic areas: Artificial Intelligence, data processing and management; Automation, robotics and sensor technology; Circular Economy; Education and Rural development; Energy and bioenergy; Integrated and sustainable Farming systems; New application technologies and mechanisation; Post-harvest technologies; Smart farming / Precision agriculture; Soil, land and water engineering; Sustainable production in Farm buildings

Mind the Gap: Developments in Autonomous Driving Research and the Sustainability Challenge

Scientific knowledge on autonomous-driving technology is expanding at a faster-than-ever pace. As a result, the likelihood of incurring information overload is particularly notable for researchers, who can struggle to overcome the gap between information processing requirements and information processing capacity. We address this issue by adopting a multi-granulation approach to latent knowledge discovery and synthesis in large-scale research domains. The proposed methodology combines citation-based community detection methods and topic modeling techniques to give a concise but comprehensive overview of how the autonomous vehicle (AV) research field is conceptually structured. Thirteen core thematic areas are extracted and presented by mining the large data-rich environments resulting from 50 years of AV research. The analysis demonstrates that this research field is strongly oriented towards examining the technological developments needed to enable the widespread rollout of AVs, whereas it largely overlooks the wide-ranging sustainability implications of this sociotechnical transition. On account of these findings, we call for a broader engagement of AV researchers with the sustainability concept and we invite them to increase their commitment to conducting systematic investigations into the sustainability of AV deployment. Sustainability research is urgently required to produce an evidence-based understanding of what new sociotechnical arrangements are needed to ensure that the systemic technological change introduced by AV-based transport systems can fulfill societal functions while meeting the urgent need for more sustainable transport solutions

Predicting plant environmental exposure using remote sensing

Wheat is one of the most important crops globally with 776.4 million tonnes produced in 2019 alone. However, 10% of all wheat yield is predicted to be lost to Septoria Tritici Blotch (STB) caused by Zymoseptoria tritici (Z. tritici). Throughout Europe farmers spend £0.9 billion annually on preventative fungicide regimes to protect wheat against Z. tritici. A preventative fungicide regime is used as Z. tritici has a 9-16 day asymptomatic latent phase which makes it difficult to detect before symptoms develop, after which point fungicide intervention is ineffective. In the second chapter of my thesis I use hyperspectral sensing and imaging techniques, analysed with machine learning to detect and predict symptomatic Z. tritici infection in winter wheat, in UK based field trials, with high accuracy. This has the potential to improve detection and monitoring of symptomatic Z. tritici infection and could facilitate precision agriculture methods, to use in the subsequent growing season, that optimise fungicide use and increase yield. In the third chapter of my thesis, I develop a multispectral imaging system which can detect and utilise none visible shifts in plant leaf reflectance to distinguish plants based on the nitrogen source applied. Currently, plants are treated with nitrogen sources to increase growth and yield, the most common being calcium ammonium nitrate. However, some nitrogen sources are used in illicit activities. Ammonium nitrate is used in explosive manufacture and ammonium sulphate in the cultivation and extraction of the narcotic cocaine from Erythroxylum spp. In my third chapter I show that hyperspectral sensing, multispectral imaging, and machine learning image analysis can be used to visualise and differentiate plants exposed to different nefarious nitrogen sources. Metabolomic analysis of leaves from plants exposed to different nitrogen sources reveals shifts in colourful metabolites that may contribute to altered reflectance signatures. This suggests that different nitrogen feeding regimes alter plant secondary metabolism leading to changes in plant leaf reflectance detectable via machine learning of multispectral data but not the naked eye. These results could facilitate the development of technologies to monitor illegal activities involving various nitrogen sources and further inform nitrogen application requirements in agriculture. In my fourth chapter I implement and adapt the hyperspectral sensing, multispectral imaging and machine learning image analysis developed in the third chapter to detect asymptomatic (and symptomatic) Z. tritici infection in winter wheat, in UK based field trials, with high accuracy. This has the potential to improve detection and monitoring of all stages of Z. tritici infection and could facilitate precision agriculture methods to be used during the current growing season that optimise fungicide use and increase yield.Open Acces