MinkSORT: A 3D deep feature extractor using sparse convolutions to improve 3D multi-object tracking in greenhouse tomato plants

The agro-food industry is turning to robots to address the challenge of labour shortage. However, agro-food environments pose difficulties for robots due to high variation and occlusions. In the presence of these challenges, accurate world models, with information about object location, shape, and properties, are crucial for robots to perform tasks accurately. Building such models is challenging due to the complex and unique nature of agro-food environments, and errors in the model can lead to task execution issues. In this paper, we propose MinkSORT, a novel method for generating tracking features using a 3D sparse convolutional network in a deepSORT-like approach to improve the accuracy of world models in agro-food environments. We evaluated our feature extractor network using real-world data collected in a tomato greenhouse, which significantly improved the performance of our baseline model that tracks tomato positions in 3D using a Kalman filter and Mahalanobis distance. Our deep learning feature extractor improved the HOTA from 42.8% to 44.77%, the association accuracy from 32.55% to 35.55%, and the MOTA from 57.63% to 58.81%. We also evaluated different contrastive loss functions for training our deep learning feature extractor and demonstrated that our approach leads to improved performance in terms of three separate precision and recall detection outcomes. Our method improves world model accuracy, enabling robots to perform tasks such as harvesting and plant maintenance with greater efficiency and accuracy, which is essential for meeting the growing demand for food in a sustainable manner

Development and evaluation of automated localization and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3D multi-object tracking

Accurate representation and localization of relevant objects is important for robots to perform tasks. Building a generic representation that can be used across different environments and tasks is not easy, as the relevant objects vary depending on the environment and the task. Furthermore, another challenge arises in agro-food environments due to their complexity, and high levels of clutter and occlusions. In this paper, we present a method to build generic representations in highly occluded agro-food environments using multi-view perception and 3D multi-object tracking. Our representation is built upon a detection algorithm that generates a partial point cloud for each detected object. The detected objects are then passed to a 3D multi-object tracking algorithm that creates and updates the representation over time. The whole process is performed at a rate of 10 Hz. We evaluated the accuracy of the representation on a real-world agro-food environment, where it was able to successfully represent and locate tomatoes in tomato plants despite a high level of occlusion. We were able to estimate the total count of tomatoes with a maximum error of 5.08% and to track tomatoes with a tracking accuracy up to 71.47%. Additionally, we showed that an evaluation using tracking metrics gives more insight in the errors in localizing and representing the fruits.Comment: Pre-print, article submitted and in review proces

TrimBot2020: an outdoor robot for automatic gardening

Robots are increasingly present in modern industry and also in everyday life. Their applications range from health-related situations, for assistance to elderly people or in surgical operations, to automatic and driver-less vehicles (on wheels or flying) or for driving assistance. Recently, an interest towards robotics applied in agriculture and gardening has arisen, with applications to automatic seeding and cropping or to plant disease control, etc. Autonomous lawn mowers are succesful market applications of gardening robotics. In this paper, we present a novel robot that is developed within the TrimBot2020 project, funded by the EU H2020 program. The project aims at prototyping the first outdoor robot for automatic bush trimming and rose pruning.Comment: Accepted for publication at International Sympsium on Robotics 201

Automated Signal Processing Applied to Volatile-Based Inspection of Greenhouse Crops

Gas chromatograph–mass spectrometers (GC-MS) have been used and shown utility for volatile-based inspection of greenhouse crops. However, a widely recognized difficulty associated with GC-MS application is the large and complex data generated by this instrument. As a consequence, experienced analysts are often required to process this data in order to determine the concentrations of the volatile organic compounds (VOCs) of interest. Manual processing is time-consuming, labour intensive and may be subject to errors due to fatigue. The objective of this study was to assess whether or not GC-MS data can also be automatically processed in order to determine the concentrations of crop health associated VOCs in a greenhouse. An experimental dataset that consisted of twelve data files was processed both manually and automatically to address this question. Manual processing was based on simple peak integration while the automatic processing relied on the algorithms implemented in the MetAlign™ software package. The results of automatic processing of the experimental dataset resulted in concentrations similar to that after manual processing. These results demonstrate that GC-MS data can be automatically processed in order to accurately determine the concentrations of crop health associated VOCs in a greenhouse. When processing GC-MS data automatically, noise reduction, alignment, baseline correction and normalisation are required

Automation and robotics in greenhouses

This chapter provides an overview of the state of the art of automation technology in protected cultivation and looks ahead to future directions for achieving further progress in this field. The chapter provides a generic description of the greenhouse crop production process and then uses it as a reference for reviewing the state of the art in automation and robotics. The chapter explains those tasks in protected cultivation that have already been automated and identifies those tasks that are predominantly still the domain of human labour. The chapter outlines the requirements for the technology capable of doing these tasks. The chapter describes the ongoing research in automation and robotics in protected cultivation and concludes with a description of the challenges facing high-tech systems in protected cultivation

Coverage trajectory planning for a bush trimming robot arm

A novel motion planning algorithm for robotic bush trimming is presented. The algorithm is based on an optimal route search over a graph. Differently from other works in robotic surface coverage, it entails both accuracy in the surface sweeping task and smoothness in the motion of the robot arm. The proposed method requires the selection of a custom objective function in the joint space for optimal node traversal scheduling, as well as a kinematically constrained time interpolation. The algorithm was tested in simulation using a model of the Jaco arm and three target bush shapes. Analysis of the simulated motions showed how, differently from classical coverage techniques, the proposed algorithm is able to ensure high tool positioning accuracy while avoiding excessive arm motion jerkiness. It was reported that forbidding manipulation posture changes during the cutting phase of the motion is a key element for task accuracy, leading to a decrease of the tool positioning error up to 90%. Furthermore, the algorithm was validated in a real-world trimming scenario with boxwood bushes. A target of 20 mm accuracy was proposed for a trimming result to be considered successful. Results showed that on average 82% of the bush surface was affected by trimming, and 51% of the trimmed surface was cut within the desired level of accuracy. Despite the fact that the trimming accuracy turned out to be lower than the stated requirements, it was found out this was mainly a consequence of the inaccurate, early stage vision system employed to compute the target trimming surface. By contrast, the trimming motion planning algorithm generated trajectories that smoothly followed their input target and allowed effective branch cutting.</p