Fruit Detectability Analysis for Different Camera Positions in Sweet-Pepper

For robotic harvesting of sweet-pepper fruits in greenhouses a sensor system is required to detect and localize the fruits on the plants. Due to the complex structure of the plant, most fruits are (partially) occluded when an image is taken from one viewpoint only. In this research the effect of multiple camera positions and viewing angles on fruit visibility and detectability was investigated. A recording device was built which allowed to place the camera under different azimuth and zenith angles and to move the camera horizontally along the crop row. Fourteen camera positions were chosen and the fruit visibility in the recorded images was manually determined for each position. For images taken from one position only with the criterion of maximum 50% occlusion per fruit, the fruit detectability (FD) was in no case higher than 69%. The best single positions were the front views and looking with a zenith angle of 60° upwards. The FD increased when a combination was made of multiple viewpoint positions. With a combination of five favourite positions the maximum FD was 90%

Robotic Harvesting of Fruiting Vegetables: A Simulation Approach in V-REP, ROS and MATLAB

In modern agriculture, there is a high demand to move from tedious manual harvesting to a continuously automated operation. This chapter reports on designing a simulation and control platform in V-REP, ROS, and MATLAB for experimenting with sensors and manipulators in robotic harvesting of sweet pepper. The objective was to provide a completely simulated environment for improvement of visual servoing task through easy testing and debugging of control algorithms with zero damage risk to the real robot and to the actual equipment. A simulated workspace, including an exact replica of different robot manipulators, sensing mechanisms, and sweet pepper plant, and fruit system was created in V-REP. Image moment method visual servoing with eye-in-hand configuration was implemented in MATLAB, and was tested on four robotic platforms including Fanuc LR Mate 200iD, NOVABOT, multiple linear actuators, and multiple SCARA arms. Data from simulation experiments were used as inputs of the control algorithm in MATLAB, whose outputs were sent back to the simulated workspace and to the actual robots. ROS was used for exchanging data between the simulated environment and the real workspace via its publish-and-subscribe architecture. Results provided a framework for experimenting with different sensing and acting scenarios, and verified the performance functionality of the simulator

Three-dimensional reconstruction of plant shoots from multiple images using an active vision system

The reconstruction of 3D models of plant shoots is a challenging problem central to the emerging discipline of plant phenomics – the quantitative measurement of plant structure and function. Current approaches are, however, often limited by the use of static cameras. We propose an automated active phenotyping cell to reconstruct plant shoots from multiple images using a turntable capable of rotating 360 degrees and camera mounted robot arm. To overcome the problem of static camera positions we develop an algorithm capable of analysing the environment and determining viewpoints from which to capture initial images suitable for use by a structure from motion technique

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html. Dades primàries associades a l'article http://hdl.handle.net/10459.1/68505This work was partly funded by the Secretaria d’Universitats i Recerca del Departament d’Empresa i Coneixement de la Generalitat de Catalunya (grant 2017 SGR646), the Spanish Ministry of Economy and Competitiveness (project AGL2013-48297-C2-2-R) and the Spanish Ministry of Science, Innovation and Universities (project RTI2018-094222-B-I00). Part of the work was also developed within the framework of the project TEC2016-75976-R, financed by the Spanish Ministry of Economy, Industry and Competitiveness and the European Regional Development Fund (ERDF). The Spanish Ministry of Educationis thanked for Mr. J.Gené’s pre-doctoral fellowships (FPU15/03355). We would also like to thank Nufri (especially Santiago Salamero and Oriol Morreres) and Vicens Maquinària Agrícola S.A. for their support during data acquisition, and Ernesto Membrillo and Roberto Maturino for their support in dataset labelling

CROPS : high tech agricultural robots

In the EU-funded CROPS (Clever Robots for Crops) project high tech robots are developed for site-specific spraying and selective harvesting of fruit and fruit vegetables. The harvesting robots are being designed to harvest high-value crops such as greenhouse vegetables, fruits in orchards and grapes for premium wines. The CROPS robots are also developed for canopy spraying in orchards and for precision target spraying in grape vines to reduce the use of pesticides. A CROPS robot will be able to detect the fruit, sense its ripeness, then move to grasp and gently detach only the ripe fruit. For crop protection the canopy sprayer can detect contours of trees in an orchard and consequently only spraying on the trees and the precision target sprayer can detect diseases on leaves of vine grapes and only spray pesticides on the affected spots of the leaves. In the CROPS project also attention is paid to reliable detection and classification of objects and obstacles for autonomous navigation in a safe way in plantations and forests. For the several applications within the CROPS project platforms were developed. Sensing systems and appropriate vision algorithms for the platforms have been developed. For the software platform the Robot Operating System (ROS) is used. A 9 degrees of freedom (DOF) manipulator was designed and built and tested for sweet-pepper harvesting, apple harvesting and in close range spraying. The 9-DOF manipulator is modular, since the joint configuration can be adapted to the applications, e.g. 6 DOF for the close range spraying. For the different applications different end-effectors were designed and tested. The main results of the CROPS project will be the applications, the so-called demonstrators For sweet pepper a platform that can move in between the crop rows on the common greenhouse rail system which also serves as heating pipes was built and equipped with a sensing and lightning system, the manipulator and end-effectors. The complete system was tested and showed to growers in a lab situation. The apple harvesting platform is based on a current mechanical grape harvester. In discussion with growers so-called 'walls of fruit trees' have been designed which bring robots closer to the practice. This system, equipped with a sensing system the CROPS manipulator and a special end-effector, has been successfully tested in an orchard. A canopy-optimised sprayer has been designed as a trailed sprayer with a centrifugal blower. The system has been successfully tested in an orchard with a significant reduction of pesticide use. For close range target spraying the spraying robot in a greenhouse experiment with grape vines reduced the pesticide consumption with 84%