Towards a swarm robotic system for autonomous cereal harvesting

Swarm robotics is an emerging technology that has the potential to revolutionise precision agriculture by coordinating fleets of small autonomous vehicles to minimise soil damage, increase farming resolution, lower the cost of automation, and provide solutions that are intrinsically safer and more sustainable than large monolithic systems. Here, we propose a novel swarm robotic system for autonomous harvesting of cereal crops such as wheat and barley. In contrast to existing agricultural swarm robotic systems, we intend to use small autonomous versions of traditional agricultural vehicles, in an attempt to narrow the skills gap for future end-users

Robotic Technologies for High-Throughput Plant Phenotyping: Contemporary Reviews and Future Perspectives

Phenotyping plants is an essential component of any effort to develop new crop varieties. As plant breeders seek to increase crop productivity and produce more food for the future, the amount of phenotype information they require will also increase. Traditional plant phenotyping relying on manual measurement is laborious, time-consuming, error-prone, and costly. Plant phenotyping robots have emerged as a high-throughput technology to measure morphological, chemical and physiological properties of large number of plants. Several robotic systems have been developed to fulfill different phenotyping missions. In particular, robotic phenotyping has the potential to enable efficient monitoring of changes in plant traits over time in both controlled environments and in the field. The operation of these robots can be challenging as a result of the dynamic nature of plants and the agricultural environments. Here we discuss developments in phenotyping robots, and the challenges which have been overcome and others which remain outstanding. In addition, some perspective applications of the phenotyping robots are also presented. We optimistically anticipate that autonomous and robotic systems will make great leaps forward in the next 10 years to advance the plant phenotyping research into a new era

Robotics and autonomous systems for net-zero agriculture

Purpose of ReviewThe paper discusses how robotics and autonomous systems (RAS) are being deployed to decarbonise agricultural production. The climate emergency cannot be ameliorated without dramatic reductions in greenhouse gas emis-sions across the agri-food sector. This review outlines the transformational role for robotics in the agri-food system and considers where research and focus might be prioritised.Recent FindingsAgri-robotic systems provide multiple emerging opportunities that facilitate the transition towards net zero agriculture. Five focus themes were identified where robotics could impact sustainable food production systems to (1) increase nitrogen use efficiency, (2) accelerate plant breeding, (3) deliver regenerative agriculture, (4) electrify robotic vehicles, (5) reduce food waste.SummaryRAS technologies create opportunities to (i) optimise the use of inputs such as fertiliser, seeds, and fuel/energy; (ii) reduce the environmental impact on soil and other natural resources; (iii) improve the efficiency and precision of agri-cultural processes and equipment; (iv) enhance farmers’ decisions to improve crop care and reduce farm waste. Further and scaled research and technology development are needed to exploit these opportunities

Spatial combination of sensor data deriving from mobile platforms for precision farming applications

This thesis combines optical sensors on a ground and on an aerial platform for field measurements in wheat, to identify nitrogen (N) levels, estimating biomass (BM) and predicting yield. The Multiplex Research (MP) fluorescence sensor was used for the first time in wheat. The individual objectives were: (i) Evaluation of different available sensors and sensor platforms used in Precision Farming (PF) to quantify the crop nutrition status, (ii) Acquisition of ground and aerial sensor data with two ground spectrometers, an aerial spectrometer and a ground fluorescence sensor, (iii) Development of effective post-processing methods for correction of the sensor data, (iv) Analysis and evaluation of the sensors with regard to the mapping of biomass, yield and nitrogen content in the plant, and (v) Yield simulation as a function of different sensor signals. This thesis contains three papers, published in international peer-reviewed journals. The first publication is a literature review on sensor platforms used in agricultural research. A subdivision of sensors and their applications was done, based on a detailed categorization model. It evaluates strengths and weaknesses, and discusses research results gathered with aerial and ground platforms with different sensors. Also, autonomous robots and swarm technologies suitable for PF tasks were reviewed. The second publication focuses on spectral and fluorescence sensors for BM, yield and N detection. The ground sensors were mounted on the Hohenheim research sensor platform Sensicle. A further spectrometer was installed in a fixed-wing Unmanned Aerial Vehicle (UAV). In this study, the sensors of the Sensicle and the UAV were used to determine plant characteristics and yield of three-year field trials at the research station Ihinger Hof, Renningen (Germany), an institution of the University of Hohenheim, Stuttgart (Germany). Winter wheat (Triticum aestivum L.) was sown on three research fields, with different N levels applied to each field. The measurements in the field were geo-referenced and logged with an absolute GPS accuracy of ±2.5 cm. The GPS data of the UAV was corrected based on the pitch and roll position of the UAV at each measurement. In the first step of the data analysis, raw data obtained from the sensors was post-processed and was converted into indices and ratios relating to plant characteristics. The converted ground sensor data were analysed, and the results of the correlations were interpreted related to the dependent variables (DV) BM weight, wheat yield and available N. The results showed significant positive correlations between the DVs and the Sensicle sensor data. For the third paper, the UAV sensor data was included into the evaluations. The UAV data analysis revealed low significant results for only one field in the year 2011. A multirotor UAV was considered as a more viable aerial platform, that allows for more precision and higher payload. Thereby, the ground sensors showed their strength at a close measuring distance to the plant and a smaller measurement footprint. The results of the two ground spectrometers showed significant positive correlations between yield and the indices from CropSpec, NDVI (Normalised Difference Vegetation Index) and REIP (Red-Edge Inflection Point). Also, FERARI and SFR (Simple Fluorescence Ratio) of the MP fluorescence sensor were chosen for the yield prediction model analysis. With the available N, CropSpec and REIP correlated significantly. The BM weight correlated with REIP even at a very early growing stage (Z 31), and with SAVI (Soil-Adjusted Vegetation Index) at ripening stage (Z 85). REIP, FERARI and SFR showed high correlations to the available N, especially in June and July. The ratios and signals of the MP sensor were highly significant compared to the BM weight above Z 85. Both ground spectrometers are suitable for data comparison and data combination with the active MP fluorescence sensor. Through a combination of fluorescence ratios and spectrometer indices, linear models for the prediction of wheat yield were generated, correlating significantly over the course of the vegetative period for research field Lammwirt (LW) in 2012. The best model for field LW in 2012 was selected for cross-validation with the measurements of the fields Inneres Täle (IT) and Riech (RI) in 2011 and 2012. However, it was not significant. By exchanging only one spectral index with a fluorescence ratio in a similar linear model, it showed significant correlations. This work successfully proves the combination of different sensor ratios and indices for the detection of plant characteristics, offering better and more robust predictions and quantifications of field parameters without employing destructive methods. The MP sensor proved to be universally applicable, showing significant correlations to the investigated characteristics such as BM weight, wheat yield and available N.Diese Arbeit kombiniert optische Sensoren auf einer Sensorplattform (SPF) am Boden und in der Luft bei Messungen in Weizen, um die Stickstoff-(N)-Werte zu identifizieren, während gleichzeitig die Biomasse (BM) geschätzt und der Ertrag vorhergesagt wird. Erstmals wurde hierfür der Fluoreszenzsensor Multiplex Research (MP) in Weizen eingesetzt. Die Ziele dieser Dissertation umfassen: (i) Bewertung verfügbarer Sensoren und SPF, die in der Präzisionslandwirtschaft zur Quantifizierung des Ernährungszustandes von Nutzpflanzen verwendet werden, (ii) Erfassung von Daten mit zwei Spektrometern am Boden, einem Spektrometer auf einem Modellflugzeug (UAV) und einem Fluoreszenzsensor am Boden, (iii) Erstellung effektiver Nachbearbeitungsmethoden für die Datenkorrektur, (iv) Analyse und Evaluation der Sensoren für die Abbildung der BM, des Ertrags und des N-Gehaltes in der Pflanze, und (v) Ertragssimulation als Funktion von Merkmalen unterschiedlicher Sensorsignale. Diese Arbeit enthält drei Artikel, die in international begutachteten Fachzeitschriften publiziert wurden. Die erste Veröffentlichung ist eine Literaturrecherche über SPF in der Agrarforschung. Ein detailliertes Kategorisierungsmodell wird für eine allgemeine Unterteilung der Sensoren und deren Anwendungsgebiete herangenommen, die Stärken und Schwächen bewertet, und die Forschungsergebnisse von Luft- und Bodenplattformen mit unterschiedlicher Sensorik diskutiert. Außerdem werden autonome Roboter und für landwirtschaftliche Aufgaben geeignete Schwarmtechnologien beschrieben. Die zweite Publikation fokussiert sich auf Spektral- und Fluoreszenzsensoren für die Erfassung von BM, Ertrag und N. In der Arbeit wurden die Bodensensoren auf der Hohenheimer Forschungs-SPF Sensicle und der Sensor auf dem UAV in dreijährigen Feldversuchen auf der Versuchsstation Ihinger Hof der Universität Hohenheim in Renningen für die Bestimmung von Pflanzenmerkmalen und des Ertrags eingesetzt. Auf drei Versuchsfeldern wurde Winterweizen ausgesät, und in einem randomisierten Versuchsdesign unterschiedliche N-Düngestufen angelegt. Die Sensormessungen im Feld wurden mit einer absoluten GPS Genauigkeit von ±2,5 cm verortet. Die GPS Daten des UAVs wurden mittels der Nick- und Rollposition lagekorrigiert. Im ersten Schritt der Datenanalyse wurden die Sensorrohdaten nachbearbeitet und in Indizes und Ratios umgerechnet. Die Bodensensordaten wurden analysiert, und die Ergebnisse der Korrelationen in Bezug zu den abhängigen Variablen (DV) BM-Gewicht, Weizenertrag, verfügbarer sowie aufgenommener N dargestellt. Die Ergebnisse zeigen signifikant positive Korrelationen zwischen den DVs und den Sensicle-Sensordaten. Für die dritte Publikation wurden die Sensordaten des UAV in die Auswertungen miteinbezogen. Die Analyse der UAV Daten zeigte niedrige signifikante Ergebnisse für nur ein Feld im Versuchsjahr 2011. Ein Multikopter wird als zuverlässigere Luftplattform erachtet, der mehr Präzision und eine höhere Nutzlast ermöglicht. Die Sensoren auf dem Sensicle zeigten ihren Vorteil bedingt durch einen kürzeren Messabstand zur Pflanze und eine kleinere Messfläche. Die Ergebnisse der beiden Sensicle-Spektrometer zeigten signifikant positive Korrelationen zwischen dem Ertrag und den Indizes von CropSpec, NDVI (Normalised Difference Vegetation Index) und REIP (Red-Edge Inflection Point). Auch FERARI und SFR (Simple Fluorescence Ratio) des MP-Sensors wurden für die Analyse des Ertragsvorhersagemodells ausgewählt. Mit dem verfügbaren N korrelierten CropSpec und REIP hochsignifikant. Das BM-Gewicht korrelierte bereits ab einem sehr frühen Wachstumsstadium (Z31) mit REIP und im Reifestadium (Z85) mit SAVI (Soil-Adjusted Vegetation Index). REIP, FERARI und SFR zeigten hohe Korrelationen mit dem verfügbaren N, insbesondere im Juni und Juli. Die Ratios und Signale des MP Sensors sind vor allem ab Z85 gegenüber dem BM-Gewicht hochsignifikant. Durch eine Kombination von Fluoreszenzwerten und Spektrometerindizes wurden lineare Modelle zur Vorhersage des Weizenertrags erstellt, die im Verlauf der Vegetationsperiode für das Versuchsfeld Lammwirt (LW) im Jahr 2012 signifikant korrelierten. Das beste Modell für das Feld LW im Jahr 2012 wurde für die Kreuzvalidierung mit den Messungen der Versuchsfelder Inneres Täle (IT) und Riech (RI) in den Jahren 2011 und 2012 ausgewählt. Sie waren nicht signifikant, jedoch zeigten sich durch den Austausch nur eines Spektralindexes mit einem Fluoreszenzratio in einem ähnlichen linearen Modell signifikante Korrelationen. Die vorliegende Arbeit zeigt erfolgreich, dass sich die Kombination verschiedener Sensorwerte und Sensorindizes zur Erkennung von Pflanzenmerkmalen gut eignet, und ohne den Einsatz destruktiver Methoden die Möglichkeit für bessere und robustere Vorhersagen bietet. Vor allem der MP-Fluoreszenzsensor erwies sich als universell einsetzbarer Sensor, der signifikante Korrelationen zu den untersuchten Merkmalen BM-Gewicht, Weizenertrag und verfügbarem N aufzeigte

Investigation into Swarm-based Cooperative Behaviour in Execution of Open Field Agricultural Tasks

Because of the significant drop in the number of farmers and increase in the earth population, the use of autonomous farming units including unmanned tractors is becoming more and more popular. However, relying on a single autonomous farming unit to carry out the entire task on a large field is inefficient. Using multiple autonomous tractors bring more efficiency, however, without cooperation this attempt will fail (Mataric et al., 1995). This cooperation can be achieved by an appropriate task allocation and coordination mechanism between the participating units. The current trend in this field is to use direct forms of communication in any form of directional or broadcasting meaningful messages among the group. The messages assist the group to identify the state of the task, assigned workload, collision and congestion avoidance, and etc. These forms of approaches are fast and efficient when units are within the communicating signal range. In this thesis, we aim to investigate the feasibility of cooperative execution of open field farming task including spraying and ploughing while inter-team interaction is other than direct communication methods. For every task, an algorithm is suggested and an appropriate mathematical model is presented. Then, using ROS Stage simulation environment, each algorithm is implemented and multiple tests are conducted. Finally, the simulation results and the correspondent mathematical results are compared and appropriate modifications are suggested

Real-Time Sensory Information for Remote Supervision of Autonomous Agricultural Machines

The concept of the driverless tractor has been discussed in the scientific literature for decades and several tractor manufacturers now have prototypes being field-tested. Although farmers will not be required to be physically present on these machines, it is envisioned that they will remain a part of the human-automation system. The overall efficiency and safety to be attained by autonomous agricultural machines (AAMs) will be correlated with the effectiveness of information sharing between the AAM and the farmer through what might be aptly called an automation interface. In this supervisory scenario, the farmer would be able to both receive status information and send instructions. In essence, supervisory control of an AAM is similar to the current scenario where farmers physically present on their machines obtain status information from displays integrated into the machine and from general sensory information that is available due to their proximity to the operating machine. Therefore, there is reason to expect that real-time sensory information would be valuable to the farmer when remotely supervising an AAM through an automation interface. This chapter will provide an overview of recent research that has been conducted on the role of real-time sensory information to the task of remotely supervising an AAM

ABC: Adaptive, Biomimetic, Configurable Robots for Smart Farms - From Cereal Phenotyping to Soft Fruit Harvesting

Currently, numerous factors, such as demographics, migration patterns, and economics, are leading to the critical labour shortage in low-skilled and physically demanding parts of agriculture. Thus, robotics can be developed for the agricultural sector to address these shortages. This study aims to develop an adaptive, biomimetic, and configurable modular robotics architecture that can be applied to multiple tasks (e.g., phenotyping, cutting, and picking), various crop varieties (e.g., wheat, strawberry, and tomato) and growing conditions. These robotic solutions cover the entire perception–action–decision-making loop targeting the phenotyping of cereals and harvesting fruits in a natural environment. The primary contributions of this thesis are as follows. a) A high-throughput method for imaging field-grown wheat in three dimensions, along with an accompanying unsupervised measuring method for obtaining individual wheat spike data are presented. The unsupervised method analyses the 3D point cloud of each trial plot, containing hundreds of wheat spikes, and calculates the average size of the wheat spike and total spike volume per plot. Experimental results reveal that the proposed algorithm can effectively identify spikes from wheat crops and individual spikes. b) Unlike cereal, soft fruit is typically harvested by manual selection and picking. To enable robotic harvesting, the initial perception system uses conditional generative adversarial networks to identify ripe fruits using synthetic data. To determine whether the strawberry is surrounded by obstacles, a cluster complexity-based perception system is further developed to classify the harvesting complexity of ripe strawberries. c) Once the harvest-ready fruit is localised using point cloud data generated by a stereo camera, the platform’s action system can coordinate the arm to reach/cut the stem using the passive motion paradigm framework, as inspired by studies on neural control of movement in the brain. Results from field trials for strawberry detection, reaching/cutting the stem of the fruit with a mean error of less than 3 mm, and extension to analysing complex canopy structures/bimanual coordination (searching/picking) are presented. Although this thesis focuses on strawberry harvesting, ongoing research is heading toward adapting the architecture to other crops. The agricultural food industry remains a labour-intensive sector with a low margin, and cost- and time-efficiency business model. The concepts presented herein can serve as a reference for future agricultural robots that are adaptive, biomimetic, and configurable

Flex-Ro: A Robotic High Throughput Field Phenotyping System

Research in agriculture is critical to developing techniques to meet the world’s demand for food, fuel, fiber, and feed. Optimization of crop production per unit of land requires scientists across disciplines to collaborate and investigate new areas of science and tools for data collection. The use of robotics has been adopted in several industries to supplement labor, and accurately perform repetitious tasks. However, the use of autonomous robots in commercial agricultural production is still limited. The Flex-Ro (Flexible structured Robotic platform) was developed for use in large area fields as a multipurpose tool to perform monotonous agricultural tasks. This work presents the design and implementation of the control system for the Flex-Ro machine. The machine control architecture was developed for safe operation with redundant emergency stops and checks. Operators use the remote-control device to maneuver the machine in uncontrolled environments. Autonomous field coverage was developed using global positioning system (GPS) guidance. The guidance system tracked within 4 cm of the guidance line 95% of the time at a travel speed of 4 kph. Waypoint guidance was implemented and demonstrated such that Flex-Ro could be programmed to follow complex paths and curves. High-throughput plant phenotyping is a continuously developing and evolving field of plant science. The methods used to collect phenotyping data include drones, satellites, manual measurement, and ground rovers. A suite of phenotyping sensors was installed onto the Flex-Ro to cover large field areas. The system was verified in soybean research plots at the University of Nebraska-Lincoln (UNL) Spidercam phenotyping facility. Positive correlations between the Spidercam and Flex-Ro phenotyping data were established. The Flex-Ro was able to statistically distinguish between soybean variety emergence and maturity differences. The late season phenotyping data showed statistical differences between the fully irrigated versus deficit plots. Basic economic calculations estimated the cost to operate the Flex-Ro machine for field phenotyping use at approximately $5.50/ha. Advisor: Santosh K. Pitl

Flex-Ro: A Robotic High Throughput Field Phenotyping System

Research in agriculture is critical to developing techniques to meet the world’s demand for food, fuel, fiber, and feed. Optimization of crop production per unit of land requires scientists across disciplines to collaborate and investigate new areas of science and tools for data collection. The use of robotics has been adopted in several industries to supplement labor, and accurately perform repetitious tasks. However, the use of autonomous robots in commercial agricultural production is still limited. The Flex-Ro (Flexible structured Robotic platform) was developed for use in large area fields as a multipurpose tool to perform monotonous agricultural tasks. This work presents the design and implementation of the control system for the Flex-Ro machine. The machine control architecture was developed for safe operation with redundant emergency stops and checks. Operators use the remote-control device to maneuver the machine in uncontrolled environments. Autonomous field coverage was developed using global positioning system (GPS) guidance. The guidance system tracked within 4 cm of the guidance line 95% of the time at a travel speed of 4 kph. Waypoint guidance was implemented and demonstrated such that Flex-Ro could be programmed to follow complex paths and curves. High-throughput plant phenotyping is a continuously developing and evolving field of plant science. The methods used to collect phenotyping data include drones, satellites, manual measurement, and ground rovers. A suite of phenotyping sensors was installed onto the Flex-Ro to cover large field areas. The system was verified in soybean research plots at the University of Nebraska-Lincoln (UNL) Spidercam phenotyping facility. Positive correlations between the Spidercam and Flex-Ro phenotyping data were established. The Flex-Ro was able to statistically distinguish between soybean variety emergence and maturity differences. The late season phenotyping data showed statistical differences between the fully irrigated versus deficit plots. Basic economic calculations estimated the cost to operate the Flex-Ro machine for field phenotyping use at approximately $5.50/ha. Advisor: Santosh K. Pitl