Agricultural Robotics

A key goal of contemporary agriculture is to dramatically increase production of food, feed, fiber, and biofuel products in a sustainable fashion, facing the pressure of diminishing farm labor supply. Agricultural robots can accelerate plant breeding and advance data-driven precision farming with significantly reduced labor inputs by providing task-appropriate sensing and actuation at fine spatiotemporal resolutions. This article highlights the distinctive challenges imposed on ground robots by agricultural environments, which are characterized by wide variations in environmental conditions, diversity and complexity of plant canopy structures, and intraspecies biological variation of physical and chemical characteristics and responses to the environment. Existing approaches to address these challenges are presented, along with their limitations; possible future directions are also discussed. Two key observations are that biology (breeding) and horticultural practices can reduce variabilities at the source and that publicly available benchmark data sets are needed to increase perception robustness and performance despite variability

Deterministic predictive dynamic scheduling for crop-transport co-robots acting as harvesting aids

Manual harvesting of fresh-market crops like strawberries is very labor-intensive. Apart from picking fruits, pickers spend significant amounts of time carrying full trays to a collection station at the edge of the field. Small teams of harvest-aid robots that help large picking crews by transporting empty and full trays can increase harvest efficiency by reducing pickers’ non-productive walking times. However, robot sharing among the crew may introduce non-productive waiting delays between the time a tray becomes full and when a robot arrives to collect it. Reactive robot scheduling cannot eliminate mean waiting times because pickers must wait for a robot to travel the distance from the collection station to them. Predictive scheduling is better suited to this task, because if the time and location that a pickers’ tray will fill are known to the scheduler in advance, a robot can start moving toward that location before the tray becomes full; hence, waiting times due to robot travel can be reduced or eliminated. In this paper, dynamic predictive scheduling was modeled for teams of robots carrying trays during manual harvesting. The times and locations of the tray-transport requests were assumed to be known exactly (deterministic predictions). Near-optimal scheduling was implemented to provide efficiency upper-bounds for any predictive scheduling algorithms that incorporate uncertainty in the predictive requests. Robot-aided harvesting was simulated using manual-harvest data collected from a commercial picking crew. Scheduling performance was studied as a function of the number of robots – for a given crew size – with robot speed as a parameter. Additionally, the effect of the earliness of the availability of the predictions on performance was studied. Experimental results showed that both reactive and predictive scheduling did not improve the mean non-productive time significantly relative to manual harvesting, when only four robots were used. Actually, deploying fewer than four robots led to worse non-productive time. However, introducing five to eight robots decreased mean non-productive time drastically, and when ten or more robots were used, non-productive time was reduced by 64.6% (reactive scheduling) and up to 93.7% (predictive scheduling) with respect to all-manual non-productive time. The efficiency increases were 15% and 24%, respectively. It was also verified that reactive dispatching always performed worse than deterministic predictive scheduling. Also, when the robot-to-picker ratio was larger than approximately 1:3, the waiting time and efficiency plateaued, i.e., did not improve further, regardless of how early the prediction was available to the scheduler. The reason is that the mean waiting time is lower bounded by the sum of mean travel time and tray exchange time, which are both constant. Although the above results represent upper-bounds for performance – since predictions were perfect - they indicate that tray-transport robots acting as harvest aids can increase harvesting efficiency significantly when scheduled properly

Robotic Tree-Fruit Harvesting with Telescoping Arms: A Study of Linear Fruit Reachability under Geometric Constraints

Modern commercial orchards are increasingly adopting trees of SNAP architectures (Simple, Narrow, Accessible, and Productive) as the fruits on such trees are, in general, more easily reachable by human or robotic harvesters. This article presents a methodology that utilizes three dimensional (3D) digitized computer models of high-density pear and cling-peach trees, and fruit positions to quantify the linear fruit reachability (LFR) of such trees, i.e., their reachability by telescoping robot arms. Robot-canopy non-interference geometric constraints were introduced in the simulator, to determine the closest position of the arms' base frames with respect to the trees, inside an orchard row. Also, design constraints for such arms, such as maximum reach, size and type of the gripper, and range of approach directions, were introduced to estimate the effect of each of these constraints on the LFR. Simulations results showed that 85.5% of pears were reachable after harvesting consecutively, at three different approach angles ('passes') with a gripper of size 11 cm and an arm extension of 150 cm. For peaches, after three passes, 83.5% were reachable with a gripper size of 11 cm and an arm extension of 200 cm. LFR increased as the gripper's size approached the maximum fruit size and decreased thereafter. Also, retractive grippers on linear arms yielded more fruit compared to vacuum-tube type grippers. Overall, the results suggested that telescoping arms can be used to harvest certain types of SNAP-style trees. Also, this methodology can be used to guide the design of robotic harvesters, as well as the canopy management practices of fruit trees

Deterministic predictive dynamic scheduling for crop-transport co-robots acting as harvesting aids

Manual harvesting of fresh-market crops like strawberries is very labor-intensive. Apart from picking fruits, pickers spend significant amounts of time carrying full trays to a collection station at the edge of the field. Small teams of harvest-aid robots that help large picking crews by transporting empty and full trays can increase harvest efficiency by reducing pickers’ non-productive walking times. However, robot sharing among the crew may introduce non-productive waiting delays between the time a tray becomes full and when a robot arrives to collect it. Reactive robot scheduling cannot eliminate mean waiting times because pickers must wait for a robot to travel the distance from the collection station to them. Predictive scheduling is better suited to this task, because if the time and location that a pickers’ tray will fill are known to the scheduler in advance, a robot can start moving toward that location before the tray becomes full; hence, waiting times due to robot travel can be reduced or eliminated. In this paper, dynamic predictive scheduling was modeled for teams of robots carrying trays during manual harvesting. The times and locations of the tray-transport requests were assumed to be known exactly (deterministic predictions). Near-optimal scheduling was implemented to provide efficiency upper-bounds for any predictive scheduling algorithms that incorporate uncertainty in the predictive requests. Robot-aided harvesting was simulated using manual-harvest data collected from a commercial picking crew. Scheduling performance was studied as a function of the number of robots – for a given crew size – with robot speed as a parameter. Additionally, the effect of the earliness of the availability of the predictions on performance was studied. Experimental results showed that both reactive and predictive scheduling did not improve the mean non-productive time significantly relative to manual harvesting, when only four robots were used. Actually, deploying fewer than four robots led to worse non-productive time. However, introducing five to eight robots decreased mean non-productive time drastically, and when ten or more robots were used, non-productive time was reduced by 64.6% (reactive scheduling) and up to 93.7% (predictive scheduling) with respect to all-manual non-productive time. The efficiency increases were 15% and 24%, respectively. It was also verified that reactive dispatching always performed worse than deterministic predictive scheduling. Also, when the robot-to-picker ratio was larger than approximately 1:3, the waiting time and efficiency plateaued, i.e., did not improve further, regardless of how early the prediction was available to the scheduler. The reason is that the mean waiting time is lower bounded by the sum of mean travel time and tray exchange time, which are both constant. Although the above results represent upper-bounds for performance – since predictions were perfect - they indicate that tray-transport robots acting as harvest aids can increase harvesting efficiency significantly when scheduled properly

Co-robotic harvest-aid platforms: Real-time control of picker lift heights to maximize harvesting efficiency

Harvest-aid platforms are used in modern orchards to improve manual harvesting efficiency, safety, and ergonomics. Typically, workers stand at pre-set heights on a platform's multi-level deck, and each worker harvests fruits inside a canopy zone that is defined by the lowest and highest reach of the worker's arms. However, fruit distributions are non-uniform, and worker picking speeds vary, thus generating a mismatch between labor demand (incoming fruit rates) and labor supply (fruit picking rates) in each zone; this mismatch limits platform-based harvesting efficiencies. To alleviate this problem, we transformed a conventional harvesting platform into a collaborative robot (co-robot) platform. As the co-robotic platform travels forward, it estimates the incoming fruit distribution using a vision system, it measures each worker's picking speed using instrumented picking bags, and controls the heights of hydraulic lifts that move workers up and down. The model-based control algorithm maximizes the machine's harvesting speed by changing the height at which each worker harvests as a response to incoming fruit load because it matches fruit-picking labor supply and demand. Simulation experiments with pre-recorded fruit distribution data validated the approach and provided efficiency gains under various conditions. Apple-harvesting experiments were also performed in a commercial orchard, where 2307 kg of apples were picked: 1045 kg in variable-height zone harvesting mode, and 1262 kg in fixed zone harvesting mode, with workers at fixed heights that were set by the grower. Variable-height zone harvesting mode throughput was 327.6 kg/h vs. 298.8 kg/h for fixed zone harvesting mode at human-controlled platform moving speed, resulting in an improvement of 9.5%

Co-robotic harvest-aid platforms: Real-time control of picker lift heights to maximize harvesting efficiency

Harvest-aid platforms are used in modern orchards to improve manual harvesting efficiency, safety, and ergonomics. Typically, workers stand at pre-set heights on a platform's multi-level deck, and each worker harvests fruits inside a canopy zone that is defined by the lowest and highest reach of the worker's arms. However, fruit distributions are non-uniform, and worker picking speeds vary, thus generating a mismatch between labor demand (incoming fruit rates) and labor supply (fruit picking rates) in each zone; this mismatch limits platform-based harvesting efficiencies. To alleviate this problem, we transformed a conventional harvesting platform into a collaborative robot (co-robot) platform. As the co-robotic platform travels forward, it estimates the incoming fruit distribution using a vision system, it measures each worker's picking speed using instrumented picking bags, and controls the heights of hydraulic lifts that move workers up and down. The model-based control algorithm maximizes the machine's harvesting speed by changing the height at which each worker harvests as a response to incoming fruit load because it matches fruit-picking labor supply and demand. Simulation experiments with pre-recorded fruit distribution data validated the approach and provided efficiency gains under various conditions. Apple-harvesting experiments were also performed in a commercial orchard, where 2307 kg of apples were picked: 1045 kg in variable-height zone harvesting mode, and 1262 kg in fixed zone harvesting mode, with workers at fixed heights that were set by the grower. Variable-height zone harvesting mode throughput was 327.6 kg/h vs. 298.8 kg/h for fixed zone harvesting mode at human-controlled platform moving speed, resulting in an improvement of 9.5%

Robotic Tree-Fruit Harvesting with Telescoping Arms: A Study of Linear Fruit Reachability under Geometric Constraints

Modern commercial orchards are increasingly adopting trees of SNAP architectures (Simple, Narrow, Accessible, and Productive) as the fruits on such trees are, in general, more easily reachable by human or robotic harvesters. This article presents a methodology that utilizes three dimensional (3D) digitized computer models of high-density pear and cling-peach trees, and fruit positions to quantify the linear fruit reachability (LFR) of such trees, i.e., their reachability by telescoping robot arms. Robot-canopy non-interference geometric constraints were introduced in the simulator, to determine the closest position of the arms' base frames with respect to the trees, inside an orchard row. Also, design constraints for such arms, such as maximum reach, size and type of the gripper, and range of approach directions, were introduced to estimate the effect of each of these constraints on the LFR. Simulations results showed that 85.5% of pears were reachable after harvesting consecutively, at three different approach angles ('passes') with a gripper of size 11 cm and an arm extension of 150 cm. For peaches, after three passes, 83.5% were reachable with a gripper size of 11 cm and an arm extension of 200 cm. LFR increased as the gripper's size approached the maximum fruit size and decreased thereafter. Also, retractive grippers on linear arms yielded more fruit compared to vacuum-tube type grippers. Overall, the results suggested that telescoping arms can be used to harvest certain types of SNAP-style trees. Also, this methodology can be used to guide the design of robotic harvesters, as well as the canopy management practices of fruit trees

Estimation of worker fruit-picking rates with an instrumented picking bag

Estimating and recording a worker's picking rate during tree fruit harvesting can provide useful information for better workforce management, orchard platform crew management, and generation of yield maps (in combination with position). A commercial picking bag was instrumented to estimate harvested fruit weight in real-time. All electronics were placed inside an enclosure that was placed between the bag and its shoulder straps, without hindering picking motions. The electronics included two load cells to measure the forces exerted on the straps by the bag and fruits, an Arduino microcontroller, signal conditioning circuits, data storage, wireless communication components, and inertial sensors. Software was developed for data acquisition, filtering, transmission, and storage. Two calibration models were developed to estimate fruit weight. One model (model 2) used inertial sensor data to compensate for the picking bag's angle with respect to gravity direction, while the other model (model 1) did not. Dynamic calibration experiments were performed over the entire weight range of the bag (0 to 20 kg) with reference objects of known weight (baseballs and fresh apples). The weight was divided into three ranges: light load (<8 kg), medium load (8 to 13 kg), and heavy load (>13 kg). Results showed that model 1 performed slightly better in the light load range, but model 2 was superior in the medium and heavy load ranges, presumably due to bag angle compensation. The best root mean squared error over the entire range was achieved by model 2 and was 0.36 kg (1.8% of bag capacity). In an application case study, two bags were used by workers harvesting from a platform in a commercial apple orchard. From the data, the pickers' harvesting speeds were estimated, and the fruit yield distribution was calculated for one side of a tree row

Collaboration of human pickers and crop-transporting robots during harvesting – Part II: Simulator evaluation and robot-scheduling case-study

Harvest-aid robots that transport empty and full trays during manual harvesting of specialty crops such as strawberries or table grapes can increase harvest efficiency, by reducing pickers' non-productive walking times. In Part I of this work, a modeling framework, and a stochastic simulator were presented for all-manual and robot-aided harvesting. This paper reports Part II of our work, which utilized data gathered in two strawberry fields during harvesting, to estimate the stochastic parameters involved in modeling pickers, and evaluate the prediction accuracy of the simulator for all-manual picking. Then, as a case study, non-productive time and harvest efficiency were estimated for robot-aided harvesting, for various picker-robot ratios and three priority-based reactive dispatching strategies for the robots. The simulator predicted the pickers' non-productive time during all-manual harvesting, with 6.4%, 3%, and 1.2% errors for the morning, afternoon, and “all-day” harvesting shifts, respectively. Statistical testing verified that predicted non-productive times followed the same distributions as the measured non-productive times (5% significance level). Simulations robustness was assessed by using morning data to simulate afternoon harvesting and vice-versa: non-productive times distributions were predicted accurately (10% significance level). Robot-aided simulation results – using the calibrated simulator for a 25-picker crew – showed that all-manual harvest efficiencies of 81.8% and 78.2% for morning and afternoon shifts increased to 92% and 86.5%, respectively, when five robots were deployed. Different scheduling policies did not affect efficiency when more than five robots were used, because there were always enough robots to serve pickers' requests immediately. Also, harvest efficiency plateaued when more than five robots were used, as a consequence of the time needed for a robot to travel to a picker

Collaboration of human pickers and crop-transporting robots during harvesting – Part II: Simulator evaluation and robot-scheduling case-study

Harvest-aid robots that transport empty and full trays during manual harvesting of specialty crops such as strawberries or table grapes can increase harvest efficiency, by reducing pickers' non-productive walking times. In Part I of this work, a modeling framework, and a stochastic simulator were presented for all-manual and robot-aided harvesting. This paper reports Part II of our work, which utilized data gathered in two strawberry fields during harvesting, to estimate the stochastic parameters involved in modeling pickers, and evaluate the prediction accuracy of the simulator for all-manual picking. Then, as a case study, non-productive time and harvest efficiency were estimated for robot-aided harvesting, for various picker-robot ratios and three priority-based reactive dispatching strategies for the robots. The simulator predicted the pickers' non-productive time during all-manual harvesting, with 6.4%, 3%, and 1.2% errors for the morning, afternoon, and “all-day” harvesting shifts, respectively. Statistical testing verified that predicted non-productive times followed the same distributions as the measured non-productive times (5% significance level). Simulations robustness was assessed by using morning data to simulate afternoon harvesting and vice-versa: non-productive times distributions were predicted accurately (10% significance level). Robot-aided simulation results – using the calibrated simulator for a 25-picker crew – showed that all-manual harvest efficiencies of 81.8% and 78.2% for morning and afternoon shifts increased to 92% and 86.5%, respectively, when five robots were deployed. Different scheduling policies did not affect efficiency when more than five robots were used, because there were always enough robots to serve pickers' requests immediately. Also, harvest efficiency plateaued when more than five robots were used, as a consequence of the time needed for a robot to travel to a picker