An Overview of the Application of Machine Vision in Recognition and Localization of Fruit and Vegetable Harvesting Robots

Intelligent agriculture imposes higher requirements on the recognition and localization of fruit and vegetable picking robots. Due to its unique visual information and relatively low hardware cost, machine vision is widely applied in the recognition and localization of fruit and vegetable picking robots. This article provides an overview of the application of machine vision in the recognition and localization of fruit and vegetable picking robots. Firstly, the advantages, disadvantages, and the roles of different visual sensors and machine vision algorithms in the recognition and localization of fruit and vegetable picking robots are introduced, including monocular cameras, stereo cameras, structured light cameras, multispectral cameras, image segmentation algorithms, object detection algorithms, and 3D reconstruction algorithms. Then, the current status and challenges faced by machine vision in the recognition and localization of fruit and vegetable picking robots are summarized. These challenges include the stability of fast recognition under complex background interference, stability of recognition under different lighting environments for the same crop, the reliance of recognition and localization on prior information in the presence of fruit overlap and occlusions caused by leaves and branches, and the uncertainty of picking caused by complex working environments. In current research on algorithms dealing with complex background interference and various occlusion disturbances, good results have been achieved. Different lighting environments have a significant impact on the recognition and positioning of fruits and vegetables, with a minimum accuracy of 59.2%. Finally, this article outlines future research directions to address these challenges

High-Resolution Cooperate Density-Integrated Inversion Method of Airborne Gravity and Its Gradient Data

Airborne (or satellite) gravity measurement is a commonly used remote sensing method to obtain the underground density distribution. Airborne gravity gradiometry data have a higher horizontal resolution to shallower causative sources than airborne gravity anomaly, so joint exploration of airborne gravity and its gradient data can simultaneously obtain the anomaly feature of sources with different depths. The most commonly used joint inversion method of gravity and its gradient data is the data combined method, which is to combine all the components into a data matrix as mutual constraints to reduce ambiguity and non-uniqueness. In order to obtain higher resolution results, we proposed a cooperate density-integrated inversion method of airborne gravity and its gradient data, which firstly carried out the joint inversion using cross-gradient constraints to obtain two density structures, and then fused two recovered models into a result through Fourier transform; finally, data combined joint inversion of airborne gravity, and gradient data were reperformed to achieve high-resolution density result using fused density results as a reference model. Compared to the data combined joint inversion method, the proposed cooperate density-integrated inversion method can obtain higher resolution and more accurate density distribution of shallow and deep bodies meanwhile. We also applied it to real data in the mining area of western Liaoning Province, China. The results showed that the depth of the skarn-type iron mine in the region is about 900–1300 m and gives a more specific distribution compared to the geological results, which provided reliable data for the next exploration plan

A Global Path Planning Method for Unmanned Ground Vehicles in Off-Road Environments Based on Mobility Prediction

In a complex off-road environment, due to the low bearing capacity of the soil and the uneven features of the terrain, generating a safe and effective global route for unmanned ground vehicles (UGVs) is critical for the success of their motion and mission. Most traditional global path planning methods simply take the shortest path length as the optimization objective, which makes it difficult to plan a feasible and safe route in complex off-road environments. To address this problem, this research proposes a global path planning method, which considers the influence of terrain factors and soil mechanics on UGV mobility. First, we established a high-resolution 3D terrain model with remote sensing elevation terrain data, land use and soil type distribution data, based on a geostatistical method. Second, we analyzed the vehicle mobility by the terramechanical method (i.e., vehicle cone index and Bakker’s theory), and then calculated the mobility cost based on a fuzzy inference method. Finally, based on the calculated mobility cost, the probabilistic roadmap method was used to establish the connected matrix and the multi-dimensional traffic cost evaluation matrix among the sampling nodes, and then an improved A* algorithm was proposed to generate the global route

A Global Path Planning Method for Unmanned Ground Vehicles in Off-Road Environments Based on Mobility Prediction

In a complex off-road environment, due to the low bearing capacity of the soil and the uneven features of the terrain, generating a safe and effective global route for unmanned ground vehicles (UGVs) is critical for the success of their motion and mission. Most traditional global path planning methods simply take the shortest path length as the optimization objective, which makes it difficult to plan a feasible and safe route in complex off-road environments. To address this problem, this research proposes a global path planning method, which considers the influence of terrain factors and soil mechanics on UGV mobility. First, we established a high-resolution 3D terrain model with remote sensing elevation terrain data, land use and soil type distribution data, based on a geostatistical method. Second, we analyzed the vehicle mobility by the terramechanical method (i.e., vehicle cone index and Bakker’s theory), and then calculated the mobility cost based on a fuzzy inference method. Finally, based on the calculated mobility cost, the probabilistic roadmap method was used to establish the connected matrix and the multi-dimensional traffic cost evaluation matrix among the sampling nodes, and then an improved A* algorithm was proposed to generate the global route

RBF-Based Monocular Vision Navigation for Small Vehicles in Narrow Space below Maize Canopy

Maize is one of the major food crops in China. Traditionally, field operations are done by manual labor, where the farmers are threatened by the harsh environment and pesticides. On the other hand, it is difficult for large machinery to maneuver in the field due to limited space, particularly in the middle and late growth stage of maize. Unmanned, compact agricultural machines, therefore, are ideal for such field work. This paper describes a method of monocular visual recognition to navigate small vehicles between narrow crop rows. Edge detection and noise elimination were used for image segmentation to extract the stalks in the image. The stalk coordinates define passable boundaries, and a simplified radial basis function (RBF)-based algorithm was adapted for path planning to improve the fault tolerance of stalk coordinate extraction. The average image processing time, including network latency, is 220 ms. The average time consumption for path planning is 30 ms. The fast processing ensures a top speed of 2 m/s for our prototype vehicle. When operating at the normal speed (0.7 m/s), the rate of collision with stalks is under 6.4%. Additional simulations and field tests further proved the feasibility and fault tolerance of our method

Image_1_An occluded cherry tomato recognition model based on improved YOLOv7.jpeg

The typical occlusion of cherry tomatoes in the natural environment is one of the most critical factors affecting the accurate picking of cherry tomato picking robots. To recognize occluded cherry tomatoes accurately and efficiently using deep convolutional neural networks, a new occluded cherry tomato recognition model DSP-YOLOv7-CA is proposed. Firstly, images of cherry tomatoes with different degrees of occlusion are acquired, four occlusion areas and four occlusion methods are defined, and a cherry tomato dataset (TOSL) is constructed. Then, based on YOLOv7, the convolution module of the original residual edges was replaced with null residual edges, depth-separable convolutional layers were added, and jump connections were added to reuse feature information. Then, a depth-separable convolutional layer is added to the SPPF module with fewer parameters to replace the original SPPCSPC module to solve the problem of loss of small target information by different pooled residual layers. Finally, a coordinate attention mechanism (CA) layer is introduced at the critical position of the enhanced feature extraction network to strengthen the attention to the occluded cherry tomato. The experimental results show that the DSP-YOLOv7-CA model outperforms other target detection models, with an average detection accuracy (mAP) of 98.86%, and the number of model parameters is reduced from 37.62MB to 33.71MB, which is better on the actual detection of cherry tomatoes with less than 95% occlusion. Relatively average results were obtained on detecting cherry tomatoes with a shade level higher than 95%, but such cherry tomatoes were not targeted for picking. The DSP-YOLOv7-CA model can accurately recognize the occluded cherry tomatoes in the natural environment, providing an effective solution for accurately picking cherry tomato picking robots.</p