Fruit Detection and Tree Segmentation for Yield Mapping in Orchards

Accurate information gathering and processing is critical for precision horticulture, as growers aim to optimise their farm management practices. An accurate inventory of the crop that details its spatial distribution along with health and maturity, can help farmers efficiently target processes such as chemical and fertiliser spraying, crop thinning, harvest management, labour planning and marketing. Growers have traditionally obtained this information by using manual sampling techniques, which tend to be labour intensive, spatially sparse, expensive, inaccurate and prone to subjective biases. Recent advances in sensing and automation for field robotics allow for key measurements to be made for individual plants throughout an orchard in a timely and accurate manner. Farmer operated machines or unmanned robotic platforms can be equipped with a range of sensors to capture a detailed representation over large areas. Robust and accurate data processing techniques are therefore required to extract high level information needed by the grower to support precision farming. This thesis focuses on yield mapping in orchards using image and light detection and ranging (LiDAR) data captured using an unmanned ground vehicle (UGV). The contribution is the framework and algorithmic components for orchard mapping and yield estimation that is applicable to different fruit types and orchard configurations. The framework includes detection of fruits in individual images and tracking them over subsequent frames. The fruit counts are then associated to individual trees, which are segmented from image and LiDAR data, resulting in a structured spatial representation of yield. The first contribution of this thesis is the development of a generic and robust fruit detection algorithm. Images captured in the outdoor environment are susceptible to highly variable external factors that lead to significant appearance variations. Specifically in orchards, variability is caused by changes in illumination, target pose, tree types, etc. The proposed techniques address these issues by using state-of-the-art feature learning approaches for image classification, while investigating the utility of orchard domain knowledge for fruit detection. Detection is performed using both pixel-wise classification of images followed instance segmentation, and bounding-box regression approaches. The experimental results illustrate the versatility of complex deep learning approaches over a multitude of fruit types. The second contribution of this thesis is a tree segmentation approach to detect the individual trees that serve as a standard unit for structured orchard information systems. The work focuses on trellised trees, which present unique challenges for segmentation algorithms due to their intertwined nature. LiDAR data are used to segment the trellis face, and to generate proposals for individual trees trunks. Additional trunk proposals are provided using pixel-wise classification of the image data. The multi-modal observations are fine-tuned by modelling trunk locations using a hidden semi-Markov model (HSMM), within which prior knowledge of tree spacing is incorporated. The final component of this thesis addresses the visual occlusion of fruit within geometrically complex canopies by using a multi-view detection and tracking approach. Single image fruit detections are tracked over a sequence of images, and associated to individual trees or farm rows, with the spatial distribution of the fruit counting forming a yield map over the farm. The results show the advantage of using multi-view imagery (instead of single view analysis) for fruit counting and yield mapping. This thesis includes extensive experimentation in almond, apple and mango orchards, with data captured by a UGV spanning a total of 5 hectares of farm area, over 30 km of vehicle traversal and more than 7,000 trees. The validation of the different processes is performed using manual annotations, which includes fruit and tree locations in image and LiDAR data respectively. Additional evaluation of yield mapping is performed by comparison against fruit counts on trees at the farm and counts made by the growers post-harvest. The framework developed in this thesis is demonstrated to be accurate compared to ground truth at all scales of the pipeline, including fruit detection and tree mapping, leading to accurate yield estimation, per tree and per row, for the different crops. Through the multitude of field experiments conducted over multiple seasons and years, the thesis presents key practical insights necessary for commercial development of an information gathering system in orchards

Human and animal classification using Doppler radar

South Africa is currently struggling to deal with a significant poaching and livestock theft problem. This work is concerned with the detection and classification of ground based targets using radar micro- Doppler signatures to aid in the monitoring of borders, nature reserves and farmlands. The research starts of by investigating the state of the art of ground target classification. Different radar systems are investigated with respect to their ability to classify targets at different operating frequencies. Finally, a Gaussian Mixture Model Hidden Markov Model based (GMM-HMM) classification approach is presented and tested in an operational environment. The GMM-HMM method is compared to methods in the literature and is shown to achieve reasonable (up to 95%) classification accuracy, marginally outperforming existing ground target classification methods.Dissertation (MEng)--University of Pretoria, 2017.Electrical, Electronic and Computer EngineeringMEngUnrestricte

Automatic Generation of Dynamic Musical Transitions in Computer Games

In video games, music must often change quickly from one piece to another due to player interaction, such as when moving between different areas. This quick change in music can often sound jarring if the two pieces are very different from each other. Several transition techniques have been used in industry such as the abrupt cut transition, crossfading, horizontal resequencing and vertical reorchestration among others. However, while several claims are made about their effectiveness (or lack thereof), none of these have been experimentally tested. To investigate how effective each transition technique is, this dissertation empirically evaluates each technique in a study informed by music psychology. This is done based on several features identified as being important for successful transitions. The obtained results led to a novel approach to musical transitions in video games by investigating the use of a multiple viewpoint system, with viewpoints being modelled using Markov models. This algorithm allowed the seamless generation of music that could serve as a transition between two composed pieces of music. While transitions in games normally occur over a zone boundary, the algorithm presented in this dissertation takes place over a transition region, giving the generated music enough time to transition. This novel approach was evaluated in a bespoke video game environment, where participants navigated through several pairs of different game environments and rated the resulting musical transitions. The results indicate that the generated transitions perform as well as crossfading, a technique commonly used in the industry. Since crossfading is not always appropriate, being able to use generated transitions gives composers another tool in their toolbox. Furthermore, the principled approach taken opens up avenues for further research

Study of Seasonal change and Water Stress Condition in Plant Leaf Using Polarimetric Lidar Measurement

Study of vegetation is of great importance to the improvement of agriculture and forest management. Although there have been various attempts to characterize vegetation using remote sensing techniques, polarimetric lidar is a novel remote sensing tool that has shown potential in vegetation remote sensing. In this thesis, a near-infrared polarimetric lidar at 1064 nm was used to investigate the effects of seasonal change and water stress condition on plant leaves. Two variables, time and water content, affected the plant leaf laser depolarization ratio measurement. The first study focused on the maple tree in order to figure out how seasonal change affected the maple leaf depolarization. Seasonal trendline was obtained and revealed an overall downward trend over time. In the second study, the leaves from maple, lemon, and rubber trees were investigated to study the effect of water stress on the depolarization ratio. It was discovered that the leaf depolarization ratio increased for more water content and went down for less water content. In addition, leaf samples were collected in the morning, afternoon, and evening, respectively, to study the diurnal change. Statistical analysis suggested that depolarization ratio did not change significantly for the different times of a day. It was suggested that the seasonal change had a greater effect on depolarization than the diurnal change. This study demonstrates that the near-infrared polarimetric lidar system has an ability to remotely characterize the vegetation internal conditions that may not be visible to the human eyes. Furthermore, the lidar system has the potential to differentiate the various plant species based on the depolarization ratio. In conclusion, the polarimetric lidar system at 1064-nm is an effective and sensitive enough remote sensing tool which can be widely used in active remote sensing

ADAPTIVE PROCESSING ARCHITECTURE OF MULTISENSOR SIGNALS FOR LOW-IMPACT TREATMENTS OF PLANT DISEASES.

Intelligent sensing for production of high-value crops Scientific and technical quality This thesis has been realized within the CROPS project. CROPS will develop scientific know-how for a highly configurable, modular and clever carrier platform that includes modular parallel manipulators and intelligent tools (sensors, algorithms, sprayers, grippers) that can be easily installed onto the carrier and are capable of adapting to new tasks and conditions. Several technological demonstrators will be developed for high value crops like greenhouse vegetables, fruits in orchards, and grapes for premium wines. The CROPS robotic platform will be capable of site-specific spraying (targets spray only towards foliage and selective targets) and selective harvesting of fruit (detects the fruit, determines its ripeness, moves towards the fruit, grasps it and softly detaches it). Another objective of CROPS is to develop techniques for reliable detection and classification of obstacles and other objects to enable successful autonomous navigation and operation in plantations and forests. The agricultural and forestry applications share many research areas, primarily regarding sensing and learning capabilities. The project started in October 2010 and will run for 48 month. The aim of this thesis is to lay the foundations, suggesting the guidelines, of one task addressed by the CROPS project, in particular, the aim of this work is to study the application of a VIS-NIR imaging approach (intelligent sensing), based on a relatively simple algorithm, to detect symptoms of powdery mildew and downy mildew disease at early stages of infection (sustainable production of high-value crops). Also a preliminary work for botrytis detection will be shown. Concept and objectives Many site-specific agricultural and forestry tasks, such as cultivating, transplanting, spraying, trimming, selective harvesting, and transportation, could be performed more efficiently if carried out by robotic systems. However, to date, agriculture and forestry robots are still not available, partly due to the complex, and often contradictory, demands for developing such systems. On the one hand, agro-forestry robots must be of reasonable cost, but on the other, they must be able to deal with complex, dynamic, and partly changing tasks. Addressing problems such as continuously changing conditions (e.g., rain and illumination), high variability in both the products (size, and shape) and the environment (location and soil properties), the delicate nature of the products, and hostile environmental conditions (e.g. dust, dirt, extreme temperature and humidity) requires advanced sensing, manipulation, and control. Since it is impossible to model a-priori all environments and task conditions, the robot must be able to learn new tasks and new working conditions. The solution to these demands lies in a modular and configurable design that will keep costs to a minimum by applying a basic configuration to a range of agricultural applications. At least a 95% yield rate is necessary for economical feasibility of an agro-forestry robotic system. Objectives An objective of CROPS project is to develop an \u201cintelligent tools\u201d (sensors, algorithms, sprayers) that can easily be installed onto a modular and clever carrier platform. The CROPS robotic platform will be capable of site-specific spraying (targeted spraying only on foliage and selected targets). Research efforts To achieve the novel systems described above, we will focus on intelligent sensing of disease detection on crop canopy (investigating different types and/or multiple sensors with decision making models). Technology evaluation Technology evaluation of the developed systems will include the performance evaluation of the different components (e.g., capacities, success rates/misses). Progress beyond the state-of-the-art Despite the extensive research conducted to date in applying robots to a variety of agriculture and forestry tasks (e.g., transplanting, spraying, trimming, selective harvesting), limited operating efficiencies (speeds, success rates) and lack of economic justification have severely limited commercialization. The few commercial autonomous agriculture and forestry robots that are available on the market include a cow milking robot, a robot for cutting roses (RomboMatic), and various remote-controlled forest harvesters. These robots either have a low level of autonomy or are able to perform only simple operations in structured and static environments (e.g. dairy farms and plant breeding facilities). Developing capabilities for robots operating in unstructured outdoor environments or dealing with the highly variable objects that exist in agriculture and forestry is still open-ended, and one of CROPS aims is to address this problem. Current state-of-the-art Field trials have routinely shown that most crop damage due to diseases and pests can be efficiently controlled when treatments are applied timely and accurately by hand to susceptible targets (i.e., by intelligent spraying). Site-specific spraying targeted solely to trees and/or to infected areas can reduce pesticide use by 20\u201340%. An issue of relevance to targeted agriculture is the detection of diseases in field crops. Since such events often have a visual manifestation, state-of-the-art methods for achieving this goal include fluorescence imaging or the analysis of spectral reflectance in carefully selected spectral bands. While reports of these methods used separately achieved performance at 75\u201390% accuracy, attempts to combine them have boosted disease discrimination accuracy to 95%. We must note here, however, that despite these promising results, very little research has been conducted on in-field disease detection. Expected progress The diseased detection approach for precision pesticide spraying will be developed investigating image processing techniques (after a laboratory spectral evaluation and greenhouse testing) for high-precision close-range targeted spraying to selectively and precisely apply chemicals solely to targets susceptible to specific diseases/pests, with a mean 90% success rate. Local changes in spectral reflection of parts of the canopy will be used as an indication of disease. \u201cSoft-sensor\u201d for detection of ripeness and diseases (noncontact rapid sensing system) will be developed by multispectral sensor (multispectral spectral camera). These \u201csoft sensor\u201d can be used as a decision model for targeted spraying