Image processing techniques for plant phenotyping using RGB and thermal imagery = Técnicas de procesamiento de imágenes RGB y térmicas como herramienta para fenotipado de cultivos

[eng] World cereal stocks need to increase in order to meet growing demands. Currently, maize, rice, wheat, are the main crops worldwide, while other cereals such as barley, sorghum, oat or different millets are also well placed in the top list. Crop productivity is affected directly by climate change factors such as heat, drought, floods or storms. Researchers agree that global climate change is having a major impact on crop productivity. In that way, several studies have been focused on climate change scenarios and more specifically abiotic stresses in cereals. For instance, in the case of heat stress, high temperatures between anthesis to grain filling can decrease grain yield. In order to deal with the climate change and future environmental scenarios, plant breeding is one of the main alternatives breeding is even considered to contribute to the larger component of yield growth compared to management. Plant breeding programs are focused on identifying genotypes with high yields and quality to act as a parentals and further the best individuals among the segregating population thus develop new varieties of plants. Breeders use the phenotypic data, plant and crop performance, and genetic information to improve the yield by selection (GxE, with G and E indicating genetic and environmental factors). More factors must be taken into account to increase the yield, such as, for instance, the education of farmers, economic incentives and the use of new technologies (GxExM, with M indicating management). Plant phenotyping is related with the observable (or measurable) characteristics of the plant while the crop growing as well as the association between the plant genetic background and its response to the environment (GxE). In traditional phenotyping the measurements are collated manually, which is tedious, time consuming and prone to subjective errors. Nowadays the technology is involved in many applications. From the point of view of plan phenotyping, technology has been incorporated as a tool. The use of image processing techniques integrating sensors and algorithm processes, is therefore, an alternative to asses automatically (or semi-automatically) these traits. Images have become a useful tool for plant phenotyping because most frequently data from the sensors are processed and analyzed as an image in two (2D) or three (3D) dimensions. An image is the arrangement of pixels in a regular Cartesian coordinates as a matrix, each pixel has a numerical value into the matrix which represents the number of photons captured by the sensor within the exposition time. Therefore, an image is the optical representation of the object illuminated by a radiating source. The main characteristics of images can be defined by the sensor spectral and spatial properties, with the spatial properties of the resulting image also heavily dependent on the sensor platform (which determines the distance from the target object).[spa] Las existencias mundiales de cereales deben aumentar para satisfacer la creciente demanda. Actualmente, el maíz, el arroz y el trigo son los principales cultivos a nivel mundial, otros cereales como la cebada, el sorgo y la avena están también bien ubicados en la lista. La productividad de los cultivos se ve afectada directamente por factores del cambio climático como el calor, la sequía, las inundaciones o las tormentas. Los investigadores coinciden en que el cambio climático global está teniendo un gran impacto en la productividad de los cultivos. Es por esto que muchos estudios se han centrado en escenarios de cambio climático y más específicamente en estrés abiótico. Por ejemplo, en el caso de estrés por calor, las altas temperaturas entre antesis y llenado de grano pueden disminuir el rendimiento del grano. Para hacer frente al cambio climático y escenarios ambientales futuros, el mejoramiento de plantas es una de las principales alternativas; incluso se considera que las técnicas de mejoramiento contribuyen en mayor medida al aumento del rendimiento que el manejo del cultivo. Los programas de mejora se centran en identificar genotipos con altos rendimientos y calidad para actuar como progenitores y promover los mejores individuos para desarrollar nuevas variedades de plantas. Los mejoradores utilizan los datos fenotípicos, el desempeño de las plantas y los cultivos, y la información genética para mejorar el rendimiento mediante selección (GxE, donde G y E indican factores genéticos y ambientales). El fenotipado plantas está relacionado con las características observables (o medibles) de la planta mientras crece el cultivo, así como con la asociación entre el fondo genético de la planta y su respuesta al medio ambiente (GxE). En el fenotipado tradicional, las mediciones se clasifican manualmente, lo cual es tedioso, consume mucho tiempo y es propenso a errores subjetivos. Sin embargo, hoy en día la tecnología está involucrada en muchas aplicaciones. Desde el punto de vista del fenotipado de plantas, la tecnología se ha incorporado como una herramienta. El uso de técnicas de procesamiento de imágenes que integran sensores y algoritmos son por lo tanto una alternativa para evaluar automáticamente (o semiautomáticamente) estas características

Automatic Counting of Wheat Spikes from Wheat Growth Images

This study aims to develop an automated screening system that can estimate the number of wheat spikes (i.e. ears) from a given wheat plant image acquired after the flowering stage. The platform can be used to assist the dynamic estimation of wheat yield potential as well as grain yield based on wheat images captured by the CropQuant platform. Our proposed system framework comprises three main stages. Firstly, it transforms the wheat plant raw image data using colour index of vegetation extraction (CIVE) and then segments wheat ear regions from the image to reduce the influence of the background signals. Secondly, it detects wheat ears using Gabor filter banks and K-means clustering algorithm. Finally, it estimates the number of wheat spikes within extracted wheat spike region through a regression method. The framework is tested with a real-world dataset of wheat growth images equally distributed from flowering to ripening stages. The estimations of the wheat ears were benchmarked against the ground truth produced in this study by human manual counting. Our automatic counting system achieved an average accuracy of 90.7% with a standard deviation of 0.055, at a much faster speed than human experts and hence the system has a potential to be improved for agricultural applications on wheat growth studies in the future

Convolutional Neural Networks for Image-based Corn Kernel Detection and Counting

Precise in-season corn grain yield estimates enable farmers to make real-time accurate harvest and grain marketing decisions minimizing possible losses of profitability. A well developed corn ear can have up to 800 kernels, but manually counting the kernels on an ear of corn is labor-intensive, time consuming and prone to human error. From an algorithmic perspective, the detection of the kernels from a single corn ear image is challenging due to the large number of kernels at different angles and very small distance among the kernels. In this paper, we propose a kernel detection and counting method based on a sliding window approach. The proposed method detect and counts all corn kernels in a single corn ear image taken in uncontrolled lighting conditions. The sliding window approach uses a convolutional neural network (CNN) for kernel detection. Then, a non-maximum suppression (NMS) is applied to remove overlapping detections. Finally, windows that are classified as kernel are passed to another CNN regression model for finding the (x,y) coordinates of the center of kernel image patches. Our experiments indicate that the proposed method can successfully detect the corn kernels with a low detection error and is also able to detect kernels on a batch of corn ears positioned at different angles.Comment: 14 pages, 9 figure

High-throughput phenotyping technology for corn ears

The phenotype of any organism, or as in this case, plants, includes traits or characteristics that can be measured using a technical procedure. Phenotyping is an important activity in plant breeding, since it gives breeders an observable representation of the plant’s genetic code, which is called the genotype. The word phenotype originates from the Greek word “phainein” which means “to show” and the word “typos” which means “type”. Ideally, the development of phenotyping technologies should be in lockstep with genotyping technologies, but unfortunately it is not; currently there exists a major discrepancy between the technological sophistication of genotyping versus phenotyping, and the gap is getting wider. Whereas genotyping has become a high-throughput low-cost standardized procedure, phenotyping still comprises ample manual measurements which are time consuming, tedious, and error prone. The project as conducted here aims at alleviating this problem; To aid breeders, a method was devised that allows for high-throughput phenotyping of corn ears, based on an existing imaging arrangement that produces frontal views of the ears. This thesis describes the development of machine vision algorithms that measure overall ear parameters such as ear length, ear diameter, and cap percentage (the proportion of the ear that features kernels versus the barren area). The main image processing functions used here were segmentation, skewness correction, morphological operation and image registration. To obtain a kernel count, an “ear map” was constructed using both a morphological operation and a feature matching operation. The main challenge for the morphological operation was to accurately select only kernel rows that are frontally exposed in each single image. This issue is addressed in this project by developing an algorithm of shadow recognition. The main challenge for the feature-matching operation was to detect and match image feature points. This issue was addressed by applying the algorithms of Harris’s Conner detection and SIFT descriptor. Once the ear map is created, many other morphological kernel parameters (area, location, circumference, to name a few) can be determined. Remaining challenges in this research are pointed out, including sample choice, apparatus modification and algorithm improvement. Suggestions and recommendations for future work are also provided

Automatic Wheat Ear Counting Using Thermal Imagery

Ear density is one of the most important agronomical yield components in wheat. Ear counting is time-consuming and tedious as it is most often conducted manually in field conditions. Moreover, different sampling techniques are often used resulting in a lack of standard protocol, which may eventually affect inter-comparability of results. Thermal sensors capture crop canopy features with more contrast than RGB sensors for image segmentation and classification tasks. An automatic thermal ear counting system is proposed to count the number of ears using zenithal/nadir thermal images acquired from a moderately high resolution handheld thermal camera. Three experimental sites under different growing conditions in Spain were used on a set of 24 varieties of durum wheat for this study. The automatic pipeline system developed uses contrast enhancement and filter techniques to segment image regions detected as ears. The approach is based on the temperature differential between the ears and the rest of the canopy, given that ears usually have higher temperatures due to their lower transpiration rates. Thermal images were acquired, together with RGB images and in situ (i.e., directly in the plot) visual ear counting from the same plot segment for validation purposes. The relationship between the thermal counting values and the in situ visual counting was fairly weak (R2 = 0.40), which highlights the difficulties in estimating ear density from one single image-perspective. However, the results show that the automatic thermal ear counting system performed quite well in counting the ears that do appear in the thermal images, exhibiting high correlations with the manual image-based counts from both thermal and RGB images in the sub-plot validation ring (R2 = 0.75-0.84). Automatic ear counting also exhibited high correlation with the manual counting from thermal images when considering the complete image (R2 = 0.80). The results also show a high correlation between the thermal and the RGB manual counting using the validation ring (R2 = 0.83). Methodological requirements and potential limitations of the technique are discussed

High-throughput estimation of crop traits: A review of ground and aerial phenotyping platforms

Crop yields need to be improved in a sustainable manner to meet the expected worldwide increase in population over the coming decades as well as the effects of anticipated climate change. Recently, genomics-assisted breeding has become a popular approach to food security; in this regard, the crop breeding community must better link the relationships between the phenotype and the genotype. While high-throughput genotyping is feasible at a low cost, highthroughput crop phenotyping methods and data analytical capacities need to be improved. High-throughput phenotyping offers a powerful way to assess particular phenotypes in large-scale experiments, using high-tech sensors, advanced robotics, and imageprocessing systems to monitor and quantify plants in breeding nurseries and field experiments at multiple scales. In addition, new bioinformatics platforms are able to embrace large-scale, multidimensional phenotypic datasets. Through the combined analysis of phenotyping and genotyping data, environmental responses and gene functions can now be dissected at unprecedented resolution. This will aid in finding solutions to currently limited and incremental improvements in crop yields

Machine vision detection of pests, diseases, and weeds: A review

Most of mankind’s living and workspace have been or going to be blended with smart technologies like the Internet of Things. The industrial domain has embraced automation technology, but agriculture automation is still in its infancy since the espousal has high investment costs and little commercialization of innovative technologies due to reliability issues. Machine vision is a potential technique for surveillance of crop health which can pinpoint the geolocation of crop stress in the field. Early statistics on crop health can hasten prevention strategies such as pesticide, fungicide applications to reduce the pollution impact on water, soil, and air ecosystems. This paper condenses the proposed machine vision relate research literature in agriculture to date to explore various pests, diseases, and weeds detection mechanisms

TIPS: a system for automated image-based phenotyping of maize tassels

Abstract Background The maize male inflorescence (tassel) produces pollen necessary for reproduction and commercial grain production of maize. The size of the tassel has been linked to factors affecting grain yield, so understanding the genetic control of tassel architecture is an important goal. Tassels are fragile and deform easily after removal from the plant, necessitating rapid measurement of any shape characteristics that cannot be retained during storage. Some morphological characteristics of tassels such as curvature and compactness are difficult to quantify using traditional methods, but can be quantified by image-based phenotyping tools. These constraints necessitate the development of an efficient method for capturing natural-state tassel morphology and complementary automated analytical methods that can quickly and reproducibly quantify traits of interest such as height, spread, and branch number. Results This paper presents the Tassel Image-based Phenotyping System (TIPS), which provides a platform for imaging tassels in the field immediately following removal from the plant. TIPS consists of custom methods that can quantify morphological traits from profile images of freshly harvested tassels acquired with a standard digital camera in a field-deployable light shelter. Correlations between manually measured traits (tassel weight, tassel length, spike length, and branch number) and image-based measurements ranged from 0.66 to 0.89. Additional tassel characteristics quantified by image analysis included some that cannot be quantified manually, such as curvature, compactness, fractal dimension, skeleton length, and perimeter. TIPS was used to measure tassel phenotypes of 3530 individual tassels from 749 diverse inbred lines that represent the diversity of tassel morphology found in modern breeding and academic research programs. Repeatability ranged from 0.85 to 0.92 for manually measured phenotypes, from 0.77 to 0.83 for the same traits measured by image-based methods, and from 0.49 to 0.81 for traits that can only be measured by image analysis. Conclusions TIPS allows morphological features of maize tassels to be quantified automatically, with minimal disturbance, at a scale that supports population-level studies. TIPS is expected to accelerate the discovery of associations between genetic loci and tassel morphology characteristics, and can be applied to maize breeding programs to increase productivity with lower resource commitment

Leaf to panicle ratio (LPR): a new physiological trait indicative of source and sink relation in japonica rice based on deep learning

Identification and characterization of new traits with sound physiological foundation is essential for crop breeding and production management. Deep learning has been widely used in image data analysis to explore spatial and temporal information on crop growth and development, thus strengthening the power of identification of physiological traits. Taking the advantage of deep learning, this study aims to develop a novel trait of canopy structure that integrate source and sink in japonica rice. We applied a deep learning approach to accurately segment leaf and panicle, and subsequently developed the procedure of GvCrop to calculate the leaf to panicle ratio (LPR) of rice canopy during grain filling stage. Images of training dataset were captured in the field experiments, with large variations in camera shooting angle, the elevation and the azimuth angles of the sun, rice genotype, and plant phenological stages. Accurately labeled by manually annotating the panicle and leaf regions, the resulting dataset were used to train FPN-Mask (Feature Pyramid Network Mask) models, consisting of a backbone network and a task-specific sub-network. The model with the highest accuracy was then selected to check the variations in LPR among 192 rice germplasms and among agronomical practices. Despite the challenging field conditions, FPN-Mask models achieved a high detection accuracy, with Pixel Accuracy being 0.99 for panicles and 0.98 for leaves. The calculated LPR displayed large spatial and temporal variations as well as genotypic differences. In addition, it was responsive to agronomical practices such as nitrogen fertilization and spraying of plant growth regulators. Deep learning technique can achieve high accuracy in simultaneous detection of panicle and leaf data from complex rice field images. The proposed FPN-Mask model is applicable to detect and quantify crop performance under field conditions. The newly identified trait of LPR should provide a high throughput protocol for breeders to select superior rice cultivars as well as for agronomists to precisely manage field crops that have a good balance of source and sink

Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant’s phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field