Application of deep learning and machine learning workflows for field-scale phenotyping

Tassel is the male inflorescences organ of the maize plant that develops atop the plant. Coarse features of tassels, including shape and size, can influence shedding pollen, fertilization, and subsequently grain yield. Therefore, understanding tassel dynamics and characteristics as well as how it evolves during the plant growth can help the plant scientist community to increase the grain yield as a final goal. To do so, first, tassels were investigated in one time points. The tassels were cut in the field and their images were captured in a lightbox. Coarse features were measured using novel image processing approaches. 351 tassels with different genotypes were used for the experiment. Tassel length, first lowest branch length, and angle as well as central spike length were measured by applying image processing and machine learning techniques. Tassels were also classified to open and close structures to obtain accurate predictions for the traits. The results show that R2 values for the tassel length and central spike length were 0.92 and 0.80, respectively. In addition, the R2 values for the first lowest branch length and angle were 0.63 and 0.91, respectively. The R2 values for the first lowest branch length was low compared to others because locating the first lowest branch point and its corresponding branch tip was hard due to branches occlusion. This study was done to create a robust algorithm for tassel phenotyping. Challenges were figured out for better tassel phenotyping in the field. Then, we looked at a diverse panel in the field, using stationary cameras to capture 6 tassels every 10 minutes for 8 hours per day during a month. Traditional approaches for phenotyping anthesis progression are time-consuming, subjective, and labor-intensive and are thus impractical for phenotyping large populations in multiple environments. In this work, we utilize a high throughput phenotyping approach that is based on extracting time-lapse information of anthesis progress from digital cameras. The major challenge is identifying the region of the interest (i.e. location of tassels in the imaging window) in the acquired images. Camera drift, different types of weather, including fog, rain, clouds, and sun and additionally, occlusion of tassels by other tassels or leaves complicated this problem. We discussed the associated challenges for object detection and localization under noisy conditions. In addition, a framework was developed to utilize Amazon Mechanical Turk to allow turkers to annotate the images and evaluate them to create an object detection dataset. Finally, we illustrated a promising deep-learning approach to tassel recognition and localization that is based on Faster-RCNN which has shown the strong capability for detection and localization. This method was improved using a boosting method to improve the dataset. This approach is able to reliably identify a diverse set of tassel morphologies with the mAP of 0.81. Tassel flowering pattern is the most important and complex trait. Tassel maize as a male structure is responsible to produce pollen for the silk as a female organ on the same plant. The amount of pollen and shedding time is important for the breeders as well as the biologists. This study introduced an automated end-to-end pipeline by coupling various deep learning, machine learning and image processing approaches. Inbred lines from both SAM and NAM panels were grown at Curtiss farm at Iowa State University, Ames, IA. A stationary camera was installed for every two plants. Tassels architecture, weather type, tassels and camera movements are the most important challenges of the research. To address these issues, deep learning algorithms were utilized. Tassel detection, classification, and segmentation. In addition, advanced image processing approaches were used to crop the tassel main spike and track the during tassel evolution. The results showed that deep learning is a powerful tool to detect, classify and segment the tassels. The mAP for the tassel detection was 0.91. The F1-score obtained for the tassel classification was 0.93. In addition, the accuracy of semantic segmentation for creating a binary image from the RGB tassel images was 0.95. The width of the flowering was obtained using graph theory in image processing and the time and location of the flowering can be obtained from the width data over the main spike branch. In addition to tassel structures, crop growth simulation models can help farmers and breeders predict crop performance, and in maize, Leaf Appearance Rate (LAR) is an important parameter used in crop performance simulation models such as APSIM. Since breeders and biologists would like to minimize human involvement in monitoring LAR, this trait can be monitored by applying a high-throughput phenotyping system. Engineers have entered the picture in collaboration with plant scientists to establish different and robust phenotyping methods, and in this study, maize leaf appearance rate was investigated using high-throughput phenotyping approaches. We developed an imaging system for automatically capturing a time-series of images of maize plants under field conditions, with 380 RGB cameras were used to capture images from 380 rows. There were 6 plants with the same genotype in each row that had different genotypes differed row-by-row, and the images were taken for 9 hours daily at 20- minute intervals for more than one month during a growing season. An end-to-end deep learning method was then used to count the numbers of leaves in the images. The dataset for the deep learning algorithm, obtained using the Amazon Mechanical Turk platform, was created by one expert turker along with a well-trained turker. Results demonstrated that an end-to-end model with training based on the expert turker dataset performed very well, handling variation in images that included leaf occlusions and weather type. The R2 between the ground truth obtained by the expert turker and predicted values was approximately 0.73, 0.74, and 0.95 for three testing cameras. The model\u27s prediction performance demonstrated that the number of leaves increase with different slopes for different genotypes. The data can be used for further genotypic analysis

TasselNet: Counting maize tassels in the wild via local counts regression network

Accurately counting maize tassels is important for monitoring the growth status of maize plants. This tedious task, however, is still mainly done by manual efforts. In the context of modern plant phenotyping, automating this task is required to meet the need of large-scale analysis of genotype and phenotype. In recent years, computer vision technologies have experienced a significant breakthrough due to the emergence of large-scale datasets and increased computational resources. Naturally image-based approaches have also received much attention in plant-related studies. Yet a fact is that most image-based systems for plant phenotyping are deployed under controlled laboratory environment. When transferring the application scenario to unconstrained in-field conditions, intrinsic and extrinsic variations in the wild pose great challenges for accurate counting of maize tassels, which goes beyond the ability of conventional image processing techniques. This calls for further robust computer vision approaches to address in-field variations. This paper studies the in-field counting problem of maize tassels. To our knowledge, this is the first time that a plant-related counting problem is considered using computer vision technologies under unconstrained field-based environment.Comment: 14 page

In Vivo Human-Like Robotic Phenotyping of Leaf and Stem Traits in Maize and Sorghum in Greenhouse

In plant phenotyping, the measurement of morphological, physiological and chemical traits of leaves and stems is needed to investigate and monitor the condition of plants. The manual measurement of these properties is time consuming, tedious, error prone, and laborious. The use of robots is a new approach to accomplish such endeavors, which enables automatic monitoring with minimal human intervention. In this study, two plant phenotyping robotic systems were developed to realize automated measurement of plant leaf properties and stem diameter which could reduce the tediousness of data collection compare to manual measurements. The robotic systems comprised of a four degree of freedom (DOF) robotic manipulator and a Time-of-Flight (TOF) camera. Robotic grippers were developed to integrate an optical fiber cable (coupled to a portable spectrometer) for leaf spectral reflectance measurement, a thermistor for leaf temperature measurement, and a linear potentiometer for stem diameter measurement. An Image processing technique and deep learning method were used to identify grasping points on leaves and stems, respectively. The systems were tested in a greenhouse using maize and sorghum plants. The results from the leaf phenotyping robot experiment showed that leaf temperature measurements by the phenotyping robot were correlated with those measured manually by a human researcher (R2 = 0.58 for maize and 0.63 for sorghum). The leaf spectral measurements by the phenotyping robot predicted leaf chlorophyll, water content and potassium with moderate success (R2 ranged from 0.52 to 0.61), whereas the prediction for leaf nitrogen and phosphorus were poor. The total execution time to grasp and take measurements from one leaf was 35.5±4.4 s for maize and 38.5±5.7 s for sorghum. Furthermore, the test showed that the grasping success rate was 78% for maize and 48% for sorghum. The experimental results from the stem phenotyping robot demonstrated a high correlation between the manual and automated stem diameter measurements (R2 \u3e 0.98). The execution time for stem diameter measurement was 45.3 s. The system could successfully detect and localize, and also grasp the stem for all plants during the experiment. Both robots could decrease the tediousness of collecting phenotypes compare to manual measurements. The phenotyping robots can be useful to complement the traditional image-based high-throughput plant phenotyping in greenhouses by collecting in vivo morphological, physiological, and biochemical trait measurements for plant leaves and stems. Advisors: Yufeng Ge, Santosh Pitl

Semantic Segmentation of Sorghum Using Hyperspectral Data Identifies Genetic Associations

This study describes the evaluation of a range of approaches to semantic segmentation of hyperspectral images of sorghum plants, classifying each pixel as either nonplant or belonging to one of the three organ types (leaf, stalk, panicle). While many current methods for segmentation focus on separating plant pixels from background, organ-specific segmentation makes it feasible to measure a wider range of plant properties. Manually scored training data for a set of hyperspectral images collected from a sorghum association population was used to train and evaluate a set of supervised classification models. Many algorithms show acceptable accuracy for this classification task. Algorithms trained on sorghum data are able to accurately classify maize leaves and stalks, but fail to accurately classify maize reproductive organs which are not directly equivalent to sorghum panicles. Trait measurements extracted from semantic segmentation of sorghum organs can be used to identify both genes known to be controlling variation in a previously measured phenotypes (e.g., panicle size and plant height) as well as identify signals for genes controlling traits not previously quantified in this population (e.g., stalk/leaf ratio). Organ level semantic segmentation provides opportunities to identify genes controlling variation in a wide range of morphological phenotypes in sorghum, maize, and other related grain crops

New Approaches to Use Genomics, Field Traits, and High-throughput Phenotyping for Gene Discovery in Maize (\u3ci\u3eZea mays\u3c/i\u3e)

Maize is one of major crop species over the world. With lots of genetic resources and genomic tools, maize also serves as a model species to understand genetic diversity, facilitate the development of trait extraction algorithms and map candidate functional genes. Since the first version of widely used B73 reference genome was released, independent research groups in the maize community propagated seeds themselves for further research purposes. However, unexpected or occasional contamination may happen during this process. The first study in this thesis used public RNA-seq data of B73 from 27 research groups across three countries for calling single nucleotide polymorphisms (SNP). Those SNPs were applied for investigating the distance of 27 maize B73 samples from the reference genome and three major clades were defined for determining their original sources. On the other side, maize is a plant with clear plant architecture. The second study was to employ the high-throughput plant phenotyping to dissect plant phenotypes using computer vision methods. A total of 32 maize inbreds distributed from the Genomes to Fields project were captured images in daily by 4 types of cameras (RGB, Hyperspectral, Fluorescence and Thermal-IR) for approximate 1 month. Differences between computer vision measurements and manual measurements about the plant fresh biomass were evaluated. Broad-sense heritability was estimated for extracted measurements from images. The expanded types of plant phenotype from the perspective of imaging provided a broader range of opportunities for connecting phenotypic variants with genetic variants. The third study utilized the phenome-wide variants in maize Goodman-Buckler 282 association panel to scan and associate with genetic variants of annotated genes along the maize genome. Genes detected by the proposed model, Genome-Phenome Wide Association Study (GPWAS), are significantly different from conventional GWAS detected genes. GPWAS genes tend to be more functionally conserved and more similar as classical maize mutants with known functions. Results from these researches assist to answer question about the genetic purity of same maize genotype. Methods developed in this thesis can also provide the valuable reference for trait discoveries from images and candidate functional gene identification using a broad set of phenotypes. Adviser: James C. Schnabl

Maize Tassel Detection From UAV Imagery Using Deep Learning

The timing of flowering plays a critical role in determining the productivity of agricultural crops. If the crops flower too early, the crop would mature before the end of the growing season, losing the opportunity to capture and use large amounts of light energy. If the crops flower too late, the crop may be killed by the change of seasons before it is ready to harvest. Maize flowering is one of the most important periods where even small amounts of stress can significantly alter yield. In this work, we developed and compared two methods for automatic tassel detection based on the imagery collected from an unmanned aerial vehicle, using deep learning models. The first approach was a customized framework for tassel detection based on convolutional neural network (TD-CNN). The other method was a state-of-the-art object detection technique of the faster region-based CNN (Faster R-CNN), serving as baseline detection accuracy. The evaluation criteria for tassel detection were customized to correctly reflect the needs of tassel detection in an agricultural setting. Although detecting thin tassels in the aerial imagery is challenging, our results showed promising accuracy: the TD-CNN had an F1 score of 95.9% and the Faster R-CNN had 97.9% F1 score. More CNN-based model structures can be investigated in the future for improved accuracy, speed, and generalizability on aerial-based tassel detection

Assessment of inducibility and spontaneous haploid genome doubling in maize (Zea mays L.)

Maize is a staple food, fuel, and feed crop grown around the world. Doubled haploid technology allows for the quick of development of inbred lines for hybrid development. The maternal in vivo doubled haploid system has gained rapid adoption by the maize breeding sector within the last 10 years. There have been significant improvements in the doubled haploid technology, which made it commercially viable. Within the doubled haploid system, there is limited genetic information about the two important traits that control the ability of generating doubled haploids, which are inducibility and spontaneous haploid genome doubling. Better understanding of these two traits could drastically improve the efficiencies and reduce labor needs for producing doubled haploid lines. In this dissertation, the genetic control of both inducibility and spontaneous haploid genome doubling were studied. A Quantitative Trait Loci (QTL) mapping study was conducted for both traits using an F2:3 population derived from inbred A427 and CR1Ht. Inducibility QTL were identified and the improvement of inducibility is examined. A major QTL was found for spontaneous haploid genome doubling and its application to doubled haploid breeding is discussed

A high-throughput, field-based phenotyping technology for tall biomass crops

Recent advances in omics technologies have not been accompanied by equally efficient, cost-effective and accurate phenotyping methods required to dissect the genetic architecture of complex traits. Even though high-throughput phenotyping platforms have been developed for controlled environments, field-based aerial and ground technologies have only been designed and deployed for short stature crops. Therefore, we developed and tested Phenobot 1.0, an auto-steered and self-propelled field-based high-throughput phenotyping platform for tall dense canopy crops, such as sorghum (Sorghum bicolor L. Moench). Phenobot 1.0 was equipped with laterally positioned and vertically stacked stereo RGB cameras. Images collected from 307 diverse sorghum lines were reconstructed in 3D for feature extraction. User interfaces were developed and multiple algorithms were evaluated for their accuracy in estimating plant height and stem diameter. Tested feature extraction methods included: i) User-interactive Individual Plant Height Extraction based on dense stereo 3D reconstruction (UsIn-PHe); ii) Automatic Hedge-based Plant Height Extraction (Auto-PHe) based on dense stereo 3D reconstruction; iii) User-interactive Dense Stereo Matching Stem Diameter Extraction (DenS-Di); and iv) User-interactive Image Patch Stereo Matching Stem Diameter Extraction (IPaS-Di). Comparative genome-wide association analysis and ground-truth validation demonstrated that both UsIn-PHe and Auto-PHe were accurate methods to estimate plant height while Auto-PHe had the additional advantage of being a completely automated process. For stem diameter, IPaS-Di generated the most accurate estimates of this biomass-related architectural trait. In summary, our technology was proven robust to obtain ground-based high-throughput plant architecture parameters of sorghum, a tall and densely planted crop species

Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant’s phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field

High-throughput analysis and advanced search for visually-observed phenotypes

Title from PDF of title page (University of Missouri--Columbia, viewed on May 13, 2013).The entire thesis text is included in the research.pdf file; the official abstract appears in the short.pdf file; a non-technical public abstract appears in the public.pdf file.Dissertation advisor: Dr. Chi-Ren ShyuIncludes bibliographical references.Vita.Ph. D. University of Missouri--Columbia 2012."May 2012"The trend in many scientific disciplines today, especially in biology and genetics, is towards larger scale experiments in which a tremendous amount of data is generated. As imaging of data becomes increasingly more popular in experiments related to phenotypes, the ability to perform high-throughput big data analyses and to efficiently locate specific information within these data based on increasingly complicated and varying search criteria is of great importance to researchers. This research develops several methods for high-throughput phenotype analysis. This notably includes a registration algorithm called variable object pattern matching for mapping multiple indistinct and dynamic objects across images and detecting the presence of missing, extra, and merging objects. Research accomplishments resulted in a number of unique advanced search mechanisms including a retrieval engine that integrates multiple phenotype text sources and domain ontologies and a search method that retrieves objects based on temporal semantics and behavior. These search mechanisms represent the first of their kind in the phenotype community. While this computational framework is developed primarily for the plant community, it has potential applications in other domains including the medical field.Includes bibliographical references