Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant’s phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field

A high-throughput, field-based phenotyping technology for tall biomass crops

Recent advances in omics technologies have not been accompanied by equally efficient, cost-effective and accurate phenotyping methods required to dissect the genetic architecture of complex traits. Even though high-throughput phenotyping platforms have been developed for controlled environments, field-based aerial and ground technologies have only been designed and deployed for short stature crops. Therefore, we developed and tested Phenobot 1.0, an auto-steered and self-propelled field-based high-throughput phenotyping platform for tall dense canopy crops, such as sorghum (Sorghum bicolor L. Moench). Phenobot 1.0 was equipped with laterally positioned and vertically stacked stereo RGB cameras. Images collected from 307 diverse sorghum lines were reconstructed in 3D for feature extraction. User interfaces were developed and multiple algorithms were evaluated for their accuracy in estimating plant height and stem diameter. Tested feature extraction methods included: i) User-interactive Individual Plant Height Extraction based on dense stereo 3D reconstruction (UsIn-PHe); ii) Automatic Hedge-based Plant Height Extraction (Auto-PHe) based on dense stereo 3D reconstruction; iii) User-interactive Dense Stereo Matching Stem Diameter Extraction (DenS-Di); and iv) User-interactive Image Patch Stereo Matching Stem Diameter Extraction (IPaS-Di). Comparative genome-wide association analysis and ground-truth validation demonstrated that both UsIn-PHe and Auto-PHe were accurate methods to estimate plant height while Auto-PHe had the additional advantage of being a completely automated process. For stem diameter, IPaS-Di generated the most accurate estimates of this biomass-related architectural trait. In summary, our technology was proven robust to obtain ground-based high-throughput plant architecture parameters of sorghum, a tall and densely planted crop species

Generation of 360 Degree Point Cloud for Characterization of Morphological and Chemical Properties of Maize and Sorghum

Recently, imaged-based high-throughput phenotyping methods have gained popularity in plant phenotyping. Imaging projects the 3D space into a 2D grid causing the loss of depth information and thus causes the retrieval of plant morphological traits challenging. In this study, LiDAR was used along with a turntable to generate a 360-degree point cloud of single plants. A LABVIEW program was developed to control and synchronize both the devices. A data processing pipeline was built to recover the digital surface models of the plants. The system was tested with maize and sorghum plants to derive the morphological properties including leaf area, leaf angle and leaf angular distribution. The results showed a high correlation between the manual measurement and the LiDAR measurements of the leaf area (R2\u3e0.91). Also, Structure from Motion (SFM) was used to generate 3D spectral point clouds of single plants at different narrow spectral bands using 2D images acquired by moving the camera completely around the plants. Seven narrow band (band width of 10 nm) optical filters, with center wavelengths at 530 nm, 570 nm, 660 nm, 680 nm, 720 nm, 770 nm and 970 nm were used to obtain the images for generating a spectral point cloud. The possibility of deriving the biochemical properties of the plants: nitrogen, phosphorous, potassium and moisture content using the multispectral information from the 3D point cloud was tested through statistical modeling techniques. The results were optimistic and thus indicated the possibility of generating a 3D spectral point cloud for deriving both the morphological and biochemical properties of the plants in the future. Advisor: Yufeng G

Development and assessment of a multi-sensor platform for precision phenotyping of small grain cereals under field conditions

The growing world population, changing food habits especially to increased meat consumption in newly industrialized countries, the growing demand for energy and the climate change pose major challenges for tomorrows agriculture. The agricultural output has to be increased by 70% by 2050 to achieve food and energy security for the future and 90% of this increase must be achieved by increasing yields on existing agricultural land. Achieving this increase in yield is one of the biggest challenges for the global agriculture and requires, among other things, an efficient breeding of new, higher-yielding varieties adapted to the predicted climate change. To achieve this goal, new methods need to be established in plant breeding which include efficient genotyping and phenotyping approaches of crops. Enormous progress has been achieved in the field of genotyping which enables to gain a better understanding of the molecular basis of complex traits. However, phenotyping must be considered as equally important as genomic approaches rely on high quality phenotypic data and as efficient phenotyping enables the identification of superior lines in breeding programs. In contrast to the rapid development of genotyping approaches, phenotyping methods in plant breeding have changed only little in recent decades which is also referred to as phenotyping bottleneck. Due to this discrepancy between available phenotypic and genotypic information a significant potential for crop improvement remains unexploited. The aim of this work was the development and evaluation of a precision phenotyping platform for the non-invasive measurement of crops under field conditions. The developed platform is assembled of a tractor with 80 cm ground clearance, a carrier trailer and a sensor module attached to the carrier trailer. The innovative sensors for plant phenotyping, consisting of several 3D Time-of-Flight cameras, laser distance sensors, light curtains and a spectral imaging camera in the near infrared reflectance (NIR) range, and the entire system technology for data acquisition were fully integrated into the sensor module. To operate the system, software with a graphical user interface has been developed that enables recording of sensor raw data with time- and location information which is the basis of a subsequent sensor and data fusion for trait determination. Data analysis software with a graphical user interface was developed under Matlab. This software applies all created sensor models and algorithms on sensor raw data for parameter extraction, enables the flexible integration of new algorithms into the data analysis pipeline, offers the opportunity to generate and calibrate new sensor fusion models and allows for trait determination. The developed platform facilitates the simultaneous measurement of several plant parameters with a throughput of over 2,000 plots per day. Based on data of the years 2011 and 2012, extensive calibrations were developed for the traits plant height, dry matter content and biomass yield employing triticale as a model species. For this purpose, 600 plots were grown each year and recorded twice with the platform followed by subsequent phenotyping with state-of-the-art methods for reference value generation. The experiments of each year were subdivided into three measurements at different time points to incorporate information of three different developmental stages of the plants into the calibrations. To validate the raw data quality and robustness of the data collection and reduction process, the technical repeatability for all developed data analysis algorithms was determined. In addition to these analyses, the accuracy of the generated calibrations was assessed as the correlations between determined and observed phenotypic values. The calibration of plant height based on light curtain data achieved a technical repeatability of 0.99 and a correlation coefficient of 0.97, the calibration of dry matter content based on spectral imaging data a of 0.98 and a of 0.97. The generation and analysis of dry biomass calibrations revealed that a significant improvement of measurement accuracy can be achieved by a fusion of different sensors and data evaluations. The calibration of dry biomass based on data of the light curtains, laser distance sensors, 3D Time-of-Flight cameras and spectral imaging achieved a of 0.99 and a of 0.92. The achieved excellent results illustrate the suitability of the developed platform, the integrated sensors and the data analysis software to non-invasively measure small grain cereals under field conditions. The high utility of the platform for plant breeding as well as for genomic studies was illustrated by the measurement of a large population with a total of 647 doubled haploid triticale lines derived from four families that were grown in four environments. The phenotypic data was determined based on platform measurements and showed a very high heritability for dry biomass yield. The combination of these phenotypic data with a genomic approach enabled the identification of quantitative trait loci (QTL), i.e., chromosomal regions affecting this trait. Furthermore, the repeated measurements revealed that the accumulation of biomass is controlled by temporal genetic regulation. Taken together, the very high robustness of the system, the excellent calibration results and the high heritability of the phenotypic data determined based on platform measurements demonstrate the utility of the precision phenotyping platform for plant breeding and its enormous potential to widen the phenotyping bottleneck.Die stetig wachsende Weltbevölkerung, sich ändernde Ernährungsgewohnheiten hin zu vermehrtem Fleischkonsum in Schwellenländern, der stetig wachsende Energiebedarf sowie der Klimawandel stellen große Herausforderungen an die Landwirtschaft von morgen. Um eine gesicherte Lebensmittel- und Energieversorgung zu gewährleisten muss die landwirtschaftliche Produktion bis 2050 um 70% gesteigert werden, wobei 90% dieser Steigerung durch eine Erhöhung der Erträge auf bereits bestehenden landwirtschaftlichen Flächen erzielt werden muss. Diese erforderliche Ertragssteigerung ist eine der größten Herausforderungen für die weltweite Landwirtschaft und bedarf unter anderem einer effizienten Züchtung neuer, an den Klimawandel angepasster, ertragsreicherer Sorten. Um eine ausreichende Steigerung der Erträge sicherstellen zu können müssen neue Methoden in der Pflanzenzucht etabliert werden, welche auf einer effizienten Geno- sowie Phänotypisierung der Pflanzen basieren. Im Bereich der Genotypisierung gab es in den letzten Jahrzehnten große Fortschritte, wodurch ein enormer Wissenszuwachs über die molekulare Basis komplexer Merkmale erzielt werden konnte. Trotzdem ist der Bereich der Phänotypisierung als ebenso wichtig anzusehen, da genetische Untersuchungen unter anderem von der Qualität phänotypischer Daten abhängen und qualitativ hochwertige phänotypische Daten die Selektion überlegener Linien in der Pflanzenzucht verbessern können. Im Vergleich zur Genotypisierung gab es jedoch im Bereich der Phänotypisierung in den letzten Jahrzehnten nur wenig wissenschaftlichen Fortschritt. Durch dieses Missverhältnis zwischen der Qualität phänotypischer und genotypischer Informationen bleibt somit ein erhebliches Potential an neuen Erkenntnissen unentdeckt. Das Ziel dieser Arbeit war die Entwicklung und Bewertung einer Präzisionsphänotypisierungsplattform zur zerstörungsfreien Charakterisierung von Energiegetreide in der Pflanzenzucht, um den aktuell bestehenden Flaschenhals bei der Umsetzung neuer Zuchtmethoden zu weiten. Die entwickelte Plattform ist ein Gespann bestehend aus einem Hochradschlepper mit 80 cm Bodenfreiheit, einem eigens entwickelten Trägeranhänger und einem am Trägeranhänger befestigten Sensormodul. Die innovative Sensorik zur Pflanzenvermessung, bestehend aus mehreren 3D Time-of-Flight Kameras, Laserabstandssensoren, Lichtgittern und einem bildgebenden Spektralmessgerät im nahen infrarot (NIR) Bereich, sowie die gesamte Systemtechnik zur Datenaufnahme wurden vollständig im Sensormodul integriert. Zur Bedienung des Systems wurde eine Software mit graphischer Benutzeroberfläche entwickelt, die eine zeit- und ortsbezogene Aufnahme der Sensorrohdaten ermöglicht, was die Grundlage einer anschließenden Sensor- und Datenfusion zur Merkmalsbestimmung darstellt. Zur Datenauswertung wurde eine Software mit graphischer Benutzeroberfläche unter Matlab entwickelt. Durch diese Software werden alle erstellten Sensormodelle und Algorithmen zur Datenauswertung auf die Rohdaten angewendet, wobei neue Algorithmen flexibel in das System eingebunden, Sensorfusionsmodelle erzeugt und kalibriert und Pflanzenparameter bestimmt werden können. Die entwickelte Plattform ermöglicht die simultane Vermessung mehrerer Pflanzenparameter bei einem Durchsatz von über 2000 Parzellen pro Tag. Basierend auf Daten aus den Jahren 2011 und 2012 wurden umfangreiche Kalibrierungen für die Parameter Pflanzenhöhe, Trockensubstanzgehalt und Trockenmasse für Triticale erstellt. Zu diesem Zweck wurden in beiden Jahren Feldversuche mit jeweils 600 Parzellen angelegt, doppelt mit der Plattform vermessen und zur Referenzwertgenerierung im Anschluss konventionell phänotypisiert. In beiden Jahren wurden drei Messungen von jeweils 200 Parzellen zu drei verschiedenen Zeitpunkten durchgeführt, um Daten unterschiedlicher Entwicklungsstadien der Pflanzen für die Erstellung der Kalibrierungen zur Verfügung zu haben. Zur Validierung der Rohdatenqualität sowie der Robustheit der Datenreduktionsverfahren wurden zunächst für alle entwickelten Auswertungsalgorithmen basierend auf den Wiederholungsmessungen die technischen Wiederholbarkeiten bestimmt. Neben der Validierung der Rohdatenqualität wurden die Genauigkeiten der erstellten Kalibrierungen als Korrelation zwischen den Referenzwerten und den mit der Sensorplattform gemessenen Werten ermittelt. Die Kalibrierung der Pflanzenhöhe basierend auf Lichtgitterdaten erreicht eine technische Wiederholbarkeit Rw2 von 0.99 und einen Korrelationskoeffizienten Rc² von 0.97, die Kalibrierung des Trockensubstanzgehalts basierend auf Spectral-Imaging Daten ein Rw2 von 0.98 und ein Rc² von 0.97. Bei der Erstellung der Trockenmasse Kalibrierung konnte gezeigt werden, dass durch eine Fusion verschiedener Sensoren und Datenauswertungen eine signifikante Verbesserung der Messgenauigkeit erreicht werden kann. Die Kalibrierung der Trockenmasse basierend auf Daten der Lichtgitter, Laserabstandssensoren, 3D Time-of-Flight Kameras und des Spectral-Imaging erreicht ein Rw2 von 0.99 und ein Rc² von 0.92. Die hervorragenden technischen Wiederholbarkeiten, sowie die exzellenten Genauigkeiten der entwickelten Kalibrierungen verdeutlichen die herausragende Eignung der entwickelten Plattform, der integrierten Sensoren und der entwickelten Datenaufnahme- sowie Datenauswertesoftware zur zerstörungsfreien Phänotypisierung von Getreide unter Feldbedingungen. Der hohe praktische Nutzen der Plattform für die Pflanzenzucht sowie für genetische Studien konnte durch die wiederholte Phänotypisierung einer DH Population mit 647 doppelhaploiden Triticale Linien in vier Umwelten aufgezeigt werden. Die Pflanzen wurden mit der Plattform an drei verschiedenen Zeitpunkten phänotypisiert und die erzeugten Daten zeigten eine sehr hohe Heritabilität für Biomasse. Die Kombination dieser phänotypischen mit genotypischen Informationen in einer Assoziationskartierungsstudie ermöglichte die Identifizierung von Regionen im Genom welche für quantitative Merkmale (QTL) kodieren. So konnten z.B. Regionen auf mehreren Chromosomen identifiziert werden, welche die Biomasse beeinflussen. Des Weiteren konnte durch Auswertung der wiederholten Messungen der Nachweis erbracht werden, dass die Biomasseentwicklung durch sich zeitlich ändernde genetische Mechanismen beeinflusst wird. Die erreichte sehr hohe Robustheit des Systems, die exzellenten Kalibrierungsergebnisse und die hohen Heritabilitäten der mit der Plattform bestimmten phänotypischen Daten verdeutlichen die hervorragende Eignung des Systems zur Anwendung in der Pflanzenzucht und das enorme Potential der entwickelten Technologie zur Weitung des aktuell bestehenden Phänotypisierungs-Flaschenhalses

A multi-sensor system for high throughput field phenotyping in soybean and wheat breeding

Collecting plant phenotypic data with sufficient resolution (in both space and time) and accuracy represents a long standing challenge in plant science research, and has been a major limiting factor for the effective use of genomic data for crop improvement. This is particularly true in plant breeding where collecting large-scale field-based plant phenotypes can be very labor intensive and costly. In this paper we reported a multi-sensor system for high throughput phenotyping in plant breeding. The system comprised five sensor modules (ultrasonic distance sensors, thermal infrared radiometers, NDVI sensors, portable spectrometers, and RGB web cameras) to measure crop canopy traits from field plots. A GPS was used to geo-reference the sensor measurements. Two environmental sensors (a solar radiation sensor and air temperature/relative humidity sensor) were also integrated into the system to collect simultaneous environmental data. A LabVIEW program was developed to control and synchronize measurements from all sensor modules and stored sensor readings in the host computer. Canopy reflectance spectra (by portable spectrometers) were post processed to extract NDVI and red-edge NDVI spectral indices; and RGB images were post processed to extract canopy green pixel fraction (as a proxy for biomass). The sensor system was tested in a soybean and wheat field trial. The results showed strong correlations among the sensor-based plant traits at both early and late growing season. Significant correlations were also found between the sensor-based traits and final grain yield at the early season (Pearson’s correlation coefficient r ranged from 0.41 to 0.55) and late season (r from 0.55 to 0.70), suggesting the potential use of the sensor system to assist in phenotypic selection for plant breeding. The sensor system performed satisfactorily and robustly in the field tests. It was concluded that the sensor system could be a powerful tool for plant breeders to collect field-based, high throughput plant phenotyping data

A Novel LiDAR-Based Instrument for High-Throughput, 3D Measurement of Morphological Traits in Maize and Sorghum

Recently, imaged-based approaches have developed rapidly for high-throughput plant phenotyping (HTPP). Imaging reduces a 3D plant into 2D images, which makes the retrieval of plant morphological traits challenging. We developed a novel LiDAR-based phenotyping instrument to generate 3D point clouds of single plants. The instrument combined a LiDAR scanner with a precision rotation stage on which an individual plant was placed. A LabVIEW program was developed to control the scanning and rotation motion, synchronize the measurements from both devices, and capture a 360◦ view point cloud. A data processing pipeline was developed for noise removal, voxelization, triangulation, and plant leaf surface reconstruction. Once the leaf digital surfaces were reconstructed, plant morphological traits, including individual and total leaf area, leaf inclination angle, and leaf angular distribution, were derived. The system was tested with maize and sorghum plants. The results showed that leaf area measurements by the instrument were highly correlated with the reference methods (R2 \u3e 0.91 for individual leaf area; R2 \u3e 0.95 for total leaf area of each plant). Leaf angular distributions of the two species were also derived. This instrument could fill a critical technological gap for indoor HTPP of plant morphological traits in 3D

A Low-cost Depth Imaging Mobile Platform for Canola Phenotyping

To meet the high demand for supporting and accelerating progress in the breeding of novel traits, plant scientists and breeders have to measure a large number of plants and their characteristics accurately. A variety of imaging methodologies are being deployed to acquire data for quantitative studies of complex traits. When applied to a large number of plants such as canola plants, however, a complete three-dimensional (3D) model is time-consuming and expensive for high-throughput phenotyping with an enormous amount of data. In some contexts, a full rebuild of entire plants may not be necessary. In recent years, many 3D plan phenotyping techniques with high cost and large-scale facilities have been introduced to extract plant phenotypic traits, but these applications may be affected by limited research budgets and cross environments. This thesis proposed a low-cost depth and high-throughput phenotyping mobile platform to measure canola plant traits in cross environments. Methods included detecting and counting canola branches and seedpods, monitoring canola growth stages, and fusing color images to improve images resolution and achieve higher accuracy. Canola plant traits were examined in both controlled environment and field scenarios. These methodologies were enhanced by different imaging techniques. Results revealed that this phenotyping mobile platform can be used to investigate canola plant traits in cross environments with high accuracy. The results also show that algorithms for counting canola branches and seedpods enable crop researchers to analyze the relationship between canola genotypes and phenotypes and estimate crop yields. In addition to counting algorithms, fusing techniques can be helpful for plant breeders with more comfortable access plant characteristics by improving the definition and resolution of color images. These findings add value to the automation, low-cost depth and high-throughput phenotyping for canola plants. These findings also contribute a novel multi-focus image fusion that exhibits a competitive performance with outperforms some other state-of-the-art methods based on the visual saliency maps and gradient domain fast guided filter. This proposed platform and counting algorithms can be applied to not only canola plants but also other closely related species. The proposed fusing technique can be extended to other fields, such as remote sensing and medical image fusion

Digital phenotyping and genotype-to-phenotype (G2P) models to predict complex traits in cereal crops

The revolution in digital phenotyping combined with the new layers of omics and envirotyping tools offers great promise to improve selection and accelerate genetic gains for crop improvement. This chapter examines the latest methods involving digital phenotyping tools to predict complex traits in cereals crops. The chapter has two parts. In the first part, entitled “Digital phenotyping as a tool to support breeding programs”, the secondary phenotypes measured by high-throughput plant phenotyping that are potentially useful for breeding are reviewed. In the second part, “Implementing complex G2P models in breeding programs”, the integration of data from digital phenotyping into genotype to phenotype (G2P) models to improve the prediction of complex traits using genomic information is discussed. The current status of statistical models to incorporate secondary traits in univariate and multivariate models, as well as how to better handle longitudinal (for example light interception, biomass accumulation, canopy height) traits, is reviewe