Beyerer, Jürgen

Gruna, Robin

Krause, Julius

Krähmer, Andrea

Längle, Thomas

Maier, Georg

Tron, Nanina

English

KITopen

Detection of pyrrolizidine alkaloid containingherbs using hyperspectral imaging in theshort-wave infraredJulius Krause1, Nanina Tron2, Georg Maier1, Andrea Krähmer2,Robin Gruna1, Thomas Längle1, and Jürgen Beyerer1,31 Fraunhofer IOSB, Karlsruhe, Institute of Optronics, System Technologiesand Image Exploitation, Visual Inspection Systems,Fraunhoferstr. 1, 76131 Karlsruhe, Germany2 Julius Kühn-Institut (JKI), Federal Research Centre of Cultivated Plants,Institute for Ecological Chemistry, Plant Analysis and Stored ProductProtection, Königin-Luise-Str. 19, 14195 Berlin, Germany3 Karlsruhe Institute of Technology (KIT), Vision and Fusion Laboratory(IES), Haid-und-Neu-Str. 7, 76131 Karlsruhe, GermanyAbstract Plants containing pyrrolizidine alkaloids (PA) are un-wanted contaminants in consumer products such as herbal teadue to their toxicity to humans. The detection of these plants ortheir components using hyperspectral imaging was investigated,with focus on application in sensor-based sorting. For this, 431hyperspectral images of leafs from three common herbs (pepper-mint, lemon balm, stinging nettle) and the poisonous commongroundsel were acquired. By using a convolutional neural net-work, a mean F1 score of 0.89 was obtained for the classificationof all four plant products based on the individual spectra. Tovalidate the neural network, significant wavelengths were deter-mined and visualized in an attribution map.Keywords Hyperspectral Imaging, convolutional neural net-work, Pyrrolizidine alkaloids, sensor-based sorting45J. Krause et. al.1 IntroductionPyrrolizidine alkaloids (PA) are secondary plant substances that aretoxic to the genome and the liver. They can pose a health hazard, asthey are often found as a contaminant in food or medicinal plant prod-ucts. Just a few plants per hectare e.i. of Senecio are sufficient to con-taminate the product non marketable. Hence, regular field monitoringand removal of appropriate weeds is necessary. This is a considerablepersonnel effort that can hardly be afforded economically by the grow-ers. Additionally, for correct identification of toxic herbs and to preventcontamination of the products, the growers need well trained personal.Therefore, methods to identify and remove toxic contaminants afterharvest are highly demanded.Sensor-based sorting is a machine vision application that has foundindustrial application in various fields. An accept-or-reject task is exe-cuted by deflecting single particles from a material stream. The mainfields of application of sensor-based sorting are recycling, e.g., remov-ing materials from glass shard streams harmful to the melting processsuch as stones and ceramic glass [1]. Another field of application is theprocessing of industrial minerals, mainly to remove unwanted ganguefrom ore, e.g., copper-gold ore [2]. For ensuring product safety forfoodstuff and agricultural products, these methods are used for thedetection and removal of fungus-infected wheat kernels [3]. Hence,sensor-based sorting of crops also represents an opportunity to safethe harvest potentially contaminated with PA.In this paper, three of the economically most important herb culturesin Germany (peppermint, lemon balm, stinging nettle) and the mostcommon contaminant of these cultures (common groundsel) were char-acterized by near infrared spectroscopy. Optical spectroscopy in nearand short-wave infrared is particularly suitable for the detection anddifferentiation of organic products. Using hyperspectral camera sys-tems, material streams can be monitored and foreign substances can beseparated after detection by implementing the techniques in a sensor-based sorting system. Targeting this application, hyperspectral datawas acquired for the mentioned herbs and their frequent contamina-tion in an experimental sensor-based sorting system.46Detection of pyrrolizidine alkaloid containing herbs2 Material and methodsThis section describes the experimental sensor-based sorting systemthat was used to acquire the data. In addition, insights into the result-ing data set are given. Finally, the novel data analysis method for thetask at hand is described.2.1 Experimental sensor-based sorting platform with hyperspectralcamera systemIn the course of this study, data for different plants are acquired us-ing an experimental sorting platform that is equipped with an hyper-spectral camera system, see Fig. 2.1. A thorough description of theexperimental platform is provided in [4]. The spectral sensitivity ofthe InGaAs sensor of the line-scanning hyperspectral camera lies inthe range of approximately 1200 to 2200 nm that is sampled into 256spectral bands and provides 320 pixels locally at a maximal tempo-ral resolution of approximately 300 hz. The implemented illuminationconsists of two arrays of halogen spotlights, which are locally targetedat the scan-line of the camera. Transportation of the material is real-ized by means of a conveyor belt that runs at 1.1 ms−1. The materialis observed directly after being discharged from the belt, i. e., during afree flight phase. An array of pneumatic nozzles is located behind thisscan-line and serves the purpose of material separation. It consists of16 nozzles that can be triggered individually, each of which covering awidth of 10 mm.2.2 DatasetThe plant material for the measurements was cultivated in beds at theJKI Berlin. The medicinal cultures were harvested twice, each timeat the beginning of flowering, as the concentration of essential oils ishighest then, using a cutting height of approx. 15 cm. For lemon balm(Melissa officinalis) and peppermint (Mentha x piperita ’Multimentha’) 24each and for stinging nettle (Urtica dioica) 12 and for groundsel (Seneciovulgaris) 10 single plants were harvested.For training, the data set was split by random sample selection. Aratio of 75% was assigned as training and of 25% as validation data.47J. Krause et. al.Figure 2.1: Photo of the experimental sensor-based sorting platform, equipped with thehyperspectral camera system.Table 5.1: Description of the resulting dataset.name trivialis samples spectraSenecio vulgaris groundsel 84 44683Urtica dioica stinging nettle 98 34447Melissa officinalis lemon balm 141 17747Mentha piperita peppermint 108 15970The entire data set can be described with a matrix X := {xi}i=1..N of Nspectra and a matrix Y := {yi}i=1..N of N labels. Each spectrum xi ∈RQ contains the reflectance of Q = 256 spectral bands. The labels yi ∈RC are coded as one-hot vectors with C = 4 classes, which correspondto the plant types. An exemplary visualization of the data is shown inFig. 2.2.48Detection of pyrrolizidine alkaloid containing herbsFigure 2.2: Shown are the mean spectra of all four plant species. Through the intensitydifferences at certain wavelengths, a classification can be done.2.3 Data analysis using the convolutional neural network AnniNetA new neural network architecture, AnniNet, has been developed atFraunhofer IOSB. This architecture is particularly suitable for the anal-ysis of near-infrared spectra. AnniNet consists of three components,namely an encoder network for feature extraction, a decoder networkto improve feature extraction and a classification network that esti-mates the class membership of a spectrum. All three networks aretrained in parallel through multi-task learning. The feature extractionstructure does not require any type of spectral data pre-processing.The individual parts of AnniNet and multi-task learning are presentedin the following.Encoder networkThe encoder layer is used for feature extraction. For this purpose, alayer with different one-dimensional convolution kernels is trained andapplied to the spectrum. This layer can be compared with wavelet-based methods that are already successfully used in chemometrics [5].It has been shown that wavelets are successful in reducing noise [6]and suppressing background effects [7]. This enabled improved re-sults [8], even the transfer of chemometric models was improved [9].The features determined by means of the convolutional layer are sent to49J. Krause et. al.a pooling layer. This layer has a very practical effect besides reducingthe parameters for the following layers. The feature extraction becomesmore robust against shifts in the wavelength. This is especially a dis-turbing effect known as smiley keystone distortion in hyperspectralcameras.Decoder networkEnd-to-end learning in artificial neural networks with high dimen-sional inputs such as hyperspectral measurements requires a largeamount of labeled training samples due to a high count of weightswhich are iteratively adjusted. Autoencoders are used to learn the rep-resentation of a dataset in a unsupervised manner. Adding a decodernetwork creates such an autoencoder within AnniNet and improvestraining results without the need for additional labels.Classification networkTo evaluate the overlapping information from the individual absorp-tion bands, fully connected (dense) layers were used. In correspon-dence with the exponential behavior of absorption processes (Beer-Lambert law), the Scaled Exponential Linear Units (SELU) activationfunctionf (x) ={λα(ex − 1), if x < 0λx, otherwise(2.1)was chosen for the first dense layer. Finally, a fully connected layerwith the softmax activation function is applied. The output of the clas-sification network is a normalized probability density function of theestimated class memberships.50Detection of pyrrolizidine alkaloid containing herbsFigure 2.3: AnniNet consists of three components. First, an encoder network extractsspectral features using a convolutional layer. During training, a decoder net-work is used as an autoencoder to improve feature extraction. The classifica-tion network analyses the non-linear and overlapping spectral features.Multi-task-learningUsing multi-task-learning, all four classes and the reconstruction of theautoencoder were trained in parallel. The categorical cross entropyCCE = − 1NN∑i=1C∑j=1yi,j log ŷi,j (2.2)was used as a loss function to optimize the classification results for allclasses C. The reconstruction of the spectral data using the autoencoder51J. Krause et. al.was optimized by minimizing the Root Mean Square ErrorRMSE =1NN∑i=1√∑Qj=1(x̂i,j − xi,j)2n(2.3)for the autoencoder, which improves the feature extraction.3 Results and discussionThe classification was carried out using AnniNet without any spectralpre-processing. To validate the results in the following, only spectrafrom samples that were not included in the training were used. Tocheck the plausibility of the classification results, a sensitivity analysiswas performed and compared with the spectral data.3.1 Classification resultsThe results of the classification are shown in the confusion matrix inFig. 3.1. For all classes, the classification achieves a rather high accu-racy. The application of a majority vote for the respective samples hasled to a completely correct mapping in the validation data set. Thequality of the classification results can be quantified by the so-called F1scoreF1 =tptp + 0.5(fp + tp), (3.1)which is the harmonic mean of precision and recall. The evaluationcalculation is based on the false (fp) and true (tp) positive classificationresults. For the present four-class classification, the F1 score was deter-mined for the individual classes and averaged. The score determinedin this way is F1 = 0.89.3.2 Spectral attribution mapNeural networks cannot be interpreted by humans due to their highnumber of parameters and high-dimensional transformations. How-ever, when evaluating hyperspectral data, it is interesting to see which52Detection of pyrrolizidine alkaloid containing herbsFigure 3.1: The normalised confusions matrix shows the probabilities of the estimatedclass memberships in the individual fields. The estimation of the class mem-bership by single spectra is mostly correct. It is important here that by major-ity vote of all spectra of a sample, all test samples could be correctly classified.absorption bands have an influence on the classification result. Onepossibility for investigating neural networks is the creation of a so-called attribution map. For this purpose, individual areas of the spec-trum are successively masked and then the classification quality is eval-uated. For the classification task at hand, such attribution maps weredetermined and are visualized in Fig. 4.1. As can be seen, the classifi-cation result is mainly depend on data in the range between 1400 nmand 1900 nm. This range is between the absorption’s by water andcan therefore be considered robust. In addition, areas are selected orweighted differently for the different class memberships, which is an-other indicator of robust classification.4 SummaryLeaves from four different plants were recorded with a hyperspectralcamera built in a senor-based sorting system. The data were used with-out spectral pre-processing to train a neural network designed for thisclassification task. The trained network was able to predict individualspectra of the validation data of all four classes at a high accuracy. Us-53J. Krause et. al.ing a majority vote per sample, all validation samples were correctlyclassified. An attribution map was used to investigate which spectralranges dominate the classification decision. The result shows differ-ent spectral ranges for the individual classes, apart from absorption bywater.Figure 4.1: The attribution maps show the classification error caused by masking outindividual spectral channels, highlighting the importance of information inthe range of 1400 nm and 1900 nm.AcknowledgementThe authors of this work were supportet by the joint project ”Detek-tion und Entfernung von Pyrrolizidinalkaloid-haltigen Unkräutern ausKulturpflanzen nach der Ernte – PA-NIRSort” which is funded by theGerman Federal Ministry of Food and Agriculture and by the Facha-gentur für Nachwachsende Rohstoffe e. V. (FNR) (FKZ 220132165) onthe basis of a resolution of the German Bundestag and the FraunhoferCenter for Machine Learning within the Fraunhofer Cluster of Excel-lence Cognitive Internet Technologies CCIT.References1. G. Bonifazi and S. Serranti, “Imaging spectroscopy based strategiesfor ceramic glass contaminants removal in glass recycling,” WasteManagement, vol. 26, no. 6, pp. 627–639, Jan. 2006. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S0956053X0500164954Detection of pyrrolizidine alkaloid containing herbs2. T. Phiri, H. J. Glass, and P. Mwamba, “Development of a strategy andinterpretation of the NIR spectra for application in automated sorting,”Minerals Engineering, vol. 127, pp. 224–231, Oct. 2018. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S08926875183035223. S. R. Delwiche, T. C. Pearson, and D. L. Brabec, “High-SpeedOptical Sorting of Soft Wheat for Reduction of Deoxynivalenol,” PlantDisease, vol. 89, no. 11, pp. 1214–1219, Nov. 2005. [Online]. Available:http://apsjournals.apsnet.org/doi/abs/10.1094/PD-89-12144. G. Maier, F. Pfaff, C. Pieper, R. Gruna, B. Noack, H. Kruggel-Emden,T. Längle, U. D. Hanebeck, S. Wirtz, V. Scherer et al., “Experimental eval-uation of a novel sensor-based sorting approach featuring predictive real-time multiobject tracking,” IEEE Transactions on Industrial Electronics, vol. 68,no. 2, pp. 1548–1559, 2020.5. C. B. Singh, R. Choudhary, D. S. Jayas, and J. Paliwal, “Wavelet analysisof signals in agriculture and food quality inspection,” Food and BioprocessTechnology, vol. 3, no. 1, pp. 2–12, 2010.6. C. R. Mittermayr, S. G. Nikolov, H. Hutter, and M. Grasserbauer, “Waveletdenoising of Gaussian peaks: A comparative study,” Chemometrics and Intel-ligent Laboratory Systems, vol. 34, no. 2, pp. 187–202, 1996.7. H. Tan, S. T. Sum, and S. D. Brown, “Improvement of a standard-free methodfor near-infrared calibration transfer,” Applied Spectroscopy, vol. 56, no. 8, pp.1098–1106, 8 2002.8. X. Fu, G. Yan, B. Chen, and H. Li, “Application of wavelet transforms toimprove prediction precision of near infrared spectra,” Journal of Food Engi-neering, vol. 69, no. 4, pp. 461–466, 2005.9. J. Trygg and S. Wold, “PLS regression on wavelet compressed NIR spectra,”Chemometrics and Intelligent Laboratory Systems, vol. 42, no. 1-2, pp. 209–220,1998.55

Detection of pyrrolizidine alkaloid containing herbs using hyperspectral imaging in the short-wave infrared

https://publikationen.bibliothek.kit.edu/1000130827/106913288

Detection of pyrrolizidine alkaloid containing herbs using hyperspectral imaging in the short-wave infrared

Abstract

Similar works

Full text

Available Versions

KITopen