Popova, Anna

Reischl, Markus

Rettenberger, Luca

Rieken Münke, Friedrich

English

KITopen

Friedrich Rieken Münke*, Luca Rettenberger, Anna Popova, and Markus ReischlA Lightweight Framework for SemanticSegmentation of Biomedical ImagesAbstract: We introduce a lightweight framework for seman-tic segmentation that utilizes structured classifiers as an alter-native to deep learning methods. Biomedical data is knownfor being scarce and difficult to label. However, this frame-work provides a lightweight, easy-to-apply, and fast-to-trainapproach that can be adapted to changes in image materialthough efficient retraining. Moreover, the framework is able toadapt to various input sizes making it robust against changesin resolution and is not tied to specialized hardware, whichallows efficient application on standard laptops or desktopswithout GPUs. We benchmark two distinct models, a singlestructured classifier and an ensemble of structured classifiers,against a U-Net, evaluating overall performance and train-ing speed. The framework is versatile and can be applied tomulti-class semantic segmentation. Our study shows that theproposed framework can effectively compete with establisheddeep learning methods on diverse datasets in terms of perfor-mance while reducing training time immensely.Keywords: Semantic Segmentation, Structured Classifier,Structured Random Forest, Deep Learning, Machine Learn-ing, Benchmark1 IntroductionComputer vision has become increasingly important in a widerange of fields, from self-driving cars to medical imaging. Oneof the most fundamental tasks in computer vision is semanticsegmentation, which involves labeling each pixel in an imageto identify different objects and regions of interest. In particu-lar, biomedical image segmentation is widely used to analyzeexperimental results or to support the diagnosis of diseases.While deep learning approaches, such as Convolutional Neu-ral Networks (CNNs), show state-of-the-art performance on*Corresponding author: Friedrich Rieken Münke, Institute forAutomation and Applied Informatics, Karlsruhe Institute ofTechnology, Eggenstein-Leopoldshafen, Germany, e-mail:friedrich.muenke@kit.eduLuca Rettenberger, Markus Reischl, Institute for Automation andApplied Informatics, Karlsruhe Institute of Technology,Eggenstein-Leopoldshafen, GermanyAnna Popova, Institute of Biological and Chemical Systems -Functional and Molecular Systems, Karlsruhe Institute of Technol-ogy, Eggenstein-Leopoldshafen, Germanya variety of datasets, they are complex and computationallyexpensive [9]. These methods often require large amounts ofannotated data to train, which can be a significant hurdle inbiomedical applications where labeling is difficult and time-consuming [12]. In addition, biomedical data can be highlyvariable, with differences in color, resolution, annotator noiseand other modalities between datasets. This variability makesit challenging for CNNs to generalize across different datasets,which may require retraining the model for each new dataset.To address these challenges, we propose a lightweight and thusfast to train framework for semantic segmentation based onstructured random forests.Structured random forests are a technique used in com-puter vision for assigning labels to each pixel in an image toidentify regions of interest [3]. This approach is similar to aCNN, where a kernel defines the neighborhood for each pixel,and is used to select features per pixel for the final classifi-cation. Structured random forests have been used for variousimage segmentation tasks, including edge detection [3], braintumor segmentation [15], and road crack segmentation [13]. Itis possible to further increase performance by combining mul-tiple structured classifiers [1].Our objective is to use this method to enable biomedicalresearchers to analyze and interpret their image data, enablingfaster and efficient evaluations of experiments and supportingdiagnoses of diseases.2 MethodThe proposed framework integrates structured segmentationalgorithms as modular components to enable the creation of ef-ficient and powerful multi-class semantic segmentation mod-els. We showcase the capabilities of the framework by present-ing two models designed to compete with a U-Net in terms ofperformance while maintaining fast training times.The first model (SC: Structured Classifier Model) consistsof one structured classifier. The classifier processes the imageat its original resolution classifying each pixel based on is sur-rounding pixels. This classifier comprises two components: akernel and a regular classifier. The kernel determines the pix-els that are taken into consideration for classification, while theregular classifier makes the final decision. The kernel gathersthe pixel values surrounding a given pixel and organizes themDE GRUYTER Current Directions in Biomedical Engineering 2023;9(1): 190-193 Open Access. © 2023 The Author(s), published by De Gruyter.  This work is licensed under the Creative Commons Attribution 4.0 International License. https://doi.org/10.1515/cdbme-2023-1048190F. Münke et al., A Lightweight Framework for Segmentation of Biomedical Imagesinto a single feature vector. The position of each value in thefeature vector corresponds to its relative position to the pixel.In our case, the kernel considers all pixels within a radius of5 pixels. This model has a limited capacity since it is not ableto learn features at a large scale. The second structured classi-fier model (ED: Encoder-Decoder Model) scales the previousmodel to enable the model to process larger context in an im-age. Instead of using a single classifier for segmentation, themodel utilizes six structured classifiers as an ensemble, whichare stacked on top of each other. The concept is depicted inFig. 1.Fig. 1: The ED model is an ensemble of six classifiers, whereeach classifier provides it predictions for the next classifiers asfeatures. Each classifier has its defined image scale and is onlytrained on one third of all images.The predictions from all previous classifiers serve as in-put for the next one. The classifiers learn features of differentscales by downscaling the image by a factor of two for thefirst three classifiers and then increasing its size to the originalresolution for the last three classifiers. Each individual classi-fier is trained on one-third of the dataset, which is randomlysampled, to reduce training time and prevent overfitting. Bothproposed models are able to process input images of any sizeduring training and inference and support colored and gray-scale images. In our case, both models process gray-scale im-ages by default to further boost their speed. We use an extratree classifier [5] to classify the pixels for all models.The models are developed to use the same data formatas deep learning-based methods, ensuring compatibility withvarious image labeling methods and frameworks specificallydesigned for deep learning applications. Moreover, these mod-els can be easily adjusted to process either gray-scale or colorimages.1Our benchmark model is a U-Net [10] with aResNet18 [6] backbone and the encoder pretrained on theImagenet dataset [11]. To train the U-Net, we used commonaugmentation techniques such as rotation, flipping, cropping,blurring, salt-and-pepper noise, and brightening. Our proposedmodels are not using augmentations techniques to simplify1 For a more detailed explanation and the implementation, refer to https://github.com/FMuenke/structured_segmentationtheir training procedure. We split the dataset into trainingand validation sets using a 90-10 split, and we selected theweights that performed best on the validation dataset as our fi-nal model. If there was no improvement after 100 epochs, westopped the training. The U-Net was trained with an ADAMoptimizer, a learning rate of 10−4 and the binary-cross-entropyloss. The U-Net processes colored images, due to its pretrainedencoder. We score the models by training time in minutes andthe segmentation performance in F1-Score (Dice Coefficient).Each model was trained three times to validate against effectsof randomness and the average result is used as a representa-tive value. The training time was measured on a MacBook Pro(2020) with an 2,3 GHz Quad-Core Intel Core i7 CPU and 16GB Ram.3 DatasetsIn Tab. 1, we introduce biomedical datasets as general bench-mark for evaluating semantic segmentation algorithms. InFig. 2 we present exemplary images and discuss the diversechallenges. Each dataset is prepared to present a binary seg-mentation task to the models separating between foregroundand background.Dataset Train Test DomainCryoNuSeg [7] 148 30 NucleiLIVECell [4] 2.640 176 CellDerma ISIC [2] 2.000 600 Skin LesionSeafood [14] 8.550 450 Seafood TypesSphere [8] 201 38 Tumor SpheroidsTab. 1: An overview over all benchmark datasets used to show-case the functionality of the proposed framework and the distribu-tion of training and test images.The CryoNuSeg [7] dataset is the first annotated dataset ofHematoxylin and Eosin (H&E)-stained images for nuclei seg-mentation from 10 different organs. Its diverse textures makeit a challenging dataset that requires adaptation to the appear-ance of different organs. The LIVECell [4] dataset is a high-quality dataset of phase-contrast images for cell segmentation.Due to time constraints, we use a subset of the original datasetfor training, where the training dataset consists of fixed-sizecrops (256px x 256px) while the test set has a different resolu-tion (704px x 520px). Structural difficulty is common in otherdatasets and underscores the importance of input size flexibil-ity. The Derma ISIC [2] dataset focuses on skin lesion anal-ysis and melanoma detection, with nine diagnostic categoriesand varying image sizes in the training and test datasets. This191F. Münke et al., A Lightweight Framework for Segmentation of Biomedical Images(a) CryoNuSeg [7] (b) LIVECell [4] (c) Derma ISIC [2] (d) Seafood [14] (e) Sphere [8]Fig. 2: Five examples of images from each benchmark dataset for semantic segmentation, showcasing the variability in object size andarea, textures, and shape features present in the datasets.dataset poses a difficult challenge due to its varying appear-ance and large differences in the melanoma scale in the image.The Seafood [14] dataset comprises nine seafood types, and itsimages are heavily augmented with rotation, resulting in thelargest dataset. This dataset is distinct from others due to itsunique combination of color and texture information. Finally,the Sphere [8] dataset consists of tumor spheroids captured ina high-throughput Droplet Microarray (DMA). This datasetsposes a unique challenge since there is only one segmentationtarget and it often overlaps with the borders of the capturingsetup.4 ResultsWe evaluate the models on the introduced datasets. We mea-sure the performance by computing the F1-Score on the testdataset, the results are shown in Tab. 2. The training time ismeasured in seconds and reported in Tab. 3.PerformanceModelCryoNuSeg LIVECell ISIC Seafood SphereU-Net 59.08 81.97 82.51 92.81 92.52SC 59.90 92.50 57.44 64.79 79.92ED 60.77 93.04 69.62 84.55 90.24ED-c 56.70 92.93 71.65 91.14 83.81Tab. 2: Summary of the performance in F1-Score on all datasets.ED-c represents the ED model with color processing enabled.On the CryoNuSeg dataset, both of our models achievedbetter results than the U-Net by approximately 1% and 2%.However, this dataset is challenging due to its complexity anddiversity, resulting in lower performance across all algorithms.On the LIVECell dataset, both proposed models outperformedthe U-Net by approximately 11%, and perform similarly toeach other. Although the ED model has the capability to an-alyze more context, we believe it was not necessary in thiscase since the segmentation targets were small in size. Thedataset features a unique characteristic of different resolutionsbetween the test and training images. Our proposed modelsare able to easily accommodate inputs of varying sizes, result-ing in no impact on their performance. In contrast, the U-Netwas unable to adjust to this variation. On the ISIC dataset, theU-Net model exceeds the performance of the ED model byapproximately 13% and the SC model by approximately 25%.The diversity of the ISIC dataset, which includes varying im-age capture setups and a range of melanoma types, is likelythe reason for the notable differences in performance amongthe models. It’s worth noting that the melanomas in the datasetalso vary greatly in scale, which could be another contributingfactor. On the seafood dataset, the U-Net outperformed ourproposed models by approximately 8% and 28%. Since bothproposed models were processing gray-scale images, crucialinformation was lost. On the Spheroid dataset, the U-Net againperformed better, outperforming our proposed models by ap-proximately 2% and 13%. Additionally, we are evaluating themodel ED-c, which is able to process colored images. The re-sults show a positive impact on the F1-Score for the Seafood(+6.59%) and ISIC (+2.03%) dataset, while the impact is smallor negative for the datasets CryoNuSeg, LIVECell and Sphere.We also evaluated the speed of training, the results areshown in Tab. 3. Both of our proposed models showed notablyfaster training times compared to the U-Net, with the simplerSC model being about 100 times faster and the ED model per-forming about 10 times as well on all evaluated datasets. Dur-ing inference the U-Net is able to process an image in 113ms,while the SC model requires 924ms and the ED model takes1699ms. While the inference is slower it is still able to processimages with reasonable speed.When comparing the performance of the SC and ED mod-els, the ED model outperforms the SC model. Particularly,the ED model displays superior performance on the last threedatasets, which contain larger objects or require a broader con-text to identify the primary target against background objects.192F. Münke et al., A Lightweight Framework for Segmentation of Biomedical ImagesHowever, the difference in datasets with numerous smallerobjects such as CryoNuSeg and LIVECell was not as pro-nounced. It is important to note that this improvement comesat the expense of training speed, as the SC model is 10 timesfaster than the ED model. Thus, we recommend using the SCmodel for segmenting many small objects, and the ED modelwhen a larger image context is necessary.Overall, our proposed models were able to outperform theU-Net on two out of five datasets and provide competitive per-formance on the Sphere and Seafood dataset. We also demon-strated that for complex datasets with important color features,the performance can be further improved by activating coloredinputs. Additionally, we proof that our framework enables fasttraining cycles and is able to quickly adapt to new datasets.Training TimeModelCryoNuSeg LIVECell ISIC Seafood SphereU-Net 53.5 886.0 702.2 27665.0 71.4SC 0.5 1.7 4.8 8.1 0.3ED 5.2 26.4 23.9 79.6 1.3ED-c 7.3 45.4 43.6 97.0 3.4Tab. 3: Summary of the training times on all datasets in minutes.ED-c represents the ED model with color processing enabled.5 ConclusionOur paper presents a lightweight framework for semantic seg-mentation. Biomedical data is notoriously scarce and hard tolabel, making it challenging to train deep-learning models thatcan generalize to new datasets. Our framework overcomes thischallenge by enabling researchers to quickly generate special-ized models for each specific dataset. This allows the frame-work to be easily adapted to any changes in image mate-rial that may arise. Through our experiments, we evaluatedtwo different models in terms of training speed and F1-Scoreperformance. Our findings demonstrate that the frameworkcan achieve impressive results on four out of five benchmarkdatasets with diverse characteristics, and that its fast trainingtime, up to 100 times faster than the U-Net on a CPU, providesa feasible alternative to popular deep-learning frameworks inscenarios where GPU resources are limited or unavailable. Werecommend the use of the ensemble of structured classifiersmodel when larger image context is needed and the singlestructured classifier model when segmenting many small ob-jects.Future works may involve enhancing the framework withimage augmentation techniques, mechanisms for automaticdetection of the need for colored image processing, and quan-tifying the minimum number of images required for good re-sults to reduce unnecessary labeling effort.References[1] Amiri S, et al. Bayesian Network and Structured RandomForest Cooperative DeepLearning for Automatic Multi-labelBrain Tumor Segmentation. In: 10th International Conferenceon Agents and Artificial Intelligence (ICAART 2018). 2018,183–190.[2] Codella NCF, et al. Skin Lesion Analysis Toward MelanomaDetection: A Challenge at the 2017 International Symposiumon Biomedical Imaging (ISBI), Hosted by the InternationalSkin Imaging Collaboration (ISIC) 2018.[3] Dollar P, et al. Structured Forests for Fast Edge Detection.In: Proceedings of the IEEE International Conference onComputer Vision (ICCV). 2013.[4] Edlund C, et al. LIVECell—A large-scale dataset for label-free live cell segmentation. Nature Methods 2021;18.[5] Geurts P, et al. Extremely randomized trees. Machine Learn-ing 2006;63.[6] He K, et al. Deep Residual Learning for Image Recognition.arXiv 2015;1512.03385.[7] Mahbod A, et al. CryoNuSeg: A Dataset for Nuclei InstanceSegmentation of Cryosectioned H&E-Stained Histologi-cal Images. Computers in Biology and Medicine 2021;132:104349.[8] Popova A, et al. Facile One Step Formation and Screening ofTumor Spheroids Using Droplet-Microarray Platform. Small2019;15.[9] Rettenberger L, et al. Self-Supervised Learning for An-notation Efficient Biomedical Image Segmentation. IEEETransactions on Biomedical Engineering 2023;1:1–11.[10] Ronneberger O, et al. U-Net: Convolutional Networksfor Biomedical Image Segmentation. CoRR 2015;abs/1505.04597.[11] Russakovsky O, et al. ImageNet Large Scale Visual Recog-nition Challenge. IJCV 2015;115:211–252.[12] Schilling M, et al. KaIDA: a modular tool for assisting imageannotation in deep learning. Journal of Integrative Bioinfor-matics 2022;19:20220018.[13] Shi Y, et al. Automatic Road Crack Detection Using Ran-dom Structured Forests. IEEE Transactions on IntelligentTransportation Systems 2016;17:3434–3445.[14] Ulucan O, et al. A Large-Scale Dataset for Fish Segmen-tation and Classification. In: 2020 Innovations in IntelligentSystems and Applications Conference (ASYU). 2020, 1–5.[15] Zhang J, et al. Segmentation of Perivascular Spaces UsingVascular Features and Structured Random Forest from 7TMR Image. In: Machine Learning in Medical Imaging. 2016,61–68.193

A Lightweight Framework for Semantic Segmentation of Biomedical Images

https://publikationen.bibliothek.kit.edu/1000162504/151384710

A Lightweight Framework for Semantic Segmentation of Biomedical Images

Abstract

Similar works

Full text

Available Versions

KITopen