Ahuja, Niket

Reischl, Markus

Rettenberger, Luca

Scherr, Tim

Schilling, Marcel P.

English

KITopen

Marcel P. Schilling*, Niket Ahuja, Luca Rettenberger, Tim Scherr, and Markus ReischlImpact of Annotation Noise onHistopathology Nucleus Segmentationhttps://doi.org/10.1515/cdbme-2022-1051Abstract: Deep learning is often used for automated diagno-sis support in biomedical image processing scenarios. Anno-tated datasets are essential for the supervised training of deepneural networks. The problem of consistent and noise-free an-notation remains for experts such as pathologists. The vari-ability within an annotator (intra) and the variability betweenannotators (inter) are current challenges. In clinical practiceor biology, instance segmentation is a common task, but acomprehensive and quantitative study regarding the impactof noisy annotations lacks. In this paper, we present a con-cept to categorize and simulate various types of annotationnoise as well as an evaluation of the impact on deep learningpipelines. Thereby, we use the multi-organ histology imagedataset MoNuSeg to discuss the influence of annotator vari-ability. We provide annotation recommendations for cliniciansto achieve high-quality automated diagnostic algorithms.Keywords: Instance Segmentation, Annotator Variability,Deep Learning, Image Processing1 IntroductionDeep Neural Networks (DNNs) achieve high-quality resultsin complex image processing tasks, i.e., cell segmentation inbiology or medicine [1, 4]. Though, supervised DNNs requireannotated images by experts such as pathologists or biologists.Noisy annotations, i.e., partially false or wrong annota-tions are an open problem. In general, annotation noise canbe divided into variability within an annotator (intra) and be-tween different annotators (inter) [3]. There are various rea-sons for annotation noise like different levels of knowledge orexperience of annotators, fatigue, limited time budget, varyingdecision boundaries, or different hardware devices used duringannotation. In clinical practice, division of labor is common toreduce the effort for individual annotators. However, annota-tion noise in terms of consistency issues may affect the dataset.Karimi et al. [3] give a general overview w.r.t. noisy an-notation in image processing. Various methods to handle noisyannotations, i.e., detecting and correcting or reducing the im-*Corresponding author: Marcel P. Schilling, Institute for Au-tomation and Applied Informatics, Karlsruhe Institute of Tech-nology, Hermann-von-Helmholtz-Platz 1, 76344 Eggenstein-Leopoldshafen, Germany, e-mail: marcel.schilling@kit.edupact during training of DNNs, are discussed. In addition, theimpact of noisy annotation for classification [6] or semanticsegmentation [8, 9] is investigated. Löffler et al. [5] exam-ine the impact of wrong segmentation results w.r.t. the perfor-mance of tracking algorithms. Following the arguments in [3],biomedical problems, often consisting of only a small numberof samples, are prone to noisy annotations. Northcutt et al. [6]argue that noisy annotations are disadvantageous since therecould be negative effects on the DNN training or metrics maybecome unreliable yielding perhaps a wrong model selection.However, less consideration is given to instance segmen-tation, although, this problem often occurs in biomedical DLapplications. A concept for a comprehensive study w.r.t. an-notation noise in the case of instance segmentation is miss-ing. Recommendations for experts regarding annotation noise,what to pay special attention to during annotation to get ahigh quality DNN, can be very useful. Further, for data sci-entists, the question of whether more accurate annotations canimprove DNN performance remains open.Hence, we propose a detailed categorization of noisy an-notation in instance segmentation tasks utilizing a syntheticdataset. Further, we present methods for simulating intra-and inter-annotator variability. Our developed concept is com-pleted by examining the influence of annotator variability onDL pipelines. Besides, we implement the concept and con-tribute an evaluation of annotator variability w.r.t. the multi-organ histology image dataset MoNuSeg [4] dealing with nu-cleus instance segmentation. Hence, we provide recommenda-tions for researchers, i.e. clinicians or biologists, what is nededto pay special attention to obtain high-performing DNNs.2 Material and Methods2.1 Categorization of Annotation Noisein Instance SegmentationFirst, we comprehensively categorize annotation noise in in-stance segmentation using a synthetic dataset in combinationwith various corruptions as shown in Figure 1. Our work buildson the ideas concerning annotation noise in semantic segmen-tation presented in [8].Annotators may draw "oversized" or "undersized" masks.The corruption "contour" occurs in contour-based annotationcdbme_2022_8_2.pdf   197 8/29/2022   5:45:42 PM197DE GRUYTER Current Directions in Biomedical Engineering 2022;8(2): 197-200Open Access. © 2022 The Author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License.M.P. Schilling et al., Impact of Annotation Noise on Histopathology Nucleus Segmentation(a) Sample (b) Ground Truth (c) Oversized (d) Undersized (e) Contour (f) Merged(g) Split (h) Jitter (i) Holes (j) Approximation (k) ForgottenFig. 1: Synthetic Dataset and Corruptions. A sample (a) and corresponding ground truth annotation (b) are given. Different instancesare encoded by different colors. Corruptions are depicted in (c)-(k) using white line patterns to visualize affected instances.tools with auto-filling in the case where no loop closure isformed. "Merging" neighboring instances or "splitting" an in-stance into two segments are further noise types. In addition,small "jitter" instances or "holes" within an instance can beenumerated as noise in practical annotation procedures. Ne-glecting the exact shape of an instance, the corruption "approx-imation" transforms segments into ellipses. Moreover, "forget-ting" to annotate instances altogether can also be referred to asannotation noise type.In general, all those types of annotation noise could occurrandomly in terms of intra-annotator variability. However, tak-ing inter-annotator variability into account, systematic errorsmatter which applies only to the subset { "oversized", "under-sized", "contour", "approximation" } of corruptions.2.2 Concept for Simulating andEvaluating Annotation NoiseFigure 2 presents an overview of our proposal to investigateannotation noise in instance segmentation tasks. We assumean initially clean dataset D with no noise.In the case of intra-annotator variability, corruptions areadded yielding the noisy dataset D̃intra. The parameter β con-trols the ratio of affected instances per images, e.g. β = 0.2 inthe synthetic dataset (cf. Figure 1) since two of ten instancesare corrupted.In terms of inter-annotator variability, we randomly splitthe dataset into two subsets AD and BD to simulate differentannotators in the division of labor case. Different corruptionsare added through which AD̃ and BD̃ result. It needs to be re-Fig. 2: Concept Overview. An initially clean dataset D is corruptedi) controlled by the instance corruption ratio β (intra-annotator vari-ability) or ii) by randomly splitting, inserting two different corruptiontypes, and merging (inter-annotator variability) leading to noisydatasets D̃intra/inter. DNNs are trained on those noisy datasets andevaluated on a clean test dataset Dtest taking the performancemeasure AJI+ into account.marked that all instances are affected by the corruption in thisscenario (β = 1). The reason for this is the consideration ofsystematic annotation noise. After merging the noisy dataset,D̃inter is obtained.Subsequently, DNNs are trained using noisy datasets. Weconsider a clean test dataset Dtest to examine the impact ofnoisy annotated datasets on the DNN performance. The met-ric Aggregated Jaccard Index AJI+ based on the work in [2]is considered to evaluate both segmentation and detection per-formance of the DNN.cdbme_2022_8_2.pdf   198 8/29/2022   5:45:42 PM198M.P. Schilling et al., Impact of Annotation Noise on Histopathology Nucleus SegmentationFig. 3: MoNuSeg Dataset. Two crops including ground truth in-stances (contours, instances encoded by different colors) aredepicted [4]. The crops originate from different organs.3 Results3.1 DatasetWe use our proposed concept in combination with theMoNuSeg dataset [4] to investigate the quantitative impact ofannotation noise in instance segmentation tasks (cf.Figure 3).The dataset addresses the instance segmentation of cell nu-clei in diverse tissue images. The training dataset is com-posed of 30 hematoxylin and eosin histology images (1000pxx 1000px) of seven different organs including 21.623 nuclei.The test dataset covers 14 images.3.2 Architecture, Training, andImplementationWe use a U-Net [7] to predict Euclidean distance maps withsubsequent seed-based watershed post-processing for seg-menting instances. Smooth L1 loss serves as an objectivefunction for the Adam optimizer. Early stopping and learn-ing rate scheduling are considered. The used random train-validation split ratio is 80/20. For the training, we generate256px x 256px crops of the original images using a slidingwindow. Min-max-normalization to a range [0,1] is applied tothe raw data. Data augmentations (flipping, random crop & re-size, brightness/contrast adjustment, rotate/shift/scale, Gaus-sian noise/blur) are used to extend the training dataset. Hyper-parameters are obtained using random search.The DNN is implemented in PyTorch Lightning. The im-age processing libraries OpenCV and scikit-image are used fornoise simulation. DNN training is performed on multiple clus-ter nodes equipped with Intel Xeon Platinum 8368 CPU andfour NVIDIA A100 Tensor Core GPU, respectively.To avoid initialization effects, we repeat the experimentsfive times using various random seeds. Mean metrics includingspread in form of standard deviation are shown.30 40 50 60 70 80 90 100OversizedUndersizedContourMergeSplitJitterHolesApproximationForget63.2663.6463.4663.5765.4864.8165.1964.7963.958.8862.1258.863.9161.963.4465.3765.7661.7146.6453.8236.3464.1364.9864.7364.7765.1239.61AJI+ in %Corruptionsβ = 0.10β = 0.25β = 0.50Fig. 4: Impact of Intra-Annotator Variability: Mean AJI+ (standarddeviation visualized with black bars) w.r.t. Dtest is given for differentnoisy datasets D̃intra and corruption rates β. The baseline on theclean dataset D (AJI+ = 64.61±1.19 %) is depicted in gray.3.3 Impact on Deep LearningIntra-Annotator VariabilityFigure 4 shows the impact of the introduced corruptions fordifferent corruption rates β = {0.10, 0.25, 0.50}. The corrup-tions of "approximation", "jitter", and "holes" lead to no per-formance degradation and even in some cases to performanceimprovements. The AJI+ in the case of "split" or "merge" re-mains mostly in the baseline performance interval apart froma single outlier. However, the corruptions "forget", "contour","undersized", and "oversized" show strong performance re-duction which is related to the instance corruption ratio β. Asthis is a preliminary study, additional t-tests are not performed.Inter-Annotator VariabilitySimulating the division of labor case, the results of investi-gations in the context of inter-annotator variability are shownin Table 1. The combination of "approximation" corruptionand no annotation noise leads to ≈ 0.5% performance boostw.r.t. AJI+. However, all other combinations of corruptionsyield a strong decreasing AJI+. In addition, the case of a par-tially clean dataset ("none") shows better performance mea-sures compared to corrupting both parts of the dataset.cdbme_2022_8_2.pdf   199 8/29/2022   5:45:42 PM199REFERENCESTab. 1: Impact of Inter-Annotator Variability. Mean AJI+ (± stan-dard deviation) w.r.t. Dtest is compared to noisy datasets D̃inter.CorruptionsAJI+ in %Annotator A Annotator BNone None (Baseline) 64.61 ± 1.19Oversized Undersized 43.59 ± 2.10Oversized Contour 33.15 ± 4.26Oversized Approximation 45.37 ± 3.00Oversized None 45.51 ± 3.08Undersized Contour 5.53 ± 2.52Undersized Approximation 53.14 ± 3.83Undersized None 55.23 ± 3.68Contour Approximation 40.16 ± 13.78Contour None 32.35 ± 7.61Approximation None 65.08 ± 0.564 DiscussionTaking intra-annotator variability into account, the main mes-sage is that annotation noise does not generally lead to de-creasing performance in instance segmentation. The most crit-ical types of noise are forgetting instances, undersized or over-sized instances, and contour corruption. Against expectation,merging/splitting instances seems no major issue if correct an-notations dominate the total dataset. In addition, there might beperformance improvements in the case of some minor corrup-tions, i.e., jitter, approximating the shape, or holes. On closerinspection, this appears to be a kind of annotation augmen-tation that may explain the better generalization of the DNN.Besides, this also means for experts that this kind of annotationnoise poses no problem for the DNN training.The results of inter-annotator variability show that differ-ent annotation styles impede the DNN training in mostly everycase. Hence, in the division of labor cases, a consistent annota-tion policy between all annotators is of great significance. Lessperformance degradation in cases of only a partially corrupteddataset makes sense since the training can profit from partiallycorrect labels. It should be noted, however, that contour cor-ruption in the case of variability between annotators will notoccur in practice to the extent simulated.5 ConclusionAnnotator variability is already examined in classificationand semantic segmentation. We categorized different formsof annotator variability in instance segmentation for the firsttime and proposed a concept to investigate its impact onDL pipelines. Minor annotation noise of a single annotatorleads to no quantifiable performance drop. Hence, the ac-curacy requirements for annotators can be relaxed in thiscase. Nevertheless, the corruptions of forgetting instances, un-dersized/oversized instances, and missing filling of instancesshould be avoided to achieve high-quality DNNs. We demon-strated that different annotation styles are a major issue w.r.t.DNN performance. The agreement of annotation policies isnecessary in division labor cases. Future work are investiga-tions w.r.t. automated detection of noisy annotations and thepotential of annotation augmentation to improve DL perfor-mance.Acknowledgment: This work was supported in by theHoreKa Supercomputer through the Ministry of Science,Research, and the Arts Baden-Württemberg and by theHelmholtz Association Initiative and Networking Fund on theHAICORE@KIT partition.Author StatementResearch funding: This work was funded by the KIT FutureFields II Project "Screening Platform for Personalized Oncol-ogy". Conflict of interest: Authors state no conflict of interest.References[1] Juan C. Caicedo et al. “Nucleus Segmentation across Imag-ing Experiments: The 2018 Data Science Bowl.” In: NatureMethods 16.12 (2019), pp. 1247–1253.[2] Simon Graham et al. “Hover-Net: Simultaneous Segmen-tation and Classification of Nuclei in Multi-Tissue HistologyImages.” In: Medical Image Analysis (2019), p. 101563.[3] Davood Karimi et al. “Deep Learning with Noisy Labels: Ex-ploring Techniques and Remedies in Medical Image Analy-sis.” In: Medical Image Analysis 65.5 (2020), p. 101759.[4] Neeraj Kumar et al. “A Multi-Organ Nucleus SegmentationChallenge.” In: IEEE Transactions on Medical Imaging 39.5(2020), pp. 1380–1391.[5] Katharina Löffler et al. “A Graph-Based Cell Tracking Algo-rithm with Few Manually Tunable Parameters and AutomatedSegmentation Error Correction.” In: PLOS ONE 16.9 (2021),pp. 1–28.[6] Curtis Northcutt et al. “Pervasive Label Errors in Test SetsDestabilize Machine Learning Benchmarks.” In: Neural Infor-mation Processing Systems Track on Datasets and Bench-marks. Vol. 1. 2021.[7] Olaf Ronneberger et al. “U-Net: Convolutional Networks forBiomedical Image Segmentation.” In: Medical Image Com-puting and Computer-Assisted Intervention. Vol. 9351. 2015,pp. 234–241.[8] Marcel P. Schilling et al. “Automated Annotator VariabilityInspection for Biomedical Image Segmentation.” In: IEEEAccess 10 (2022), pp. 2753–2765.[9] Shaode Yu et al. “Robustness Study of Noisy Annotationin Deep Learning Based Medical Image Segmentation.” In:Physics in Medicine & Biology 65.17 (2020), p. 175007.cdbme_2022_8_2.pdf   200 8/29/2022   5:45:42 PM200

Impact of Annotation Noise on Histopathology Nucleus Segmentation

https://publikationen.bibliothek.kit.edu/1000150405/149254045

Impact of Annotation Noise on Histopathology Nucleus Segmentation

Abstract

Similar works

Full text

Available Versions

KITopen