46 research outputs found
Crowdsourcing Airway Annotations in Chest Computed Tomography Images
Measuring airways in chest computed tomography (CT) scans is important for
characterizing diseases such as cystic fibrosis, yet very time-consuming to
perform manually. Machine learning algorithms offer an alternative, but need
large sets of annotated scans for good performance. We investigate whether
crowdsourcing can be used to gather airway annotations. We generate image
slices at known locations of airways in 24 subjects and request the crowd
workers to outline the airway lumen and airway wall. After combining multiple
crowd workers, we compare the measurements to those made by the experts in the
original scans. Similar to our preliminary study, a large portion of the
annotations were excluded, possibly due to workers misunderstanding the
instructions. After excluding such annotations, moderate to strong correlations
with the expert can be observed, although these correlations are slightly lower
than inter-expert correlations. Furthermore, the results across subjects in
this study are quite variable. Although the crowd has potential in annotating
airways, further development is needed for it to be robust enough for gathering
annotations in practice. For reproducibility, data and code are available
online: \url{http://github.com/adriapr/crowdairway.git}
Expected exponential loss for gaze-based video and volume ground truth annotation
Many recent machine learning approaches used in medical imaging are highly
reliant on large amounts of image and ground truth data. In the context of
object segmentation, pixel-wise annotations are extremely expensive to collect,
especially in video and 3D volumes. To reduce this annotation burden, we
propose a novel framework to allow annotators to simply observe the object to
segment and record where they have looked at with a \$200 eye gaze tracker. Our
method then estimates pixel-wise probabilities for the presence of the object
throughout the sequence from which we train a classifier in semi-supervised
setting using a novel Expected Exponential loss function. We show that our
framework provides superior performances on a wide range of medical image
settings compared to existing strategies and that our method can be combined
with current crowd-sourcing paradigms as well.Comment: 9 pages, 5 figues, MICCAI 2017 - LABELS Worksho
Crowd disagreement about medical images is informative
Classifiers for medical image analysis are often trained with a single
consensus label, based on combining labels given by experts or crowds. However,
disagreement between annotators may be informative, and thus removing it may
not be the best strategy. As a proof of concept, we predict whether a skin
lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd
annotations of visual characteristics of that lesion. We compare using the mean
annotations, illustrating consensus, to standard deviations and other
distribution moments, illustrating disagreement. We show that the mean
annotations perform best, but that the disagreement measures are still
informative. We also make the crowd annotations used in this paper available at
\url{https://figshare.com/s/5cbbce14647b66286544}.Comment: Accepted for publication at MICCAI LABELS 201
Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis
Neural networks promise to bring robust, quantitative analysis to medical
fields, but adoption is limited by the technicalities of training these
networks. To address this translation gap between medical researchers and
neural networks in the field of pathology, we have created an intuitive
interface which utilizes the commonly used whole slide image (WSI) viewer,
Aperio ImageScope (Leica Biosystems Imaging, Inc.), for the annotation and
display of neural network predictions on WSIs. Leveraging this, we propose the
use of a human-in-the-loop strategy to reduce the burden of WSI annotation. We
track network performance improvements as a function of iteration and quantify
the use of this pipeline for the segmentation of renal histologic findings on
WSIs. More specifically, we present network performance when applied to
segmentation of renal micro compartments, and demonstrate multi-class
segmentation in human and mouse renal tissue slides. Finally, to show the
adaptability of this technique to other medical imaging fields, we demonstrate
its ability to iteratively segment human prostate glands from radiology imaging
data.Comment: 15 pages, 7 figures, 2 supplemental figures (on the last page
Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis
Neural networks promise to bring robust, quantitative analysis to medical
fields, but adoption is limited by the technicalities of training these
networks. To address this translation gap between medical researchers and
neural networks in the field of pathology, we have created an intuitive
interface which utilizes the commonly used whole slide image (WSI) viewer,
Aperio ImageScope (Leica Biosystems Imaging, Inc.), for the annotation and
display of neural network predictions on WSIs. Leveraging this, we propose the
use of a human-in-the-loop strategy to reduce the burden of WSI annotation. We
track network performance improvements as a function of iteration and quantify
the use of this pipeline for the segmentation of renal histologic findings on
WSIs. More specifically, we present network performance when applied to
segmentation of renal micro compartments, and demonstrate multi-class
segmentation in human and mouse renal tissue slides. Finally, to show the
adaptability of this technique to other medical imaging fields, we demonstrate
its ability to iteratively segment human prostate glands from radiology imaging
data.Comment: 15 pages, 7 figures, 2 supplemental figures (on the last page