Medical segmentation models are evaluated empirically. As such an evaluation
is based on a limited set of example images, it is unavoidably noisy. Beyond a
mean performance measure, reporting confidence intervals is thus crucial.
However, this is rarely done in medical image segmentation. The width of the
confidence interval depends on the test set size and on the spread of the
performance measure (its standard-deviation across of the test set). For
classification, many test images are needed to avoid wide confidence intervals.
Segmentation, however, has not been studied, and it differs by the amount of
information brought by a given test image. In this paper, we study the typical
confidence intervals in medical image segmentation. We carry experiments on 3D
image segmentation using the standard nnU-net framework, two datasets from the
Medical Decathlon challenge and two performance measures: the Dice accuracy and
the Hausdorff distance. We show that the parametric confidence intervals are
reasonable approximations of the bootstrap estimates for varying test set sizes
and spread of the performance metric. Importantly, we show that the test size
needed to achieve a given precision is often much lower than for classification
tasks. Typically, a 1% wide confidence interval requires about 100-200 test
samples when the spread is low (standard-deviation around 3%). More difficult
segmentation tasks may lead to higher spreads and require over 1000 samples.Comment: 10 page