Uncertainty categories in medical image segmentation: a study of source-related diversity

Anatomical segmentations of magnetic resonance imaging (MRI) scans are commonly used to support a number of clinical and research tasks using either manual or automated methods. While optimising overall measures of success (e.g., Dice/F1 scores) is important for any segmentation task, the uncertainties associated with image segmentations have become a salient field of enquiry for researchers; to (i) improve the quality and interpretability of structural delineations, and (ii) improve trust when applying automated techniques to clinical practice [1]. Whilst it is not uncommon in the literature to implicitly consider a single uncertainty distribution, different sources for uncertainties exist (e.g., epistemic and aleatoric). Therefore, a thorough treatment of uncertainties arising from different sources is needed to understand their usefulness, distinctiveness and stability.

Researchers have investigated uncertainties for some time but, more recently, specific epistemic and aleatoric sources of uncertainty have been addressed for deep convolutional networks. Here, two complementary estimation paradigms have emerged; namely, test-time dropout (TTD) for model or epistemic uncertainty and test-time augmentation (TTA) for data or aleatoric uncertainty. The proposed use of Monte Carlo dropout at test-time by Gal and Ghahramani [2] overcomes the computational problems of traditional Bayesian approaches, although other approximate models of uncertainty, including Monte Carlo batch normalisation [3] and Markov Chain Monte Carlo methods [4] have also been proposed. The use of TTD, however, has become more widespread amongst researchers than these other methods. To measure aleatoric uncertainties, Ayhan and Berens proposed the use of TTA [5], with Wang and colleagues proposing a systematic approach to assess aleatoric uncertainties in medical imaging [6]. Building upon these paradigms, much research attention has now turned to the incorporation of uncertainties to improve model performance [7, 8, 9, 10].

As the use of uncertainties in imaging pipelines is becoming increasingly common, it is important to perform explorations around which sources of uncertainty are the most beneficial, the most stable and reliable, and the most interpretable. A key question related to this is whether a single uncertainty source and distribution is sufficient for a general use case, or whether multiple sources and distributions should be used. Jungo and colleagues [11, 12] have addressed issues around reliability and model confidence of epistemic uncertainties for some TTD model architecture strategies and learned aleatoric uncertainty ($\sigma^{2}$); where $\sigma^{2}$ is determined using a method proposed by Kendall and Gal [13]. While these and other works (e.g., [14, 15]) address uncertainty reliability and model calibration generally, they do not consider different levels of TTD parameters or how aleatoric uncertainties from different underlying sources manifest in observations. Additionally, further important work on modelling uncertainty related to annotator variation has emerged. This utilises latent space encoding techniques derived from the Variational Autoencoder framework [16] to directly model segmentation distributions using a Bayesian approach [17, 18]. These methods provide a mathematically rigorous model for capturing complex voxel dependencies, and though they are currently designed to produce one distribution, they could be modified to capture a number of distinct uncertainty sources, in order to report the most appropriate information for each individual use case.

We assess the range and stability of uncertainty measures for epistemic and aleatoric categories by considering various TTD and TTA parameter settings. The parameter space for assessing epistemic uncertainty with TTD is typically less complex than that for aleatoric categories using TTA. While there are different approaches for how dropout is applied structurally to a model, once this is determined, the key variables are the probabilities of the applied dropout. In contrast, TTA has both a number of augmentation types that can be applied (some particularly relevant to MRI), and a number of parameter settings for each type of augmentation. In addition, as is done in model calibration analyses (e.g., [14, 15]), we have estimated a prediction error likelihood for the underlying segmentation model, to test how similar it is to other uncertainty types.

For brevity, we report results only for tumor core tissue using voxelwise entropy values as our measure of uncertainty. These results generalise to using the variance as the uncertainty measure, as well as to other tumor tissue categories. The U-net performance (Dice=0.74) is not the focus but is comparable with the mean of BraTS 2020 challenge submissions.

It is clear from these results that uncertainty distributions for epistemic (TTD) and aleatoric (TTA) sources are distinctly different from each other and that the parameter types and values within them also have a substantial impact on the uncertainty distributions. This indicates that more than one category of uncertainty should be modeled and estimated for a variety of use cases, such as to (i) aid interpretability, (ii) capture a diversity of uncertainty sources, (iii) indicate reliability, or (iv) incorporate into uncertainty-aware networks. Although reported results only used entropy, these conclusions also hold for variance and other tissue classes. We chose to show entropy values as they are better for distributions that are not unimodal, which were likely to be produced. In this early work we chose not to include all uncertainty sources (e.g., inter-rater variability) or compare to further techniques such as ensembles or Bayesian-based network architectures, but rather demonstrate the diversity of uncertainties generated by two common methods - especially accounting for MR artefacts with TTA.

In some instances, the interquartile range approached the full range of possible entropy values. This shows that the parameters used for TTD or TTA uncertainty estimation are important. For more extreme dropout rates the TTD uncertainty estimates are not useful for any purpose and so researchers should err on the side of using reduced dropout probability settings. Furthermore, the calculated uncertainties may gradually change from being useful to non-informative, based on the gradual changes in correlations found at lower dropout values, although further testing would be required to establish this generally.

Aleatoric uncertainties spanned a range of distinctly different maps, as shown by the correlation matrices, indicating that a number of different data augmentation types need to be included to capture aleatoric uncertainty. The augmentation parameters also clearly have an effect, and when pushed to higher values can sometimes result in widespread low-grade uncertainties across images. This could be used to establish a practical upper-bound for these parameter settings.

Careful consideration of different parameter settings is needed when using TTD and TTA estimation methods, and researchers should consider that there is no single uncertainty map, but rather a set of different distributions governed by the TTD and TTA hyper-parameter values and types. In order to have reliable, repeatable, interpretable measures of uncertainty it is important to specify the types of augmentation and all hyper-parameters. When using uncertainties with uncertainty-aware networks, we expect incorporating maps from multiple categories to be beneficial. Finally, while a predicted error map may share similarities with some uncertainty cases, they measure at most one aspect of uncertainty and possibly something different again, and are thus not a replacement for a set of maps from different uncertainty categories. Researchers should carefully consider parameter settings for each method, as figure 3 demonstrates how larger parameter settings can produce unhelpful, unrepresentative uncertainties, indicating that upper-bounds should be established in each case (e.g., TTA cases: bias-field and combined).

In this study one major limitation is the size and nature of the dataset. BraTS is a well studied and characterised dataset, which contains several different segmentation tasks, and thus allowed us to verify that the findings generalised to different tissue types and also different uncertainty measures. However, to establish these results more generally, further studies are required.

In conclusion, we have shown that there are multiple sources of uncertainties that generate distinctly different uncertainty maps (see figure 2), and that these should be incorporated into all uncertainty work. The epistemic and aleatoric uncertainty estimates, as obtained through TTD and TTA respectively, are sensitive to their hyper-parameter settings and to obtain repeatable maps it is important to specify the hyper-parameters involved. These settings need to be thought through carefully when building and running networks that produce uncertainty estimates. Furthermore, we found that there is a richness in the set of uncertainty estimations that is not captured by a single distribution, and this should be considered when feeding maps into uncertainty-aware networks, or when using them to aid interpretability or indicate reliability. Therefore, although a chosen distribution or summary may capture uncertainties to some degree, it is unlikely to provide comprehensive estimates across all regions where uncertainties could manifest; which could be vital to ensuring clinical efficacy in challenging situations.