Confidence Contours: Uncertainty-Aware Annotation for Medical
Semantic Segmentation

We recruited 30 undergraduate students majoring in the biological sciences to each annotate 40 images with CCs and 40 images with the singular method as a baseline. Every image was annotated by 3 different annotators, producing 3 CC and 3 singular annotations. Annotators were briefed and evaluated on the relevant radiological and medical background before annotating. We received IRB approval for this study under ID 16518. While our annotators were students, we consider them expert annotators as the quality of our annotations generally matched that of the LIDC dataset produced by radiologists. There was no statistically significant difference between the level of disagreement comparing our annotations and LIDC annotations versus comparing LIDC annotations with each other. This similarity in skill level may be explained by our task only being the simpler segmentation rather than the more challenging localization task.

The same study procedure was used for the FoggyBlob dataset with 15 students from general areas of study.

To counteract bias from learning effects, we counterbalanced the order of annotation method (singular and Confidence Contours) for annotation studies on both datasets. All participants were compensated $\$17$ per hour, higher than the local minimum wage at the time of study. Of 45 total annotators, 69% identify as female and 31% identify as male. In total, we collected 3,600 annotations across 600 images.¹¹1See https://github.com/andre-ye/confcontours-data for annotation dataset.

After annotating the assigned image set for each annotation method, annotators completed a NASA TLX questionnaire on task load (Hart and Staveland 1988). The survey recorded the following dimensions on a 10-point Likert scale: mental demand, physical demand, temporal demand, performance, effort, and frustration. We also measured the time annotators spent per annotation through logging in our annotation tool.

Annotators generally report that producing singular annotations requires lower overall load than producing CCs (Table 1). This is expected because creating CCs requires physically more user input than the singular method. Notably, however, annotators did not experience statistically significant differences in self-perceived performance level across both tasks ($p<0.05$), suggesting that CC annotation is learnable. Moreover, there are no statistically significant differences between the two annotation methods except in the mental demand and frustration dimensions for FoggyBlob. This suggests that the significance of task-load differences may be dependent on the complexity of the context. Last, the absolute magnitude of the differences between singular and CC annotations are at most 1.3 points out of a 10-point scale in all dimensions.

We measure the average time taken to produce CCs versus singular annotations after the first 20 annotations, to account for the period where the annotator is learning the tool and task. On average, annotators spent 27 sec to create a singular annotation and 44 sec to create a CC annotation per image in LIDC. For FoggyBlob, the time spent was 25 sec and 45 sec, respectively. Across all annotators, the annotation time for each sample using CCs is 68.50% and 75.32% more than using the singular method for the LIDC and FoggyBlob datasets, respectively. Thus, having one annotator create a CC annotation is faster than having at least two annotators each create a singular annotation and then use their disagreement to infer uncertainty.

As another dimension of representative capacity, we evaluate how consistently annotators create the upper and lower bounds. Given that annotator disagreement has been shown to have strong relationships with sources of structural visual uncertainty, we use disagreement as a proxy for CCs’ ability to represent uncertainty. We compute the disagreement between a set of annotations as the average pairwise discrete Frechét distance, a common measure of curve similarity (Wylie 2013). In choosing a particular measurement, we are primarily concerned with the annotator behavior on the inclusion or exclusion of particular ambiguous structures along the surface of the curves. Area-based IoU measurements inflate the role of unambiguous regions of high agreement, so we use Frechét distance instead. Across both datasets, we observe a statistically significant reduction in disagreement between min contours as opposed to singular annotations (Table 2). For LIDC, we find no statistically significant difference between disagreement between max contours and singular annotations, whereas we do for FoggyBlob. This is possibly due to the synthetic nature of the FoggyBlob dataset. On the other hand, in real-world datasets, different annotators may disagree on whether an ambiguous structure should be included into the max contour. In general, we find that annotators tend to agree more on regions of high confidence than regions of low confidence.

One function of Confidence Contours (CCs) is as a means for one annotation to encompass the range of multiple singular annotation responses we might otherwise observe across a group of different annotators. Existing work (Cheplygina and Pluim 2018) shows that disagreements among annotators are often centered in regions of visual ambiguity, which cast uncertainty on the identification of relevant structures in the image. CC annotations should similarly be drawn to accommodate the same sources of uncertainty. Additionally, in other annotation modalities, it has been shown that single annotators can often anticipate the distribution of responses of their peers (Chung et al. 2019).

We can view any segmentation representation $s$ as a partition of an image into three types of points: $s^{+}$—points certainly associated with the subject-of-interest, $s^{-}$—points certainly not associated with the subject-of-interest, and $s^{?}$—points that may or may not be associated with the subject-of-interest. Under this view, we can intuitively see that one segmentation $s_{a}$ bounds another $s_{b}$ if $s_{a}^{+}\subseteq s_{b}^{+}$ and $s_{a}^{-}\subseteq s_{b}^{-}$. Of course, in practice, segmentation representations are rarely expected to perfectly bound another. To understand representative capacity, we want to quantify the error at which a segmentation fails to bound another. To do this we define the following error metrics:

$$L^{+}(s_{a},s_{b})=|\{\forall p:p\in s_{a}^{+}\land p\notin s_{b}^{+}\}|$$

$$L^{-}(s_{a},s_{b})=|\{\forall p:p\in s_{a}^{-}\land p\notin s_{b}^{-}\}|$$

Intuitively, $L^{+}$ measures the degree to which $s_{a}^{+}$ fails to be a subset of $s_{b}^{+}$ (error in $s_{a}$ serving as a lower bound for $s_{b}$, or put simply, “underflow”) and $L^{-}$ measures the degree to which $s_{a}^{-}$ fails to be a subset of $s_{b}^{-}$ (error in $s_{a}$ serving as an upper bound for $s_{b}$, “overflow”). Together, they holistically represent error to which $s_{a}$ bounds $s_{b}$.

Of course, more practically, we would like to quantify the representative capacity of a new segmentation representation $s_{a}$ in relation to a reference set of segmentations $S$. This can be done by computing the expected error²²2We use $\neg s^{-}$ to denote $s^{?}\cup s^{+}$.:

$$\mathbb{L}^{+}(s_{a},S)=\mathbb{E}_{s\in S}[L^{+}(s_{a},s)/|s_{a}^{+}\cup s^{+}|]$$

$$\mathbb{L}^{-}(s_{a},S)=\mathbb{E}_{s\in S}[L^{-}(s_{a},s)/|\neg s_{a}^{-}\cup\neg s^{-}|]$$

Note that we introduce a normalization factor in the above expressions to account for the different sizes of $s^{+}$; this also bounds both metrics between 0 and 1 (inclusive).

We calculate the representative capacity of CCs ($\mathbb{L}^{+}_{\text{CC}}$ and $\mathbb{L}^{-}_{\text{CC}}$), using our singular annotations as a reference set. As a baseline, we compute the representative capacity of singular annotations ($\mathbb{L}^{+}_{\text{base}}$ and $\mathbb{L}^{-}_{\text{base}}$) against the same reference set. The data for evaluating singular annotations is constructed by taking a held-out annotation that was not used in the reference set.

We find that, compared to the baseline, CCs demonstrate more representative capacity, as evidenced by statistically significantly lower underflow and overflow across both datasets (Table 3). This representative capacity is clearly visualized in Figure 4. We also found that, for LIDC and FoggyBlob respectively, 42.96% and 50.16% of instances had $\mathbb{L}^{+}\leq 0.05$ (a trivial level of error) and 40.20% and 45.45% of instances had $\mathbb{L}^{-}\leq 0.05$. In addition to these observations, we found that for individual instances $s_{a}$ experience less overflow than underflow ($\mathbb{L}^{-}(s_{a})\leq\mathbb{L}^{+}(s_{a})$) to a statistically significant degree ($p=0.0236<0.05$ for LIDC, $p=0.0158<0.05$ for FoggyBlob) for LIDC and FoggyBlob respectively.

Moreover, we explored whether the degree of uncertainty in CCs correlates with the uncertainty observed from disagreements in sets of singular annotations. For each image, this is calculated as $|s^{?}_{\text{CC}}|$ a single CC annotation and $\mathbb{E}[|\neg s^{-}_{\text{Base}}|]-\mathbb{E}[|s^{+}_{\text{Base}}(x)|]$ for an ensemble of singular annotations. We find that there is a statistically significant correlation between these metrics across both LIDC ($\rho=0.5868$, $p<0.001$) and FoggyBlob ($\rho=0.4343$, $p<0.001$).

These results suggest that $s^{?}_{\text{CC}}$ represents bounds on the range of singular annotations; that is, we would expect a singular annotation drawn by some annotator to fall within $s^{?}_{\text{CC}}$. Overall, the results suggest that a single CC annotation represents many uncertainty-relating structural properties across multiple singular annotations.

We also explored whether CCs can be learned effectively by a variety of downstream segmentation models. Conceptually, these models should predict both the min and the max contour masks simultaneously. We simply train models to predict a two-channel mask for convenience, although there are several alternative methods to do so, such as training two separate models. Therefore, any general segmentation model can be trivially modified to support CC labels. The optimization objective for a model $M$ using loss function $\mathcal{L}$ on CC-annotated data becomes:

$$\min_{M}[\mathcal{L}(M_{\text{min}}(x),y_{\text{min}})+\mathcal{L}(M_{\text{max}}(x),y_{\text{max}})]$$

We evaluated performance across four model architectures commonly used for medical segmentation: U-Net (Ronneberger, Fischer, and Brox 2015), attention U-Net (Oktay et al. 2018), DeepLab (Chen et al. 2016) with a MobileNetV2 backbone (Sandler et al. 2018), and PSPNet (Zhao et al. 2016). To account for variability in training outcomes, we perform grid search over hyperparameters for each architecture: for U-Net and attention U-Net we tested configurations of batch size $\in$ {8, 16, 32, 64}, initial filters $\in$ {8, 16, 32}, and encoder-decoder pathway block count $\in$ {2, 3, 4, 5}; for DeepLab – batch size $\in$ {8, 16, 32, 64}, filters $\in$ {8, 16, 32}; for PSPNet – batch size $\in$ {8, 16, 32, 64}, initial filters $\in$ {8, 16, 32}, and block count $\in$ {1, 2, 3}. Model architectures were scaled down to accommodate for our smaller dataset and image size. For each architecture and hyperparameter set, we train one instance on singular annotations and another on CC annotations until convergence. We use the Adam optimizer and the dice loss optimization objective across all instances. Following standard practice, reference masks are generated from singular annotations using 50% (majority) consensus; CC annotations are not aggregated (i.e., each image is associated with multiple labels). We employ an extensive augmentation pipeline, including affine transformations, blurring, and sharpening.

We investigate whether general segmentation models can successfully model CC annotations. To adapt an existing general segmentation model to learn CCs, we can map CC labels into two-channel masks. A model trained on these masks conceptually behaves as two segmentation models with heavily shared weights, which predict the min (‘min-subnetwork’) and the max (‘max-subnetwork’) contours separately. Each individual subnetwork is formally equivalent in structure to a general model trained on singular-type annotations. In our modeling experiments, we observe no statistically significant difference ($p>0.05$) between the converged loss of singular annotations and either the ‘min-subnetwork’ or the ‘max-subnetwork’, across all 144 ($48+48+12+36$) modeling trials. This shows that it is not more difficult for a wide range of general segmentation models to learn to predict CC labels than singular ones.

While we observe no performance losses when training on CCs or similar labels, we experience significant improvements in the interpretability of the uncertainty predictions. Figure 5 visualizes predictions from different models across multiple samples from LIDC. Rather than predicting singular annotations, which do not provide explicit information about uncertainty, models trained on CCs explicitly report areas of high and low annotator confidence. Moreover, as opposed to previously mentioned model-based uncertainty map generation methods such as Bayesian segmentation models which produce smooth, unthresholded maps, our models’ discrete predictions provide clearly interpretable uncertainty thresholds directly corresponding to human annotations. We note that while alternative approaches such as elicitation and candidate generation produce uncertainty representations through aggregating multiple singular predictions (a kind of “disagreement” uncertainty), using CCs reproduces uncertainty as assessed by a single annotator (a kind of “ambiguity” uncertainty).

We recruited five experts with backgrounds in medical imaging for thirty-minute interviews (Table 4). First, we asked the experts to look at annotations from 6–9 (depending on time) high-disagreement samples in the LIDC dataset, explain how they interpreted the disagreement, and show how they would have annotated the image. Second, we showed annotators 5 samples along with three model predictions—standard (singular binary mask), CC, and Bayesian—and the original image without additional context (similarly to what is shown in Figure 5). Experts were asked to interpret what the different masks meant and which they felt best corresponded to the original image. Last, we explained how each of the three model predictions were derived and asked experts to reflect on how they might use each of the models in practice, with the ability to flip through 15 additional samples. Each expert was compensated $50.

Throughout the analysis, we use ‘Bayesian’ when the expert is talking about a feature of the predictions which is dependent on the specific method of Bayesian segmentation, and ‘continuous maps’ when the expert is talking about continuity of uncertainty values, which is a feature of many alternative uncertainty representation methods. Therefore, statements on continuous maps can be generalized to all approaches which present uncertainty in a similarly unthresholded manner.

Experts noticed that the max contour of Confidence Contours annotations tended to pick up on higher-sensitivity information than singular annotations. In several cases, an ambiguous protrusion was connected to a high-certainty mass; while both the standard and the Bayesian predictions neglected the ambiguous area, the max contour of the CC prediction encapsulated it. P1 made the accurate observation that “the ground truth of the mask can only give you so much… this [singular annotation] is probably what the human labeler is labeling for [continuous annotations] anyway.” That is, Bayesian maps are a reformulation of the same data and model used to produce singular masks. P2, who was less trustful of models and almost always preferred to work with the images directly, said that they preferred Confidence Contours out of all three possible masks because it gave them the most information.

By ‘structural uncertainty,’ we mean that the uncertainty does not stem strictly from blurriness, stains, or other visual impediments, but rather concerns a concrete structure in the image. This structure may be uncertain in part because of visual impediments, but also due to its relationships with other structures (which requires domain knowledge to identify). Structural uncertainty is often demonstrated by discrete disagreement between singular annotations, in which annotators either include or exclude a particular structure with no ‘in between.’

Experts identified structural uncertainty in many LIDC samples and explained differences between singular annotations as experts following different reasoning schemes rather than perception-level differences (“she just happened to see X or adopt Y sensitivity level”). Across the experts, we did not observe consensus on the best annotations. Confidence Contours works well in these cases, because uncertainty is built into the structure of the problem and needs to be accounted for in measurements in a clear way. P5 argued that “[CCs] is going to be advantageous in situations where there are clustered nodules or a solid ground-glass nodule that has more of a halo which also needs to be measured.” Moreover, experts’ ability to rationalize other experts’ annotations provides further support that experts are able to create min and max contours which accurately bound the range of uncertainty.

However, when an image features low structural uncertainty, annotators are content with singular annotation masks because they provide all the information that is needed. P1 noted that “Just doing a binary segmentation will give you a very crude estimate, but it’s the most easy to just identify.” P5 concurs: “[its] benefit is in the simplicity.” Moreover, experts note that singular annotations suffice when the expert only needs to measure features in which disagreement from structural uncertainty tends not to matter, such as the rough location of a nodule.

In contrast, experts stated that the unthresholded nature of continuous maps makes it difficult to efficiently and reliably make judgements. This aligns with the challenges expressed in the literature discussed earlier. P1 commented: “The problem with [continuous maps] if I were looking at it just with my eye is that it’s really difficult to tell the certainty level… it would be nice to have some range or threshold”. On the other hand, with Confidence Contours, “You can get a really clean outline of the node, and you also have some idea of the uncertainty.” P4 remarked that “[continuous maps] could be helpful, but you have to check the color scales, look at the numbers, so it would be slower… if I had to pick one [segmentation map], it is probably [CCs].” P4 also noted later that calibrating color scales across samples and ensuring that they are comparable would be another difficulty of practically using continuous maps. P5 observed, “It’s cool that [continuous maps] are more of a continuous metric of what is in the nodule and what is not in the nodule, but for conventional clinical reporting, it’s difficult to know where to draw the line, which makes it kind of hard to use.” On the other hand, the double-thresholded nature of CCs often more intuitively captures the distribution of uncertainty in such problems.

Experts also noted that continuous maps can obscure the shape of the region of interest, as the more ambiguous but significant edges may be decayed. P5 brought up that the Bayesian approach makes it less easy to see what is going on, noting that “[Singular annotations] and [Confidence Contours] do a pretty good job of showing—here is a shape, and here is the more solid area in the shape… so I probably prefer [singular annotations] and [CCs] over [continuous masks].” P3, who was very focused on the shapes of the nodes, disliked continuous maps because it does not clearly present this important discretized feature. On the other hand, the thresholded nature of both singular and CCs annotations enables a more definite presentation of the nodule shape.

Some experts found the two-level sensitivity representation method more ‘natural’ when comparing it to their own process of interpretation of an image. P5 mentioned that “you may get into situations where you report two separate measurements, a solid measurement and a ground glass measurement. [CCs] makes it easier to automate that.” We further explore this in the Discussion section (“Greater representation of uncertainty range”).

P5 also commented that “[CCs] is easier to understand because I feel like the [min] contour is something which is more reliable that you can fall back on, and you can use the [max] contour if it makes sense to, given the situation. But you don’t get those two channels of information in [continuous maps].” That is, when making a judgement, one can begin from the min contour and move outwards as necessary. For instance, P3—who we observed tended to draw very strict boundaries at low sensitivity—said that CCs segmentations most reflected the information in the original image because it gave them the choice to focus on the min contour and to ignore the max contour. In this way, the expert has more control over their judgement of the image due to the intepretability of the contours, whereas in singular annotations they are given no alternatives and in continuous maps little guidance.

Indeed, continuous maps can lead experts without a background in AI astray. We noticed that P3, who was not familiar with AI, misinterpreted continuous maps as corresponding to intensity levels (brightness) in the original image. While for lung nodule segmentation, intensity does correlate well with certainty, they are different concepts: notably, a pixel/voxel can have low intensity but high certainty, due to its spatial relations to other structures. Experts familiar with AI were able to speak about these two concepts separately.