Towards Reliable Medical Image Segmentation by utilizing Evidential Calibrated Uncertainty

In last years, most methods used for medical image segmentation were based on Convolutional Neural Networks (CNNs). Attributed to the elegant design of skip connections in U-Net [7, 15], a similar structure V-Net was presented for 3D image segmentation based on a volumetric. Then, other U-Net variants [53, 36] focused on getting better feature representations about the targets. The representative Attention-UNet employed attention gates in the skip connection to increasing the model sensitivity for varying shapes and sizes of organs/tumors. Zhou et al. [55] further rethinked the skip connections and optimal depth of the U-Net and presented UNet++ for semantic and instance segmentation. Currently, the success of Transformer [44] in image classification has received attention in the field of medical image segmentation. TransUNet+ [26] applied Vision Transformer [6] for medical image segmentation, introducing redesigned skip connections to enhance skip features and improve global attention. Wang et al. [48] developed the Transformer for brain tumor segmentation based on 3D CNN immediately. Moreover, Zhou et al. present nnFormer [54], a 3D transformer designed specifically for volumetric medical image segmentation. In addition to utilizing a fusion of interleaved convolution and self-attention operations, nnFormer introduces a self-attention mechanism that operates both locally and globally within the image volumes. Despite the increasing number of works leveraging various deep learning models to enhance the accuracy of medical image segmentation, clinicians are eager to understand the reliability of segmentation results. Meanwhile, these models seldom achieve robust predictions under noisy conditions.

Recently, uncertainty quantification has been widely investigated for many existing medical image segmentation tasks. Non-Bayesian methods include dropout-based [35], deep ensemble-based [28], and deterministic-based [34] are widely circulated for the medical image segmentation. Wang et al. [45] proposed a test-time augmentation-based aleatoric uncertainty for brain tumor segmentation by Monte Carlo sampling. Mehrtash et al. [28] presented model ensembling for confidence calibration of the FCNs. Mukhoti et al. [34] extend the deep deterministic uncertainty method [33] using feature space densities for the semantic segmentation. Besides, Bayesian approaches including Variational inference methods [19, 1] are proposed to estimate the uncertainty of medical images. To gain uncertainty from a large number of plausible hypotheses, Kohl et al. [19] associated U-Net with a conditional variational autoencoder. They compared it with other outperformed the existing methods, such as U-Net Dropout and U-Net ensemble on a lung abnormality segmentation task. Although the above methods quantify the uncertainty of medical image segmentation, they prioritize more accurate segmentation results over obtaining calibrated uncertainty and robust segmentation results. More importantly, these methods may require significant changes to the network structure, making them less adaptable to integration into arbitrary network architectures, and involving excessive additional computations during the uncertainty estimation process.

1) Deep evidential feature extraction. U-Net [7, 15], its variants [31, 36] and transformer [54] have seen recent widespread used across medical image modalities. We thus employed one of them as our backbone for capturing contextual information. For U-Net based methods, the backbones only performed down-sampling three times to achieve a balance between GPU memory usage and segmentation accuracy. For the transformer-based method, we follow its design [54]. DEviS can freely choose any backbone to extract image features. We only use its decoder output feature vector without the softmax layer. For a random image $x$ in a medical image domain $\mathbb{X}$, this process can be defined as:

Where ${f}\left(\cdot\right)$ is any network backbone without the softmax layer. In this study, we assessed the performance of the different general backbones (U-Net [7], V-Net [31], Attention-UNet [36], nnU-Net [15]) and nnformer [54]. For typical medical image segmentation tasks [9, 23, 48], the predictions are usually carried by the softmax layer as the final layer. As mentioned in [43, 12], the softmax layer has a tendency to display high confidence even for wrong predictions. DEviS mitigates this issue by utilizing the softplus activation function layer instead of the softmax layer. Therefore, following the output of any neural network, a softplus activation function layer ($Softplus$) is applied to ensure that the network output is non-negative, which is considered as evidence for the backbone feature output.

2) Dirichlet distribution for medical image segmentation. We then obtain a Dirichlet distribution from the network output, which is considered as the conjugate prior of the multinomial distribution [16]. This provides a predictive distribution for medical image segmentation and derives uncertainty from this distribution. For $\forall x\in\mathbb{X}$, the projected probability distribution of multinomial opinions is defined by:

where $\boldsymbol{b}_{x}$, $\boldsymbol{r}_{x}$ and $\boldsymbol{U}_{x}$ are the belief mass distribution, base rate distribution and the uncertainty mass distribution over $x$, respectively. Here, the variable x represents any dimensionality medical image, including two-dimensional, three-dimensional, or even multi-dimensional images. Then, Dirichlet PDF $D({{\boldsymbol{p}}_{x}}\mid{{\boldsymbol{\alpha}}_{x}})$ can be used to represent probability density over ${{\boldsymbol{p}}_{x}}$, which is given by:

where Dirichlet distribution with parameters ${{\boldsymbol{\alpha}}_{x}}=\left[{\alpha_{x}^{1},\ldots,\alpha_{x}^{C}}\right]$ is considered as basic belief assignment. $B({\boldsymbol{\alpha}_{x}})$ is the $C$-dimensional multinomial beta function, and ${{S}_{x}^{C}}$ is the C-dimensional unit simplex, given by:

The total strength ${\boldsymbol{\alpha}}_{x}$ can be denoted as:

${\bf{W}}$ denotes the weight of the base rate distribution ${{\bf{r}}_{x}}$. To simplify equations (1) and (4), we consider the base rate distribution ${{\bf{r}}_{x}}$ to be ${\bf{1}}$ and ${\bf{W}}$ to be ${\bf{1}}$. That is ${{\boldsymbol{\alpha}}_{x}}={{\bf{E}}_{x}}+{{\boldsymbol{1}}}$.

3) Evidential uncertainty estimation. One of the generalizations of Bayesian theory for subjective probability is the Dempster-Shafer Theory (DST) [4]. The Dirichlet distribution is formalized as the belief distribution of DST over the discriminative framework in the SL [16]. For medical image segmentation, we define the DEviS framework through SL [16], which derives the probability and uncertainty of different class segmentation problem based on deep evidential features. In Fig. 1(a), we illustrate the uncertainty estimation process 3D $C$-class medical image segmentation task. Given a 3D image input $x$ and evidential feature output ${{\bf{E}}_{x}}{\rm{=}}\left\{{E_{x}^{1},\ldots,E_{x}^{C}}\right\}$, where ${E_{x}^{C}}$, $c=1,2\ldots,C$ denotes the evidence for $c$-th class. The SL provides a belief mass ${{\bf{b}}_{x}}=\left\{{b_{x}^{c}}\right\}_{c=1}^{C}$ and an uncertainty mass ${{\bf{u}}_{x}}$ for different classes of segmentation results. The $C+1$ mass values are all non-negative and their sum is one. Specifically, for the $(i,j,k)$-th pixel, it has the following definition:

where $b_{{}_{i,j,k}}^{c}\geq 0$ and ${u_{i,j,k}}\geq 0$ denote the probability of the $(i,j,k)$-th pixel for the $c$-th class and the overall uncertainty of the $(i,j,k)$-th pixel in $x$, respectively, where ${u_{i,j,k}}\in{\bf{u}}_{x}$ and $b_{{}_{i,j,k}}^{c}\in{b_{x}^{C}}$. SL then associates the evidence ${{e_{i,j,k}^{c}}}$ having the Dirichlet distribution with the parameters ${\alpha_{i,j,k}^{c}}={{e_{i,j,k}^{c}}}+1$, where ${{e_{i,j,k}^{c}}}\geq 0$ and ${{e_{i,j,k}^{c}}}\in{E_{x}^{C}}$. After that, the belief mass and the uncertainty of the $(i,j,k)$-th pixel can be denoted as:

where $S^{c}=\sum\limits_{c=1}^{C}{\alpha_{i,j,k}^{c}}=\sum\limits_{c=1}^{C}{\left({e_{i,j,k}^{c}+1}\right)}$ denotes the Dirichlet strength. The more evidence of the $c$-th class obtained by the $(i,j,k)$-th pixel, the greater its probability. Vice versa.

For clarity, we provide a example that demonstrates the formulation in the context of a three-category 3D medical image segmentation task. Let us consider the evidence output ${{\bf{e}}_{i,j,k}}=\left[{40,1,1}\right]$ for the $(i,j,k)$-th pixel. The corresponding Dirichlet distribution parameter is ${{\bf{\alpha}}_{i,j,k}}=\left[{41,2,2}\right]$. In accordance with the SL theory [16], the three categories of subjective opinions are represented by ${{{b}}_{i,j,k}^{1}}=\left[40,0,0\right]$, ${{{b}}_{i,j,k}^{2}}=\left[0,1,0\right]$, and ${{{b}}_{i,j,k}^{3}}=\left[0,0,1\right]$, respectively. Additionally, the uncertainty mass is given by ${{u}_{i,j,k}}=\left[0.067\right]$. Hence, the point ${\cal M}$ on the simplex represents the opinions of the $(i,j,k)$-th pixel as ${\cal M}{\rm{=}}\left\{{{{b}}_{i,j,k}^{1},{{b}}_{i,j,k}^{2},{{b}}_{i,j,k}^{3},{u_{i,j,k}}}\right\}$. This observation indicates that there is a significant amount of evidence supporting the classification of the $(i,j,k)$-th pixel as the first class, with an associated uncertainty value of 0.067 for the pixel. This formulation can be extended to all pixels in $x$, allowing the estimation of the confidence for each individual pixel. The subjective opinion of image $x$ is then obtained as $\mathbb{M}=\left[{{\bf{b}}_{x}^{1},\ldots,{\bf{b}}_{x}^{C},{{\bf{u}}_{x}}}\right]$.

4) Calibrated uncertainty penalty. Though DEviS provides a means for directly learning evidential uncertainty without sampling, the uncertainty might not be totally calibrated to enhance the reliability of model predictions. As pointed out in  [32, 20], a reliable and well-calibrated model should be uncertain in its predictions when being inaccurate, and be confident for the opposite case. Also, calibrated uncertainty will also help in detecting the OOD data to caution AI researchers. More importantly, our CUP relied on the theoretically sound loss-calibrated approximate inference framework [21] as utility-dependent penalty term for obtaining well-calibrated uncertainty in medical domain. To this end, this paper introduces a loss function called CUP, which considers the relationship between accuracy and uncertainty in medical image segmentation. This is an optimization method used to calibrate uncertainty, aiming to maintain lower uncertainty for accurate predictions and higher uncertainty for inaccurate predictions during the training process, thereby achieving well-calibrated uncertainty. This enables the backbone to improve segmentation performance, in addition to learn to provide well-calibrated uncertainty. The CUP is defined as:

where ${N_{AC}}$, ${N_{AU}}$, ${N_{IC}}$ and ${N_{IU}}$ denote the number of samples for the network output category of Accurate and Certain (A&C), Accurate and Uncertain (A&U), the Inaccurate and Certain (I&C) and the Inaccurate and Uncertain (I&U), respectively (Examples of probability simplex can be seen in Fig. 2 b). We define proxy functions to approximate them as given by:

Ideally, we anticipate the model to exhibit certainty in its predictions when accurate, and to yield high uncertainty estimates when making inaccurate predictions. Therefore, we encourage DEviS to learn more A&C samples in the early training period and provide more I&U samples later in training. Correspondingly, we seek to impose penalties on I&C samples during the initial phase of training and on A&U samples during the later stage (Training process of CUP can be seen in Fig. 2 b). Following [20], we propose an uncertainty calibration loss function in Eq. 20. More details about the training process of calibrated uncertainty and uncertainty calibration loss are presented in  B.

To apply DEviS to clinical tasks to screen out reliable data, we propose a Uncertainty award filtering strategy (Fig. 2 c). This will benefit the predictive accuracy of ID and OOD data. According to [10], a well-calibrated with confidence reflects the reliability of the model. Inspired by this, We first formulate an Uncertainty-based Calibration Error (UCE) to judge the reliability of the data according to the accuracy and uncertainty of the data. For a given validation set ${\mathbb{V}}=\left\{{{x_{1}},\ldots,{x_{m}}}\right\}$ and its uncertainty estimation ${{\mathbb{U}}_{\mathbb{V}}}=\left\{{{\bf{\tilde{u}}}\left({{x_{1}}}\right),\ldots,{\bf{\tilde{u}}}\left({{x_{m}}}\right)}\right\}$, the UCE of each image ${x_{m}}$ can be denoted as follows:

where $acc(x_{m})$ and $\exp\left({-U\left({{x_{m}}}\right)}\right)$ denote the accuracy and confidence of the $m$-th image data. The ${\bf{\tilde{u}}}\left(\cdot\right)$ represents the mean value of the uncertainty estimate corresponding to the ground truth of the medical image segmentation. As such, we consider the largest UCE sample to be the tolerable maximum for ID data or high-quality data, expressed as follows:

Therefore, we derive the adaptive uncertainty threshold ${\bf{U}}_{m}^{*}$ from the uncertainty distribution of the validation dataset ${\mathbb{V}}$, ensuring the selection of the most reliable threshold. For any OOD or low-quality image $x_{m}$ of segmentation task, the reliability of the data can be judged according to the following formula:

$0$ and $1$ denote the unreliable data and reliable data, respectively. It should be noted that for OOD or low-quality data, the model is likely to filter the data it considers reliable or not. Unreliable data may still require further diagnosis by clinical experts.

Due to the imbalance of organ/tumor, our DEviS is first trained with cross-entropy loss function, which is defined as:

where ${y_{x}^{c}}$ and $p_{x}^{c}$ are the label and predicted probability of the $m$-th sample for class $c$. Then, SL associates the Dirichlet distribution with the belief distribution under the framework of evidence theory for obtaining the probability of different classes and uncertainty of different voxels based on the evidence collected from backbone. Therefore, Eq. 14 can be further improved as follows:

where $\psi\left(\cdot\right)$ denote the $digamma$ function. ${{\boldsymbol{b}}_{x}}$ is the class assignment probabilities on a simplex. To guarantee that incorrect labels will yield less evidence, even shrinking to 0, the KL divergence loss function is introduced as below:

where $\Gamma\left(\cdot\right)$ is the $gamma$ function. ${{\tilde{{\alpha}}}_{x}^{c}}={y_{x}^{c}}+\left({1-{y_{x}^{c}}}\right)\odot{{{\alpha}}_{x}^{c}}$ denotes the adjusted parameters of the Dirichlet distribution, which is used to ensure that ground-truth class evidence is not mistaken for 0. Therefore, the evidential deep learning process can be defined as:

where ${\lambda}$ is the balance factor and set to be 0.02 [40]. The gradient Analysis for ${{\cal L}_{E}}$ can be found in  A. Furthermore, the Dice score is an important metric for judging the performance of organ/tumor segmentation. Therefore, we use a soft Dice loss to optimize the network, which is defined as:

where ${y_{x}^{c}}$ and ${p_{x}^{c}}$ are the label and probability of the target. $\varsigma$ is the balance factor. So, the segmentation loss function ${{\cal L}_{S}}$ can be define as follows:

Then, according to Sec. 3.1 of calibrated uncertainty penalty, the loss function for well-calibrated uncertainty can be defined as follows:

To guide the model optimization at the early stage of network training, $\left({1-{\beta_{t}}}\right)$ is noted as the annealing factor, which is defined by ${\beta_{t}}{\rm{=}}{\beta_{0}}{e^{\left\{{-({\rm{In}}{\beta_{0}})t/T}\right\}}}$. $T$ and $t$ are the total epochs and the current epoch, respectively. The value of ${\beta_{0}}$ is set to 0.01. More loss-calibrated approximate inference framework analysis in CUP loss can be referred to  B. Finally, the overall evidential calibrated training loss function of our proposed network can be defined as follows:

1) Experimental Setup: Our proposed network is implemented in PyTorch and trained on NVIDIA GeForce RTX 2080Ti. We adopt the Adam to optimize the overall parameters. The initial learning rates for different datasets are set to be 0.0002 (ISIC2018), 0.001 (LiTS2017), and 0.002 (BraTS2019). The poly learning strategy is used by decaying each iteration with a power of 0.9. The maximum of the epoch is set to 200. The batch sizes for the lesion segmentation, live segmentation, and brain tumor segmentation are set to 16, 4, and 2. All the following experiments adopted a five-fold cross-validation strategy to prevent performance improvement caused by accidental factors. For the ISIC2018 dataset, we used the data augmentation by random cropping, flipping, and random rotation as same as [9]. For the LiTS2017 dataset, we only used the data augmentation by random flipping. For the BraTS2019 dataset, the data augmentation techniques are similar as [48]. For the clinical application of OOD detection, we conducted the experiments on the Johns Hopkins OCT dataset and the Duke OCT dataset with DME. For the clinical application of data quality indicator, we conducted the experiments on the FIVES dataset. In these applications, the initial learning rate for the dataset are set to be 0.0001. The poly learning strategy is used by decaying each iteration with a power of 0.9. The maximum of the epoch is set to 100. The batch sizes for the layer-segmentation and voxel-segmentation from OCT are set to 8. The following will provide additional detailed information about the ISIC2018, LiTS2017, and BraTS2019 datasets used, as well as information regarding degraded image data. It also elaborates on the datasets utilized in clinical applications.

2) ISIC2018 [42] dataset. First, the public available International Skin Imaging Collaboration (ISIC) 2018 dataset is validated for reliable 2D medical image segmentation task. Following the [9], a total of 2594 images and their ground truth are randomly divided into a training set, validation set, and test set, containing 1814, 260 and 520 images, respectively. To verify the robustness and credibility of different models under OOD conditions, we add different levels of Gaussian noise and random masks to the test set, and perform 5-fold cross-validation for final results. First, we added the standard deviation of the Gaussian noise ranging from [0.1, 0.2, 0.3, 0.4, 0.5] to the original data with normalization. Then, the strategy of random mask with 8 pixel-size like [13] ranging from [0.1, 0.25, 0.4] are deployed for the original data.

3) LiTS2017 [2] dataset. Second, Liver Tumor Segmentation (LiTS) Challenge 2017 is validated for reliable 3D medical image segmentation task. It contains the public 131 and 70 contrast-enhanced 3D abdominal CT scans. Following [23], we resampled the overall samples to the same resolution $16\times 256\times 256$ with the spacing of $0.69\times 0.69\times 1$, and randomly divided them into training set and test set containing 105 (nearly 985 volumes) and 26 cases (nearly 245 volumes), respectively. For the OOD condition, we also add different levels of Gaussian noise, Gaussian blur and random mask to the test data of 3D volumes. Gaussian noise is added to the normalized data with standard deviation of the ranging from [0.05, 0.1, 0.2, 0.3, 0.4]. Gaussian blur is added to the test data with variance varying from 11 to 35 and kernel sizes of 10 to 20, specifically ranging from [(11, 10), (15, 10), (23, 20), (35, 20)]. The strategy of random mask with 8 pixel-size like [13] ranging from [0.1, 0.25, 0.4] are also deployed for the original data.

4) BraTS2019 dataset. More importantly, the Brain Tumor Segmentation (BraTS) 2019 challenge [30] with varying degradation conditions(such as noise, blur and mask) are constructed for reliable multi-modality 3D medical image segmentation tasks. Four modalities of brain MRI scans with a volume of $240\times 240\times 155$ are used. 335 cases of patients on BraTS2019 with ground-truth are randomly divided into train dataset, validation dataset, and test dataset with nearly 265, 35, and 35 cases, respectively. The three tumor sub-compartment labels are combined to segment the whole tumor and all inputs are uniformly adjusted to $128\times 128\times 128$ voxels during the training. The outputs of our network contain 4 classes, which are background (label 0), necrotic and non-enhancing tumor (label 1), peritumoral edema (label 2), and GD-enhancing tumor (label 4). Similarly, in order to verify the reliability uncertainty estimation and robust segmentation results of the model under OOD data, three changes were made to the test set, namely Gaussian noise, Gaussian blur, and random mask. Gaussian noise is added to the normalized data with standard deviation of the ranging from [0.5, 1.0, 1.5, 2.0]. Gaussian blur is added to the test data with variance varying from 3 to 9 and kernel sizes of 3 to 9, specifically ranging from [(3, 3), (5, 5), (7, 7), (9, 9)]. The strategy of random mask with 8 pixel-size like [13] ranging from [0.1, 0.25, 0.4, 0.6] are also deployed for the original data.

5) Johns Hopkins OCT & Duke-OCT-DME datasets. In the first application, Johns Hopkins OCT (JH-OCT) dataset and Duke OCT dataset with Diabetic Macular Edema (Duke-OCT-DME) are used for the OOD detector. 35 cases of patients on JH-OCT with ground-truth are randomly divided into train dataset, validation dataset and test dataset with nearly 25, 5 and 5 cases, respectively. The 5 cases of test dataset on JH-OCT are used for ID detection. In particular, the 10 cases from the Duke-OCT-DME are used as another test dataset for OOD detection. Every case is uniformly adjusted to $128\times 1024$ voxels during the training and testing.

6) FIVES dataset. In the second application, the Fundus Image Vessel Segmentation (FIVES) dataset is used for the quality indicator. In the FIVES dataset, each image was evaluated for four qualities: normal, lighting and color distortion, blurring, and low-contrast distortion. In this experiment, we define normal images as high-quality data and images under other conditions as low-quality data. During the experimental process, DEviS was initially trained on the FIVES dataset, which comprises 300 slices of high-quality images. Subsequently, the performance of DEviS was evaluated on a mixed dataset from FIVES, consisting of 300 slices comprising both high and low-quality images. This mixed dataset comprised 159 high-quality slices and 141 slices of low-quality images. Throughout both the training and testing stages, each case was consistently adjusted to dimensions of $565\times 584$ voxels, ensuring uniformity across the dataset.

1) Compared Methods: In this study, we seamlessly integrated our proposed method into both U-Net-based and transformer-based frameworks to rigorously evaluate its robustness and reliability pre and post-integration. Comparative analyses were conducted against state-of-the-art methods employing uncertainty estimation techniques. The U-Net variants encompassed traditional U-Net [7], V-Net [31], Attention-UNet [36], and nnU-Net [15], while the transformer-based approach was predominantly represented by nnFormer [54]. Additionally, our approach underwent comprehensive comparisons with uncertainty estimation methodologies, including U-Net-based Monte Carlo dropout sampling (DU) [18], U-Net-based ensemble (UE) [22], Probabilistic-UNet (PU) [19], Bayesian QuickNAT (BQNAT) [39], Test-Time Augmentation (TTA) [46] and our conference version [57]. These comparisons were undertaken to elucidate the superiorities and limitations of our proposed method in the context of existing cutting-edge techniques.

2) Evaluation Metrics: The following metrics are employed for quantitative evaluation. (a) The Dice score (Dice) and (b) Average symmetric surface distance (ASSD) is adopted as an intuitive evaluation of segmentation accuracy. Given the absence of ground truth for the uncertainty estimate, we utilize the same evaluation metrics as referenced in  [11, 17] to assess its performance. (c) Expected calibration error (ECE) [11, 17] and (d) Uncertainty-error overlap (UEO) [11, 17] are used as evaluation of uncertainty estimations.
Dice score. Dice measures the overlap areas between the prediction map ${\bf{R}}$ and ground truth mask ${\bf{G}}$. It can be represented by:

ASSD. ASSD calculates the accuracy of segmented boundaries between the point sets of prediction ${S_{R}}$ and the point sets of ground truth ${S_{G}}$. It can be defined as:

where $d\left({r,{S_{G}}}\right)={\min_{g\in{S_{G}}}}\left({\left\|{r-g}\right\|}\right)$ represents the minimum Euclidean distance from point $r$ to all the points in $S_{G}$.
ECE. ECE approximates the calibration gap between confidence ${conf\left({{B_{m}}}\right)}$ [11] and accuracy ${acc\left({{B_{m}}}\right)}$ [11]. It can be expressed as:

where $M$ is the number of interval bins. ${B_{m}}$ denotes the set of indices of samples whose prediction confidence falls into the interval. $N$ means the number of samples. ECE closer to zero means better calibration uncertainty.
UEO. UEO measures the overlap between the segmentation error ${{\bf{R}}_{e}}$ and the thresholded uncertainty ${{\bf{U}}_{t}}$, which can be denoted as:

where a higher UEO (close to one) indicates a better calibration.

To illustrate this, we tackled three challenging medical image segmentation tasks using different datasets: (1) 2D skin lesion images from the ISIC2018 dataset, (2) 3D liver CT images from the LiTS2017 dataset, and (3) multi-modal 3D MRI images from the BraTS2019 dataset. These tasks encompass the segmentation of various tissues and tumors, aiming to achieve accurate and robust segmentation performance across different levels of normal and degraded (noise, blur, and random mask) conditions in medical images, alongside reliable and efficient uncertainty estimation.

To demonstrate the effectiveness of the efficiency, we provide more insight into the computational cost performances of well-known uncertainty estimation methods (DU [18], UE [22], PU [19], TTA [46] and BQNAT [39]) on the datasets of ISIC2018, LiTS2017, and BraTS2019 with Gaussian noise by ${\sigma}{\rm{=}}\left\{{0.3,0.2,1.5}\right\}$ (Fig. 3 (b), Fig. 4 (b) and Fig. 5 (b)). Among the evaluated methods, UE and TTA are the least computationally efficient, yet they yield slightly better segmentation results compared to DU. BQNAT demonstrates slightly higher calculation efficiency than DU, with comparable segmentation performance to TTA. Although PU has shown improvements in both computing efficiency and segmentation results, it still requires additional sampling time for testing. After applying DEviS, fundamental deep learning models such as U-Net, Attention-UNet, V-Net, nnUNet, and Transformer segmentation methods represented by nnFormer exhibit noticeable improvements in both testing time and FLOPs compared to the previously mentioned uncertainty quantification methods. Additionally, they deliver more robust segmentation outcomes post-integration.

Moreover, we present comprehensive numerical analyses of computational efficiency, segmentation performance, and uncertainty estimation reliability. These can be found in the Tab. 1. Compared to existing uncertainty estimation methods, the integrated DEviS models manifest diminished computational efficiency while showcasing enhanced segmentation robustness. Furthermore, superior calibration outcomes are evident through metrics like ECE and UEO, signifying advancements in reliable medical image segmentation and uncertainty estimation. Specifically, in the 2D skin lesion segmentation task, models such as UE, DU, TTA, and BQNAT, which necessitate multiple predictions, exhibit slightly higher computation times, while the PU method demonstrates substantially less. The U-Net integrated with DEviS demonstrates significantly improved computation time, approximately 198, 88, and 2 times faster than the top three methods UE, BQNAT, and PU, respectively. From Tab. 1., it can be inferred that in the multi-modal 3D brain tumor segmentation task, as input data size and model complexity increase, the computation time of UE escalates, whereas TTA, BQNAT, and PU require relatively less time. In comparison to the U-Net integrated with DEviS, TTA, BQNAT, and PU methods necessitate approximately 687, 399, and 9 times more computation time, respectively. Notably, the nnFormer integrated with DEviS exhibits the highest computational efficiency, with computation times approximately 1060, 616, and 13 times faster than TTA, BQNAT, and PU methods, respectively.

Furthermore, concerning segmentation performance and uncertainty reliability, this paper evaluates segmentation efficacy using the Dice and ASSD metrics. As uncertainty lacks ground truth, the same ECE and UEO metrics as in the literature are employed to indirectly assess the reliability of uncertainty estimation. As delineated in Tab. 1., under noisy conditions, U-Net, Attention-UNet, V-Net, and nnFormer integrated with DEviS demonstrate heightened accuracy and reliability in comparison to existing uncertainty estimation methods.

1) Out-of-distribution detector: As shown in Fig. 2 (d), the proposed DEviS method is demonstrated to be applicable for OOD detection. It is essential that image processing systems identify any OOD samples in clinical settings. Uncertainty estimation quantifies the uncertainty of the in-distribution (ID) and OOD data to detect inputs that are far outside the training data distribution. DEviS can thus be used to alert clinicians to areas where lesions may be present in OOD data. Further, we equipped DEviS with UAF to distinguish ID and OOD samples.

We conducted OOD experiments on the Johns Hopkins OCT dataset and Duke OCT dataset with Diabetic Macular Edema (DME). As shown in Fig. 6 a, we first observed a slight improvement in results for mixed ID and OOD data after using DEviS. Then, we found significant differences in the performance of the segmentation between the with or without UAF. Additionally, there were also marked differences in the distribution of uncertainty between the ID and OOD data, especially adding the UAF module as shown in Fig. 6 b. As depicted in Fig. 6 c (i), We then employed Uniform Manifold Approximation and Projection (UMAP) to visually assess the integration of our method. In the spatial clustering results of the base network framework, we observed overlapping of ID and OOD data batches. However, after integrating DEviS, we observed improved batch-specific separation of ID and OOD data, particularly for the ID data. Furthermore, the integration of UAF with DEviS effectively eliminated the OOD, resulting in a more pronounced batch effect. Additionally, we first presented the uncertainty estimation map corresponding to UMAP in the Fig. 6 c (ii). It is evident from the map that the boundary region between different batches exhibits significantly higher uncertainty. More intuitively, the segmentation results and uncertainty maps of ID and OOD data can refer to Fig. 8 a. These results combine to show that DEviS with UAF provides a solution for filtering out abnormal areas where lesions may be present in OOD data.

2) Image quality indicator: As shown in Fig. 2 e, it demonstrates that DEviS can serve as an indicator for representing the quality of medical images. Uncertainty estimation is an intuitive and quantitative way to inform clinicians or researchers about the quality of medical images. DEviS guides image quality quantitatively through the distribution of uncertainty values and qualitatively through the degree of explicitness of the uncertainty map. Furthermore, our developed UAF module aids in the initial screening of low-quality and high-quality data. High-quality data can be directly employed in clinical practice, while low-quality data necessitates expert judgment before utilization.

In what follows, we apply DEviS with UAF to indicate the quality of data for real-world applications. The FIVES datasets are used for quality assessment experiments. We initially classified samples into three categories based on their quality labels: high quality, high & low quality, and low quality. We observed distinct performance variations among these categories (Fig. 7 a (i)). To further demonstrate its ability to indicate image quality, we delved into a combination of high and low-quality data to filter out high-quality data. Before the application of UAF, we identified 159 high-quality and 141 low-quality data samples. Upon implementing UAF, the distribution shifted, resulting in 153 high-quality and 61 low-quality data samples. This transition led to a remarkable increase in the proportion of high-quality data from 53% to 71%. Notably, the task at hand posed a greater challenge in assessing data quality compared to the detection of OOD data, as all data sources originated from the same distribution. We also found a significant performance boost with UAF in Dice and ECE metrics. (Fig. 7 a (ii)). Additionally, we investigated the distribution of uncertainty to discern differences between different qualities data (Fig. 7 b (i)). Moreover, the uncertainty distribution of high and low mixed quality with UAF was closer to the low-quality data (Fig. 7 b (ii)). The spatial clustering results of mixed-quality images were visualized using UMAP in the Fig. 7 c. Prior to incorporating our algorithm, some batch-specific separation was observed, albeit with partially overlapping regions (Fig. 7 c (i) 1st and 4th columns). However, upon integrating DEviS with UAF, a slight batch effect was observed (Fig. 7 c (i) 2nd, 3rd, 5th and 6th columns). Additionally, the UMAP visualization with uncertainty map exhibited uncertainty warnings for partially overlapping points, with noticeably high uncertainties along the edges of prediction errors (Fig. 7 c (ii) (1, 2)). Moreover, the segmentation results and uncertainty map of low-quality and high-quality images exhibited in Fig. 8 b, providing a more intuitive representation of the quality disparity. These results demonstrate that DEviS with UAF can serve as an image quality indicator to fairly value personal data in healthcare and consumer markets. This would help to remove harmful data while identifying and collecting higher-value data for diagnostic support.

DEviS proves valuable in clinical applications through its dual functionality. As an adept OOD detector, DEviS, coupled with UAF, effectively distinguishes OOD data, enhancing reliability in clinical image processing. Moreover, DEviS serves as a robust image quality indicator, offering both quantitative and qualitative assessments of medical image quality. The integration of UAF further refines this assessment, demonstrating its capability to filter high-value data and improve performance metrics. These functionalities position DEviS as a versatile tool for detecting abnormal areas in OOD data and indicating image quality, vital for diagnostic support in healthcare and other applications.