Annotation-Efficient Learning for Medical Image Segmentation based on Noisy Pseudo Labels and Adversarial Learning

Recent advances in medical image segmentation are based on CNNs [1], such as U-Net [20, 21] and V-Net [22]. Some more powerful networks including Attention U-Net [23], U-Net++ [24] and H-DenseUNet [25] further improved the segmentation performance for several tasks. However, most existing works require a large set of manual annotations for training.

Existing annotation-efficient methods for medical image segmentation mainly include weakly- and semi-supervised learning, unsupervised domain adaptation and unsupervised learning [7].

For weakly-supervised deep learning, Feng et al. [8] used image-level labels to train a pulmonary nodule classification network, and used its activation map for segmentation of lung nodules. Rajchl el al. [9] combined GrabCut [26] and iterative training for fetal brain segmentation using bounding box annotations. Semi-supervised learning methods only require a subset of training images to be labeled [27]. Bai et al. [10] proposed to alternatively update the network parameters and the segmentation for the unlabeled data for cardiac MR image segmentation. Nie et al. [11] used a region attention and adversarial learning-based method for this purpose. Other methods such as self-ensembling [28] and consistency loss [29] have also been proposed for semi-supervised segmentation. Unsupervised domain adaptation is practically appealing where no label is available for the target domain with available annotations for the source domain. With the great success of CycleGAN [30] in unpaired image-to-image translation, many approaches [14, 15] used CycleGAN to transform target domain images to the source domain for the segmentation. Jiang et al. [14] used CycleGAN with tumor-aware loss to transform CT images to MRI images for lung cancer segmentation where annotations for MRI images were not available. Chen et al. [15] synergistically exploited feature alignments with image transformation to deal with the domain adaptation between CT and MRI for cardiac substructure and abdominal organ segmentation.

For training segmentation models without any human annotations, Moriya et al. [18] used deep feature representations of training patches and clustering for unsupervised segmentation. Moriya et al. [19] employed adversarial learning with categorical latent variables for unsupervised segmentation of micro-CT images. However, these methods hardly use prior information of the segmentation target and their performance is far from learning from human annotations.

Shape models have been widely used to improve the robustness of segmentation methods [31]. For example, spherical priors like SPHARM [32] was proposed for brain structure analysis. Sparse shape composition [33] was proposed to model complex shape structures with dictionary learning. Recently, Safar et al. [34] proposed to learn shape priors for object segmentation via neural networks. Oktay et al. [35] encouraged models to follow the global anatomical properties of the underlying anatomy via learnt non-linear representations of the shape. Some other examples of employing shape models include Voxelmorph [36] for registration and DeepSSM [37] for characterization and classification. Differently from these works, we employ VAE and adversarial learning to add an implicit high-level shape constraint on the segmentation output so that it follows the distribution of our set of auxiliary masks unpaired with training images.

Learning from noisy labels has been increasingly investigated recently due to the challenges to obtain high-quality annotations, and many existing works focus on image classification tasks [38, 39]. For example, some novel loss functions, such as the Mean Absolute Error (MAE) [38], Generalized Cross Entropy [40] and noise-robust Dice loss [41], have been proposed to deal with noisy labels. Rusiecki et al. [42] proposed a trimmed cross entropy loss to exclude samples with large training errors. For medical image segmentation with noisy labels, Zhu et al. [43] proposed a strategy to evaluate the relative quality of training labels and thus only the good ones are used to tune the network parameter. Mirikharaji et al. [44] assigned lower weights to pixels with abnormal loss gradient direction. However, these methods require a set of clean labels for training. Karimi et al. [45] used an iterative label update method to deal with simulated noisy labels, but its effectiveness on real noisy labels has not been investigated.

With the auxiliary masks that are unpaired with our unannotated training images, we take advantage of their high-level shape information through adversarial training to constrain a generator $G_{A}$ so that $G_{A}$ generates pseudo labels for training images that have the same shape distribution as the auxiliary masks. As shown in Fig. 1(b), given a medical image $I$ from domain $\mathbf{I}$ and an auxiliary mask $S$ from domain $\mathbf{S}$, we use a pseudo label generator $G_{A}$ to translate $I$ into a binary mask $S^{\prime}=G_{A}(I)$ (i.e., pseudo label) corresponding to $I$, and $S^{\prime}$ is translated back to a medical image $I_{S}=G_{B}(S^{\prime})$ by the image generator $G_{B}$. On the contrary, an auxiliary mask sample $S$ from domain $\mathbf{S}$ is translated by $G_{B}$ into a pseudo medical image $I^{\prime}=G_{B}(S)$ corresponding to $S$, and $I^{\prime}$ is translated back to a binary mask $S_{I}=G_{A}(I^{\prime})$. A cycle consistency loss which prevents the generators from producing a result that is irrelevant to the input between $I$ and $I_{S}$ (between $S$ and $S_{I}$, similarly) is calculated as:

$$\mathcal{L}_{cycle}(G_{A},G_{B},\mathbf{I})=\mathbb{E}_{I\sim p_{data}(\mathbf{I})}\Big{[}||I-G_{B}(G_{A}(I))||_{1}\Big{]}$$

(1)

where $p_{data}(\mathbf{I})$ is the distribution of domain $\mathbf{I}$. An adversarial loss [30, 48] is used to encourage $S^{\prime}$ to match the distribution of $S$:

$$\begin{split}\mathcal{L}_{adv}(G_{A},D_{B})=\mathbb{E}_{I\sim p_{data}(\mathbf{I})}\Big{[}(1-D_{B}(G_{A}(I)))^{2}\Big{]}+\\ \mathbb{E}_{S\sim p_{data}(\mathbf{S})}\Big{[}D_{B}(S)^{2}\Big{]}\end{split}$$

(2)

where $p_{data}(\mathbf{S})$ is the distribution of domain $\mathbf{S}$ and the $D_{B}$ is a patch-based discriminator that distinguishes each patch of its input as a real or fake patch from domain $\mathbf{S}$. Our adversarial loss is least square adversarial loss [49] proposed for overcoming vanishing gradient problem of the original GAN loss [50]. $D_{B}$ serves to evaluate the quality of the pseudo label $S^{\prime}$. Similarly, we use another discriminator $D_{A}$ to distinguish its input as a real or fake medical image. The original discriminator in [50] outputs a single scalar that only indicates whether the input mask is real or fake as a whole, without giving details of non-local regions. In contrast, a patch-based discriminator [51] can better indicate the quality of subregions in the generator’s output. Therefore, $D_{A}$ and $D_{B}$ are implemented by patch-based discriminators in this paper.

Unlike those of regular images, pixel values in the pseudo segmentation label are not complex and sparse, which can be converted into a more compact representation by a latent vector. For example, an ellipse-like mask can be well represented by a low-dimensional vector specifying the size, position and orientation of the ellipse. Distinguishing the compact low-dimensional latent vectors additionally has a potential to obtain better performance than distinguishing the raw pseudo segmentation labels in a very high dimension only. Therefore, we propose to convert binary masks $S$ and $S’$ to their latent vector representations $z_{r}$ and $z_{f}$ respectively using a VAE [52]. We then apply a discriminator $D_{V\!AE}$ with three linear layers and leaky ReLU to distinguish them. VAE [52] is an encoder-decoder network, where the encoder network maps an input to a low-dimensional latent vector, and the decoder network attempts to reconstruct the input. We regularize the encoder by forcing the latent vector to follow a Gaussian distribution with a mean of zero and a variance of one.

As the role of VAE is to convert a segmentation mask in the 2D image space into a compact representation by a latent vector, we pretrain the VAE with the auxiliary masks. For pre-training, it takes an auxiliary mask as input and its decoder reconstructs the auxiliary mask as output. A KL divergence loss and an L2 loss were combined with an Adam optimizer for training. After pre-training, we fix the VAE and employ its encoder to obtain a compact representation of an input segmentation mask, which is sent into our $D_{V\!AE}$. The adversarial loss for $D_{V\!AE}$ can be written as:

$$\begin{split}\mathcal{L}_{V\!AE}(z_{f},z_{r})=\mathbb{E}_{z_{f}\sim p_{data}(Z_{F})}[(1-D_{V\!AE}(z_{f}))^{2}]+\\ \mathbb{E}_{z_{r}\sim p_{data}(Z_{R})}[D_{V\!AE}(z_{r})^{2}]\end{split}$$

(3)

where $Z_{F}$, $Z_{R}$ are sets of $z_{f}$, $z_{r}$, respectively. $p_{data}(Z_{F})$, $p_{data}(Z_{R})$ are the distributions of $Z_{F}$, $Z_{R}$, respectively.

The overall loss of our method is summarized as:

$$\begin{split}G_{A}^{*},G_{B}^{*}=arg\mathop{min}\limits_{G_{A},G_{B}}\mathop{max}\limits_{D_{A},D_{B},D_{V\!AE}}\lambda_{V\!AE}L_{V\!AE}(z_{r},z_{f})\\ +\mathcal{\lambda}_{adv}[\mathcal{L}_{adv}(G_{A},D_{B})+\mathcal{L}_{adv}(G_{B},D_{A})]\\ +\mathcal{\lambda}_{cycle}[\mathcal{L}_{cycle}(G_{A},G_{B},\mathbf{I})+{L}_{cycle}(G_{A},G_{B},\mathbf{S})]\end{split}$$

(4)

where $\lambda_{V\!AE}$, $\lambda_{adv}$ and $\lambda_{cycle}$ control the relative weights of the three terms, respectively.

In a standard GAN [30], the discriminator gives a feedback to the generator by the loss function with back-propagation, which can only be used for training, i.e., indirect and implicit feedback. As the patch-based discriminator $D_{B}$ indicates whether a patch of pseudo label $S^{{}^{\prime}}$ is real or fake and also learns the typical representation features [53], the feature map of $D_{B}$ has a potential to explicitly guide $G_{A}$ to get better results. Besides, as the discriminator easily outperforms the generator, the generator $G_{A}$ could learn better and faster when calibrated by the feature map of $D_{B}$. Therefore, we propose a Discriminator-guided Generator Channel Calibration (DGCC) module to boost the performance of $G_{A}$. As shown in Fig. 1(b), we use four DGCC modules to calibrate the features of $G_{A}$ at four scales, respectively.

In our DGCC, leveraging the discriminator’s feedback leads to recurrent loop connections. Let $T$ represent the total turn number in the loop connections, as shown in Fig. 2. At turn 1, the generator $G_{A}$ does not have feedback from the discriminator $D_{B}$. At the following few turns, we take $D_{B}$’s embedding feature map right before the output layer at turn $t$ as our feedback information:

$$h_{t}=D_{B}(S^{\prime}_{t})$$

(5)

where $h_{t}\in\mathbb{R}^{C\times h\times w}$, and $C$, $h$, $w$ are the channel number, height and width of the embedding feature map of $D_{B}$, respectively. $S^{\prime}_{t}$ is the $S^{\prime}$ at the turn $t$. Then we apply a global average pooling ($P$) to obtain the average feature for each channel and use Sequeeze-and-Excitation (SE) [54] consisting of two convolution layers to obtain an attention coefficient vector $\beta^{s}_{t}$ with a length of $C$ that equals to the channel number of $G_{A}$’s feature map at scale $s$ of turn $t$:

$$\beta^{s}_{t}=W_{2}\cdot\delta(W_{1}\cdot P(h_{t}))$$

(6)

where $\delta$ refers to ReLU, $W_{1}\in\mathbb{R}^{\frac{C}{r}\times{C}}$ and $W_{2}\in\mathbb{R}^{{C}\times\frac{C}{r}}$ are $1\times 1$ convolution layers and $r$ is set to 4 according to common practice [54]. Let $u^{s}_{t+1}$ be a certain feature map at scale $s$ in the decoder of $G_{A}$ before calibration at turn $t+1$ of the recurrence, the corresponding calibrated feature map $\hat{u}^{s}_{t+1}$ at turn $t+1$ of scale $s$ is:

$$\hat{u}^{s}_{t+1}=\beta^{s}_{t}u^{s}_{t+1}+u^{s}_{t+1}$$

(7)

where a residual connection is used to facilitate the training. The new mask ${S}^{{}^{\prime}}_{t+1}$ obtained by $G_{A}$ is:

$${S}^{{}^{\prime}}_{t+1}=G_{A}(I,\beta^{s}_{t})$$

(8)

For testing, we take ${S}^{{}^{\prime}}_{T}$ at turn $T$ of the recurrent connection as the pseudo segmentation label obtained by $G_{A}$.

For optic disc segmentation, we utilized the Digital Retinal Images for Optic Nerve Head (optic disc and cup) Segmentation Database (DRIONS-DB) [59] and retinal image dataset for optic nerve head segmentation (Drishti-gs) [60, 61]. DRIONS-DB consists of 110 colour digital retinal images with a size of 600 $\times$ 400. Drishti-gs consists of 101 colour digital retinal images with a varying image size. Each image in these two datasets had annotations of optic disc by two and four experts, respectively. We averaged these multiple segmentation contours along the radical direction for a given image as the ground truth. As these two datasets are relatively small, we merged them into a single dataset for experiments. As the segmentation target is relatively small and located near the center of the image, we cropped the images at the center to 60$\%$ of the original size. For fetal head segmentation, we used the HC18 dataset¹¹1http://doi.org/10.5281/zenodo.1322001 [58] containing 999 2D ultrasound images of the fetal head in the standard plane for experiment. The ultrasound images were acquired from 551 pregnant women in all trimesters of the pregnancy, and all the cases did not exhibit any growth abnormalities. The size of each 2D ultrasound image was 800 $\times$ 540 with a pixel size ranging from 0.052 mm to 0.326 mm. For these two applications, the images were randomly split into 70$\%$, 10$\%$, 20$\%$ for training, validation and testing, respectively. We abandoned the ground truth of the training set for our annotation-efficient learning. Each image was resized to $288\times 288$, randomly cropped to 256 $\times$ 256, and the intensity was normalized into the range of [-1, 1].

We first evaluate the effectiveness of our pseudo label generation method. For an ablation study of our $D_{V\!AE}$, we started with the baseline of training a CycleGAN [30] without $D_{V\!AE}$ and DGCC. Table 2 shows quantitative evaluation results of the output of $G_{A}$ on the validation set with different latent vector length of the VAE. It can be observed that the best performance is achieved when the length of latent vector is 32. Fig 5 shows that the optimal value of hyper-parameter $\lambda_{V\!AE}$ is 1.0, and the performance of $G_{A}$ does not change much when $\lambda_{V\!AE}$ is around 1.0. We also compared our $D_{V\!AE}$ with three counterparts: 1) the baseline not using $D_{V\!AE}$, 2) our $D_{V\!AE}$ without $D_{B}$, 3) replacing the VAE with beta-VAE [62] where the latent vector length was 32, denoted as $D_{V\!AE}$(beta). Quantitative evaluation based on the testing set in Table 1 shows that our $D_{V\!AE}$ outperformed the counterparts, and compared with not using $D_{V\!AE}$, it improved the average Dice from 0.909 to 0.918 for optic disc segmentation and from 0.904 to 0.918 for fetal head segmentation, respectively. Fig. 3 demonstrates the effectiveness of our VAE-based discriminator on the output of $G_{A}$.

We further evaluated the effectiveness of our multi-scale DGCC module by ablation studies. We compared it with two variants: 1) DGCC(l) that only calibrates the low-resolution feature map obtained by the bottleneck of $G_{A}$; 2) DGCC(h) that only calibrates the high-resolution feature map before the last convolution block of $G_{A}$. The quantitative evaluation results of these variants combined with our baseline model are shown in Table 3. It can be observed that our multi-scale DGCC has a higher performance than DGCC(l) and DGCC(h), which demonstrates that multi-scale calibration performed better than single-scale calibration of the pseudo label generator $G_{A}$. We also compared the calibrated result at $t=2$ of our DGCC with the result at $t=1$ (i.e., before calibration) of our DGCC module, which is denoted as DGCC($t=1$). The quantitative results in Table 3 and qualitative results in Fig. 4 show that the calibration helps to reduce and even remove some noise in the output of $G_{A}$. In addition, Fig. 3 and Table 3 show that combining our $D_{V\!AE}$ with DGCC outperforms the other variants.

To validate our noise-robust iterative method to learn from noisy pseudo labels obtained by $G_{A}$, we compared the following variants: 1) U-Net (baseline) that learns from the pseudo labels using a standard Dice loss without considering the existence of noise; 2) U-Net (MAE) that uses MAE loss [38] for training; 3) U-Net ($\mathcal{L}_{gc}$) that uses generalized cross entropy loss [40] for training; 4) U-Net trained with Dice loss from samples selected by our LQSS, which is referred to as U-Net (LQSS). These four methods only train the model once without iterative training, and were further compared with: 5) U-Net (LQSS + IT) that refers to U-Net (LQSS) followed by iterative training with Dice loss; and 6) U-Net (LQSS + IT + wDice) that refers to U-Net (LQSS) followed by iterative training with our noise-weighted Dice loss. For the last two variants, the round number determined by the validation set was 3 and 4 for optic disc segmentation and fetal head segmentation, respectively. The quantitative evaluation results are shown in Table 4, which shows that LQSS obtained better performance than the baseline, and using iterative training and noise-weighted Dice loss further improves the segmentation accuracy. Fig. 8 shows that our LQSS is able to reject low-quality pseudo labels with some noise, e.g., over segmentation with false positives. Note that in Fig. 8(a), the second rejected case of has a higher contrast than the first accepted case, which shows our LQSS does not tend to only select easy samples. Fig. 6 demonstrates the refinement of pseudo labels at different rounds of training stage. Fig. 9 shows the performance at different rounds of our iterative method to learn from noisy pseudo labels obtained by $G_{A}$. It shows that the performance increased at the beginning and reached a plateau after two rounds for optic disc and three rounds for fetal head, and that noise-weighted Dice loss is better than Dice loss during the iterative training. We compared our ellipse-based shape prior with circle-based shape prior to obtain the pseudo labels, and they are denoted as U-Net (baseline) and U-Net (baseline)^∘, respectively. Results in Table 4 show that modeling optic disc and fetal head as ellipses largely outperform modeling as circles.

Our method was compared with U-Net (manual) that represents training U-Net with manual annotations, and it was compared with three existing unsupervised segmentation methods: 1) Joshi et al. [56] that uses circular Hough transform and snake model for optic disc segmentation; 2) The deep representation and adversarial learning-based method proposed by Moriya et al. [19]; 3) The method of Perez-Gonzalez et al. [17] that uses optimal ellipse detection and texture maps for fetal head segmentation. We also compared our method with two existing weakly-supervised methods: 1) Kervadec et al. [64] that employs a differentiable penalty in loss function to enforce inequality constraints; 2) Lu et al. [63] that trains a U-Net model with the foreground segmentation map generated by an improved constraint CNN and GrabCut. Table 4 shows that our method largely outperformed existing unsupervised and weakly supervised methods for these two objects.

In the iterative training process, we also replaced our pseudo labels obtained by $G_{A}$ with those obtained by existing unsupervised methods, i.e., [56] for optic disc and [17] for fetal head, which is denoted as [56] + IT + wDice and [17] + IT + wDice, respectively. Table 4 demonstrates that when pseudo labels obtained by [17] or  [56] are used, our iterative training still leads to a large performance improvement. However, the performance is worse than using pseudo labels obtained by our $G_{A}$ for the iterative training process. Table 4 also shows that the result of our method has no significant difference from that of learning from human annotations for optic disc segmentation, and the performance gap is also subtle for fetal head segmentation. Visual comparison in Fig. 10 shows that the result of our method is comparable to that of learning from human annotations and Fig. 11 shows that our method performs well when dealing with images with weak boundary information like fetal head segmentation.

For lung segmentation, we used the Japanese Society of Radiological Technology (JSRT) dataset [65] that contains 247 posterior-anterior chest X-ray images with expert segmentation masks. The original images size was $2048\times 2048$ with a pixel spacing of 0.715 mm $\times$ 0.715 mm. To learn without the annotations of JSRT images, we obtain auxiliary lung masks from the public Montgometry County X-Ray Set (MCXS) [66]. It contains 138 posterior-anterior CXR images, of which 80 images are normal and 58 images are abnormal with manifestations of tuberculosis. We used images in the JSRT dataset as domain $\mathbf{I}$ and lung masks in the MCXS dataset as domain $\mathbf{S}$ for our annotation-efficient learning.

For liver segmentation, we utilized the data form ISBI 2019 CHAOS Challenge [67], which contains unpaired 20 CT volumes and 20 MRI volumes with expert segmentation masks. The CT volumes have an in-plane size of 512 $\times$ 512 and slice thickness of 1.5 mm. The MRI volumes have a size of 256 $\times$ 256 with varying slice thickness, and we re-sampled the slice thickness to 1.5 mm. Both CT and MRI images were cropped near the liver region in 3D, and slices without liver were excluded. We aimed to segment the liver from CT images (domain $\mathbf{I}$) with the help of auxiliary liver masks (domain $\mathbf{S}$) from the MRI images.

For these two applications, we randomly split the images of JSRT dataset and CT volumes of CHAOS dataset into 70$\%$ for training, 10$\%$ for validation and 20$\%$ for testing, respectively. And we abandoned the ground truth of the training images for our annotation-efficient learning. Each image was resized to $288\times 288$, randomly cropped to 256 $\times$ 256, and the intensity was normalized into the range of [-1, 1].

We first evaluate the effectiveness of our pseudo label generation method. For ablation studies, we started with a baseline of training a CycleGAN [30] without $D_{V\!AE}$ and DGCC. It was compared with baseline+$D_{V\!AE}$, baseline+$D_{V\!AE}$ without $D_{B}$, baseline+DGCC, and baseline+$D_{V\!AE}$+DGCC. Table 5 lists quantitative evaluation results of the output of $G_{A}$ for our testing set. It shows that our $D_{V\!AE}$ improved the average Dice score from 0.823 to 0.839 for lung segmentation and 0.853 to 0.869 for liver segmentation compared with the baseline. Using $D_{V\!AE}$ and DGCC at the same time outperformed the other variants, with an average Dice score of 0.848 for lung segmentation and 0.889 for liver segmentation. Fig. 12 shows a visual comparison between these variants. It can be observed that the output of $G_{A}$ trained by the baseline method contains some noise. By using $D_{V\!AE}$ that introduces a high-level shape constraint, the noise is reduced. DGCC also helps to improve the quality of $G_{A}$’s output compared with the baseline. The last column of Fig. 12 shows that a combination of $D_{V\!AE}$ and DGCC obtained better results than the others.

To validate our noise-robust iterative method to learn from noisy pseudo labels obtained by our generator $G_{A}$, we first compared the following variants: 1) U-Net (baseline) that learns from the pseudo labels using a standard Dice loss without considering the existence of noise; 2) U-Net trained with Dice loss from samples selected by our LQSS, which is referred to as U-Net (LQSS). These two methods only train the model once without iterative training, and were further compared with: 3) U-Net (LQSS + IT) that refers to U-Net (LQSS) followed by iterative training with Dice loss (five rounds); and 4) U-Net (LQSS + IT + wDice) that refers to U-Net (LQSS) followed by iterative training with our noise-weighted Dice loss (five rounds). The quantitative evaluation results are shown in Table 6. It can be observed that the LQSS obtained better performance than the baseline, and using iterative training and noise-weighted Dice loss can further improve the segmentation accuracy. As shown in Table 6, the iterative training with our noise-weighted Dice loss improved the segmentation Dice score from 0.907 to 0.926 for the lung, and from 0.908 to 0.933 for the liver, respectively. The results show that iteration process is important for learning from noisy pseudo labels. Fig. 13 demonstrates the pseudo labels refined at different rounds of training stage, and it can be observed that the quality of pseudo labels are gradually improved during the training rounds. Fig. 15 shows the performance of iterative training on the testing set. It demonstrates that the performance increased at the beginning and reached a plateau after four rounds, and also shows that noise-weighted Dice loss is better than Dice loss during the iterative training.

As our method uses auxiliary masks from a publicly available third-party dataset, we compared our method with applying the U-Net trained with the third-party dataset directly to our testing images, which is denoted as U-Net (no adapt.). As shown in Table 6, U-Net (no adapt.) has a poor performance for lung segmentation. This is because JSRT images are from normal persons while MCXS contains some abnormalities, and the two datasets have different intensity distributions, leading to a large domain shift. Similarly, U-Net (no adapt.) also performs poorly for the liver segmentation, due to the domain shift between MRI images and CT images.

For comparison with training from full supervision, we trained U-Net with manual annotations of the JSRT dataset for lung segmentation and CT volumes of CHAOS dataset for liver semgentation, which is denoted as U-Net (manual). We also compared our method with an existing unsupervised method proposed by Moriya et al. [19], a weakly-supervised method proposed by Kervadec et al. [64] and an unsupervised domain adaptation proposed by Wu et al. [68] that uses a characteristic function distance metric based on characteristic functions of distributions to enable explicit domain adaptation. Table 6 shows that our method largely outperformed that of Moriya et al. [19], Kervadec et al. [64] and Wu et al. [68]. What’s more, both Table 6 and Fig. 14 show that the difference between our method and U-Net (manual) is subtle.