Generator versus Segmentor: Pseudo-healthy Synthesis

The training workflow of the proposed GVS is shown in Figure 1. The generator gradually synthesizes healthy-like images by iteratively alternating Step A and Step B. The specific steps are as follows.

where $\mathcal{L}_{mse}$ denotes pixel-wise $\mathcal{L}_{2}$ loss. The total training loss of $\mathbf{G}$ is:

where $\lambda$ denotes a hyperparameter that represents the trade-offs between the healthiness and identity, and it is subject to $\lambda>0$.

This section further considers whether it is reasonable to set the same visual similarity in normal and pathological regions. Apparently, it is reasonable to keep visual consistency in normal regions between pathological and synthetic images. However, it is contradictory to remove lesions and keep pixel values within pathological regions. In order to alleviate this contradiction, we assume that potential normal tissues for lesion regions and normal tissues in the same pathological image have similar pixel values. Based on this, we improve the residual loss as follows:

where $\odot$ represents the pixel-wise multiplication, and $\overline{x}_{pn}$ denotes a matrix filled with the average value of normal tissue in the same image $x_{p}$ and has the same size as $x_{p}$; $\lambda_{1}$ denotes a hyperparameter that controls the power of visual consistency in the lesion regions, and $0<\lambda_{1}<1$ since the potential normal tissues are close but not equal to the average value of normal tissues.

The generalization ability of segmentor is further considered. During the training, the pathological regions are gradually transformed into healthy-like ones. As shown in Figure 2(b), the major part of the lesion region has been well transformed, so these pixels should be labeled as ’healthy.’ However, the basic GVS still views all the pixels in this region as lesions. In other words, the well-transformed parts are forced to fit the false label, which weakens the generalization capacity of neural networks [17]. As shown in Figure 2(b), we observe that predictions of segmentor substantially deviate from labels, which suggests the poor generalization of the segmentor. To meet this challenge, we present a novel pixel-level weighted cross-entropy loss for lesion segmentation. The difference map between pathological and synthetic images can be used as an indicator to measure the transformation degree:

where $N$ denotes the number of pixels. The weight $w$ associated with difference maps are defined as:

where $m=\text{Normalization}(x_{p}-\mathbf{G}(x_{p}))$ denotes the normalized difference map. In this work, $w[w<0.1]=0.1$ because the minimum value does not represent perfect-transformation, and it is necessary to keep a subtle penalty.

The complete GVS is developed by replacing $\mathcal{L}_{R}$ with $\mathcal{L}_{R+}$, and then replacing $\mathcal{L}_{s1}$ with $\mathcal{L}_{wce}$. Please note that the GVS refers to the complete GVS in the next subsections.

In this section, two metrics, $\mathbb{S}_{dice}$ and $iD$, are introduced to evaluate the healthiness and identity, respectively.

Zhang et al. [17] revealed an interesting phenomenon that convergence time on the false/noisy labels increases by a constant factor compared with that on the true labels. Similar to this, aligning well-transformed pixels (i.e., these pixels can be viewed as healthy ones) and lesion annotations is counterfactual and hampers the convergence. Thus, how healthy the synthetic images look is negatively related to the convergence time. Inspired by this, we present a new metric to assess the healthiness, which is defined as the accumulated dice score throughout the training process: $\mathbb{S}_{dice}=\sum_{e=1}^{epochs}dice_{e}$, where $epochs$ denotes the number of training epochs, and $dice_{e}$ denotes the dice evaluated on the training data at the $e$-th epoch.

Here, whether the $\mathbb{S}_{dice}$ can correctly assess the healthiness of synthetic images is testified. To this end, we calculate the $\mathbb{S}_{dice}$ of GVS(0.1), GVS(0.4), GVS(0.7), GVS(1.0), and original images. Note that the GVS(a) denotes the synthetic images generated by the GVS trained on $a\%$ training data. Generally, the GVS(a) should have more healthy appearances than the GVS(b) when $a>b$. The results show that the $\mathbb{S}_{dice}$ can correctly reflect the healthiness in order. In addition, we also discover that the $\mathbb{S}_{dice}$ is unstable in the small value area, which may result in false results when synthetic images have similar healthy appearances. One possible reason for this situation is that the random parameter initialization may lead to different results at each trial.

Following Xia et. [15], the identity is expressed as the structural similarity calculated on normal pixels, which is defined as $iD=\text{MS-SSIM}[(1-y_{t})\odot\mathbf{G}(x_{p}),(1-y_{t})\odot x_{p}]$, where the $\odot$ denotes the pixel-wise multiplication, and the $\text{MS-SSIM}()$ representes a masked Multi-Scale Structural Similarity Index [14].

We compare our GVS with existing pathology-deficiency based methods, including the VA-GAN [3], ANT-GAN [12], and PHS-GAN [15]. The VA-GAN¹¹1https://github.com/baumgach/vagan-code and PHS-GAN²²2https://github.com/xiat0616/pseudo-healthy-synthesis are implemented using their official codes, and the ANT-GAN is implemented on the code provided by the authors. Note that the VA-GAN uses image-level labels, and the ANT-GAN and PHS-GAN utilize pixel-level labels. Our experimental setting keeps consistent with the last two. Next, we analyze the performance of all methods qualitatively and quantitatively.

Qualitative results are shown in Figure 4. We first analyze the identity by comparing the reconstruction performance in the normal regions. The proposed method achieves higher reconstruction quality than the other methods (better see the difference maps). For example, compared with the existing methods, our method better preserves the high-frequency details (i.e., edges). Overall, the VA-GAN could not keep the subject identity and loses a part of lesion regions in some cases. The PHS-GAN and ANT-GAN preserve the major normal regions but lose some details. Among all the methods, the proposed method achieves the best subject identity. Then, we further analyze the healthiness, which can be judged by comparing whether pathological and normal regions are harmonious. The synthetic images generated by VA-GAN are visually unhealthy due to the poor reconstruction. The PHS-GAN and ANT-GAN remove most of the lesions, but some artifacts still remain. The GVS achieves the best healthiness due to indistinguishable pathological and normal regions.

We report quantitative results in Table 1. The first metric, $iD$, is used to assess the identity. Our GVS achieves a $iD$ value of 0.99, outperforming the second place, PHS-GAN, by 0.02, which is an evident improvement when the SSIM is large. The proposed $\mathbb{S}_{dice}$ is used to assess healthiness. Since the VA-GAN can not reconstruct the normal tissue, as shown in the third column in Figure 4), its $\mathbb{S}_{dice}$ value is meaningless and not considered. In addition, compared with baseline (i.e., original images), the $\mathbb{S}_{dice}$ of the ANT-GAN and PHS-GAN decline from 26.23 to 23.32 and 23.11, respectively. The proposed method further improves the $\mathbb{S}_{dice}$ to 21.75.

To verify the claim that the $\mathcal{L}_{wce}$ can alleviate the poor generalization of the segmentor, we respectively calculate the segmentation performance of the segmentors trained by the GVS and GVS w/o $\mathcal{L}_{wce}$. As shown in Figure 5, the GVS achieves a higher average dice score and lower variance compared to the GVS w/o $\mathcal{L}_{wce}$, which confirms that the $\mathcal{L}_{wce}$ could effectively improve the generalization ability of the segmentor. Similar conclusions are also derived from the visual examples in Figure 5. Predictions of segmentor trained by GVS w/o $\mathcal{L}_{wce}$ deviate from the labels severely. After adding the $\mathcal{L}_{wce}$, the predictions are more accurate. Note that only relying on the $\mathcal{L}_{wce}$ cannot solve the generalization problem of the segmentor entirely (see the third example in Figure 5). Hence, this problem needs further exploration in the future. Furthermore, benefiting from the better generalization of the segmentor, the healthiness attains further improvement ($23.66\rightarrow 21.75$ in Table 1). We also conduct the ablation study on the improved residual loss, and results are shown in Table 1. We observe that the $\mathbb{S}_{dice}$ increases from $21.75$ to $22.11$ after replacing $\mathcal{L}_{R}$ with $\mathcal{L}_{R+}$.

The proposed GVS is also evaluated on a public CT dataset, LiTS. The results show that the proposed method can maintain the identity and transform the high-contrast lesions well, as shown in the top row in Figure 6, However, the synthetic results of low-contrast images still exist subtle artifacts in the pathological regions, as shown in the bottom row in Figure 6. We conjecture that the reason may be that the segmentor cannot accurately detect low-contrast lesions, which further results in poor transformation.