Hierarchical Conditional Semi-paired Image-to-image Translation for Multi-task Image Defect Correction on Shopping Websites

To cover multiple defects with a single model, we propose an attention-guided conditional generator as shown in Fig.2. It hierarchically consumes the high-level defect group and specific defect type. For example, image defects such as non-white background and watermark are background-related and incorrect sunglasses angle is angle-related. The high-level defect group $g$ (0 = background-related, 1 = angle-related) helps the generator decide whether to focus on distinguishing the background and foreground, or identifying the angle of the object. Furthermore, although non-white background and watermark belong to the same group, watermark usually only occupies a small region. Adding specific defect type $\eta$ (0 = non-white background, 1 = watermark, 2 = incorrect sunglasses angle) further helps the model tell the difference of the intra-group tasks. Inspired by [21], the red block in Fig. 2 illustrates the procedure of inserting $g$ into the model. Let $I(x)\in\mathbb{R}^{c^{\prime}\times h\times w}$ denote the feature map of an image $x$, we first project $g$ into a $c^{\prime}$-dimensional vector through a linear transformation with $tanh$ activation, and perform spatial duplication to broadcast it into the same dimension as $I(x)$. Subsequently, we use $M_{c}(x)\in\mathbb{R}^{c^{\prime}\times h\times w}$ computed as $M_{c}(x)=\tanh\left(\operatorname{Conv}_{1}(I(x)\odot H_{g}(x))\right)$ to measure the relevance of each element in $I(x)$ to $g$, where $\odot$ is the element-wise multiplication, and $\operatorname{Conv}_{1}$ is a convolution layer with 1 × 1 kernel. Then $M_{g}^{\prime}(x)\in\mathbb{R}^{h\times w}$, the summation of $M_{c}(x)$ across all the channels, represents the relevance of each spatial position in $I(x)$ to $g$. Finally, we denote $I_{1}(x)\in\mathbb{R}^{c^{\prime}\times h\times w}$ as the element-wise multiplication between each channel of $I(x)$ and $M_{g}^{\prime}(x)$ as the updated feature map, which scales each position of $I(x)$ by its relevance to $g$. The similar process of inserting specific defect type $\eta$ is shown in the green box. The conditional labels $g$ and $\eta$ enable the model to cover multiple defects while focusing on the defect-related image regions.

Another challenge on shopping websites is the lack of high-quality paired data. Although we can synthesize paired data, we cannot always guarantee the quality. For example, given an image of non-white background, we can synthesize its paired image of white background by detecting the foreground object and changing the background color to white. However, the synthesized image is not accurate if the foreground object is semi-transparent, of similar color of the background, or placed on another object (see the first line in Fig 6). The I2I model trained using such paired data will memorize the patterns of those low-quality pairs. To mitigate such effect, we leverage the cycle loss of unpaired data to ensure that the transformed images can be converted back.

The training loss consists of the following components. First, adversarial losses in Eq.1 and Eq.2 ensure the generated images look realistic. Specifically, we adopt the relativistic discriminator [22], i.e., $D_{Y}^{Rel}(y_{1},y_{2})=\operatorname{sigmoid}(C(y_{1})-C(y_{2}))$, to measure the probability that $y_{1}$ looks more realistic than $y_{2}$, where $C$ refers to the non-transformed output of the discriminator. Eq.1 trains the discriminator to favor a real image $y$ against a generative image $\hat{y}$, while Eq.2 trains the generator to generate a $\hat{y}$ that looks more realistic than $y$.

$\begin{aligned} &\mathcal{L}_{GAN}^{D}\left(G_{XY},D_{Y}^{Rel},X,Y\right)=\mathbb{E}_{y,\hat{y}}\left[\log D_{Y}^{Rel}(y,\hat{y})\right]\end{aligned}$

(1)

$\begin{aligned} &\mathcal{L}_{GAN}^{G}\left(G_{XY},D_{Y}^{Rel},X,Y\right)=\mathbb{E}_{y,\hat{y}}\left[\log D_{Y}^{Rel}(\hat{y},y)\right]\end{aligned}$

(2)

We denote the sum of these two losses as $\mathcal{L}_{GAN}^{XY}$, i.e., the total adversarial loss from domain $X$ to $Y$. Similarly, we denote $\mathcal{L}_{GAN}^{YX}$ as the total adversarial loss from $Y$ to $X$. Second, for paired data, we use reconstruction loss (Eq.3) to minimize the distance between an image in the target domain and the image generated from the paired image in the source domain.

$\displaystyle\scalebox{0.83}{$\mathcal{L}_{L1}\left(G_{XY},G_{YX}\right)=\mathbb{E}_{x}\|\left(G_{XY}(x)-y\|_{1}\right.+\mathbb{E}_{y}\left\|G_{YX}(y)-x\right\|_{1}$}$

(3)

For unpaired data, we use cycle loss (Eq.4) to ensure the generated images can be transformed back to the source domain, which prevents the model from overfitting to the paired images of low quality.

$\begin{aligned} \mathcal{L}_{\mathrm{cyc}}(G_{XY},G_{YX})&=\mathbb{E}_{x}\left[\|G_{YX}(G_{XY}(x))-x\|_{1}\right]\\ &+\mathbb{E}_{y}\left[\|G_{XY}(G_{YX}(y))-y\|_{1}\right]\end{aligned}$

(4)

Besides, we add identity loss (Eq. 5), which encourages the generators to be close to an identity mapping when images from the target domain are fed to the generators [23, 8].

$\mathcal{L}_{idt}\left(G_{XY},G_{YX}\right)=\mathbb{E}_{x}\|\left(G_{YX}(x)-x\|_{1}\right.+\mathbb{E}_{y}\left\|G_{XY}(y)-y\right\|_{1}$

(5)

At last, the total loss is defined as

$\mathcal{L}_{total}=\lambda_{1}\left(\mathcal{L}_{GAN}^{XY}+\mathcal{L}_{GAN}^{YX}\right)+\lambda_{2}\mathcal{L}_{L1}+\lambda_{3}\mathcal{L}_{cycle}+\lambda_{4}\mathcal{L}_{idt}$

(6)

We resize each image to 256 x 256. For both datasets, we train the models for 300 epochs using Adam optimizer [12] with batch size 1. Following Pix2Pix [3], we set an initial learning rate of 0.0002, which is fixed for first 150 epochs and decays linearly to zero afterwards. We set $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$ to be 1, 150, 10 and 10 following WS-I2I [14]. WS-I2I is a semi-paired I2I model which leverages the same loss functions as in this paper. The difference is WS-I2I cannot cover multiple tasks due to the lack of guidance from $g$ and $\eta$. We use Frechet Inception Distance (FID) [29] as the evaluation metric, which measures the distance between the distributions of generated images and real images. Lower FID score means better performance.

On the combined four paired public data, we compare the proposed model with the baseline method Pix2Pix[3] and the state-of-the-art method MoNCE [12]. Since there are no unpaired data, we set $\lambda_{3}$ and $\lambda_{4}$ to $0$. Table 1 (Paired) shows that our method is consistently better than Pix2Pix and MoNCE. As shown Figure 3, when trained on multiple tasks, MoNCE tends to transform an image from the source domain of a task to the target domain of another task (e.g., transforming a google map to a segmentation instead of an aerial-photo) due to the lack of guidance from $g$ and $\eta$. On the combined four unpaired public data, we compare our method with the baseline method CycleGAN [8] and MoNCE [12]. Since there are no paired data, we set $\lambda_{2}$ to $0$. We observe similar pattern as the paired data as shown in Table 1 (Unpaired) and Fig 4.

On the Image Defects Dataset to correct three image defects including non-Wbg, WM and in-SA, to avoid the model from memorizing the patterns of the low-quality synthesized pairs of non-Wbg and WM images, we configure our model to be semi-paired by combining the reconstruction loss (Eq. 3) of paired data with the cycle loss (Eq. 4) and the identity loss (Eq. 5) of non-Wbg and WM images. Therefore instead of MoNCE [12], we compare the proposed model with WS-I2I [14], the state-of-the-art semi-paired I2I model to the best of our knowledge. As shown in Table 2, our proposed model performs better in all three tasks. Fig 5 shows some examples where our proposed model can transform the images correctly while WS-I2I cannot.

We conduct the ablation study in Table 3 to demonstrate the effect of adding high-level group $g$, specific defect type $\eta$ and using unpaired data. We can conclude that hierarchically adding $g$ and $\eta$ into the model can significantly improve the model performance (by comparing the FID scores in the first line with the second and third line in Table 3). The results also demonstrate the benefit of leveraging unpaired data (by comparing the first line and the fourth line in Table 3). We visualize the benefit of using unpaired data when some of the paired data are of low quality in Fig 6. When the object is semi-transparent, of similar color of the background, or placed on another object (first line in Fig 6), the synthesized paired images will be of low quality, which will negatively affect the model performance if trained using only paired data (second line in Fig 6). In comparison, our proposed model (third line in Fig 6) is robust against such situation.