Unsupervised Synthetic Image Refinement via Contrastive Learning and Consistent Semantic-Structural Constraints

SimGAN [2] was one of the pioneering papers that improve the realism of synthetic images using unlabeled real data, while preserving their annotation information with an adversarial network. It added an L1 self-regularization loss in the image pixel space to keep the annotations. Based on this method, Atapattu et al. [15] proposed an improved version that incorporated a perceptual loss term to further enhance the realism of the refined images. However, their methods only applied to gray-scale images with simple structures. Their performance in complex-scene datasets is limited.

CyclGAN [4], DualGAN [5], and DiscoGAN [3] assumed a bijective translation function and used the cycle consistency loss to enforce the consistency between images in the source domain, $X$, and the reconstructed domain, $F(G_{XY}(X))$, and vice versa. In this setting, $G$ is the mapping from the source domain, $X$, to the target domain, $Y$, and $F$ is the inverse mapping from the target domain, $Y$, to the source domain, $X$. These methods require an additional generator/discriminator pair. Additionally, they are restricted to bijective image translation. [16, 8, 11]. As a result, they may work for synthetic image refinement since the source and target domains may not contain the same scenes or content.

One-sided image translation methods were proposed as alternatives to the cycle-consistent constraint. DistanceGAN [6] learns the one-sided mapping by ensuring a strong correlation between pairwise distances calculated in the source and target domain. CcGAN [7] predefines a geometric transformation $f(.)$ so that $f^{-1}(f(x))=f(f^{-1}(x))=x$ and enforces geometry-consistency between an input image and its counterpart transformed by $f(x)$ in the new domain (i.e. $f(G_{XY}(x))\approx G_{XY}(f(x))$).

A few CL methods have been proposed to improve image translation tasks recently. CUT [8, 17] utilized mutual information maximization to learn a cross-domain similarity between patches from the source and the target domains. NEGCUT [11] trained a generator to produce negative samples online and replace the random negative samples, which effectively brought the positive and query samples closer. F-LSeSim [17] minimized fixed self-similarity and learned self-similarity to keep scene structures. The first loss is computed as a spatially-correlative map between input and output patches, and the second loss is computed as the contrastive loss among input, augmented input and target domain patches.

Several loss functions have been proposed to improve th CL model performance. One famous one is the InfoNCE loss [18], which describes the relation of correlated (positive) samples over both correlated and uncorrelated (negative) samples. Minimizing this loss function, while maximizing the lower bound of their mutual information, makes the feature space of positive samples more correlated and that of negative samples less correlated. This technique and the corresponding loss function have been popular, e.g., [19, 8, 20, 21].

Variants of InfoNCE have been investigated to further improve the performance. To give an example, the negative-positive coupling (NPC) effect reduces learning efficiency due to vanishing gradients caused by the loss function. This occurs when randomly selected negative patches are uncorrelated with positive ones. The decoupled infoNCE loss function [9] can address this problem by removing the positive pair term in the denominator. Another idea to improve the performance is mining hard negative samples, e.g., using adversarial training on negative samples [12] or applying a more informative negative sampling strategy based on the von Mises Fisher distribution [13].

We aim to design a CL-based method to refine synthetic images so that they appear more realistic to humans. The system consists of three parts.

• •

  As shown in the middle row of Figure 1, input synthetic images are fed into a generator with an encoder/decoder structure to generate refined output images. Afterward, the refined images are fed back into the encoder with the same weights.

• •

  As shown in the bottom row of Figure 2, the encoded features from different layers of synthetic and refined images are fed into a 2-layer MLP to yield a shared embedding space, denoted by $\mathcal{H}$. Since deeper layers in the encoder have larger receptive fields, the spatial positions of deeper features correspond to larger patches in synthetic images. Hence, different patches correspond to different feature vectors in the MLP feature space, and deeper layers’ features correspond to larger patches, Next, we propose the semantic-structural relation consistency (SSRC) loss to constrain semantic and structural distortions. Furthermore, we use the hard negative decoupled infoNCE loss (hDCE) to sample hard negative patches.

• •

  As shown in the top row of Figure 1, as synthetic images learn attributes from real-world captured images, the real-world images and refined images are fed to a discriminator to reduce their differences.

We adopt the following notation for the rest of this section. Let synthetic images $X\in\mathcal{X}$, refined images $\hat{Y}\in\mathcal{\hat{Y}}$ and real-world images $Y\in\mathcal{Y}$, where $\mathcal{X}$, $\mathcal{Y}$ and $\mathcal{\hat{Y}}$ are the source, target, and destination domains. Patches in their correspondings images are represented by $x\in X$, $y\in Y$ and $\hat{y}\in\hat{Y}$. Embedding vectors from MLP feature space of different patches are $z\in x$, $w\in y$ and $\hat{w}\in\hat{y}$.

The semantic-structural relation consistency (SSRC) loss contains two complementary components: 1) semantic relation consistency (SRC) and 2) structure consistency constraint (SCC). The SRC loss, $L_{SRC}$, is computed in the feature space to alleviate the semantic distortion, while the SCC loss, $L_{SCC}$, is computed in the pixel space to reduce the structural distortion. The SSRC loss is given by the weighted sum of $L_{SRC}$ and $L_{SCC}$:

We will derive $L_{SRC}$ and $L_{SCC}$ below.

If the negative patches are uncorrelated with the query patch (i.e., easy negative samples), the gradient of InfoNCE will be small due to a large denominator, which potentially causes gradient vanishing and affects learning efficiency. This is known as negative-positive coupling (NPC). It is desired to mine negative patches that are semantically correlated with the query patch (i.e., hard negative samples) and use a different loss function to alleviate the NPC effect. We re-sample negative samples with the von MisesFisher distribution [13] to obtain different latent classes and achieve large semantic similarity (inner product) between query and negative patches. Mathematically, we have

where parameter $\beta$ is a concentration parameter that controls the similarity of hard negative samples with query samples.

As discussed in Section 2, decoupled infoNCE (DCE) alleviates the NPC effect by removing the positive pair term in the denominator of InfoNCE [9]. With the negative sampling strategy [24], we can finetune the loss function furthermore as

where $N$ is the number of negative patches and $\tau$ is the temperature parameter that controls the strength of penalties on hard negative samples.

To ensure the distribution similarity between refined images in the target domain and real-world images in the destination domain, we include the standard adversarial loss in the loss function to obtain the ultimate loss. The standard adversarial loss is

The ultimate full objective fundtion is in form of

where $L_{SSRC}$, $L_{hDCE}$, and $L_{GAN}$ are given in Eqs. (1), (8), and (9), respectively, and $\lambda_{hDCE}$ and $\lambda_{GAN}$ are weights.

To quantitatively evaluate the performance of our method in refining synthetic images and its effectiveness in improving downstream semantic segmentation tasks, we train a segmentation network on the CityScapes dataset and then feed the refined synthetic data to it to predict the segmentation label maps. Then, we compute the accuracy and Intersection over Union (IoU) scores by comparing the predicted maps with ground-truth label maps of the synthetic dataset. High scores indicate its semantic consistency between input and output and thus benefit to segmentation tasks.

To be consistent with previous work in [14], we refine the first 500 images of the test set in the GTA5 dataset and feed them to FCN-8s [30] as the evaluation network pre-trained on CityScapes by pix2pix [31]. The network prediction is the segmentation label maps, and we compute the scores with groundtruth label map of GTA5. We compare the performance of our method with other methods in Table 1. As shown in the column “GTA5 $\rightarrow$ Cityscapes” of the table, our method offers the state-of-the-art performance with a substantial performance gap.

To further demonstrate the effectiveness of our method, we evaluate the domain mappers in CityScapes utilizing FCN scores. Specifically, we feed the ground truth segmentation label maps to our model to generate predicted images. Then, we use the generated images as the input to FCN-8s to predict the segmentation label maps. We compute the evaluation metrics between the predicted and ground truth labels. As shown in the column “Cityscapes parsing $\rightarrow$ Image” of Table 1, our method achieves state-of-the-art performance in this evaluation.

Figure 3 shows four representative testing images generated by four different methods, including Ours, CUT, CUT with the SCC loss only, and CUT with the SRC loss only. Our method offers superior visual results by preserving both semantic and structural consistency between input and output. In contrast, the other three benchmarking methods exhibit more semantic distortions, such as predicting the sky as trees (or buildings), or distorting other regions. Clearly, our proposed SSRC loss outperforms benchmarking methods using only the SCC loss or only the SRC loss.

The effectiveness of our method is also demonstrated on the more challenging Synthia dataset. Five refined images are shown in Figure 4. The refinement of Synthia to CityScapes is a challenging task as it involves differences in viewpoints, classes, and objects. For example, we have the top-down view in Synthia and the ground view in CityScapes. More humans and objects exist in the Synthia dataset. Our method can maintain realistic-looking images that resemble those in the CityScapes dataset after refinement.