Memory-guided Unsupervised Image-to-image Translation

To read the appropriate style values, we compute the similarity between each $\mathbf{c}_{p}$ and $\mathbf{k}$, resulting in a read-weight matrix $\alpha^{x\ (\text{or}\ y)}\in\mathbb{R}^{P\times N}$ :

where $\mathbf{c}_{p}$ denotes individual features ($p=1,\cdots,P$) of size $1\times 1\times C$, and $P$ is the total number of pixels in $\mathbf{c}$. $d(\cdot,\cdot)$ is defined using cosine similarity as follows:

Inspired by [27], we read the memory item by taking a weighted average of the cross-domain values:

This step is repeated for all $\mathbf{c}_{p}^{x\ (\text{or}\ y)}$, and produces an enhanced and aggregated style feature map $\hat{\mathbf{s}}^{y\ (\text{or}\ x)}$²²2Specifically, $(\mathbf{s}_{1},\cdots,\mathbf{s}_{k})$ are separately processed with the corresponding $N_{k}$ memory items assigned to class $k$ (see Fig. 3), and then merged into $\hat{\mathbf{s}}$ using their original coordinates in $\mathbf{s}$.. Through (4), our model can transfer class-aware and spatially varying style information across domains ($\mathcal{X}\leftrightarrow\mathcal{Y}$) by referring to their content characteristics. The translated images ($\hat{\mathbf{I}}^{x},\hat{\mathbf{I}}^{y}$) can be obtained according to (1).

To enrich the memory items, we also select and store class-aware features into the memory while removing redundant features from the memory. Similar to the read operation, we calculate an update-weight matrix $\beta^{x\ (\text{or}\ y)}\in\mathbb{R}^{P\times N}$ between $\mathbf{c}$ and $\mathbf{k}$:

where we apply the softmax function along the $\mathbf{c}$-direction, as opposed to (2). The update-weight matrix $\beta$ is used to assign the extracted content $\mathbf{c}$ and style features $\mathbf{s}$ to the relevant memory item. The items $(\mathbf{k}_{n},\mathbf{v}^{x}_{n},\mathbf{v}^{y}_{n})$ are updated using ($\mathbf{c}_{p},\mathbf{s}_{p}$) weighted by $\beta$ as follows:

We utilize both ($\mathbf{c}^{x}_{p},\mathbf{c}^{y}_{p})$ to update $\mathbf{k}_{n}$ because it records the shared content representations. In contrast, the domain-specific values ($\mathbf{v}^{x}_{n},\mathbf{v}^{y}_{n})$ are updated individually. We train the memory with a large number of images and ground-truth object annotations, thus enabling the most representative and discriminative features to be stored.

At test time, we compute $\alpha$ for all memory keys $\mathbf{k}$ without considering class information and retrieve the style values using (2) and (4). We find that this strategy still works well because our memory is discriminatively trained using ground-truth object annotations.

Our networks were implemented based on DRIT³³3https://github.com/HsinYingLee/DRIT with PyTorch [34] and trained on one single NVIDIA TITAN RTX GPU. Every network weights of each layer are initialized by a Gaussian distribution with a zero mean and a standard deviation of 0.001. The Adam solver [18] was employed for optimization, where $\beta_{1}=0.9$, $\beta_{2}=0.999$. The batch size was set to 1. The initial learning rate was set to 0.0001 and 1, kept for first 30 epochs, and linearly decayed to zero over the next 30 epochs. We set the number of memory items as 20 and its channel $C$ as 256. We resize the short side of images to 360 pixels and crop it to $360\times 360$ to train our framework. The hyperparameters $\{\lambda^{self},\lambda^{cyc},\lambda^{adv}_{c},\lambda^{adv}_{d},\lambda^{KL},\lambda^{latent}\}$ for I2I translation network are set the same as DRIT [50], and $\{\lambda_{k}^{con},\lambda_{v}^{con}\}$ are empirically determined 1 and 0.5. Our code will be made publicly available.

We conduct experiments on three datasets.

(1) INIT dataset [38] consists of 155K street scene images, including 4 domain categories (sunny, night, rainy, and cloudy). It provides instance bounding box and object class annotations for car, person, and traffic sign. We set the number of memory items for each class as 5, 3, and 2, and for the background as 10. Following INIT [38], we use 85$\%$ images for training and 15$\%$ images for testing. We conduct three translation experiments for sunny$\leftrightarrow$night, sunny$\leftrightarrow$rainy, and sunny$\leftrightarrow$cloudy.

(2) KITTI object detection benchmark [7] and Cityscapes dataset [4] are used to demonstrate that our method can help with domain adaptation. KITTI benchmark [7] contains 7,481 images for training and 7,518 images for testing, and it provides the bounding boxes for 6 object classes. Cityscapes dataset [4] is widely exploited for semantic segmentation, which consists of 5,000 images with pixel-level annotations for 30 classes. These datasets are used to conduct the domain adaptation for object detection (KITTI $\rightarrow$ Cityscapes case). To integrate two datasets’ the object classes, we set the common 4 object classes as person, car, truck, and bicycle. We build 3 memory items for each class, including 8 background memory items.

We perform the evaluation on the following methods.

• •

  CycleGAN [49] and UNIT [23] are the typical unsupervised I2I translation methods.

• •

  MUNIT [13] and DRIT [22] are the multi-modal unsupervised I2I translation methods that are extensions of CycleGAN [49] and UNIT [23]. Especially, we exploit DRIT [22] as our baseline model.

• •

  INIT [38] and DUNIT [1] are the existing instance-level unsupervised I2I methods. These methods are compared only for quantitative evaluation and not included in the qualitative comparison, since their code (parameters) is not publicly available.

Following the experimental protocol of existing unsupervised I2I translation methods, we evaluate our methods with Inception Score (IS) [37], Conditional Inception Score (CIS) [13], and LPIPS Metric [48].

Fig. 4 shows a qualitative comparison of the state-of-the-art methods. We observe that the multi-modal I2I methods MUNIT [13] and DRIT [22] fails to capture instance details and boundaries well. As these methods do not have any access to semantic information, they tend to translate instances to the other semantic styles (e.g., translating buildings into the sky). Our method produces the most visually appealing images with more vivid details. Thanks to the proposed class-aware memory network, it shows high capacity to better understand the semantic instances and employ the appropriate local style representation for object classes. We compare the translated results with instance-level I2I method DUNIT [1] in Fig. 5. Our result yields sharper and distinctive instances and more realistic images. Lastly, we visualize the multimodal translated results in Fig. 6. We use the stored key $\mathbf{k}$ in the memory and randomly sampled values $(\mathbf{v}^{x},\mathbf{v}^{y})$. It can be observed that the degree of color (\egroad, sky) changes across these images.

We conducted a user study to compare subjective quality of the translated results. For each translation, we randomly select 10 images from INIT validation to set up a total of 80 images for comparison. From 25 participants, we asked to rank all the methods in terms of the image quality and style diversity of the translated image, and we received a total of 2,000 votes. Fig. 7 shows the results, and our method ranks first in 77.9$\%$ for the image quality and 64.5$\%$ for the style diversity.

Table  1 shows the IS [37] and CIS [13], and Table 2 shows average LPIPS metric [48]. The IS measures the diversity of output images based on the Inception V3 model [41]. The CIS quantifies the quality and diversity of output conditioned on a single image. Additionally, the LPIPS metric [48] measures the translation diversity by calculating the similarity between two different deep features from the pre-trained AlexNet [19]. The results indicate significant performance gains with our method in all metrics. It further highlights the contribution of class-aware memory network to the improved performance.

We test our method for the domain adaptive object detection. Using the Faster-RCNN [35] trained on images in the source domain, we evaluate the detection performance of the translated images from source to target domain. Following DUNIT [1], we conduct experiments on the KITTI object detection benchmark [7] as the source domain and Cityscapes dataset [4] as the target domain. We compare the performance to state-of-the-art domain adaptation methods [14, 2, 17, 36] and DUNIT [1] with instance consistency loss (w/ IC) and without (w/o IC). Note that the instance consistency loss enforces the consistency constraints between results detected from original and translated image. We report the mean average precision (mAP) for the detected objects in Tab. 3. Our method performed well in the domain adaptive object detection tasks without explicitly using the object detection network. Unlike DUNIT [1] that improves performance by applying direct constraints on detected results, our method can recognize the semantic information contained in images thanks to our highly discriminative class-aware memory network. Consequently, it allows the image that are translated into an appropriate object style while preserving its inherent semantic information. Furthermore, it demonstrates that our method can realize more complex domain adaptation tasks.

The results using the single memory (in Figure 8 (b), (c)) cannot preserve the instance boundaries well, and even small instances disappear into the background. On the other hand, the results using the class-aware memory (in Figure 8 (d), (e)) show clear and well-preserved instance structures. The quantitative results from Table 4 also indicate that the translated images using the class-aware memory are more realistic.

We observe that the results using the feature contrastive loss (in Figure 8 (c), (e)) are more vivid and represent a style that is appropriate for each instance compared to the results using the feature triplet loss (in Figure 8 (b), (d)). To investigate its effect, we visualize the distribution of the content features, which are learned with the triplet loss in Fig. 9 (a) and with the contrastive loss in Fig. 9 (b). Specifically, we project the embedded content features from the test images into 2-dimensional space using t-SNE [24]. The color indicates the memory items, which means that the points with the same color are mapped to the same item. The contrastive loss is more effective in separating and clustering the feature semantically. Therefore, it enhances the diversity and discriminative power of our memory items.