Cross-View Image Synthesis with Deformable Convolution and Attention Mechanism

Cross-view image synthesis aims to translate images between two distinct views, such as synthesizing ground images from aerial images, and vice versa. This problem has aroused great interest in the computer vision and virtual reality communities, and it has been widely studied in recent years [2, 3, 4, 5, 6, 7, 8, 9]. Earlier work used encoder-decoder convolutional neural networks (CNNs) to study the viewpoint code included in the bottleneck representation for urban scene synthesis [10] and 3D object transformations [11]. Besides, when the view fields have little overlap or objects are occluded, and similar objects in one view may be completely different from another view (i.e., view invariance issues), this task will be more challenging. For example, the aerial view of a building (i.e., the roof) tells very little about the color and design of the building seen from the street-view. The generation process is generally easier when the image contains a single object in a uniform background. In contrast, when the scene contains multiple objects, generating other view becomes much more challenging. This is due to the increase in underlying parameters that contribute to the variations (e.g., occlusions, shadows, etc). An example scenario, addressed here, is generating street-view (a.k.a ground level) image of a location from its aerial (a.k.a overhead) image. Fig. 1 illustrates some corresponding images in the two different views.

To solve this challenging problem, Krishna and Ali [6] proposed a conditional GAN model that jointly learns the generation in both the image domain and the corresponding semantic domain, and the semantic predictions are further utilized to supervise the image generation. Although this method has been interestingly explored, there are still unsatisfactory aspects of the generated scene structure and details. Moreover, Tang et al. [12] recently proposed the multi-channel attention selection generation adversarial network (SelectionGAN), which can learn conditional images and target semantic maps together, and the automatically learned uncertainty map can be used to guide pixel loss to achieve better network optimization. However, we observe that there are still unsatisfactory aspects in the generated scene structure and details. For example, for the outline boundaries of some objects, there are obvious wrong marks and unclear.

To tackle this challenging problem, we add deformed convolution to the U-net network to improve the network’s ability to extract features of objects at different scales. At the same time, we use the attention mechanism [13] to refine the feature map to obtain a more detailed feature map for generating more realistic images. A large number of experiments show that our model can produce better results than state-of-the-art models, i.e., Pix2Pix [2], X-Fork [6], X-Seq [6] and SelectionGAN [12].

In summary, our contributions of this paper are as follows:

• •

  We employed the attention mechanism to refine the feature map to generate more realistic images for the challenging cross-view image translation tasks.

• •

  We also embed deformable convolutions in the U-net network to improve the network’s ability for extracting features of objects at different scales.

• •

  An additional loss function is added to improve the network training, thereby achieving a more stable optimization process.

Existing work on viewpoint transformation has been performed to synthesize novel views of the same object [14, 15, 16]. For example, Zhou et al. [16] proposed models learn to copy pixel information from the input view and uses them to retain the identity and structure of the object to generate a new view. Tatarchenko et al. [15] trained a network of codes to obtain 3D representation models of cars and chairs, which were subsequently used to generate different views of unseen images of cars or chairs. Dosovitskiy et al. [14] learned to generate models by training 3D renderings of cars, chairs, and tables, and synthesize intermediate views and objects by interpolating between views and models. Zhai et al. [17] explored the semantic layout of predicting ground images from their corresponding aerial images. They synthesized ground panoramas using the predicted layouts. Previous work on aerial and ground images has addressed issues such as cross-view co-localization [18, 19], ground-to-aerial geo-localization [20] and geo-tagging the cross-view images [21].

Compared with existing methods such as Restricted Boltzmann Machines [22] and Deep Boltzmann Machines [23], generative adversarial networks (GANs) [24] have shown the ability to generate better quality images [25, 26, 27, 28]. The vanilla GAN model [24] has two important components, i.e., the generator $G$ and the discriminator $D$. The generator $G$ aims to generate realistic from the noise vector, while $D$ tries to distinguish between real image and image generated by $G$. Although it has been successfully used to generate high visual fidelity images [26, 29, 30, 31], there are still some challenges such as how to control the image generation process under specific settings. To generate domain-specific images, the conditional GAN (CGAN) [28] has been proposed. CGAN usually combines vanilla GAN with some external information.

Krishna and Ali [6] proposed two structures (X-Fork and X-Seq) based on Conditional GANs to solve the task of image translation from aerial to street-view using additional semantic segmentation maps. Moreover, Tang et al. [12] proposed the multi-channel attention selection generation adversarial network (SelectionGAN), which consists of two generation stages. In the first stage, a cyclic semantically guided generation sub-net was proposed. This network receives images and conditional semantic maps in one view, while synthesizing images and semantic maps in another view. The second stage uses the rough predictions and learned deep semantic features of the first stage, and uses the suggested multi-channel attention selection the module performs fine-grained generation.

The network structure we proposed is based on the SelectionGAN model, which consists of three generators (i.e., $G_{i}$, $G_{a}$, $G_{s}$), two discriminators (i.e., $D_{1}$, $D_{2}$), and an attention mechanism module. The network structure can be divided into two stages, as shown in Fig. 2.

In the first stage, an image $I_{a}$ of one perspective and a semantic map $S_{g}$ of another perspective are input to the generator $G_{i}$ to generate an image $I_{g}^{\prime}$ of another perspective and the feature map $F_{i}$ of the last convolution layer. Then the generated image $I_{g}^{\prime}$ is input into the generator $G_{s}$ to generate the corresponding semantic map $S_{g}^{\prime}$.

In the second stage, the feature maps $F_{i}$ and $F_{s}$ generated in the first stage are refined through the attention mechanism module to obtain the refined feature maps $F_{i}^{\prime}$ and $F_{s}^{\prime}$. Next, they are combined with the image $I_{a}$ and the generated image $I_{g}^{\prime}$ and inputted to the generator $G_{a}$ to generate a refined image $I_{g}^{\prime\prime}$ as the final output. This refined image $I_{g}^{\prime\prime}$ is then input to the generator $G_{s}$ to generate the corresponding semantic map $S_{g}^{\prime\prime}$.

Note that we use only one generator $G_{s}$ in both the first and second stages, since the purpose is to generate a corresponding semantic image from an image.

We compute top-1 and top-5 accuracies in Table 1. As we can see, for lower resolution images ($64{\times}64$) our method outperforms the existing leading cross-view image translation methods. For higher resolution images ($256{\times}256$), our method also achieves the best results on top-1 and top-5 accuracies. This shows the effectiveness of our method and the necessity of the proposed modules.

Moreover, we provide results of SSIM, PSNR, and KL scores Table 2. We observe that the proposed method is consistently superior to other leading methods, validating the effectiveness of the proposed method.

In this paper, we propose a novel generative adversarial network based on deformable convolution and attention mechanisms for solving the challenging cross-view image generation task. We propose a novel attention mechanism to refine the feature maps, thus improving the ability of feature representation. We also embed deformed convolution in our generator to improve the network’s ability for extracting object features at different scales. Moreover, a novel semantic-guide adversarial loss is proposed to improve the whole network training, thus achieving a more robust and stable optimization. Extensive experimental results show that the proposed method obtains better results than state-of-the-art methods.