Semantic-Aware Network for Aerial-to-Ground Image Synthesis

Our aerial-to-ground synthesis network, i.e., generator, is composed of an aerial encoder $E_{a}:I_{a}\rightarrow f_{a}$ for mapping aerial image into feature space, a semantic-attentive feature transformation module $T_{ag}:f_{a}\rightarrow f_{ag}$ for modeling the structural changes of feature, and a decoder $G_{g}:f_{ag}\rightarrow I_{g}^{\prime}$ for synthesizing the ground image. During training, we adopt an auxiliary ground encoder $E_{g}:I_{g}\rightarrow f_{g}$ that maps ground image into feature space, and a pre-trained segmentation network $S_{\text{seg}}$ that extracts semantic map from ground images.
Semantic-Attentive Feature Transformation Module. The proposed semantic-attentive transformation module $T_{ag}$ is illustrated in Fig. 2(a). It learns a structural transformation from $f_{a}$ to $f_{ag}$, where $f_{ag}$ has structure aligned to $I_{g}$. Rather than solely depending on an implicit learning of the transformation [10], we perform initial coarse alignment using polar transformation [17] to generate $f_{p}$.

Since objects with different classes are likely to locate in different areas [18] (e.g., sky occupies the upper part of an image whereas road occupies the bottom), we employ semantic-attentive transformation that separately handles alignment of objects in different class. Specifically, we generate channel attentions [13] $\{M_{i}\}_{i=1}^{c}$ for $c$ semantic classes and apply them to $f_{p}$. For further alignment, we feed them into subsequent warping blocks $\{D_{i}\}_{i=1}^{c}$, each of which consists of a deformable convolution layer [19] and a convolution layer. By summing all the attentively aligned features $\{f_{ag}^{i}\}_{i=1}^{c}$, the final semantic-attentive transformed feature $f_{ag}$ is obtained.
Auxiliary Networks. The ground encoder $E_{g}$, along with $G_{g}$, operate as an autoencoder [20], i.e., $E_{g}$ extracts feature $f_{g}$ from $I_{g}$ and $G_{g}$ reconstructs the ground image $I_{g}^{\text{rec}}$. It encourages $E_{g}$ to extract rich ground-specific feature $f_{g}$ with sufficient representations to reconstruct ground images. We use $f_{g}$ as a reference for $f_{ag}$, thereby guide $E_{a}$ to extract representative features for synthesizing $I^{\prime}_{g}$, as well as $T_{ag}$ to better model the structural transformations without 3D information [12]. We use separate parameters for $E_{a}$ and $E_{g}$ to encourage higher flexibility and capacity in feature extraction [21].

A pre-trained semantic segmentation network $S_{\text{seg}}$ takes ground image $I_{g}$ as an input and outputs a semantic segmentation mask $S_{g}$. We use it to improve the semantic-awareness of the generator through semantic-aware losses, presented in the following section.

Semantic-Aware Synthesis Loss. In order to synthesize plausible ground images, objects across various classes should be considered. Here, we observe uneven number of pixels for different objects in the cross-view image datasets [14, 15] as reported in Fig. 5. This is mainly due to the different sizes of the objects and a prevailing number of scenes without a man-made object. It leads the regular L1 loss between the ground-truth and synthesized images to be dominated by the prevailing objects. Consequently, the model often fails to synthesize the objects across various semantic class.

To alleviate this problem, we balance the losses across the semantic classes by exploiting $S_{g}$. Concretely, we compute L1 loss for each class independently using the segmentation mask and average them with their own number of pixels as illustrated in Fig. 2(b). Our novel semantic-aware synthesis loss is defined as

$$\mathcal{L}_{\text{Syn}}=\sum\limits_{i=1}^{c}{\frac{w_{i}}{N_{i}}\left\|S_{g}^{i}I_{g}-S_{g}^{i}I_{g}^{\prime}\right\|_{1}},$$

(1)

where $N_{i}$ and $S_{g}^{i}$ are the number of pixels and class-mask for class $i$. We further use $w_{i}$ as class balancing weight [22, 23] to handle a prevailing number of scenes without a man-made object.
Semantic-Aware Feature Loss. Similar to semantic-aware synthesis loss, we use downsampled $S_{g}$ and apply separate loss functions for each semantic-attentive branch in the transformation module as

$$\vspace{-3pt}\mathcal{L}_{\text{Fea}}=\sum\limits_{i=1}^{c}{\frac{w_{i}}{N_{i}}\left\|S_{g}^{i}f_{g}-S_{g}^{i}f_{ag}^{i}\right\|_{1}},$$

(2)

so that each channel attention is learned to highlight class-specific features and enables an effective learning of the transformation.
Semantic Consistency Loss. To enforce the semantic consistency in the synthesized image, we constrain the semantic differences between the synthesized image and the ground truth image, defined as

$$\mathcal{L}_{\text{Sem}}=\left\|S_{g}-S_{g}^{\prime}\right\|_{1},$$

(3)

where $S^{\prime}_{g}$ is output of $S_{\text{seg}}$ with $I^{\prime}_{g}$ as input. Different from the previous works [1, 5, 12], the parameters in $S_{\text{seg}}$ are fixed when training the generator. It further encourages the synthesized objects to have similar appearance as the ground truth images $S_{\text{seg}}$ is trained on.
Ground Image Autoencoding Loss. In order to train $E_{g}$ to extract ground-representative features $f_{g}$, and $G_{g}$ to synthesize realistic ground images upon the feature, we adopt L1 reconstruction loss between $I_{g}$ and $I_{g}^{\text{rec}}$ as

$$\mathcal{L}_{\text{AE}}=\left\|I_{g}-I_{g}^{\text{rec}}\right\|_{1}.$$

(4)

Adversarial Loss. Similar to the previous works for image synthesis [24, 25], we encourage the synthesized images to be indistinguishable from the real images by adopting a discriminator $D$ and applying an adversarial loss as

$$\mathcal{L}_{Adv}=\mathbb{E}_{I_{g}}\log D(I_{g})+\mathbb{E}_{I_{g}^{\prime}}\log(1-D(I_{g}^{\prime})).$$

(5)

Network Architecture. We adopt the architecture from Zhu et al. [25] for our generator and discriminator. Specifically, we compose the encoders and decoder with 4 and 5 residual blocks [26] respectively. We use SegNet [27] architecture for $S_{\text{seg}}$.
Datasets. We conduct experiments on two commonly used cross-view image datasets, CVUSA [14] and CVACT [15], each containing 35,532/8,884 and 35,532/92,802 train/test image pairs. We apply pre-processing to adjust aerial images into a similar scale and exclude ground image areas with large panoramic distortions. Then we resize the aerial and ground images into $256\times 256$ and $128\times 512$. For both datasets, we use $S_{\text{seg}}$ trained on CVUSA dataset, with the segmentation maps provided by [10] as pseudo label which contains four classes of sky, man-made, road and vegetation.
Training Details. We set the class balancing weights $w_{i}$ in (1) and (2) as $0.5,2,1,$ and $1$ for sky, man-made, road, and vegetation classes, respectively. We aim to handle the scarcity of the man-made class and avoid the network being overfitted to the ground-truth sky representation. We set the loss weights in (6) as $\lambda_{\text{Syn}}=10,\lambda_{\text{Fea}}=2,\lambda_{\text{Sem}}=2$, and $\lambda_{\text{AE}}=5$. We use Adam optimizer [28] with momentum parameters 0.5 and 0.999, and fixed learning rate of 0.0002. The network parameters are initialized with normal distribution with zero mean and 0.02 standard deviation. We do not perform any data augmentation and train our network for 30 epochs with batch size of 4. All the experiments are conducted using Pytorch [29] library, on a single NVIDIA RTX 2080Ti X GPU.
Evaluation Protocols. For quantitative evaluations, we follow the protocols presented in [5]. We measure the visual quality of the synthesized images by Peak-Signal-to-Noise Ratio (PSNR), Structural-Similarity Index (SSIM), and Sharpness Difference (SD) with ground-truth image. We also measure the realism and diversity of the synthesized images by Inception Score (IS), Top-k prediction accuracy, and KL divergence. We additionally evaluate the pixel-wise semantic consistency of the synthesized images using mean Intersection-over-Union (mIoU).

We compare the proposed method with Pix2Pix [24], X-Fork [5], and X-Seq [5]. Since these methods handle input and output images of same shapes, we apply few modifications. We change the bottleneck kernel size from (4, 4) to (1, 4) and use unconditional discriminator. We also remove the skip connections in Pix2Pix [24]. Except the above modifications, we follow the original settings.

We present qualitative results in Figs. 3 and 4. Compared to the previous methods [24, 5], we observe that our method shows the most visually plausible results. Specifically, our results show the clearest image appearance and consistent layout with the ground-truth images. It demonstrates that our semantic-attentive feature transformation module successfully aligns the intermediate features to the ground layout by focusing on every different semantic class. We also observe that the proposed method synthesizes plausible result across various objects, while others failed (e.g., buildings). This confirms that our semantic-aware loss functions allow the network to handle objects across various classes.

Quantitative results are presented in Tables 1 and 2. The proposed method outperforms the previous methods in all the quantitative measures except for PSNR. Although our PSNR results are slightly lower compared to X-Fork [5], we achieve higher visual quality scores (KL and IS), showing that our method generates more realistic images. In addition, we can see that our method results in the highest mIoU score which verifies that our framework generates the most semantically consistent images with the ground truth.

To investigate the importance of the key components in our framework, we conduct experiments on our method without $T_{ag}$, without {$\mathcal{L}_{\text{Syn}}$, $\mathcal{L}_{\text{Fea}}$}, and without $\mathcal{L}_{\text{Sem}}$ on CVUSA dataset. Results are presented in Table 3 and Fig. 6.

For the setup w/o $T_{ag}$, we only apply polar transformation to $f_{a}$ and do not use any loss function for the transformed feature. For the setup w/o {$\mathcal{L}_{\text{Syn}}$, $\mathcal{L}_{\text{Fea}}$}, we use regular L1 loss for images and features. We observe that the networks with those setups generate the synthesized images that have unclear structure alignment and are not realistic, compared to our full framework. This observation coincides with the quantitative results, indicating that both components largely affect the results and should be applied cooperatively. For the setup w/o $\mathcal{L}_{\text{Sem}}$, both the qualitative and quantitative results show minor difference from our full framework. It demonstrates that our transformation module, together with semantic-aware loss functions sufficiently align the structure and enhance semantic awareness to the network, thereby enforcing the semantic consistency in the synthesized image.