Given an input image $X\in\mathbb{R}^{{h}\times w\times c}$, extension filling map $M\in\mathbb{R}^{{h}\times w\times 1}$ and smooth structure $S\in\mathbb{R}^{{h}\times w\times c}$ [11], our method intends to generate a visually convincing image $\widehat{Y}\in\mathbb{R}^{h\times w^{{}^{\prime}}\times c}$ where $w^{{}^{\prime}}$ is desired width of the generated image. SieNet follows the coarse-to-fine two-stage pipeline [12]. As shown in Fig. 2 (a), an uncovered smooth structure $S_{gen}\in\mathbb{R}^{{h}\times w^{{}^{\prime}}\times c}$ is produced by the structure generator first. Then the generated structure feature $S_{gen}$ would combine with image $X$ and filling map $M$, acting as the input of second generator, i.e., content generator, to generate outstretched image $\widehat{Y}$. Besides, the discriminator $D$ to ensure the output of generator and actual data conform to a certain distribution.

As illustrated in Fig. 2 (b), the encoder of structure generator downsamples the input with 8$\times$ scale. Then to capture multi-scale information two residual blocks are designed to further proceed the output of encoder. Finally, by three nearest-neighbor interpolation, the output of residual blocks is upsampled to desired resolution. The detailed structure of content generator is similar to that of structure generator, except adding two residual blocks before last two upsampling operations. The discriminator $D$ follows the structure and protocol of that of BicycleGAN [13, 14] .

To endue encoder with predicting ability, we propose a boundary sensitive convolution, i.e., adaptive filling convolution. Given a convolutional kernel with $K$ sampling locations (e.g., $K=9$ in a $3\times 3$ convolution), let $w_{k}$ and $m_{k}\in[0,1]$ denote the weight of padding convolution and mask modulation for $k$-th position. $x(p)$ and $y(p)$ represent the feature map at location $p$ from input and output respectively. Therefore, the filling convolution could be formulated as follows:

In practical learning , the scheme of our filling convolution is shown as Fig. 3. Two separate convolutions model padding weight $w_{k}$ and mask modulation $m_{k}$ respectively where $m_{k}$ is defaulted to 1. Our filling convolution could predict the pixel outside the image boundary. When kernel center locates just outside known content, only scant real pixels would join the patch calculation. Our padding convolution would predict it by known information with $w_{k}$, and another skip connection from input to output would keep characteristic value to avoid high variance. Besides, mask modulation $m_{k}$ works as a adaptive balance factor of these two branches, to guarantee the smoothness and characteristic of prediction. We insert a $7\times 7$ filling convolution into the bottleneck of encoder, therefore, encoder could possess the ability of capturing context and the ability of predicting unknown content.

Classical adversarial mechanism could constrain the whole generator toward the real case. However, this constraint is implicit to each part of generator. Especially in two-stage GAN [15, 16, 17] for image outpainting, long range encoding may lead to insufficient inferring ability for decoder. Considering the predicting ability of encoder brought by filling convolution, we further add explicit constraint in encoder to push the features of covered and uncovered image to be common. Namely, prior knowledge of uncovered image is learned by encoder. In addition, we also take advantages of adversarial mechanism in our model to constrain the generated results to be consistent with ground truth [18, 19]. Through these two thoughtful constraints brought by our siamese adversarial mechanism, the predicting burden is well regulated.

Let $G_{strut}$ denotes structure generator and $E_{cont}$ denotes encoder of content generator, $I=[X,M,S]$ is the input of the whole network. The output $F$ of long range encoder $E_{cont}(G_{strut}())$ could be formulated as:

Therefore, let superscripts ${}^{{}^{\prime}}$ and ^gt denote whom is generated by covered input and ground truth input respectively, the siamese loss $L_{siamese}$ is ${\ell 2}$ loss of $F^{{}^{\prime}}$ and $F^{gt}$:

In order to generate realistic results, we introduce adversarial loss $L_{adv}$ in the structure generator:

We introduce $L_{distance}$ loss to predict the distance between the structures $S_{gen}^{{}^{\prime}}$ and $S^{gt}$:

Besides, the perceptual loss $L_{perctual}$ and the style loss $L_{style}$ [20] are applied into our network. $L_{perctual}$ is defined as:

$\phi_{i}$ is the activation map of the $i$-th layer of the pre-trained network. In our work, $\phi_{i}$ is activation map from the layers of relu1-1, relu2-1, relu3-1, relu4-1 and relu5-1 of the ImageNet pre-trained VGG-19. These activation maps are also used to calculate the style loss to measure the covariance between the activation maps. Given a feature map size C_j $\ast$ H_j $\ast$ W_j, the style loss $L_{style}$ is calculated as follows.

$G^{\phi}$ is the the reconstructed Gram metric based on activation map $\phi_{j}$. Totally, the overall loss $L_{total}$ is:

where $\lambda_{dist}$, $\lambda_{adv}$, $\lambda_{p}$, $\lambda_{s}$ and $\lambda_{sie}$ are set 5, 1, 0.1, 250 and 1 respectively.

Our method is flexible for two-direction and single-direction outpainting. For two-direction outpainting, our method is evaluated on three datasets, i.e., Cityscapes [21], paris street-view [2] and beach [9]. Single-direction evaluation is made on Scenery dataset [22]. Besides, we take structural similarity (SSIM), peak signal-to-noise ratio (PSNR) as our evaluation metrics.

In training and inference, the images are resized to $256\times 256$. We train our model using Adam optimizer [23] with learning rate $0.0001$, beta1 $0$ and beta2 $0.999$. The batch size is set to 8 for paris-street view and beach, 2 for Cityscapes, and 16 for Scenery. The total iteration of joint training is $10^{6}$ for four datasets.

To validate the effectiveness of filling convolution and siamese adversarial mechanism in our network, We make the ablation experiment in Tab. I. Only using filling convolution or siamese adversarial mechanism could get a significant improvement over classical encoder-decoder. And the combination of them could achieve the best performance, indicating the regulation of predicting burden is beneficial for the task. We also visualize the results of street and nature scenery to measure the quantitative capability of them. As shown in Fig. 4, the combination of these proposed two parts could recover the details and generate realistic results in easy nature and complicated street cases.

Our method is compared with existing methods including Image-Outpainting [9], Outpainting-srn [8], Edge-connect [24] and NS-outpainting [22]. As shown in Tab. II, for two-direction outpainting, our method outperforms Image-Outpainting [9], Outpainting-srn [8], Edge-connect [24], and achieves state-of-the-art performance in three datasets. For single-direction, the great margin of performance between our method and NS-outpainting [22] indicates the superiority and generality of our SiENet. Besides, we also provide visual comparison of various methods in Fig. 5 and visual performance of our method on four datasets in Fig 6. Notably, the images generated by our method are similar to the ground truth than that of existing methods.