ReGO: Reference-Guided Outpainting for
Scenery Image

For an image $I$ from training set $X$, the corresponding sketch and the reference image are two necessary auxiliary data for our system. The sketch is treated as a conditional clue to guide the synthesis procedure as shown in Fig. 1, while the reference image is responsible to provide rich detailed features for the predicted new content.

To obtain the sketch, we first extract the edge map using the HED edge detector [17], and binarize the generated edge map with a pre-defined threshold (0.6 as adopted in our experiment), to acquire the binary sketch $S\in\mathbb{R}^{H\times W\times 1}$. In the training stage, we adopt the original sketches of the missing parts from training samples as inputs and force the network to restore the ground-truth parts. At the testing stage, users could input the manually drawn free-form sketches to synthesize desired results.

To obtain reference images for the input $I$, we first extract feature representations using a pre-trained Places365 [18] model, and then apply the cosine similarity to identify the similar samples. Based on our observation, the visual neighbors often share similar content with the target image and have more beneficial pixels, thus they are reasonable to be treated as the reference images. In our practice, we select multiple neighbors for each sample and randomly pick up one in each training iteration as the reference image $G$. Besides, since we only need to replenish the details of synthesized part, the right half of the reference image is only considered for further processing, as shown in Fig. 4.

With the reference image and the guiding sketch, the responsibilities of our ACS module are two-fold, i.e. , compensating the texture details of the synthesized new content and fusing the conditional information from the guiding sketch. Consequently, an image-guided convolution is designed for detail enriching, and a sketch fusing block is employed to integrate the sketch clue. Additionally, we further apply a seaming block to smooth the boundary between the original left feature and the predicted new representation. The details of each block are subsequently presented in the following.

Formally, let $F_{L}\in\mathbb{R}^{h\times w\times c}$ be the features encoded from the image to be extended, $F_{R}$ represents the hidden features for the predicted new content. $F_{L}$ and $F_{R}$ form the complete hidden features of the overall image $F\in\mathbb{R}^{h\times 2w\times c}$. And the features of $G$, which are encoded from a reference image encoder, are denoted as $F^{G}\in\mathbb{R}^{h\times w\times c}$. The designed image-guided convolution aims to complement $F_{R}$ by extracting helpful information from reference features $F^{G}$. A group of dynamic filters are conditionally produced based on the features of input and reference images, making the network adaptively collect the beneficial content from the reference image. To be specific, a dynamic kernel is produced based on the concatenation of $F^{G}$ and $F_{L}$ via a simple feed-forward procedure:

where $k\in\mathbb{R}^{3\times 3\times c}$, 3 and $c$ indicates the kernel size and channel number, respectively. The $\Psi$ can be modeled as the neural network, which takes the features of the reference and the input and recurrently use the convolution with stride=2, batch normalization and ReLU¹¹1ReLU is not applied in the last layer to get the features.

The dynamic kernel in Eq. 1 targets on providing guidance to distill the content of the reference image. To adaptively pick up the beneficial pixels and restrain the unhelpful content, we conduct the channel-wise normalization to update the dynamic kernel:

Thus, the distilled $i$-th channel map can be obtained as follow:

where the $*$ denotes the convolution operation, $P(\cdot)$ is an operation to repeat $k^{n}_{:,:,i}$ c times along the channel dimension. All of the $\widetilde{F}^{G}_{:,:,i}|^{c}_{i=1}$ are channel-wise stacked to get the distilled feature map $\widetilde{F}^{G}\in\mathcal{R}^{h\times w\times c}$.

The dynamic kernel in Eq. 2 focuses on automatically highlighting the beneficial features and suppressing the profitless content, we expect the network could assign lower enough weights for the profitless featurs while give higher weights for the helpful contents via the softmax operation. And the following distilled convolution in Eq. 3 attempts to greedily summarize the profitable semantic regions across feature channels from each dynamic kernel’s perspective. With such a procedure, the helpful features from the reference image are effectively aggregated into the map $\widetilde{F}^{G}$, which are further added to the feature $F_{R}$ to achieve the compensatory purpose: $F_{R}^{*}=F_{R}+\widetilde{F}^{G}$.

Besides the reference image, the input image itself could also provide valuable pixels. It is because the synthesized part contains the same objects as the input image with high probability. Therefore, the features from the input image are also integrated and the $F_{R}^{*}$ is updated as follow:

where $\rho(\cdot)$ is the horizontally flip operation, $\sigma(\cdot)$ denotes the sigmoid function, which is introduced to learn a dynamic feature selection mechanism.

Concretely, only the right half sketch $S^{r}\in\mathbb{R}^{H\times W/2\times 3}$ serves as the guiding clues, and its feature maps $F^{s}$ are first encoded by a sketch encoder $E^{S}$. Then, the compressed sketch features are channel-wise concatenated with the complemented feature $F_{R}^{*}$, and fed forward a residual block [21] style structure to get the fused output $F_{R}^{s}$.

The reference image adopted by ReGO is only expected to contribute texture details, its style should not be reflected on the synthesized content. To reduce the artifacts, the model should 1) only transfer the texture details from the reference image, 2) keep a consistent style between the given part and the generated part. Given the above considerations, we find the hinge based ranking is quite in line with our requirement, we take the synthesized part and the input image as the positive pair while treat the reference image as the negative sample, then the hinge ranking loss could conduct on their style representations to enforce the style of the input and the new content to be closer than the reference image.

Following previous practices [22, 24, 23], we utilize the second-order statistics of convolutional feature as style representation. Particularly, the style features of generated part $\hat{I}^{r}\in\mathbb{R}^{H\times W/2\times 3}$, which is only the right half of the image reconstruction $\hat{I}\in\mathbb{R}^{H\times W\times 3}$, is given by the Gram matrix $R^{d}\in\mathcal{R}^{N_{d}\times N_{d}}$:

where $M^{d}_{i}$ is the vectorised $i$-th feature map in layer $d$ from a convolutional neural network like VGG19 [27], $N_{d}$ indicates the channel number of layer $d$.

Analogously, the style representations of the reference image and the input image can be extracted, and our style ranking loss is defined as:

where $L^{d}$ and $G^{d}$ represent the Gram matrixs of the left input and the reference image, respectively, $SM(\cdot,\cdot)$ is the cosine similarity, $\alpha\in\mathcal{R}$ is the scalar margin, and $[\cdot]_{+}=max(\cdot,0)$.

By including the feature correlations of multiple layers, the multi-scale style representations are obtained, and the total style loss can be calculated accordingly:

where $D$ is the index collection of selected activation layers, and $w_{d}$ is the trade-off weight. In our experiments, the activated output of layer relu_Y_1(Y=1,2,3,4,5) of VGG19 network [27] are taken for style representation, i.e. , $|D|=5$. The designed style ranking loss is equipped to the generator loss to train the network.

For our study, the style ranking loss is equipped to the generator loss, and the weights of style ranking loss in multiple layers are all set as 0.2, i.e. , $w_{d}=0.2$. During the training stage, five neighbors are employed in our baseline methods, and the impact of neighbor number will be discussed in the following experiment. At the testing stage, only the most similar neighbor is used to synthesize the outpainting. Besides outpainting models, we also include three state-of-the-art inpainting models for comparison, i.e. DeepFillv2 [33], CoModGAN [35], and LaMa [34]. For LaMa and CoModGAN, we mask the right half images and introduce the sketch as an additional channel to train the network, while DeepFillV2 is a sketch-guided inpainting model, and it’s trained by only restoring the right half image. To make a fair comparison, the loss functions, the hyperparameters and the training details all follow the same settings of their original papers. The sketch augmentation strategy [9] is also employed for all methods to enhance the free-form outpainting.

Following Wang’s [9] setting, three metrics, i.e. , Fr$\acute{\rm e}$chet Inception Distance (FID) [29], the Inception Score (IS) [30] and Mean Satisfactory Degree (MSD) [9], are employed for evaluation. To evaluate the free-form outpainting results, we randomly select 555 images from test data and replace the original sketches with manually drawn free-form ones, 89 different types of sketches are collected in total. 20 volunteers are invited to label the free-form outpainting results as three levels: 0-poor, 1-ordinary and 2-good, and the mean value of all labels on selected images are taken as the mean satisfaction degree (MSD), which is taken for subjective comparison since there is no groundtruth available. Comparing to the FID and the IS, MSD directly reflects the performance in practical situations, therefore, it is a critical metric to evaluate the generalization ability on free-form sketches.

The performance of both sketch-guided outpainting and random outpainting on datasets NS6K and NS8K is reported in Table I, where $\text{ReGO}_{\text{NSIO}}$ denotes the NSIO model with our proposed ReGO equipped. From Table I, we can observe that our proposed ReGO module could simultaneously enhance the sketch-guided outpainting and the random outpainting.

In our ReGO, the searched neighbors of the input serve as the reference, while the input image itself is also an intuitive choice since it naturally shares the highest similarity with the input. To study the effects of two types of references, we conduct experiments with BDIE backbone on NS6K dataset to investigate the performance difference. Let $\text{ReGO}_{\text{BDIE}}$-SR (Self-Reference) denotes the $\text{ReGO}_{\text{BDIE}}$ with the input as reference. In our experiment, $\text{ReGO}_{\text{BDIE}}$-SR could also achieve acceptable performance on image rebuilding according to the original sketches, however, it shows poor generality when encountering the free-style sketches. For example, the FID of $\text{ReGO}_{\text{BDIE}}$-SR could reach 10.561 on NS6K and surpasses the method BDIE, however, its MDS for free-form outpainting is only 1.012, which is much worse than $\text{ReGO}_{\text{BDIE}}$. We guess the barren sketch layout and content pattern cause somewhat overfitting, consequently, the model trained with self-reference could not well generalize to the free-style outpainting. In contrast, When the neighbors serves as the reference, the model could see diverse training pairs, as a result, the trained model could perform well on both the image rebuilding and free-form outpainting.

The results are also reported in Table I, we can observe that abandoning the original guiding sketches significantly damnify the performance of the sketch-guided outpainting systems. For example, the $\text{ReGO}_{\text{SGIO}}$ with guiding sketches could reach 10.104 FID on NS6K, while its FID w/o the guiding sketches deteriorates to 15.396, this is because the systems are trained with the original sketches. The performance of SGIO with our ReGO module is comparable with its original method SGIO [9]. For the random prediction methods, $\text{ReGO}_{\text{NSIO}}$ performs slightly worse than the NSIO [3], while $\text{ReGO}_{\text{BDIE}}$ is more outstanding comparing to the STOA outpainting (BDIE [2]) and inpainting methods (LaMa [34]). From Table I, we can see that even though no guiding sketches are provided, the methods with our ReGO module could also produce comparable results with the original methods. With the designed ACS module, we can develop an unified framework that could simultaneously deal with the random prediction and the sketch-guided outpainting. what’s more, the BDIE with ReGO module could achieve the SOTA performance on both tasks.

As shown in Table II, when the reference image is introduced, the FID of BDIE could be improved from 11.02 to 10.99 and the MSD could also be boosted, which reveals compensating the texture details from the neighbors is a promising idea. However, only the reference image does not make the performance outstanding enough. With the proposed ACS module adopted, the network could adaptively block the profitless content and distill the beneficial pixels, and we can observe a significant performance improvement where the MSD and the FID could reach 1.221 and 10.269, respectively. When the style ranking loss is further equipped to ensure the style consistency, the performance step further. The model with three parts simultaneously utilized achieves the best performance, and when a new mechanism is equipped, the performance gets improved, which validates the contributions of each component.

Fig. 5 exhibits the visual results of the ablation comparison on image rebuilding and free-form outpainting. From Fig. 5, the contribution of each component can be clearly observed.

Table III shows the performance under IS, FID, and MSD, where $\text{ReGO}_{\text{BDIE}}$-Reg indicates $\text{ReGO}_{\text{BDIE}}$ using the $l_{2}$ style reconstruction loss instead of the style ranking loss. From Table III, the $\text{ReGO}_{\text{BDIE}}$ with the proposed style ranking loss performs much better on both image restoring and free-form outpainting. The reasons for such results may stem from the overfitting. Although the left half and the right half parts are from the same image, the style representations captured by the Gram matrices are still different. Therefore, stiffly conducting such a regression procedure is easy to cause overfitting. Besides, the main goal of our style ranking loss is to block the style of the reference image being reflected on the extended content, the style consistency between the input and the synthesized part is not necessary to enhance, since it can be achieved by the pixel-wise reconstruction and the adversarial training [2, 3, 9].

The performance tendencies with five different reference numbers are shown in Fig. 6, where the FID is scaled by the logarithmic function. As shown in Fig. 6, there are two important observations. First, Comparing to employing only one reference image, using multiple references could enhance the model generality and train a more robust generation model. The FID of $\text{ReGO}_{\text{BDIE}}$ with only one reference image is 10.728, when the reference number increases to 5, the FID could be improved to 10.052. Second, more reference images do not make the performance step further, which is clear from the performance tendencies when the reference number ranges from 5 to 20. The situation with five reference images achieves the best performance on average.

From Fig. 7 I, the comparison methods, LaMa [34], SGIO [9] and BDIE [2], could extend reasonable pixels for the input image, but the predicted content is blurry and lacks textural details, which makes the overall image not authentic enough. While $\text{ReGO}_{\text{BDIE}}$ could produce texture-rich outpainting results. The results of random prediction are exhibited in Fig. 7 II, comparing to the competing methods, $\text{ReGO}_{\text{BDIE}}$ could also successfully synthesize the results with more textural details when no sketches are fed, and the synthetic images are even more satisfactory than the method original designed for the random prediction, i.e. , BDIE [2]. From Fig. 7, we could find that the guiding sketch is not one of requisite inputs for our system, when the users do not provide the guiding sketch, ours system could also produce satisfactory random outpainitng results.

From the above, the proposed framework allows users to harvest three types of results: random outpainting, free-form outpainting from manually drawn sketches and controllable outpainting using sketch from another image. Therefore, our proposed method is with higher practical value. We also show some failure cases in Fig. 11, the model fails to predict reasonable pixels or generate natural enough contents for the guiding sketches. In our experiment, we find most of failed samples are from those whose guiding shapes are in the top part of the images, we guess the reason stems from the barren sketch patterns of training images. This reveals that there are still many challenges standing for this task. In the future, we will continue to explore this task and attempt to address more issues of this field.