Outpainting by Queries

Transformer Encoder Our encoder is a standard ViT [9]. Inspired from ViT, the input image is first converted to several non-overlapping patches represented as a sequence of patch tokens $\mathrm{X}_{p}$. The encoder module embeds the patch tokens through a linear projection $\mathbf{E}$ with the added positional embeddings $\mathbf{E}_{pos}$. Then the encoder processes the set of patch tokens via a series of Transformer Blocks with a length of $N$. The transformer-based encoder can be described as follows:

where $D$ is the hidden dimension of transformer block, $\mathrm{FFN}$ is a feed forward network, $\mathrm{LN}$ denotes layer normalization, $\mathrm{h}_{n}$ are the intermediate tokens’ representations, $\mathrm{h}_{enc}$ denotes the output patch tokens of the transformer encoder, and MSA represents the multi-headed self-attention.

Query Expansion Module The proposed QEM is designed to speed up the convergence of pure transformer by generating the expanded queries for the transformer decoder. We predict the decoders’ queries conditioned on encoders’ features, and take advantage of CNN’s inductive bias to accelerate the convergence. As shown in Fig. S3(b), the input tokens $\mathrm{h}_{enc}$ are first reshaped to the feature map with the size of $\frac{H}{P}\times\frac{W}{P}\times D$. Then the reshaped feature maps are extrapolated with extra $\frac{M}{P}$ pixels along width and height, where the padded tokens are generated by Multi-layer Perceptual (MLP) with uniform input noise. After that, we utilize stacked residual blocks [18] equipped with deformable convolutional layers [48] to process the queries, which is commonly practiced to capture local and long-term dependencies. Finally, the expanded queries are extracted and transformed as sequence, followed by one Normalization Layer and one Linear Layer. This process can be described as:

Transformer Decoder Inspired from the original transformer [41], the decoder equips one extra sub-layer which performs the multi-head cross attention (MCA) similar to the encoder with two sub-layers. Specifically, in MCA the queries come from the previous decoder layer and the keys and values come from the output of the encoder. This allows each position in the decoder to attend over all positions in the input sequence, leading to significant improvements of the generating performance. The process can be described as follows:

The multi-headed cross-attention in Eq. 10 aggregates information from $H$ cross attention heads, as follows:

Patch Smoothing Module The linear module is prone to generate artifacts if predicting output patches using predefined patch size of $P\times P$. The reason is that the output tokens are processed independently without explicit constraints. These arbitrary grid partitions could make the image contents discontinuous across the border edge of each patch. In order to mitigate this issue, we allow some overlaps among image patches. For each border edge of one patch, we extend it by $o$ pixels generating the output image patch size as $(P+2o)\times(P+2o)$. This operation involves the decoder with the neighboring patches’ content having a better sense of locality in the transformer architecture, thus enabling the output sequence to have same length but less effect as the predefined grids. PSM can be described as:

where $\mathcal{S}$ is a function to place the extrapolated overlapped patches around the input image, and average the pixel values in the overlapped areas.

Patch-wise Reconstruction Loss We utilize an L2 distance between the sequence of ground truth image patches $\{\mathrm{y}_{p}^{L+1},\mathrm{y}_{p}^{L+2},\cdots,\mathrm{y}_{p}^{L+R}\}$ and the sequence of predicted image patches $\{\mathrm{x}_{p}^{L+1},\mathrm{x}_{p}^{L+2},\cdots,\mathrm{x}_{p}^{L+R}\}$:

where the patch size is $(P+2o)\times(P+2o)$. We engage a per-patch normalization to enhance the patch contrast locally, where the mean and std of the image patches are pre-computed.

Perceptual Loss Perceptual loss provides a supervision on the intermediate features that can help retain more semantic information. Following previous work [10, 21, 24], we extract the features from a VGG-19 [38] network pretrained on ImageNet [7], which is denoted as $\phi$. The perceptual loss is devised as follows:

where the superscript $j$ is the index of feature map scales from $\phi$, and $w^{j}$ is set to $1/32,1/16,1/8,1/4,1$ as the scale decreases.

Adversarial Loss We utilize the same multi-scale PatchGAN discriminator $D$ used in pix2pixHD [42] except that we replace the least squared loss term [32] with the hinge loss term [27]. Since the PatchGAN discriminator has a fixed receptive field of patch, we take the whole generated images instead of image patches to train the GAN. The extrapolated images generated by our QueryOTR should be indistinguishable from real images by the discriminator. Given the extrapolated images $\mathrm{\hat{x}}\sim\mathbb{P}_{g}$ generated by QueryOTR and real images $\mathrm{y}\sim\mathbb{P}_{y}$, the adversarial loss for the discriminator is

Additionally, the adversarial loss for the generator is

We jointly train the hybrid transformer generator and CNN discriminators and optimize the final objective as a weighted sum of the above mentioned loss terms:

where $\lambda_{rec}$, $\lambda_{perceptual}$ are weights controlling the importance of loss terms. In our experiments, we set $\lambda_{rec}=5$, and $\lambda_{perceptual}=10$.

Quantitative Result Tab. S1 shows quantitative results. Our QueryOTR outperforms the competition on almost all metrics on 1-step and multi-step outpainting. In particular, QueryOTR shows obvious superiority in all entries compared with CNN-based methods, e.g., SRN, NSIPO, and IOH. These results show that transformer structure succeeds in capturing global dependencies for image outpainting compared with CNN’s inductive biases. Meanwhile, our QueryOTR outperforms the very competitive Swin-based Uformer which uses an image-to-image translation approach for image extrapolation, mainly because our query-based method allows to generate image patches attended to all the visual locations, yielding a better perceptual consistency. It is noted that our results for $1\times$ outpainting are very close to the IS upper-bound for all the datasets, indicating realistic image generation and good perceptual consistency. Extra results of replacing the center region with input sub-images are in the supplementary.

Qualitative Result Some examples of visual results on all the datasets are shown in Fig. S4. Our QueryOTR effectively extrapolates the images by querying the global semantic-similar image patches. Seen from the $1\times$ outpainting results, our QueryOTR could generate more realistic images with vivid details and enrich the contents of the generated regions marked in white box. Furthermore, our method could weaken the sense of edges between the generated regions and input sub-image. Compared with other baselines, our QueryOTR could generate water containing more realistic ripples in the $3^{rd}$ row and intact trees in the $5^{th}$ row of Fig. S4, which could be seen in the white dotted box. In the $7^{th}$ row of Fig. S4, the whole skyscraper generated by QueryOTR indicates the success of our query-based method which predicts the detailed contents with global information by queries. In the $9^{th}$ row, our method could capture the global information of the green background on the corner marked in the white box. More visual results could be seen in the supplementary material.

Transformer Encoder and Decoder We compare the impact of the pretrained ViT-based encoder and the number of transformer decoder layers $M$. As shown in Tab. S2(a), utilizing a pretrained ViT encoder contributes to the improvements of FID and IS by $2.418$ and $0.204$, respectively. The main reason is that the small datasets might not be sufficient to train the model for performance saturation. The pretrained ViT encoder is capable of capturing the long-term dependencies, which may benefit the patch prediction. Additionally, our QueryOTR performs optimally in both FID and IS when the number of decoder layers is set to 4. Further increasing the depth of decoder indefinitely will not improve the performance of our QueryOTR.

Loss Terms We investigate the impact of patch-wise reconstruction loss $\mathcal{L}_{rec}$ and perceptual loss $\mathcal{L}_{perceptual}$ in Tab. S2(b). We first train the model with only adversarial loss, which is equivalent to training the model unpaired, resulting in a FID of $38.009$ and IS of $3.433$. On the basis of adversarial training, using either $\mathcal{L}_{rec}$ or $\mathcal{L}_{perceptual}$ could improve the overall performance. Fig. S5(c) and (d) show that high-frequency checkerboard artifacts occur when trained without $\mathcal{L}_{rec}$, and the details cannot be generated without $\mathcal{L}_{perceptual}$.

QEM We ablate the impact of QEM and its internal key components. In the experiments, we do not use a pretrained encoder to avoid reducing the difficulty of training learnable queries. Since training pure transformer may require larger datasets and longer time, it is hard for learnable queries to converge well on Scenery dataset, resulting in a high FID (see Tab. S2(c)) and blurry image patches (see Fig. S5(a)). On the other hand, the proposed QEM generates queries conditioned on input images, significantly improving FID by 14.227. Meanwhile, generating queries with noise slightly improves the patch diversity, and deformable convolution enables an active long distance modeling for query generation.

PSM Tab. S2(d) demonstrates the effect of the proposed PSM and per-patch normalization. Although using a single linear layer can generate vivid image patches, the connections between patches are unnatural, as shown in Fig. S5(b). Per-patch normalization could improve the reconstruction of high-frequency by enhancing the local contrast of patches, leading an improvement of FID 20.872. Meanwhile, PSM significantly alleviates the checkerboard artifacts caused by per-patch prediction, and improves the overall perceptual quality of the extrapolated images. PSM alleviates checkerboard artifacts via explicit constraints, while perceptual loss penalizes image discontinuity from a semantic perspective. PSM appears more effective and direct than perceptual loss. If both are applied, even better performance can be obtained.