2S-ODIS: Two-Stage Omni-Directional Image Synthesis by Geometric Distortion Correction

VQVAE (Vector Quantized Variational AutoEncoder) [17] has been proposed to improve the generated image blurriness in VAE [11] by representing image patches with quantized latent vectors based on vector quantization. Furthermore, the adversarial loss has been introduced in VQVAE to make the generated images clearer, called VQGAN [7]. In this method, Transformer [18] has been used to sequentially predict image patches from neighbor patches based on auto-regressive prediction. The patches are represented with the quantized latent vectors called VQGAN codes to generate clear images with low computational cost. To improve the slow inference in VQGAN due to the sequential predictions of patches, MaskGIT [4] has been proposed by predicting multiple patches simultaneously. Although MaskGIT has succeeded in improving the inference speed, it has been difficult to generate high quality images in the high resolution. To solve this problem, Muse [3] has been proposed using a two-stage structure, where a low-resolution image is generated at the first stage, and then is refined to generate a higher-resolution image at the second stage. In our proposed method, this two-stage structure was adopted for the omni-directional image synthesis in the high resolution.

Several methods have been proposed for synthesizing omni-directional images from NFoV images. Okubo and Yamanaka [10] have proposed a method of generating omni-directional images based on conditional GAN from a single NFoV image with the class label. Hara et al. [8] have also proposed a method based on the symmetric property in the omni-directional images using GAN and VAE. Another work to synthesize omni-directional images is Guided ImmerseGAN[6], which generates omni-directional images from a NFoV image with the modulation guided by a given class label, which does not have to be the true class of the input NFoV image. In the work of OmniDreamer [2], VQGAN has been applied to the omni-directional image synthesis by using Transformer for auto-regressive prediction. In this method, VQGAN encoder and decoder have to be fine-tuned on an omni-directional image dataset since the geometric distortion in ERP has to be represented in the latent codes of VQGAN. Text2Light [5] also uses VQGAN with auto-regressive prediction for the generation, though only text information is taken as the input instead of the NFoV image. Nakata et al. [13] have proposed a method to increase the diversity of generated omni-directional images based on MLP-Mixer by efficiently propagating the information of the NFoV image embedded at the center in ERP. AOGNet [12] has generated omni-directional images by out-painting an incomplete 360-degree image progressively with NFoV and text guidances jointly or individually. This has been realized using auto-regressive prediction based on the stable-diffusion backbone model. Due to the nature of sequential auto-regressive prediction, it takes long inference time.

In our proposed method, the pre-trained VQGAN model was used without fine-tuning on the omni-directional image dataset, since multiple NFoV images are synthesized based on the pre-trained VQGAN and then integrated into omni-directional images. By removing the step of VQGAN training, the overall training of the model was drastically shortened than the previous method with VQGAN such as OmniDreamer [2]. In addition, the proposed method is based on simultaneous synthesis of multiple NFoV images in different directions, whose inference was faster than auto-regressive prediction such as OmniDreamer [2] and AOGNet [12].

The proposed method consists of the two-stage structure, where a global coarse omni-directional image in ERP is synthesized in a low resolution (256$\times$512 pixels) at the first stage without geometric distortion correction, and then is refined at the second stage by integrating the multiple synthesized NFoV images in different directions based on the geometric distortion correction, producing a high-quality omni-directional image in ERP in a high resolution (1024$\times$2048 pixels). At both stages, the pre-trained VQGAN was utilized without fine-tuning on the omni-directional image dataset.

As a preliminary experiment, an omni-directional image was reconstructed in ERP or in multiple NFoV images using pre-trained VQGAN encoder and decoder without the fine-tuning, as shown in Fig. 2. It can be seen from the figure that the reconstruction in ERP cannot correctly reproduce the texture in the region toward the ground (blue frame) and the continuity in the region at both edges (yellow frame), although it can reproduce the region at center in ERP (red frame). On the contrary, all the regions can be correctly reproduced in the extracted NFoV images. This indicates that the pre-trained VQGAN model can be applied without fine-tuning if it is applied to NFoV images.

Thus, the generated omni-directional image in ERP at the first stage in the proposed method includes distortions since the pre-trained VQGAN cannot represent the texture and continuities in the omni-directional images in ERP. However, these distortions are correctly compensated at the second stage by synthesizing the multiple NFoV images which can be correctly reproduced by the pre-trained VQGAN model. If only the second stage is used in the proposed method, it is difficult to synthesize multiple NFoV images simultaneously with global compatibility. Therefore, the two-stage structure was adopted in the proposed method to produce globally plausible coarse omni-directional image at the first stage.

The structure of the proposed method is shown in Fig. 3. At the first stage, the low resolution model produces the low resolution codes, which are converted into patches of omni-directional images in ERP using the pre-trained VQGAN decoder. At the second stage, the high resolution model produces the high resolution codes, which are corresponding to the patches in the NFoV images in multiple directions (26 directions in our implementation) in an omni-directional image with overlapping. These 26 directions for the NFoV images were the same directions as the normal vectors in the faces of a rhombicuboctahedron. The field of view was set to 60 degrees for all directions. The generated NFoV images with the size of 256$\times$256 pixels were integrated into an omni-directional image with the size of 1024$\times$2048 pixels in ERP.

For synthesizing an omni-directional image in the inference, the low-resolution codes are first generated using the sampling strategy proposed in MaskGIT [4] at the first stage from the conditional image where an input NFoV image is embedded at the center in ERP, as shown in Fig. 3(a). In MaskGIT, the generation is started with ‘Masked low resolution codes’ which is filled with the [MASK] code, the VQGAN code which indicates that it is masked, for all locations. Then, the low resolution model predicts the probabilities for all the [MASK] locations in parallel, and samples a VQGAN code based on its predicted probabilities over all possible VQGAN codes for each location. The location with the low probability is replaced with the [MASK] code again, and the VQGAN codes are resampled by predicting the probabilities using the low resolution model. This process is repeated in $T$ steps. At each iteration, the model predicts all VQGAN code simultaneously but only keeps the most confident ones. The remaining VQGAN codes are replaced with the [MASK] code and re-predicted in the next iteration. The mask ratio during the iterations is determined by $\cos\left(\frac{\pi}{2}\frac{t}{T}\right)$, where $t$ indicates the current iterations in the total steps $T$. This mask ratio is monotonically decreasing from 1 to 0 with respect to $t$, which ensures that most of the locations are masked during the early stage in the iterations to prevent producing the inconsistent codes.

At the second stage, the high-resolution codes are generated using the high resolution model, where the generation process is almost same as the low resolution model. The difference from the low-resolution model is that the model accepts the low-resolution image generated at the first stage as an additional conditional image, and generates NFoV images, as shown in Fig. 3(a). To integrate the generated NFoV images into an omni-directional image, the overlapped regions are merged with weights depending on the distance from the centers of the NFoV images. Specifically, let $x_{i}$ and $x_{j}$ be the two overlapping pixel values, and let $d_{i}$ and $d_{j}$ be the distances from the centers of the NFoV images at each position. The integrated pixel value $y$ is given by

where $w_{i}=1-\frac{di}{\max_{k}(d_{k})}$, $w_{j}=1-\frac{dj}{\max_{k}(d_{k})}$.

The network architecture used in the low-resolution model and the high resolution model is shown in Fig. 4. The layer structure was adapted from MaxViT [16], although Transformer has been used in MaskGIT [4] and Muse [3]. The 8-layer MaxViT models were used in both the low-resolution model and the high-resolution model. In the low-resolution model, the padding in MBConv was replaced to the circular padding to encourage the continuity at the edges in ERP, whereas it was remained to the zero padding in the high-resolution model. The block attention was applied within each divided region at the low-resolution model and within each NFoV image in the high-resolution model, as shown in Fig. 4. The grid attention was also applied globally in sparse at the low-resolution model as in the original MaxViT model, whereas it was applied among same locations over the NFoV images at the high-resolution model.

The low-resolution and high-resolution models are independently trained, as shown in Fig. 3(b). The objective of the training is to make the low-resolution and high-resolution models predict plausible VQGAN codes at [MASK]-code locations for each inference step. For the low-resolution model, the inputs are ‘randomly masked low-resolution codes’ and a conditional image which emulates the NFoV image embedded at the center in ERP (Fig. 3 a). The mask ratio in the randomly masked low-resolution codes is set to $\cos\left(\frac{\pi}{2}r\right)$, where $r$ is sampled from a uniform distribution $[0,1)$, since this emulates the single iteration in the inference. Examples of the conditional image in the training are shown in Fig. 5. They are prepared by randomly masking an original omni-directional image in ERP to emulate the conditional image in the inference. Since they are not limited to the single NFoV image embedded in ERP, the trained model can be applied to various in-painting and out-painting tasks, as described in 5.2. The low-resolution model is trained to predict the original VQGAN codes in the real omni-directional image at [MASK]-code locations, so that the cross entropy is used as the loss function to train the model.

For the high-resolution model, the inputs are ‘randomly masked high-resolution codes’ for multiple NFoV images (26 NFoV images in our implementation), conditional NFoV images converted from the conditional image in ERP, and the reconstructed low-resolution NFoV images converted from the low-resolution omni-directional image reconstructed using the pre-trained VQGAN encoder and decoder. The ‘randomly masked high-resolution code’ and the conditional image in ERP are prepared in the same manner for the low-resolution model. The low-resolution omni-directional image is required in the inputs of the high-resolution model to emulate the low-resolution image generated at the first stage. The high-resolution model is trained to predict the original VQGAN codes in the NFoV images converted from the real omni-directional image at [MASK]-code locations based on the cross-entropy loss function.

Although the first and second stages are sequentially processed in the inference, they are independently trained in parallel, as shown in Fig. 3(b). This property is advantageous to shorten the required training time if multiple GPUs (Graphics Processing Units) can be used, although a single GPU was used in our implementation.

The proposed method, 2S-ODIS, was quantitatively evaluated, compared with the conventional methods, OmniDreamer [2], CNN-based cGAN [10], MLPMixer-based cGAN [13], and LAMA [15], as shown in Table 1. The models in the proposed method was trained for 4 days (2 days for low-resolution model and 2 days for high-resolution model). For comparison, the result with the models trained for 2 days (1 day + 1 day for low-resolution and high-resolution models) was also shown in the table. The field of view for a NFoV image embedded in an input conditional image was set to 126.87 and 112.62 degrees for width and height in the experiments, respectively. It can be seen from the table that the proposed method achieved higher performance than the other conventional methods. Although the highest performance was achieved in the proposed method trained for 4 days, the performance already outperformed the other methods even with the 2-day training. To see this more clearly, the evaluation metrics during the training of the proposed method is shown in Fig. 6, where the performance exceeded OmniDreamer by 2 days and converged in around 4 days. Thus, the proposed method drastically shortened the training to 2-4 days from 14 days in OmniDreamer including the fine-tuning of the VQGAN model. In addition, the inference in the proposed method was much faster than in OmniDreamer: 1.54 seconds and 39.33 seconds for synthesizing each omni-directional image in the proposed method and OmniDreamer, respectively. This is because the proposed method is based on the simultaneous VQGAN-code prediction instead of the sequential auto-regressive prediction.

For the qualitative comparison, the examples of the synthesized omni-directional images are shown in Fig. 7. It is clear that the proposed method generated globally plausible and locally detained omni-directional images, while OmniDreamer [2] sometimes failed in generating continuous images, especially along the edges in the input conditional images. To see if the proposed method can generate NFoV images without the distortion, the NFoV images toward the ground were extracted from the synthesized omni-directional images, since the geometric distortion is large at the poles in ERP. The examples of the NFoV images toward the ground are shown in Fig. 8. The left column shows the NFoV images at the first stage, while the middle column shows the NFoV images at the second stage. Since the pre-trained VQGAN code cannot represent the geometric distortion in ERP, the straight lines were not appropriately reproduced at the first stage. However, they were compensated at the second stage, which can be clearly seen in the sample images. On the other hand, OmniDreamer cannot appropriately reproduce the NFoV images toward the ground. Thus, it was confirmed that the proposed method synthesized omni-directional images without geometric distortion.

As explained in 3.3, the models in the proposed method were trained with various conditional images in Fig. 5. They were not limited to the single NFoV image embedded in ERP, so that the proposed model can be applied various in-painting and out-painting tasks. For example, the model can be applied to the in-painting task to remove objects or people in the omni-directional image taken by a 360-degree camera, as shown in the top row in Fig. 9. Another example is the task to fill in the ground region of an omni-directional image as shown in the middle row in Fig. 9, since the omni-directional image often includes a hand or a camera stand at the bottom region in ERP. The last example in Fig. 9 is to synthesize an omni-directional image which includes two NFoV images such as front and rear cameras of a smartphone. Although OmniDreamer failed in synthesizing the omni-directional images in these situations, the proposed method generated high-quality omni-directional images. The quantitative results shown in Table 2 also indicate that the proposed method achieved higher performance than OmniDreamer. Although the diversity of the synthesized images was higher in OmniDreamer than the proposed method, it may be due to generating random images as shown in Fig. 9.

Ablation studies were conducted to investigate the effectiveness of each component in the proposed method: the low-resolution and high-resolution models in the 2-stage structure. The results are shown in Table 3. (1) is the proposed method with the 2-stage structure, while (2) and (3) are the results with 1-stage structure only using the low-resolution model and the high-resolution model, respectively. As can be seen from the table, the 2-stage structure was indispensable for the high-quality image synthesis.

Moreover, it was examined that the low-resolution VQGAN codes generated at the first stage were directly used at the second stage instead of the low-resolution image generated at the first stage, since the 2-stage structure in Muse [3] uses the low-resolution VQGAN codes directly. The result is shown in Table 3 (4). It was confirmed from the result that the low-resolution image was better to use at the second stage than the low-resolution VQGAN codes directly. This may be because the low-resolution image is compressed by CNN to properly extract the global information generated at the first stage.