FuseSR: Super Resolution for Real-time Rendering through Efficient Multi-resolution Fusion

To align pixels sharing the same screen space position, upsampling and pooling are two common strategies, where the former aligns at HR level and the latter aligns at LR level. Considering that the performance of convolutional networks degrades sharply as the input resolution increases, the upsampling alignment contradicts with the need of real-time performance, while pooling is a more feasible choice.

Pooling operations are commonly used in neural networks, including maximum pooling and average pooling. However, these pooling operations inevitably damage the spatial details such as HR edges and textures, which are key information for SR quality improvements. Therefore, instead of lossy pooling operations, we adopt pixel unshuffling operation [Shi et al., 2016; Gharbi et al., 2016] to align an HR feature to LR without information loss. We find that pixel unshuffling can losslessly shrink an HR feature map into LR space, converting pixel-wise spatial information into channel-wise deep information. Specifically, this operation divides the HR map into blocks of size $r\times r$ ($r$ is the downscaling factor) and concatenates the features of all pixels within each block to form pixels of the LR version, i.e. transforming a map of shape $[C,H*r,W*r]$ into an LR map of shape $[C*r*r,H,W]$ without information loss.

We propose an H-Net architecture with a pixel unshuffling-shuffling pair, to efficiently align and fuse LR and HR information to faithfully preserve HR details during the alignment and fusion. The architecture of H-Net is shown in Fig. 3:

1. (1)

   We use pixel unshuffling $\mathcal{P}_{D}$ to downscale $G^{HR}$ and concatenate it with other LR inputs.

2. (2)

   The concatenated feature is fed into a fusion network backbone $\mathbf{F}$. Note that $\mathbf{F}$ runs at LR level, which prevents significant HR computational overhead.

3. (3)

   The output of $\mathbf{F}$ is transformed to HR space using pixel shuffling $\mathcal{P}_{U}$ to obtain the HR output $\hat{L}_{D}^{HR}$.

Formally, the process can be expressed as follows:

where $[\cdot,\cdot]$ represents concatenation, and $\mathbf{E}$ is the encoder used to obtain the LR feature map (Section 4). The name “H-Net” is motivated by the shape of the network with two HR ends and an LR bottleneck, resembling an “H”.

With the use of pixel unshuffling, our method can faithfully preserve the HR details, thus facilitating high-quality SR outcomes. Moreover, since neighboring pixels are usually highly correlated, aggregating neighboring pixels of the HR map together via $\mathcal{P}_{D}$ allows obtaining more compact implicit representations in the following fusion network, which is beneficial for reducing data redundancy and enhancing efficiency. Our experiments show that our alignment strategy even achieves better quality than the naïve HR upsampling alignment strategy (see Section 5.4.2).

The encoder module takes LR irradiance and G-buffers as input and outputs an LR feature map as part of the input of our H-Net (Eq. 5). Following existing works , we also utilize LR historical frames to encourage temporal consistency. The architecture of the encoder is shown in the bottom left corner of Fig. 2, and please refer to our supplementary material for details on how we encode LR current frame and historical frames.

Formally, the encoder takes the LR irradiance along with G-buffers of current frame and 2 previous frames as input, and output 2 LR feature maps $F_{s}^{LR}$ and $F_{i}^{LR}$:

where $i$ is the current frame index, $F_{s}^{LR}$ is the output of the first layer of $\mathbf{E}$ used for skip connection later, and $F_{i}^{LR}$ is the final output as an input of the fusion network.

In our fusion network $\mathbf{F}$, the LR feature from encoder $F_{i}^{LR}$, and the unshuffled HR auxiliary features $G^{HR}$ are concatenated and fed into a subsequent network $\mathbf{F}$ to obtain an LR fused feature map $F_{f}^{LR}$:

In this module, we skip-connect the first layer output $F_{s}^{LR}$ of the encoder $\mathbf{E}$ with the fused feature map $F_{f}^{LR}$ and then upscale this concatenated feature map via the pixel shuffling operation $\mathcal{P}_{U}$. Finally, the upscaled feature map is processed by a single-layer convolutional network to obtain the RGB prediction of demodulated irradiance $\hat{L}_{D}^{HR}$ (Section 3.1):

After obtaining $\hat{L}_{D}^{HR}$, we can compute the final color prediction $\hat{I}^{HR}$ by remodulating with $F_{\beta}^{HR}$ as described in Eq. 4.

We directly apply $L_{1}$ loss between HR ground truth $I^{HR}$ and the predicted HR image $\hat{I}^{HR}$ to supervise the training:

Introduced by Johnson et al. [2016], we use a pretrained VGG-16 network to compute perceptual loss to enhance semantic consistency:

where $\mathrm{VGG}_{i}$ represents the i-th layer of pre-trained VGG network, and $V$ denotes the set of selected layers.

Following NSRR [Xiao et al., 2020], we also incorporate structural similarity index (SSIM) [Wang et al., 2004] as an additional loss function:

To summarize, the total loss is the weighted combination of all losses mentioned above:

We empirically set $\lambda_{\mathrm{p}}=0.5$ and $\lambda_{\mathrm{s}}=0.05$ in our experiments.

Our model is implemented and trained using PyTorch [Paszke et al., 2019]. All training and testing are performed on a single NVIDIA RTX 3090 GPU. For the best trade-off between performance and quality, we evaluate two implementations of our FuseSR model, namely Ours and Ours ↯. “Ours” version provides full network implementation with optimal quality. “Our ↯” version is more performance-oriented. It cuts off the history reusing module in the encoder (Section 4.1), reduces the number of hidden channels of all layers in fusion network F by half, and replaces all convolutional layers in F with faster DWS layers [Sandler et al., 2018] to improve the performance at a slight sacrifice of quality.

We have constructed a large-scale dataset using four virtual scenes selected from Unreal Engine marketplace¹¹1https://unrealengine.com/marketplace We select 2 scenes from Unreal Engine 4 (Kite and Showdown), and 2 scenes from Unreal Engine 5 (Slay and City). These scenes contain extensively complex geometry and lighting conditions with dynamic objects. Each scene contains 1080 continuous frames, of which 960 frames are used for training and 120 are used for testing. To show our method’s ability for high-quality rendering, the 2 scenes from UE 5 are rendered by real-time raytracing to achieve photorealistic appearances. We choose 4K ($3840\times 2160$) as the resolution of the target HR frame and the LR frames are downscaled according to scale factor. All frames are generated using UE 4 [2020b] and 5 [2021] with customized shaders to compute the required G-buffers, pre-integrated BRDF, and motion vectors.

Our evaluation comprises two aspects: performance and quality. We report the runtime of networks in milliseconds (ms) as a straightforward metric in performance. In terms of quality, we report two widely-used image metrics: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) [Wang et al., 2004]. For both PSNR and SSIM, higher is better.

We test G-buffer generation time by recording the rendering state via the UE built-in profiling tool. For each scene, we measure the average G-buffer time on a sequence of dynamic frames and take 10 samples to cover as many scenarios as possible. Notably, G-buffer generation time is affected by various factors and there is a variance between certain frames within four orders of magnitude ranging from 0.01 ms to less than 10 ms. In general, the average G-buffer time is significantly lower than network time (as reported in the first row of Table 2). In modern game engines, simplification techniques, such as LOD or new techniques like Nanite in UE5, can further reduce the overhead of G-Buffer generation to a stably low cost.

The ultimate goal of SR for real-time rendering is to reduce computational overhead by avoiding native rendering of HR images by the rendering engine. The performance of native rendering is affected by multiple factors including resolution and anti-aliasing options. In Fig. 5, we compare the runtime of HR native rendering with the total runtime of rendering an LR image and then super-resolving it into HR by our method. We report native rendering runtimes with different anti-aliasing levels (w/o AA, FXAA, and SSAA), and our results with different network configurations (Ours and Ours ↯) and different SR upscaling factors (4x and 8x). Results show that integrating our method into the rendering pipeline indeed reduces computational overhead compared to native rendering.

The native rendering overhead typically depends on the complexity of scenes and lighting conditions, while the inference time of SR network basically remains constant among varying scenes. Integrating SR into rendering pipelines can increase the potential of designing more complex scenes for modern games without noticeable degradation of rendering performance.