FreqNet: A Frequency-domain Image Super-Resolution Network with Discrete Cosine Transform

Numerous methods have proven the effectiveness of the CNN-based pipeline on image super-resolution tasks. The pioneering work was done by Dong et al.[5], their proposed SRCNN for image SR achieved superior performance against previous works. Kim et al. proposed VDSR[16] and DRCN[17] by introducing residual learning to ease the training difficulty and significantly improve accuracy. Tai et al. introduced recursive blocks in DRRN[18] and memory blocks in MemNet[19]. A faster network structure FSRCNN[20] was proposed to accelerate the pipeline of SRCNN. Ledig et al.[7] introduced ResNet[6] to construct a deeper network, SRResNet, for image SR. They also proposed SRGAN with perceptual losses[21] and generative adversarial network (GAN)[14] for photo-realistic SR. Such GAN based model was then introduced in ESRGAN[22], which confirmed that dropping the batch normalization layers can result in better performance. Although SRGAN and ESRGAN can alleviate the blurring and over smoothing artifacts, their predicted results may not be faithfully reconstructed and produce unpleasing artifacts. By removing unnecessary modules in conventional residual networks, Lim et al.[8] proposed EDSR and MDSR, which achieve significant improvement. Zhang et al.[12] introduce channel attention to residual block. Blind-SR methods have also received increasing attention recently, aiming at complex degradation models in real scenarios by estimating degradation kernel using an extra module[23]. The methods of [24],[25] and [26] achieved state-of-the-art performance in real-world scenario with multiple modelling strategies.

However, all these CNN-based methods operate on the spatial domain. The information on the frequency domain is not directly used, though the recovery of high-frequency information is precisely the target of the image super-resolution task.

Projecting image to frequency domain provides a new perspective for various computer vision tasks. Remarkable performance has been achieved in some frequency-domain works. Works of [27] ,[28] and [29] jointly train auto-encoder-based networks on compression and inference tasks with frequency-domain input. [30] extracts features from the frequency domain to classify images. [31] proposes a model conversion algorithm to convert the spatial-domain CNN models to the frequency domain. [32] propose a method of learning in the frequency domain using DCT-based sparse image representations, proving that we can use frequency-domain information directly in current CNN models without a complex model transition procedure. [33] further translate the DCT representation into a sequence of DCT channel, spatial location, and DCT coefficient triples, and achieve state-of-art performance on image generation and restoration tasks with a Transformer-based auto-regressive architecture.

The essence of the SR task is to recover the information of high-frequency channels in the image. Hence, frequency-domain features are informative for HR reconstruction and can potentially enhance the performance with proper methods. However, there is no existing SR method using the characteristics of DCT feature maps. Hence, we propose a super-resolution pipeline on the DCT domain, which we will present in detail in the following section.

Precisely, the overall architecture is given in figure 1. In section 3.1, we introduce the image conversion process that projects the spatial image to spatial domain. In section 3.2 and 3.3, we explain the architecture and components of FreqNet in detail. The propsed frequency-domain loss function will be presented in section 3.4.

Following the JPEG codec, we first transform the original RGB images to zero-centered normalized YCrCb color space, containing a brightness component Y (luma) and two color components Cb and Cr (chroma). Then we upsample the LR image to make it the same size as the HR image.

As shown in figure 1, our FreqNet contains two parallel data flows: the Spatial Extraction Network(SEN) and the Frequency Reconstruction Network(FRN), in order to make use of both domains’ information.

The SEN takes up-scaled LR image $I_{LR}$ as input. Only one convolutional layer is used to extract the shallow feature $F_{shallow}$ from the LR input. $F_{shallow}$ is then passed through the Reconstruction Trunk(RT), which contains a sequence of multiple Residual Groups(RG)[12] and Deformable Residual Groups(DRG) to convert the spatial feature maps into frequency domain features $F_{freq}$. Then we feed $F_{freq}$ to Shrinking Trunk(ST), which consists of 4 down-sampling convolution layers with $stride=2$, to gradually shrink the scale of features maps while maintaining the channels. The final output $M_{SR_{1}}$ is one component of target HR DCT feature maps. The overall process can be interpreted as:

Where $H_{shallow}$ denotes the first convolution operation, $H_{RT}$ and $H_{ST}$ denote the RT and ST structure.

The FRN is purely operated on frequency domain. We take the pre-processed LR DCT feature maps $M_{LR}$ as input, through the frequency-domain reconstruction trunk(FRT), which contains a sequence of depth-wise residual groups(DWRG) and RG to obtain the other component of target HR DCT feature maps, noted as $M_{SR_{2}}$. A skip-connection is added to take advantage of the similarities between input and the target, thus drawing attention towards the difference on high-frequency channels. The overall process can be interpreted as:

Where $H_{FRT}$ denotes the FRT structure.

The outputs of two sub-network have the same size, and a weighted element-wise sum is applied to get the final output:

Where the $W_{1}$ and $W_{2}$ are the pre-defined weights for two components.

The output $M_{SR}$ is further fed to a 2-stage inverse Discrete Cosine Transform(iDCT) module, which is an inverse flow of data-processing pipeline we defined in 3.1. We first project the $M_{SR}$ back to its original range of values by performing denormalization on each channel. Then, in stage-1, we reform the $H/R\times W/R\times R^{2}$ DCT feature maps back to DCT blocks of size $R\times R$, and the rest of $32\times 32$ block is filled with information from LR DCT blocks. Then we use iDCT to get the final SR image in stage-2.

Inspired by the success of residual groups(RG) in [12], we take it as the basic module of our network. As shown in figure 4, an RG is a sequence of $n$ residual blocks(RB)[8] with an in-group skip connection between the input and output features. The original RB can be interpreted as:

Where $H_{Conv}$ denotes a convolution layer, $F_{input}$ is the feature from last block and $F_{output}$ is the feature towards next layer. As described in 3.1.3, the final output of network should be of zero-centered distribution, thus we replace ReLU layer by LeakyReLU with a high negative slope to fit our case.

The definition of our frequency-domain loss function $L_{freq}$ is critical to the performance of our network. Commonly, the loss function of super-resolution task is based on pixel-wise Mean Square Error(MSE), as minimizing spatial MSE also maximizes the peak signal-to-noise ratio(PSNR). However, solutions from MSE optimization can achieve high PSNR while lacking high-frequency content, which results in unsatisfying perceptual quality with overly smooth textures[7].

For our frequency domain super-resolution, this problem can be solved in a intuitive method. Since the target is a series of frequency-domain feature maps with semantic meaning assigned to each channel, we can allocate different weights to each frequency channel while computing the loss, thus explicitly guide the network to focus on the reconstruction of selected high-frequency channels. Following [36], we further replace the MSE backbone by Charbonnier Loss that can better handle the outliers, which are more likely to appear in frequency-domain samples. The proposed frequency-domain loss function $L_{freq}$ is calculated as:

Where $L_{Char}$ is the backbone of Charbonnier Loss, $W_{map}$ and $H_{map}$ denotes the width and height of output feature maps, $\beta_{c}$ denotes the weight assigned to channel $c$ and $M$ denotes the frequency-domain feature maps, as we previously define in equation 10.

Our experimental settings about datasets, degradation models, evaluation metric and training settings are declared below:

Datasets and degradation model. Following [37] We use 800 training images from DIV2K dataset[37] as training set. For testing, we use four standard benchmark datasets: Set5[38], Set14[39], BSD100[40] and MANGA109[41]. We conduct experiments with Bicubic degradation model.

Evaluation Metrics. The SR results are evaluated with PSNR on the luminance channel(Y channel) of transformed YCrCb space. We also propose a frequency-domain reconstruction metric(FRM) on the luminance channel that measures the quality of high-frequency feature reconstructed:

Training Settings. We crop 800 training images into mini patches. Respectively, the size of cropped LR image is $32\times 32$ and the size of cropped HR image is $256\times 256$. The relative location of each pair of LR and HR patches is strictly identical. Our model is trained by ADAM optimizor[42], with $\beta_{1}=0.9$, $\beta_{2}=0.99$ and $\epsilon=10^{-}8$. We implement Cosine Learning Rate(CosLR) strategy, which periodically adjust the learning rate at $t_{th}$ epoch of $i_{th}$ period with the equation:

Where the $\eta_{max}$ is $10^{-}4$ and $\eta_{min}$ is $10^{-}7$, the number of epochs in each period is 30. We use PyTorch[43] to implement our method with Nvidia Geforce RTX 2080 ti GPU.

The channel-wise weights allocation of our proposed loss function $L_{freq}$(Equation 15) will be discussed in detail in section4.3.

We quantitatively compare our method with 8 State-of-the-art methods, including SRCNN[5], FSRCNN[20], EDSR[8], EDN[9], RRDB[22] and its perceptual-driven method ESRGAN[22], MSRResNet[7] and its perceptual-driven method MSRResNet-GAN[7]. We further perform visual comparisons with these two GAN-based methods and their PSRN-oriented version to demonstrate the perceptual quality and fidelity of the output from our model.

We study the effects of proposed Deformable Residual Group, Depth-wise Residual Group and the Frequency-domain Loss Function.

As the output of our proposed model is a group of separated frequency-domain feature maps, we can easily merge our output with the output of other SR models, thus realize the enhancement on selected high-frequency channels. We first perform a similar process as 3.1 to convert the output from other SR model $I_{SR_{o}ri}$ to its frequency-domain feature maps group $F_{SR_{o}ri}$, then we replace certain channels in $F_{SR_{o}ri}$ with the corresponding channels in $F_{output}$ from FreqNet to get the merged output $F_{merge}$.

Specifically, we can merge our output with GAN-based SR models. As shown in Figure 8, we merge the output of MSRResNet-GAN[22] with the output of our model, for image “ARMS.png” in “Manga109”[41], the results are presented in Y-channel. The excessive artifacts from GAN can be corrected by channel replacement, and the reasonable high-frequency information that doesn’t ruin the fidelity can be preserved. This method is practical when the output is blurred due to the difficulty of prediction.