Perception-Distortion Balanced Super-Resolution:
A Multi-Objective Optimization Perspective

Most of the SR methods in this category [2, 3, 11, 5] employ the pixel-wise loss functions, such as the $\ell_{2}$ or $\ell_{1}$ norm of the prediction error, while focusing on the design of network architectures to improve the SR performance. The seminal work of SRCNN [2] introduced a convolutional neural network (CNN) with three layers to perform the SR task. Afterward, many SR models have been developed. Kim et al. [6] trained a deeper and wider network to improve the representational ability of CNN. Deep residual connections were introduced to improve deep SR network learning performance [41, 42, 10]. Tong et al. [12] proposed to use dense skip connections in a very deep network. The recursive network et al. [6] was designed to improve SR performance without introducing new parameters for additional convolutions. Very recently, the transformer networks [43, 44, 45, 46] have been successfully employed for SR, resulting in better distortion-based measures such as PSNR. Because $\ell_{2}$ or $\ell_{1}$ loss tends to generate blurry SR outputs, the SSIM loss [15] and perceptual loss [16] have been proposed to regularize the local structures of images. These two losses can improve the visual quality of SR outputs without harming too much the distortion measures. However, they cannot generate image details to make the SR image perceptually more realistic.

To make the reconstructed SR images perceptually more realistic, SRGAN [7] combined the adversarial loss with distortion-oriented losses (e.g., $\ell_{1}$ loss) in training. Following SRGAN, ESRGAN [8] used the VGG features before activation and employed the RRDB backbone [8] as the generator model, improving the perceptual quality. DualFormer [31] leveraged the spectral and spatial discriminators for identifying high-frequency and low-frequency information, respectively. While GAN-based methods can generate new image details, they may introduce many unnatural visual artifacts because of unstable adversarial training. Therefore, many of the following works aim to reduce the GAN-generated artifacts from the perspectives of frequency division [27, 26, 28, 47, 48], the weight of layers in perceptual loss [30, 49], or different statistical properties [13, 29]. In addition, some works [22] aim to optimize the no-reference image quality metrics, which can be non-differentiable. For example, RankSRGAN [22] designed a differentiable rank-content loss to rank the quality of predictions with NIQE [50] or Ma [51] index.

EA [35, 36] is a classical search-based optimization technique, where individuals interactively evolve from a population. The EA operators mainly consist of crossover [52], mutation, and selection. Due to the global search and gradient-free properties, EA has proven its effectiveness in many applications [53, 54, 55]. It has recently been applied to DNN learning [56, 57, 58]. Guo et al. [56] applied evolutionary architecture search to construct a simplified supernet with all architectures optimized simultaneously. Yu et al. [57] searched over a big single-stage model that contains small children models. EAGAN [58] applied an efficient two-stage EA-based network search framework to learn GANs. Another application is network compression. For example, Zhou et al. [59] established filter pruning as a multi-objective optimization problem and proposed a knee-guided EA algorithm to trade-off between the scale of parameters and performance. In addition, EA algorithm can be used as the network optimizer. ESGD [60] alternately applied SGD and EA in one framework to improve the average performance of DNN. Zhang et al. [61] proposed a hierarchical cluster-based suppression algorithm to remove similar weights and improve population diversity. GEMONN [62] determined the search direction in the EA optimizer by the gradient of weights to improve the efficiency of EA for training DNN.

To optimize a network, EA is usually coupled with a gradient-based optimizer to solve large-scale optimization problems [63]. Most of the previous EA-based DNN learning works [60] optimize a single objective. In this work, we formulate the perceptually realistic SR task as a multi-objective optimization problem and propose a hybrid EA and Adam algorithm to solve it.

Most of the GAN-based SR methods [7, 8] train the generator model by weighting the pixel-wise $L_{pix}$ loss, perceptual loss $L_{percep}$ and adversarial loss $L_{adv}$ as follows:

where the $L_{pix}$ (i.e., $\ell_{1}$ loss) optimizes the reconstruction fidelity, and $L_{percep}$ and $L_{adv}$ improve the perceptual quality. The weights $\alpha_{1}$, $\alpha_{2}$ and $\alpha_{3}$ are often empirically set to 0.01, 1 and 0.005, respectively.

The Adam optimizer [32] is dominantly used to optimize the above loss function. Unfortunately, the gradient-based Adam is difficult to minimize the perception- and distortion-oriented losses simultaneously due to their opposite gradient descent directions, while an ideal weight setting is hard to be found given an infinite searching space. Therefore, most traditional SR models turn to pursue either fidelity or perceptual quality yet sacrifice the other. As illustrated in Fig. 1, when the weights are biased towards distortion- or perception-oriented losses, the SR results will be over-smoothed or include visually unpleasant artifacts. When similar weights are set to balance perception and distortion, the optimization might be stagnant at the local minimum as no explicit gradient descent directions can be found. Note that we apply gradient normalization [64] to avoid the trivial solution.

In this paper, rather than linearly combining the distortion- and perception-oriented losses into a single objective function, we formulate the perceptually realistic SR task as a multi-objective optimization problem as follows:

where objectives $f_{1}$ and $f_{2}$ respectively focus on the reconstruction fidelity and the perceptual quality of the SR results. In the following, we propose to couple EA [36] with Adam [32] to address the multi-objective optimization problem in Eq. (2). EA and Adam help with the optimization process for the divergence and convergence, respectively.

The algorithm of the proposed hybrid EA-Adam optimizer is summarized in Table I. EA and Adam alternately optimize the $N$ SR models with different perception-distortion preferences. We employ the general framework of GAN-based SR learning, which is shown in Fig. 2(a). The generator network learns the mapping between the LR and HR images, and the discriminator network discriminates between the true and fake images. The two networks compete with each other to ensure that the generated images are of high quality and more natural.

The Adam step is illustrated in Fig. 2(b). Initialized by a pre-trained model with $\ell_{1}$ loss, the other $N-1$ generator-discriminator pairs are updated by Adam separately, as shown in the Adam step of Table I. We fix the first model across the training process as it has the best reconstruction fidelity by training with pure pixel-wise $\ell_{1}$ loss. Then, the EA step starts, as illustrated in Fig. 2(c). The adversarial training facilitates the model convergence in the Adam step to obtain better objective scores, while the EA step focuses on the model divergence, enabling the optimized models uniformly distributed on the performance plane spanned by the two objective indices. Therefore, the discriminators are only updated in the Adam steps to enhance convergence.

In the EA step, a population of generator models is optimized together. Following MOEA-D [36], the parents are first chosen from the neighboring population $\mathcal{B}$ or the whole population $\mathcal{G}$, which focuses on the local or global evolution. The offspring is generated by the selected parents via crossover and mutation operators. The crossover is used to interact with the individuals in a population with each other so that useful information can be effectively propagated internally. We employ the SBX crossover [52], which is defined as follows:

Here, $\{\theta\}_{I,1,2}$ indicate the parameters of the corresponding models, $\beta$ is a weighting variable that relies on a random number $r\in[0,1]$, where $\beta={{{(r\times 2)}^{1/(1+\eta)}}}$ if $r<0.5$ and $\beta={{{(1/(2-r\times 2))}^{1/(1+\eta)}}}$ otherwise, $\eta$ is a constant and is set to 20 as in SBX [52]. The mutation operator disturbs the optimization to jump out of the local optimum by adding a zero-mean random Gaussian noise ${\varepsilon^{(k)}}\sim{\mathcal{N}(0,0.01)}$ [60] to each of the $N$ models, i.e.,

To select a good model from the parent models and the newly generated offspring models, we measure the model performance by using an aggregation function derived from the Tchebycheff decomposition [65] as follows:

where $z_{1}^{*}$ and $z_{2}^{*}$ are the moving minimum values of $f_{1}$ and $f_{2}$ recorded until the current iteration, and weights $\lambda$ are uniformly distributed within $(0,1]$. Specifically, we set $\lambda={0,\frac{1}{{N-1}}},{\frac{2}{{N-1}},\cdots,1}$ for the $N$ models, respectively. With different aggregation loss weights $\lambda$, the multiple optimization problem can be divided into $N$ sub-problems of single-objective optimization for different perception-distortion preferences. Note that the momentum in the Adam optimizer should be initialized after the EA optimization step. Finally, a set of generator models with different perception-distortion preferences are obtained to form a PF, as shown in Fig. 1.

To illustrate the training convergence process of the EA-Adam optimization algorithm, in Fig. 3 we demonstrate the performance of optimized models on the two conflicting objectives $f_{1}$ and $f_{2}$ over an Adam-EA cycle on the B100 dataset. Different shaped symbols represent the $N=4$ optimized models, where the green color denotes the model optimized after the first step of Adam, the blue color denotes the model optimized after $T_{Adam}$ steps by Adam ($T_{Adam}=10$ in this experiment), and the red color denotes the model further optimized by EA with $T_{EA}=1$ step. We can see that Adam focuses on the convergence of the 4 models, enabling a fast search for better objective scores of them. However, due to the opposite optimization gradient directions of $f_{1}$ and $f_{2}$, Adam suffers from optimization stagnation. That is, the optimized models targeting at different perception-distortion objectives tend to converge on the same side of the plane, making it difficult to form a diverse solution space. Fortunately, the introduction of EA enhances the diversification of the $N$ models so that the optimized models are more evenly distributed in the optimization plane with improved performance.

As mentioned above, the $N$ SR models (with the same architectural topology) optimized by the proposed hybrid EA-Adam optimizer can work better than their counterparts optimized by Adam in balancing perceptual quality and distortion degree. However, it can be labor-consuming to manually select one from them to super-resolve the input LR image. On the other hand, if we apply all the $N$ models to the input image and then fuse the outputs into one image, it can be expensive and less effective in inference [39]. To solve this issue, we propose to automatically fuse the $N$ models into one stronger model to pursue further enhanced performance beneath the PF formed by the $N$ models. To achieve this goal, we first introduce a lightweight attention-based weight regression network to fuse the convolution layers of the $N$ SR models as in [38, 37, 66]. As shown at the left of Fig. 4, the input is the feature from one layer and the output is the weight of the corresponding layer in the $N$ models. The generated weights satisfy $\sum\nolimits_{k}{{w_{k}}=1},k\in\{1,\cdots,N\}$. There are 3 modules in the weight regression network. The first module consists of 3 convolution layers with Leaky ReLU activation. The kernel sizes of convolution layers are all $5\times 5$. The second module is a global average pooling layer, and the last is a mapping module, which consists of two linear layers followed by a Sigmoid activation. The first module extracts the features and the last two modules estimate the weights to fuse the convolution layers of the SR models.

We train the attention-based weight regression network with the most commonly-used $L_{GAN}$ loss as it encourages the output to be perceptually more realistic. Meanwhile, since the aggregated model linearly interpolates among the $N$ SR models that are fixed in this stage, the reconstruction fidelity can also be well preserved. We fuse the $N$ SR models in one shot and apply the fused model to all testing images, as shown at the right of Fig. 4. Here, we employ a set of $M$ validation images, i.e., $\{I_{i}\}_{i=1}^{M}$, to facilitate the fusion process. Based on the validation data, we calculate the adaptive fusion weights for each $I_{i}$, i.e., $[w_{1}^{i},w_{2}^{i},\cdots,w_{N}^{i}]$, using the trained attention-based weight regression network, and average them over the $M$ weight vectors to get a universal weight, i.e., $[\bar{w}_{1},\bar{w}_{2},\cdots,\bar{w}_{N}]$. Finally, a fused model is obtained, which inherits the advantages from the $N$ SR models in terms of both reconstruction fidelity and perceptual quality, while holding a stronger generalization capability.

Among the 800 images in the DIV2K training dataset, we utilize 700 images as training data for the weight regression network, and the remaining 100 images are used as the validation data in the network fusion process. We have not employed any test data during the training stage. Once learned, our weight regression network is fixed and applied to any given test set. Therefore, our method is zero-shot, which aligns with the approach taken by the referenced and compared SR methods.

Following prior arts [8, 13, 48], we conduct experiments with a scaling factor of $4\times$ on both synthetic (downsampled using MATLAB bicubic kernel) and real-world (degraded using RealESRGAN pipeline [69]) experiments. For the bicubic-degraded SR task, we evaluate the performance of different methods on 6 benchmarks, including Set5 [70], Set14 [71], BSD100 [72], Urban100 [73], Manga109 [74], and DIV2K100 [67]. We compute the PSNR and SSIM [15] indices on the Y channel in the YCbCr space to measure the distortion degree, and compute the LPIPS [33] and DISTS [75] indices in the RGB space to evaluate the perceptual quality. For the real-world-degraded SR task, we evaluate the performance of different methods on the RealSR [76] and DrealSR [77] benchmarks.

Backbones and compared methods: As in many previous perceptually realistic SR works [22, 21, 13, 27], we adopt SRResNet [7] and RRDB [8] as the backbone networks. SRResNet is a lightweight network proposed in SRGAN [7], while RRDB [8] is widely used in later GAN-based SR methods [21, 78, 13] for its superior performance. Specifically, we compare our SRResNet-based model with SRGAN [7] and RankSRGAN [22], and compare our RRDB-based model with ESRGAN [8], SPSR [78], LDL [13], CAL-GAN [79] and WGSR [48]. In addition, Transformer-based backbones have become popular as they can capture long range dependency. Therefore. we compare SwinIR [43] trained with the $L_{GAN}$ loss by Adam optimizer and our proposed hybrid EA-Adam optimization method to validate the effectiveness of our method on the Transformer backbone. For the methods that apply different backbones or training datasets, we compare them with our model of similar capacity and training datasets. Specifically, we compare our SRResNet-based model with CFSNet [80] and G-MGBP [20], and compare our RRDB-based model with SROOE[81], PD-ADMM [28] and DualFormer [31]. We further validate the proposed method for the real-world SR tasks, and compare the obtained ‘RealESRGAN+Ours’ model with RealESRGAN [69], IKC [82], IKR-Net [83], LDL [13] and DASR [38]. The results of the compared methods are obtained using the officially released models. Note that we do not compare our method with the distortion-oriented SR methods, because this work focuses on perception-distortion balance.

Training details: We validate the effectiveness of our method with scaling factor 4 following the previous works [7, 8]. The data augmentation and discriminator networks used in SRGAN and ESRGAN are adopted in our SRResNet- and RRDB-based models, respectively, for a fair comparison. The size of training input patches is set as 32×32. For both the hybrid EA-Adam optimizer and the attention-based weight regression network, the learning rate is set as $1e^{-5}$ for the SRResNet backbone, and $1e^{-4}$ for the RRDB and SwinIR backbones. To balance the performance and the training efficiency, we set the population size as $N=5$. The $T^{Adam}$ for the Adam optimizer is 10, the $T^{EA}$ for the EA optimizer is 1, and the total optimization epoch $T$ is 100. The probability $\delta$ of selecting parents from the neighboring population $\mathcal{B}$ is 0.7.

In this section, we compare the final fused model by our proposed hybrid EA-Adam optimizer and network fusion method against state-of-the-art methods. Table II provides the quantitative comparisons among the methods with the same backbone. Table III provides the quantitative comparisons among the methods with similar model capacity.

From Table II, we can see that for the lightweight network models with the SRResNet backbone, SRGAN gets better PSNR and SSIM scores than RankSRGAN on almost all the benchmarks, while they have higher or lower LPIPS scores on different benchmarks. Our proposed method improves the fidelity- and perceptual-oriented metrics by a large margin under all benchmarks without increasing the SR model complexity. This validates the advantages of our hybrid EA-Adam optimization strategy, which could push the PF toward the optimal point of solution space. For models trained with the RRDB backbone, we can see that our proposed method achieves better results than the competing methods ESRGAN, SPSR, LDL, CAL-GAN, and WGSR in most cases. The LDL also demonstrates competing results in several instances, especially for the perceptual quality metric on the Urban100 dataset. This advantage comes from its iterative and local discriminative strategy that can effectively inhibit visual artifacts and encourage the generation of high-frequency details. Furthermore, our method with the SwinIR backbone demonstrates significant improvement over its counterparts, which verifies that the proposed method can be applied to both CNN-based and transformer-based models.

From Table III, we can see that the proposed method achieves consistent improvement on most benchmarks in terms of both perceptual quality (LPIPS) and reconstruction accuracy (PSNR and SSIM), although the compared methods such as G-MGBP, CFSNet and SROOE have more model parameters and FLOPs (e.g., 22.42G FLOPs and 19.22G FLOPs for ‘G-MGBP’ and ‘CFSNet’, while 10.38G FLOPs for ‘SRResNet+Ours’). In addition, we test the released model of PD-ADMM, which performs favorably in terms of distortion control yet sacrifices the perceptual quality. PD-ADMM applies the same weight, i.e., [0.5, 0.5], for perception and distortion objectives. A constraint term is further introduced to ensure that the low-frequency information in two stages is equal. The ADMM optimization is applied to solve the constrained optimization problem in PD-ADMM. Since it is improper and difficult to integrate the constraint term into our proposed hybrid EA-Adam approach with $N$-optimized models, to fairly compare our method with PD-ADMM, we apply the optimization objective of PD-ADMM to EA-Adam without its constraint. One can see that our method outperforms PD-ADMM in almost all cases but with fewer FLOPs and parameters. SROOE employs an additional condition network to learn the loss aggregation weights, leading to significantly larger model size and Flops than the other methods. In contrast, our EA-Adam method is more efficient while achieving comparable results with SROOE. Our method obtains better performance than DualFormer on both perception and distortion metrics. Note that DualFormer employs a spectral and a spatial discriminator. Our EA-Adam presents a new training strategy from the optimization perspective, which can be applied to DualFormer to further enhance its performance, as shown in ‘DualFormer+Ours’ in Table III.

We provide 4× SR visual comparisons between models (with SRResNet [7], RRDB [8] and SwinIR [43] backbones) trained by our method and other state-of-the-arts models in Fig. 5, Fig. 6 and Fig. 7. We do not compare with the early published method ESRGAN due to limited space. In addition, we give the visual comparisons between PD-ADMM [28] and ‘RRDB+Ours-PD’, i.e., the RRDB model trained by our EA-Adam optimization method with the weight setting in PD-ADMM, in Fig. 8. The visual comparisons between SROOE [30], DualFormer [31] and our proposed EA-Adam optimization method are presented in Fig. 9. One can see that our method can restore more correct visual patterns (e.g., the steps in the second group of Fig. 5 and the zebra crossings in the third group of Fig. 6) while inhibiting the artifacts (e.g., the building windows in the fifth group of Fig. 9). At the same time, our model can generate many high-frequency details (e.g., the lines of buildings and wood floor in the first and third group of Figs. 7 and 8), respectively.

In this section, we conduct a series of ablation studies to validate the effectiveness of the proposed hybrid EA-Adam optimization. We first conduct experiments to compare our hybrid EA-Adam algorithm with the Adam algorithm, and compare their results with different selections of $N$. Then we compare our network fusion method with the learnable weight method. Furthermore, we study the selection of different loss functions in optimizing the weight regression network. We also study the aggregation weights of obtained models with different perception-distortion preferences. All ablation results are evaluated on the Urban100 dataset.

Table IV gives the quantitative comparison. Without loss of generality, we conduct experiments with the SRResNet backbone and evaluate on the Urban100 dataset. PSNR and LPIPS metrics are used to measure the fidelity and perceptual quality, respectively. One can see that our hybrid EA-Adam/AdamW optimizers outperform Adam/AdamW by a large margin in most cases. Both the fidelity and perceptual quality metrics are improved, validating that the proposed hybrid EA-Adam strategy can push the PF closer to the optimal point of the metric space. For most models, EA-Adam improves PSNR by more than 0.4dB over Adam and improves LPIPS by more than $10\%$. This might be caused by the gradient vanishing problem in the gradient-based optimization process so that the models can be trapped into a local minimum. In contrast, with our hybrid EA-Adam optimizer, effective interaction between different models can be introduced by the crossover operator in EA, so that the models can be stably optimized with better convergence.

On the other hand, each of the four models may perform well in one aspect, yet sacrifices the other aspect as a trade-off. The fused model by our network fusion method achieves a better balance between fidelity and perceptual quality. Specifically, although the fidelity indices of the fused model are lower than that of Models 1, 2, and 3, its perceptual performance is better than Model 4, which is trained to maximize perceptual quality. This result well aligns with the goal of fusion network, which aims to further improve the perceptual quality of the fused model without reducing much the fidelity. It is worth noting that the bad convergence of Models 1 to 3 makes the traditional Adam/Adamw optimizer gain less improvement from the network fusion process. In contrast, the proposed hybrid optimization strategy facilitates the effectiveness of network fusion via the interaction of individuals in crossover operation. In addition, we find that AdamW does not show many advantages over Adam in the SR task. This is because AdamW is mainly proposed to handle the over-fitting problem in high-level vision tasks by introducing weight decay. However, this may not fit the SR task as the optimization might be under-fitting to the large space of image details.

Fig. 10 shows the visual comparisons among SR models trained by Adam and our EA-Adam optimizers, where consistent observation with Table IV can be drawn. Specifically, models optimized by EA-Adam outperform their counterparts optimized by Adam, where the structures and patterns are more loyal to the HR and richer details are generated. It can also be observed that while each of the four models is preferred in either fidelity or perceptual quality, the fused model by our method shows comprehensive enhancement in both aspects.

To demonstrate the generalization capability of the proposed method, we further present the comparisons by using the complex real-world SR setting. Specifically, we use the generator and discriminator of RealESRGAN [69] and optimize the model with our proposed EA-Adam hybrid optimizer and the fusion strategy. The DF2K [67, 68] is used as the training dataset, and the LR images are degraded with the RealESRGAN degradation pipeline [69]. The test data are from the real-world datasets, including RealSR [76] and DRealSR [77]. The compared methods include RealESRGAN [69], IKC [82], LDL [13], IKRNet [83] and DASR [38]. The results are shown in Table X. One can see that IKC obtains the worst results since it focuses on solving blur degradation and cannot adapt to real-world SR tasks. The proposed method obtains the best perception-distortion results in the real-world SR task, demonstrating its effectiveness.

While our method can improve the training process of GAN-based SR models and improve the model performance in perception-distortion balance, it can still fail in some challenging cases. Fig. 11 shows some typical examples. If the high-frequency structures are largely destroyed in the input LR image, it is hard for our model to correct them in the SR output, as shown in Fig. 11(a). In addition, for some small-scale textures and text characters, our method may generate some visual artifacts, as shown in Figs. 11(b)) and 11(c)).