HiREN: Towards Higher Supervision Quality for Better Scene Text Image Super-Resolution

The enhancement of HR images is a challenging task, where the challenges lie in two aspects that will be detailed in the sequel. Formally, the HR image $I_{HR}$ and the corresponding HQ image $I_{HQ}$ we are pursuing are connected by a degradation model as follows:

where $\otimes$ denotes the convolution operation, $k$ is the degradation kernel, and $n$ is the additive noise that follows Gaussian distribution in real world applications [52, 53]. Different from the degradation from $I_{HR}$ to $I_{LR}$ where the kernel is determined by lens zooming, unfortunately, the degradation $k$ of $I_{HQ}$ is unknown. As shown in Fig. 1(c), such degradation can be but not limited to blurring (the 1st and 2nd cases) and low-contrast (the 3rd case). What is more, we also lack pixel-level supervision information of $I_{HQ}$. These two challenges make existing STISR methods unable to enhance $I_{HR}$. To cope with the first challenge, here we adopt blind image deblurring techniques [54, 55, 53, 52] to boost the recovery of $I_{HR}$. Specifically, as shown in Fig. 2, our HR enhancement branch consists of two components: a kernel predictor $P$ and a kernel-guided enhancement network $f_{ke}$. The kernel predictor aims at estimating the degradation kernel $k$ (i.e., $k=P(I_{HR})$ where $k\in\mathbb{R}^{d}$, and $d$ is the size of the kernel), while the kernel-guided enhancement network takes the predicted kernel and $I_{HR}$ as input to conduct a kernel-guided enhancement: $I_{HQ}=f_{ke}(I_{HR},k)$. The predicted kernel is utilized as a clue to strengthen the model’s ability to handle various degradation and boost the recovery of HR images. As for the second challenge, we introduce a pre-trained scene text recognizer $R$ to provide the supervision for generating more recognizable HQ images. And after training the HR enhancement branch $f_{HR}$, HiREN uses the trained $f_{HR}$ to generate HQ images, which are exploited for training the LR recovery branch.

As shown in Fig. 3, to generate a prediction of the degradation kernel, we first utilize convolution layers to obtain a spatial estimation of the kernel. Then, we employ global average pooling [56] to output the global prediction by evaluating the spatial mean value. Thus, we can get the prediction of the kernel of size $\mathbb{R}^{d}$, in a simple yet effective way.

As shown in Fig. 3, our kernel-guided enhancement network is designed in the following way: 1) Start with an input convolution to change the channel number from $C$ to $C^{\prime}$. 2) Repeat $N$ modified SRB blocks [7]. Each block consists of two convolution layers and one Bi-directional GRU [57] (BGRU) to handle sequential text images. At this step, we first stretch the predicted kernel $k$ to pixel shape, then concatenate the pixel kernel with the feature map extracted by convolution layers at channel dimension. 3) An output convolution is applied to getting the final enhanced HQ image $I_{HQ}$.

Here, we design the loss functions of the HR enhancement branch $f_{HR}$. As shown in Fig. 2, there are two loss functions in $f_{HR}$. The first one is the recognition loss $\mathcal{L}_{rec}$ that is used to make the enhanced image $I_{HQ}$ to be more easily recognized than $I_{HR}$. It is provided by a pre-trained recognizer $R$ and the text-level annotation of $I_{HR}$. Suppose the encoded text-level annotation is $p_{GT}\in\mathbb{R}^{L\times|\mathcal{A}|}$, where $L$ is the max prediction length of recognizer $R$, and $|\mathcal{A}|$ denotes the length of the alphabet $\mathcal{A}$. Then, the recognition loss can be evaluated by

which is the cross entropy of $p_{GT}$ and $R(I_{HQ})$. Beside the recognition loss, it is essential to keep the style of the enhanced images, which has also been pointed out in a recent work [8]. Though HR images are not trustworthy, pixel information from HR images can help the model to enhance the input images, rather than totally regenerate them, which is a much more challenging and uncontrollable task. In HiREN, we use mean-squared-error (MSE) to compute pixel loss to keep the style unchanged. Formally, we have

With the recognition loss Eq. (3) and the style loss Eq. (4), the whole loss function of the HR enhancement branch can be written as follows:

where $\alpha$ is a hyper-parameter to trade-off the two losses.

The TextZoom [7] dataset consists of 21,740 LR-HR text image pairs collected by lens zooming of the camera in real-world scenarios. The training set has 17,367 pairs, while the test set is divided into three settings based on the camera focal length: easy (1,619 samples), medium (1,411 samples), and hard (1,343 samples).

These datasets are used to check the generalization power of our model trained on TextZoom when being adapted to other datasets. In particular, three regular STR datasets are evaluated in our paper to further check the advantage of HiREN: IC15-352 [8], SVT [59], and SVTP [60]. In what follows, we give brief introductions on these datasets.

The IC15-352 dataset is first divided in [8]. This dataset consists of 352 low-resolution images collected from the IC15 [61] dataset.

Street View Text (SVT) [59] is collected from the Google Street View. The test set contains 647 images. Many images in SVT are severely suffered from noise, blur, and low-resolution.

SVT-Perspective (SVTP) [60] is proposed for evaluating the performance of reading perspective texts. Images in SVTP are picked from the side-view images in Google Street View. Many of them are heavily distorted by the non-frontal view angle. This dataset contains 639 images for evaluation.

The major metric used in this paper is word-level recognition accuracy that evaluates the recognition performance of STISR methods. Following the settings of previous works [9], we remove punctuation and convert uppercase letters to lowercase letters for calculating recognition accuracy. Besides, Floating-point Operations Per Second (FLOPS) is used to evaluate the computational cost of various methods. Following [9, 32], we only report Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) [62] as the auxiliary metrics to evaluate the fidelity performance because of the quality issue of the HR images.

Here, we evaluate our method on TextZoom. Since HiREN is a framework that can work with most existing methods, we plug HiREN to the training of several typical super-resolution methods to check the universality and effectiveness of HiREN, including one generic method SRCNN [20], two recent proposed STISR methods TSRN [7], TG [9], and one iterative-based and clue-guided STISR method TPGSR [12]. To show that HiREN can support various recognizers, we follow previous works [12, 8, 9] and evaluate the recognition accuracy on three recognizers: CRNN [36], MORAN [40] and ASTER [38]. We re-implement these methods to unify hardware, software, and evaluation protocols for fair comparison. Generally, our results are higher than those in the original papers. For example, with CRNN the averaged accuracy of TG is boosted from 48.9% to 49.6%. All the results are presented in Tab. II.

We first check the universality of HiREN. As can be seen in Tab. II, HiREN significantly boosts the recognition performance in almost all the cases, except for one case on TPGSR, which means that HiREN can work well with various existing techniques. As for the performance improvement of HiREN, taking a non-iterative method for example. The state-of-the-art TG [9] achieves 49.6%, 57.6% and 61.2% averaged accuracy respectively with the three recognizers (see the 9th row). After equipping our method HiREN, the accuracy is lifted to 51.1%, 58.6% and 61.7% (increasing by 1.5%, 1.0%, and 0.5%) respectively (see the 10th row). This demonstrates the effectiveness of our method. Results on more datasets and recognizers are given in the supplementary materials to demonstrate its universality.

It is worth mentioning that our HR enhancement branch can also be applied to weakly supervising the enhancement of LR and HR images to lift their recognition accuracies, as shown in the 3rd and 5th rows of Tab. II. This further supports the universality of our technique. Results above show the promising application potential of our method — not only work with STISR methods, but also pioneer weakly supervised enhancement of LR and HR text images.

Furthermore, to better demonstrate the universality of HiREN, we conduct more experiments on more STR datasets and recently proposed STR datasets. We first evaluate our method on three STR datasets, including IC15-352, SVT, and SVTP. We use the STISR models (TSRN, TG, TPGSR, and our techniques performed on them) developed on the TextZoom dataset to evaluate these datasets. The experimental results on IC15-352, SvT, and SVTP are given in Tab. III. As shown in Tab. III, HiREN also works well on them and achieve lifted performance in almost all the cases. In particular, the performance of TPGSR on three datasets are lifted from 66.2%, 77.4%, 62.8% to 66.8%, 78.7%, and 63.6%, respectively, which demonstrates the advantage of HiREN.

Apart from that, we also give the experimental results on more recently proposed recognizers including SEED [46] and ABINet [48]. The experimental results are given in Tab. IV. As can be checked in Tab. IV, these recent recognizers still find difficulty in recognizing low-resolution text images. For example, SEED and ABINet can only correctly read 45.8% and 61.0% of LR images, which are inferior to performance of reading HR images (i.e., 84.8% and 89.8%). Our method HiREN can also achieve boosted performance on these recognizers in almost all the cases.

We also report the results of fidelity improvement (PSNR and SSIM) on major existing methods in Tab. V. Notice that these fidelity metrics have the following limitations. On the one hand, PSNR and SSIM globally measure the similarity between SR image and the ground truth image, including both characters and background. With the goal of lifting the recognition ability and readability of the scene text images, STISR should put more emphasis on recovering characters rather than the background [9, 32]. On the other hand, as pointed out by our paper, HR images are suffering various quality issues. Ergo, it is inappropriate to measure the pixel similarity between erroneous HR images whose pixels are not trustworthy. Therefore, we only present PSNR and SSIM as auxiliary metrics to roughly draw some conclusions.

Notice that existing methods utilize SR-HR image pairs to calculate PSNR and SSIM. However, as mentioned above, the HR images are suffering from quality issues. Hence, we additionally provide the fidelity results of calculating PSNR and SSIM between SR and HQ images. The experimental results are given in Tab. V. As can be seen in Tab. V, 1) A higher PSNR does not means a higher recognition accuracy. For example, the PSNR of TG in SR-HR is inferior to that of TSRN (i.e., 21.47 v.s. 21.84) but TG performs better on recognition accuracy (i.e., 49.6% v.s. 42.1%). The reason lies in that TG is a stroke-focused technique, focusing on recovering fine-grained stroke details rather than the whole image quality including background that is minor to recognition. This is consistent with the results in [9]. 2) Comparing with the original models, after applying HiREN, the SR-HQ fidelity performance of the new models are boosted in almost all cases. 3) HiREN gets a low performance on the PSNR and SSIM of SR-HR images but obtains an improved recognition performance, which supports the quality issue of HR images.

Here, we visualize several examples in Fig. 4 to better demonstrate the performance of our technique. We can see that HiREN can help the existing methods to recover the blurry pixels better (see the 2nd $\sim$ 6th cases). In particular, a better “ee” in the 2nd and 3rd cases, ‘m’ in the 4th case, ‘f’ in the 5th case, and ‘e’ in the 6th case are obtained by our technique. Besides, in some extremely tough cases where even with the HR images the recognition is hard, HiREN can still achieve better recovery (see the 7th case). These results show the power of HiREN.

We have discussed the high performance of our technique above. In this section, we provide the results of training and inference costs to show the efficiency of HiREN. Specifically, We take TG and TPGSR as baselines and add HiREN to them and count their FLOPS during training and inference. The experimental results are presented in Tab. VI. In terms of training cost, we can see that the offline deployment of HiREN does not incur any additional cost. As for online version, we can see that the additional computational cost caused by HiREN is negligible (e.g, from 19.60G to 20.59G, only 0.99G). What is more, neither of the two variants introduce any additional inference cost. In conclusion, the offline deployment not only saves training and inference cost, but also significantly boosts the performance. These results validate the efficiency of our method.

Here, we check the design of the HR enhancement branch. As mentioned above, two techniques are developed to promote the enhancement of HR images: kernel-guided enhancement network $f_{ke}$ and the loss $\mathcal{L}_{HR}$. We conduct experiments to check their effects. The experimental results are presented in Tab. VII. Visualization of the effect of the HR enhancement branch is given in the supplementary materials.

The effect of the HR enhancement branch. Comparing the results in the 1st and 7th rows of Tab. VII, we can see that the HR enhancement branch lifts the accuracy from 66.9% to 74.1%, which proves the effect of the branch as a whole.

The effect of kernel-guided enhancement network. To check the power of the kernel-guided enhancement network, we design a variant that removes the kernel predictor. Comparing the results of the 2nd and 7th rows in Tab. VII, we can see that the variant without the kernel predictor is inferior to that with the kernel predictor (72.7% v.s. 74.1%). This demonstrates the effectiveness of the proposed kernel-guided enhancement network.

The design of loss function. Here, we check the design of the loss function used in the HR enhancement branch. We first remove the recognition loss $\mathcal{L}_{rec}$ and the style loss $\mathcal{L}_{sty}$ separately. As can be seen in the 3rd, 4th, and 7th rows in Tab. VII, comparing with the combined one, the performance of using only one single loss is degraded. Next, we check the selection of style loss. Specifically, we consider three candidates (MSE, Charbonnier and L1) for the style loss function. As can be seen in the 5th, 6th, and 7th rows of Tab. VII, MSE loss outperforms Charbonnier loss [64] and L1 loss. The reason lies in that MSE penalizes large errors and is more tolerant to small errors, which is more suitable for HiREN to enhance the blurry or missed character details and keep the style unchanged [65]. Ergo, MSE is selected as the style loss in HiREN.

Here, we provide the grid search results of the hyper-parameter $\alpha$ introduced in HiREN for balancing the two losses. The results are presented in Tab. VIII. As can be seen in Tab. VIII, the best performance is achieved when $\alpha$=0.1 and 0.05.

Here, we compare the performances of different models w/o the quality estimation module. As can be seen in Tab. IX, without $f_{QE}$, all methods are degraded, which demonstrates the effect of the quality estimation module.