Deep Image Compression Using Scene Text Quality Assessment

An overall architecture of the STIQA model is shown in Figure 3. The STIQA model crops a text region from an image, resizing it to $32\times 128$. We extract features from the text region using two networks. Firstly, we predict a sequence of probabilities for character categories using a convolutional recurrent neural network (CRNN) [23]. The character categories include a–z, 0–9, and white space. Thus, the total number of character categories is 37. The second network extracts image features using a convolution layer with a window size of $7\times 7$. Then, we fuse the sequence and the image features using the transformer [24] to produce features. The final layer comprises adaptive average pooling, fully connected layers, and a sigmoid activation function. The output is a predicted text quality ranging from [0, 1]. Higher value is high quality.

The text regions are cropped from a scene image and are resized to $32\times 128$. A standard rescale method, such as bilinear interpolation, distorts the aspect ratio of the original text. Therefore, we resize the text by expanding backgrounds if the text size is less than $32\times 128$. We use the second-lowest pixel value among the four corner pixels, and Gaussian noise is then added to the pixel value. On the other hand, if the image size exceeds $32\times 128$, we randomly crop the text to $32\times 128$. As a result, the original text image can be input into the STIQA model.

We adopted the transformer [24] to predict text quality. Figure 4 illustrates the architecture. The Transformer can find the global relationships between two different features and output them as features. Convolution cannot deal with scene text images of various shapes and angles owing to its narrow effective range. Therefore, the Transformer with a global correlation is appropriate. The Transformer has an encoder and decoder. The positional encoding processes, called FPE and RPE, are carried out separately since the input does not consider the position or semantic order. Therefore, positional information must be added in advance.

The encoder extracts correlations and features from the text strings. In addition, the encoder reshapes the size of the features. The encoder embeds the sequence of character probabilities and the image features into the same embedding space. Specifically, the self-attention operation is used to extract correlations between the probabilities. The feed-forward network performs feature refinement.

The decoder transforms the image features into those that contain text information. Cross-attention is used to extract mutual features for the encoder output and image features. In other words, the decoder associates the text information output from the encoder with the corresponding positions in the image features. With the decoder, image features include text information, which strongly influences the text quality assessment.

During training, the loss function uses the error between the ground truth and predicted labels. We use the L1 norm loss and the $\epsilon$-insensitive loss function. The $\epsilon$-insensitive loss function does not give a penalty if the error is within the allowed $\epsilon$ and gives a penalty if the error exceeds $\epsilon$. The formula for the $\epsilon$-insensitive loss function $L_{\epsilon}$ is defined in Eq. (1). $GT$ represents the ground truth label and $Pred$ the predicted value. In this experiment, $\epsilon=0.1$. The reason for introducing the $\epsilon$-insensitive loss function is to penalize the learning process further if the estimation of text quality is largely incorrect. The overall loss function $L_{all}$ is defined by Eq. (2).

The proposed image compression method uses variable-rate deep image compression [22]. The network is illustrated in Figure 5. We can perform variable-rate image compression by using a pixel-by-pixel quality map.

The SFT layer [25] is used in the condition network, encoder, and decoder. The others consist of convolution and normalization layers. The condition network fuses the information from a quality map using image features. Pixel-by-pixel quality maps enable high image quality in various areas, such as text and significant areas. Therefore, it can be used for a wide variety of tasks. The quality map has values ranging from [0, 1] to allow fine quality control. Henceforth, the value assigned to the quality map is denoted by the weight $k$.

A previous work [22] assigns the same weight value $k$ to all text regions in the image. However, the size and shape of the text vary significantly. Thus, the texts are different in readability. Therefore, we assigned different weights to each text region in this study.

We propose an image compression method using a quality map to preserve text quality assessed by STIQA. The flow of the proposed image compression method is illustrated in Figure 6. This quality map is weighted differently for each text region.

The initial quality map has weights of 0.5 on pixels of character edges and inside. Otherwise, the weight is 0.2 for the background. High weights are assigned to the edges because character edges are the most critical factors for readability. We use the Canny method to extract edges.

We compress images and assess the quality of each text in the compressed image. We update weights $k_{new}$ by the distance between the assessment results and the target text quality. The update process is defined in Eq. (3). $k_{old}$ denotes the value of the previous weights, $\lambda$ is the hyperparameter that increases the update amount. $Score_{target}$ represnets the target text quality. $Score$ is the assessed text quality. We use $\lambda=5$ and $Score_{target}=0.90$ in this work.

We iteratively perform image compression using the updated quality map. In this paper, the repetition is three times. Consequently, we can obtain a quality map that achieves high compression while maintaining text quality.

The image compression results were presented in Table 1. The results were averaged over the test images in ICDAR2013. The results showed that the proposed method was superior for all the metrics. The proposed method obtained the second-best results for the entire image. The proposed image compression method outperformed traditional image compression methods, such as JPEG. Furthermore, the proposed method is the best in terms of text quality and text detection metrics. Therefore, the results verified the effectiveness of the proposed image compression method.

Qmap outperformed ours in overall image quality because Qmap was optimized for overall images, whereas our method focuses on text readability. Image compression is to determine a bitstream for a set of pixel values, as shown in Figure 5. Qmap determines a bitstream using uniform weights for all pixels, i.e., the overall image. Thus, the overall quality is favorable. However, text quality greatly deteriorates since the number of text pixels is smaller than the other pixels. In contrast, our method modifies weights by assigning more bits to text, resulting in a bitstream maintaining text readability. As shown in Figure 7, although overall image quality is reduced slightly, text quality improved greatly.

Some of the compressed images are shown in Figure 7. JPEG and JPEG2000 were significantly degraded in both background and text quality. In addition, the naive method degrades only the background. By contrast, the proposed method achieves a higher image quality for text regions and a higher image quality for background regions compared to the other methods.

Subjective evaluation experiments were conducted using the proposed and naive methods. Fifteen participants chose better readability between two compressed images in terms of the entire image and the text region. Each participant had 50 pairs.

The results are presented in Table 2. The proposed method is superior for the entire image and text regions. The proposed method obtained more than 66% of the votes for the text region. These results demonstrate the capability of this method in maintaining text readability.

The effect of varying the hyperparameter $\lambda$ and the number of updates was also investigated. $\lambda$ is defined in Eq. (3) to determine the amount of quality map update. Table 3 shows the bit rates (bpp) and the average score of the text qualities assessed using STIQA. The bit rates and STIQA scores increased according to $\lambda$ and the number of updates. The best STIQA score was obtained when $\lambda=5$. By considering the computing time, it was determined that three update cycles were sufficient.

The results are presented in Table4. Three evaluation metrics were used: mean absolute error, Spearman’s rank correlation coefficient, and Pearson’s correlation coefficient. We used the indicators to show a relationship between humans and our model in text quality assessment. Specifically, MAE shows the gap between a model and humans. Pearson’s correlation coefficient uses scores of text quality to evaluate a linear relationship. Spearman’s rank correlation coefficient uses ranks among data to verify a monotonic relationship. Thus, we can comprehensively evaluate text quality assessment models using the three indicators.

The results showed that STIQA performed the best on the test dataset with text quality labels obtained from subjective evaluation experiments. The main difference between STIQA and other methods is the existence of a text recognition model, which may play a significant role in text quality assessment.

We evaluated the impact of the combinations of the loss functions: the L1 loss and the $\epsilon$-insensitive loss. The results are presented in Table 6. The $L_{\epsilon}$ loss was better than the $L_{1}$ loss. In addition, the best results were achieved by using the two loss functions simultaneously, and the combination with an $\epsilon$ of 0.10 was the best overall.