MSLKANet: A Multi-Scale Large Kernel Attention Network for Scene Text Removal

Fig.1 shows the overall architecture of the proposed MSLKANet. It is based on the UNet-Style architecture with encoder, decoder, and skip-connections. Furthermore, we propose large kernel spatial pyramid pooling (LKSPP) and multi-scale large kernel attention (MSLKA) for obtaining long-range dependencies between the text regions and the backgrounds. In the loss functions, we use reconstruction loss, style loss [8], and perception loss [16] to keep the same with most existing methods.

Attention mechanisms can help networks focus on important information and ignore irrelevant ones. And multi-scale feature fusion methods effectively combine image features at different scales and are now widely used to collect useful information about objects and their surroundings. However, they seem to be ignored by previous methods. Compare with other attention mechanisms(Channel Attention [13], Self-Attention [28]), Large Kernel Attention [10] combines the advantages of convolution and self-attention. It takes the local contextual information, large receptive field, linear complexity, dynamic process, spatial dimension adaptability, and channel dimension adaptability into consideration.

Under the guidance of the above ideas, we propose multi-scale large kernel attention (MSLKA) to combine large kernel decomposition and multi-scale learning for obtaining heterogeneous-scale correlations between the text regions and the background regions.

Large Kernel Attention. LKA adaptively builds the long-range relationship by decomposing a $K\times K$ convolution into a $\lceil\frac{K}{d}\rceil\times\lceil\frac{K}{d}\rceil$ depth-wise dilation convolution with dilation $d$, a $(2d-1)\times(2d-1)$ depth-wise convolution and a 1$\times$1 convolution. Through the above decomposition, it can capture a long-range relationship with slight computational cost and parameters.

Multi-Scale Large Kernel Attention. To learn the attention maps with multi-scale longe range information, we improve the LKA with the group-wise multi-scale mechanism. As shown in Fig.2, given the input feature maps $X\in\mathbb{R}^{C\times H\times W}$, we split the input into different groups. For different groups, we use different dilation rates for various perceptive fields for multi-scale longe range information. In detail, we leverage four groups of LKA with a set of dilation rates$\{2,3,4,5\}$. By setting a fix-size kernel 5 in each depth-wise $d$-dilation convolution, we get a set of large kernels $\{10,15,20,25\}$. Through the above convolutions, we get the four scale attention maps and then reweight the different groups of features. In the last, we concat the four groups of features as output.

For obtaining large perceptive field and multi-scale information, Spatial Pyramid Pooling(SPP) is another frequently-used method. Atrous Spatial Pyramid Pooling(ASPP) proposed and improved in [4, 3, 5] is a most common and efficient block in different SPP methods(SPP[11], PPM[34], ASPP). In Deeplabv2[5], the ASPP block only includes several parallel atrous convolutions with different rates for getting multi-scale information and a 1x1 convolution. Image-level pooling ars insert into ASPP for obtaining global information in Deeplabv3 [4]. In Deeplabv3+[5], they use depthwise separable atrous convolution replacing the standard atrous convolution for reducing the cost of computation. Motivated by LKA and the development of ASPP, we use decomposed large kernel convolution to substitute the depthwise separable atrous convolution for the fusion of more valid pixels in the spatial dimension while maintaining large receptive fields and lower cost of computation. In detail, our large kernel spatial pyramid pooling (LKSPP) includes three parallel decomposed large kernel convolutions with different rates(3,4,5) and a fix-size kernel 7 in each depth-wise dilation convolution for getting multi-scale information, a 1x1 convolution, and an image-level pooling as shown in Fig.3, .

We introduce three loss functions for training MSLKANet, including reconstruction loss, perceptual loss, and style loss. Given the input scene text image $I_{i}$, text-removed ground truth (gt) $I_{gt}$, and the STR output from MSLKANet is denoted as $I_{o}$.

Reconstruction Loss. First of all, we apply $L_{1}$ distance is proposed to measure the output and the ground truth in the pixel level.

Perceptual Loss. To capture the high-level semantic difference and simulate the human perception of image quality, we utilize the perceptual loss[16] in Eq.(2). $\Phi_{i}$ is the activation features of the $i$-th layer of the VGG-16 backbone.

Style Loss. We also use the style loss [14] for reducing the global level style of output $I_{o}$ and its corresponding ground-truths $I_{gt}$ as Eq.(3), where $G_{i}^{(}\Phi)$ is the Gram matrix constructed from the selected activation features in perceptual loss, and $||||_{2}$ is mean-squared distance.

In Summary, the total loss for training MSLKANet is the weighted combination of the above mentioned losses:

In experiments, $\lambda_{s}$, $\lambda_{p}$ is set to be 120,0.01, respectively.

SCUT-Syn. It is a synthetic dataset that splits 8,000 images for training and 800 images for testing. More details can be found in [33].

SCUT-EnsText. It contains 2,749 training images and 813 test images which are collected in real scenes. More descriptions refer to [17].

Evaluation Metrics:. Following the previous methods [33, 17, 20] For detecting text on the output images, we employ a text detector CRAFT[1] to calculate recall and F-score. The lower, the better. PSNR, MSE, MSSIM, AGE, pEPs, and pCEPS are adopted for measuring the equality of the output images. The higher MSSIM and PSNR, and the lower AGE, pEPs, pCEPS, and MSE indicate better results.

We train MSLKANet on the training set of SCUT-EnsText and SCUT-Syn and evaluate them on their corresponding testing sets. We follow [17] to apply data augmentation during the training stage. We apply a random rotation of a maximum degree of 10◦, random horizontal flip, and random vertical flip with a probability of 0.5 for data augmentation during training. The model is optimized by adamW optimizer with the weight decay of $1e^{-4}$ and the epison of $1e^{-3}$. The initial learning rate is set to be $1e^{-3}$. We also use the warmup strategy [9] that increases the learning rate from 0 to the initial learning rate linearly to overcome early optimization difficulties. After the learning rate warmup, we typically apply the cosine annealing strategy [19] to steadily decrease the value from the initial learning rate. The MSLKANet is trained by a single NVIDIA GPU 3090 with a batch size of 16 and the input image size of 256 $\times$ 256.

In this section, we verify the contributions of different components MSLKA and LKSPP on the SCUT-Text dataset. Our baseline model is implemented by a UNet-Style model similar to the pioneer method. In total, we conduct four experiments: 1. baseline, 2.baseline+MSLKA, 3.baseline+MSLKA+ASPP 4.baseline+MSLKA+LKSPP. All experiments use the same training and test settings.

MSLKA. MSLKA aims to obtain long-range dependencies between the text regions and the backgrounds at various granularity levels. As shown in table2, our MSLKA provides significant enrichments over the baseline on all metrics.

LKSPP. LKSPP is proposed to perceive more valid pixels in the spatial dimension while maintaining large receptive fields and lowering the cost of computation. It can also provide global information According to the results shown in table2, our LKSPP is helpful in both text removal and background in painting.

In this section, we compare our proposed MSLKANet with six state-of-the-art methods: Pix2pix[14], Scene Text Eraser[21], EnsNet[33], EraseNet[17], PERT [30] and PSSTRNet [20], on both SCUT-EnsText and SCUT-Syn datasets. Qualitative and quantitative results are illustrated in Fig.4, table3 and 4, respectively.

Qualitative Comparison. As shown in the 1st, 3rd, and 4th rows of Fig.4, our model can remove multi-scale texts, especially for the larger scale text. Compared with other state-of-the-art methods, the results of our proposed MSLKANet have significantly fewer color discrepancies and blurriness, and cleaner background. They demonstrate our model could generate more natural results on text removal and background inpainting results.

Quantitative Comparison. As shown in table 4 and 4, our method produces the best scores on most scene text removal and image inpainting equality evaluation metrics for both SCUT-EnsText and SCUT-Syn datasets. It indicates the advances of our proposed MSLKANet.

Following the important scene text removal network EnsNet[33], we also execute an experiment on the general object removal task to test the generalization ability of MSLKANet. We also evaluate MSLKANet on the well-known Scene Background Modeling (SBMnet) dataset [15], which is proposed for background estimation algorithms. The qualitative results are shown in Fig.5, our MSLKANet reconstructs a natural background. This means that our method can be generalized to remove pedestrians, cars, and fish without specific adjustments. It can certify the generalization of our proposed MSLKANet.