Learning a Sketch Tensor Space for Image Inpainting of Man-made Scenes

Partially Gated Convolutions. We adopt the Gated Convolution (GC) layers to process the masked input features, as it works well for the irregular mask inpainting tasks in  [43]. Unfortunately, GC demands much more trainable parameters than vanilla convolutions. To this end, as shown in Fig. 2 and Fig. 3(a), we propose a partially GC strategy of only utilizing three GC layers for the input and output features in both encoder and decoder models. Essentially, this is motivated by our finding that the outputs of GC mostly devoting to filtering features of masked and unmasked regions only in the encoder layers of the coarse network and the decoder layers of the refinement network in [43]. In contrast, we do not observe significantly performance improvement of using GC in the middle layers of backbones. Thus, we maintain vanilla convolutions (i.e., residual blocks) in the middle layers, to save parameters and improve the performance as empirically validated in Tab. 4.

Efficient Attention Block. Intuitively, attention is important to learn patterns crossing spatial locations in image inpainting [42, 25, 21, 39]. However, attention modules are expensive to be computed, and non-trivial to be parallelized in image inpainting [42]. To this end, we leverage Efficient Attention (EA) module among middle blocks of Fig. 2 and detailed in Fig. 3(b). Particularly, for the input feature $\mathbf{X}\in\mathbb{R}^{h\times w\times d}$, and mask $\mathbf{M}\in\mathbb{R}^{h\times w\times 1}$, we first reshape them to $\mathbb{R}^{hw\times d}$ and $\mathbb{R}^{hw\times 1}$ as the vector data respectively. Then, the process of efficient attention can be written as

$$\vspace{-0.07in}\begin{split}\mathbf{Q}&={\mathrm{softmax}}_{row}(\mathbf{W}_{q}\mathbf{X}+(\mathbf{M}\cdot-\infty))\\ \mathbf{K}&={\mathrm{softmax}}_{col}(\mathbf{W}_{k}\mathbf{X}+(\mathbf{M}\cdot-\infty))\\ \mathbf{V}&=\mathbf{W}_{v}\mathbf{X},\;\;\;\;\;\;\;\mathbf{E}=\mathbf{Q}(\mathbf{K}^{T}\mathbf{V}),\end{split}$$

(2)

where $\mathbf{W}_{q,k,v}$ indicates different learned parameter matrices; $+(\mathbf{M}\cdot-\infty)$ means masking the corrupted inputs before the softmax operation. Critically, $\mathrm{softmax}_{row}$ and $\mathrm{softmax}_{col}$ means the softmax operations on the row and column individually. Then, we achieve the output $\mathbf{E}\in\mathbb{R}^{hw\times d}$, which will be reshaped back to $\mathbb{R}^{h\times w\times d}$. Note that Eq. (2) is an approximation to vanilla attention operation as in  [31]. Typically, this strategy should in principle, reduce significantly computational cost. Since $\mathbf{E}$ is computed by $\mathbf{K}^{T}\mathbf{V}\in\mathbb{R}^{d\times d}$, rather than standard $\mathbf{Q}\mathbf{K}^{T}\in\mathbb{R}^{hw\times hw}$. In practice, the dimension $d$ is much smaller than $hw$ in computer vision tasks. Furthermore, as in Fig. 3(b), we introduce the multi-head attention [33] to further reduce the dimension from $d$ to $d^{\prime}=d/n_{head}$. Thus, $\mathbf{K}^{T}\mathbf{V}$ could aggregate the query $\mathbf{Q}$ in feature level and obtain the global context $\mathbf{E}$. Note that EA module is inspired but different from  [31], as the attention scores of corrupted regions are masked to aggregate features from uncorrupted regions.

Pyramid Decomposing Separable (PDS) Block. We have found that GAN-based edge inpainting [35] suffers from generating meaningless edges or unreasonable blanks for masked areas as shown in Fig. 6, due to the nature of sparsity in binary edge maps. Thus, we propose a novel PDS block in PSS as in Fig. 3(a). PDS leverages dilated lower-scale structures to make up the sparse problem for ST space. Particularly, assume the image feature $\mathbf{X}\in\mathbb{R}^{h\times w\times d}$, PDS firstly decouples it as

$$\vspace{-0.05in}\begin{split}\mathbf{E}_{le}&=f_{eb}(\mathrm{GateConv}(\mathbf{X})),\;\{\mathbf{E}_{l},\mathbf{E}_{e}\}=\mathrm{split}(\mathbf{E}_{le}),\\ \negthickspace\mathbf{O}_{e}&=\sigma(\mathrm{Conv}(\mathbf{E}_{e})),\;\mathbf{O}_{l}=\sigma(\mathrm{Conv}(\mathbf{E}_{l})),\end{split}$$

(3)

where $\sigma$ is sigmoid activation. We project the feature into two separated embedding spaces called line embedding $\mathbf{E}_{l}$ and edge embedding $\mathbf{E}_{e}$ with the function $f_{eb}$, which is composed with $\mathrm{Conv2D\rightarrow IN\rightarrow ReLU}$. $\mathrm{split}$ is the operation of splitting the features into two respective tensors with equal channels. Then, $\mathbf{E}_{l}$ and $\mathbf{E}_{e}$ are utilized to learn to roughly reconstruct the image $\mathbf{O}_{im}$ without corruption as

$$\vspace{-0.05in}\begin{split}\mathbf{A}&=f_{ab}(\mathbf{E}_{le})\in\mathbb{R}^{h\times w\times 1}\\ \mathbf{E}_{le}^{\prime}&=\mathbf{E}_{l}\odot(1-\mathbf{A})+\mathbf{E}_{e}\odot\mathbf{A}\\ \mathbf{O}_{im}&=\mathrm{tanh}(\mathrm{Conv}(\mathbf{E}_{le}^{\prime})),\end{split}$$

(4)

where $f_{ab}$ is $\mathrm{\mathrm{Conv2D}\rightarrow IN\rightarrow ReLU\rightarrow Conv2D\rightarrow Sigmoid}$. $\odot$ denotes the element-wise product. The prediction of coarse $\mathbf{O}_{im}$ gives a stronger constraint to the conflict of $\mathbf{E}_{l}$ and $\mathbf{E}_{e}$, and we expect the attention map $\mathbf{A}$ can be learned adaptively according to a challenging target. Furthermore, a multi-scale strategy is introduced in the PDS. For the given feature $\mathbf{X}^{(1)}\in\mathbb{R}^{64\times 64\times d}$ from dilated residual blocks of PSS, PDS predicts lines, edges, and coarse images with different resolutions as follow

$$\vspace{-0.05in}\mathbf{X}^{(i+1)},\mathbf{O}_{im}^{(i)},\mathbf{O}_{l}^{(i)},\mathbf{O}_{e}^{(i)}=\mathrm{PDS}^{(i)}(\mathbf{X}^{(i)}),$$

(5)

where $i=1,2,3$ indicate that the output maps with 64$\times$64, 128$\times$128, and 256$\times$256 resolutions respectively. Various scales of images can be in favor of reconstructing different types of structure information.

Loss Function of PSS. We minimize the objectives of PPS with two spectral norm [27] based SN-PathGAN [43] discriminators $D_{l}$ and $D_{e}$ for lines and edges respectively. And the $\mathcal{L}_{D}^{enc}$ and $\mathcal{L}_{G}^{enc}$ are indicated as

$$\vspace{-0.1in}\begin{split}\mathcal{L}_{D}^{enc}=&\mathcal{L}_{D_{l}}+\mathcal{L}_{D_{e}},\\ \mathcal{L}_{G}^{enc}=&\lambda_{a}\mathcal{L}_{adv}^{enc}+\lambda_{f}\mathcal{L}_{fm}+\sum_{i=1}^{3}\lVert\mathbf{O}_{im}^{(i)}-\mathbf{I}^{(i)}\rVert_{1},\end{split}$$

(6)

where $\mathbf{O}_{im}^{(i)}$, $\mathbf{I}^{(i)}$ are the multi-scale outputs (Eq. 5) and ground-truth images varying image size $64\times 64$ to $256\times 256$ respectively. $\mathcal{L}_{fm}$ is the feature matching loss as in [28] to restrict the $l_{1}$ loss between discriminator features from the concatenated coarse-to-fine real and fake sketches discussed below. And the adversarial loss can be further specified as

$$\vspace{-0.05in}\begin{split}\mathcal{L}&{}_{adv}^{enc}=-\mathbb{E}\left[\log D_{l}(\hat{\mathbf{O}}_{l})\right]-\mathbb{E}\left[\log D_{e}(\hat{\mathbf{O}}_{e})\right],\\ \mathcal{L}&{}_{D_{l}}=-\mathbb{E}\left[\log D_{l}(\hat{\mathbf{I}}_{l})\right]-\mathbb{E}\left[1-\log D_{l}(\hat{\mathbf{O}}_{l})\right],\\ \mathcal{L}&{}_{D_{e}}=-\mathbb{E}\left[\log D_{e}(\hat{\mathbf{I}}_{e})\right]-\mathbb{E}\left[1-\log D_{e}(\hat{\mathbf{O}}_{e})\right],\end{split}$$

(7)

where $\hat{\mathbf{O}}_{l}$, $\hat{\mathbf{O}}_{e}\in\mathbb{R}^{256\times 256\times 3}$ are got from the multi-scale PSS outputs of lines and edges, which are upsampled and concatenated with $\hat{\mathbf{O}}_{l}=[\mathrm{up}(\mathbf{O}_{l}^{(1)});\mathrm{up}(\mathbf{O}_{l}^{(2)});\mathbf{O}_{l}^{(3)}]$ and $\hat{\mathbf{O}}_{e}=[\mathrm{up}(\mathbf{O}_{e}^{(1)});\mathrm{up}(\mathbf{O}_{e}^{(2)});\mathbf{O}_{e}^{(3)}]$. To unify these outputs into the same scale, the nearest upsampling $\mathrm{up()}$ is utilized here. Accordingly, $\hat{\mathbf{I}}_{l}$, and $\hat{\mathbf{I}}_{e}$ are the concatenation of multi-scale ground-truth edge and line maps, respectively. Note that, we firstly dilate the ground-truth edges and lines with the $2\times 2$ kernel, and then subsample them to produce the low resolution maps at the scale of 64$\times$64, 128$\times$128. The hyper-parameters are set as $\lambda_{a}=0.1$, $\lambda_{f}=10$ .

Quantitative Comparisons. In this section, we evaluate results with PSNR, SSIM [36], and FID [11]. As discussed in [46], we find that $l_{2}$ based metrics such as PSNR and SSIM often contradict human judgment. For example, some meaningless blurring areas will cause large perceptual deviations but small $l_{2}$ loss [46]. Therefore, we pay more attention to the perceptual metric FID in quantitative comparisons. The results on ShanghaiTech, P2M, and York Urban are shown in Tab. 1. From the quantitative results, our method outperforms other approaches in PSNR and FID. Especially for the FID, which accords with the human perception, our method achieves considerable advantages. Besides, our method enjoys good generalization, as it also works properly in P2M, York Urban, and even P2C.

Qualitative Comparisons. Qualitative results among our method and other state-of-the-art methods are shown in Fig. 5. Compared with other methods, our approach achieves more semantically coherent results. From the enlarged regions, our schema significantly outperforms other approaches in preserving the perspectivity and the structures of man-made scenes. Moreover, we show the edge results of EC [28] and partial sketch tensor space (edges and lines) results of our method in ShanghaiTech, P2M, and P2C in Fig. 6. Results in Fig. 6 demonstrate that line segments can supply compositional information to avoid indecisive and meaningless generations in edges. Furthermore, line segments can also provide clear and specific structures in masked regions to reconstruct more definitive results in man-made scenes. Besides, for the natural scene images in P2C without any lines, our method can still outperform EC with reasonable generated edges for the masked regions. As discussed in Sec. 3.2, the proposed PDS works properly to handle the sparse generative problem.

Results of Natural Scenes. In the last three rows of Tab. 1, we present the results of ours method on the comprehensive Places2 dataset (P2C) to confirm the generalization. They are compared with GC and EC, which have achieved fine scores in man-made Places2 (P2M). Note that all metrics are improved in P2C compared with ones in P2M of Tab. 1, which demonstrates man-made scenes are more difficult to tackle. Our methods still get the best results among all competitors, and the object removal results achieve superior performance. Besides, the last two rows of Fig. 6 show that our method can generate reliable edge sketches even without lines in natural scenes. These phenomenons are largely due to two reasons: 1) There is still a considerable quantity of line segments in the comprehensive scenes, and these instances are usually more difficult than others. 2) The proposed partially gated convolutions, efficient attention, and PDS blocks can work properly for various scenes.

Human Judgements. For more comprehensive comparisons, 50 inpainted images from GC [43], EC [28], and ours are chosen from ShanghaiTech and P2M randomly. And these samples are compared by 3 uncorrelated volunteers. Particularly, volunteers need to choose the best one from the mixed pool of the inpainted images with different methods, and give one score to the respective approach. If all methods work roughly the same for a certain sample, it will be ignored. The average scores are shown in Tab. 2. Ours method is significantly better than other competitors.

Object Removal. From the last column in Fig. 5, we show the inpainting results of our MST working in the post-process of LSM with $m=0$ for the object removal. These refined images with more reasonable structures indicate that the our model can strengthen the capability of image reconstruction without redundant residues. From Fig. 7, some object removal cases are shown to confirm the practicability of the proposed method. Numerical results are presented in the last columns of Tab. 1. So, our method can achieve further improvements in all metrics compared with the vanilla image inpainting schema for the object removal, and it can correctly mask undesired lines. Therefore, the post-process of LSM can significantly improve the performance in the object removal task with the separability of line segments .

Masked Wireframe Detection. As discussed in Sec. 3.1, we retrain the HAWP with mask augmentation to ensure the robustness for the corrupted data as LSM-HAWP in ShanghaiTech dataset [14]. To confirm the effect of the augmentation, we further prepare another masked ShanghaiTech testset with the same images. The results are shown in Tab. 3. The structural average precision (sAP) metric is proposed in [49], which is defined as the area under the PR curve of detected lines with different thresholds. From Tab. 3, LSM-HAWP performs better than vanilla HAWP significantly for the masked testset. Moreover, HAWP can also gain improvements in the uncorropted testset with more than 1% in sAP, which indicates the great generality of the mask augmentation for the wireframe detection task.

Simplification of Gated Convolutions. The related ablation study of GC are shown in Tab. 4. Only replacing vanilla convolutions for input and output stages of the model with partially GC gains satisfactory improvements. But replacing all convolution layers with GC fails to achieve a further large advance, while the parameters are doubled²²2The visualization results of GC are shown in the supplementary..

Contributions of Lines, PDS, and EA. We explored the effects of lines, Pyramid Decomposing Separable (PDS) blocks and the Efficient Attention (EA) module in Tab. 5 in ShanghaiTech. Specifically, line segments, PDS blocks and the EA module are removed respectively, while other settings unchanged. As shown in Fig. 8, our method without line segments causes severe structural loss, and the one without PDS suffers from contradictory perspectivity. From Tab. 5, we can see all line segments, PDS blocks and the EA module can improve the performance in image inpainting.

Partially Gated Convolution (GC) Block. Both encoder and decoder contain two groups of partially GC blocks for the input (3 blocks) and output (2 blocks) features, and the GC is consist of GateConv2D [43] $\rightarrow$ InstanceNorm $\rightarrow$ ReLU. We also add the spectral normalization [27] to the GC used in the generator.

Dilated Residual Block. The dilated residual blocks are the same as the ones used in EdgeConnect [28] with Conv2D $\rightarrow$ InstanceNorm $\rightarrow$ ReLU $\rightarrow$ Conv2D $\rightarrow$ InstanceNorm, where the first convolution is based on $\mathrm{dilation}=2$.

Efficient Attention. The input feature of the Efficient Attention is $256$ channels, and we use the multi-head attention with $n_{head}=4$. So, the dimension $d^{\prime}$ of each group is $256/4=64$.