Image Inpainting Guided by Coherence Priors of Semantics and Textures

Deep learning-based inpainting approaches have recently been proposed by understanding the images, which can generate meaningful content for filling in the missing region. Context Encoder [22] was first proposed to employ a generative adversarial network and demonstrated its potential for inpainting tasks. Based on it, efforts have been made to enhance the inpainting performance, including introducing specific losses [6, 32, 44], building recursive architectures for progressive refinement [13, 37], and involving the structural priors as guidance in a two-stage framework for structural consistency [14, 19, 24, 38]. However, these methods lack the ability to model long-term correlations between distant context, leading to blurry textures.

To better refine the inpainted image textures, non-local algorithms are adopted to borrow distant features from a known region, which contains fine textures, to the missing region. [39] firstly proposed to compute textural affinity within the same image to fill the corrupted area with more realistic texture patches from the available area. [42] devised a pyramid of contextual attention at multiple layers to refine the textures from high-level to low-level. [17] used a coherent semantic attention layer to ensure semantic relevance between nearby filled features. [34] extends the single attention map to bidirectional attention maps and re-normalizes the features to let the decoder concentrate on filling the holes. Although these methods have delivered considerable improvements, they failed to address the semantic ambiguity as they try to measure the texture affinity across all semantics.

Recent research reveals that the semantic priors of the high-level vision tasks are useful in the guidance of low-level vision tasks [4, 16, 30]. One hot research topic is semantic-guided image generation. It covers several research directions, including image translation from segmentation maps to realistic images [10, 29] and semantic image synthesis [4, 27, 30]. To avoid the vanishing of semantic priors in the generation process, [21] proposed a spatially adaptive normalization layer to propagate the semantic information to the synthesized images. With the useful semantic priors, it can generate high-quality images.

The semantic priors have also been applied to promote many conditional low-level vision tasks and demonstrated its effectiveness in constraining the plausible solution space in the ill-posed problems. For example, they are involved in the tasks of super-resolution [31], dehazing [23], denoising [16], style transfer [18, 28] and image manipulation [9, 20]. Inspired by the successful assistance from semantic priors in conditional image generation, we also exploit the semantic guidance in image inpainting.

We build our network on an alternating-optimization architecture to utilize the coherence priors to mutually assist image inpainting and semantic segmentation for a corrupted image. Specially, we propose a multi-task learning framework by sharing features in the decoders for the two tasks (as shown in Figure 2). The encoder encodes the corrupted image and its mask into hierarchical contextual features, which are then fed into the decoder to predict the inpainted images and semantic segmentation maps across scales. Prior to feeding the encoded feature of the last layer into the decoder, we initially complete the feature via a Context Inference Module (CIM) based on the contextual inference method [15, 33].

In the decoder, the contextual feature at each scale is processed by two task-specific heads to predict the inpainted image and the segmentation map, respectively. Different from the method proposed in [15] that updates the contextual features by spatial adaptive normalization to capture the common properties of the same semantic, we propose a SWAP module to stress the realistic texture of each semantic patch by referring to the semantic relevant features from the known regions. In this way, the contextual features are learnt to represent the global structure and refined with the semantic-aware texture details.

For brevity, we adopt the following notations. $\varphi^{l}$ and $\phi^{l}$ denote the features from the encoder and from the decoder at scale $l$, respectively; $\hat{I}^{l}$ and $\hat{S}^{l}$ respectively represent the inpainted image and the predicted $k$-channel segmentation maps from the inpainting head $h(\cdot)$ and from the segmentation head $g(\cdot)$ at scale $l$, where $k$ is the total number of semantic labels; $l$ ranges from 1 (the coarsest layer) to 5 (the finest layer).

The SWAP module is designed to optimize the contextual features by enhancing the semantic authenticity of textures based on coherence priors. As shown in Figure 2(b), SWAP takes four inputs: two of them are the current-scale feature $\phi^{l}$ and the next-scale skip feature $\varphi^{l+1}$ from the encoder. The third is the predicted segmentation probability map $\hat{S}^{l}$, that is used to guide the separation of features. The last is the missing-region mask $M$. The propagation process can be formulated as follows:

where $swap(\cdot)$ is the process of refining the contextual features in SWAP.

The attention-based approaches in [25, 34, 39] resort to contextual attention to pick known regions as references to complete a missing region, which, however, cannot distinguish patches of different semantics and thereby leads to blurry boundaries and semantic confusion during attention propagation. Unlike these approaches, SWAP calculates the attention score by matching semantic-aware features of missing patches and the known patches based on the coherence priors. Specifically, we first split the contextual features $f^{l}$, which is generated from $\phi^{l}$ and $\varphi^{l+1}$, into different semantic parts according to the semantic labels in the predicted segmentation map $\hat{S}^{l}$ of the $l$-th layer. We subsequently drop the superscript $l$ for the sake of notational simplicity.

Within each semantic part, each patch’s attention score is evaluated by the patch affinity between the missing region and a known region using the normalized inner product followed by a softmax operation:

After obtaining the attention score from the known region, the feature of the $j$-th missing patch is updated by

The output feature of SWAP is generated by merging the updated features of all semantics, followed by four groups of dilated convolutions with different rates to improve the structural coherence in the final reconstructed features.

We devise new coherence losses between the semantics and textures as supervisions to guide the image inpainting and semantic segmentation to meet the following requirements: 1) the overall structure between the inpainted image and segmentation map should match each other; 2) the predicted textures of a certain semantic class should have the same distribution as that of the semantic textures in the known region. Under these considerations, we propose two coherence losses shown in Figure 3, a non-local patch coherence loss and a structure coherence loss, to respectively evaluate the patch similarity and the structural matching.

Similar to the attention propagation process, we first split $\hat{I}$ and $I$ into different semantic images and extract patches to build the corresponding semantic patch sets $P^{c}=\{p_{j}^{c}\}$ and $B^{c}=\{b_{i}^{c}\}$ according to $\hat{S}$ and $S$. For each patch in $P^{c}$, we randomly select one patch from $B^{c}$ with the same semantic, and compute the pairwise cosine distances between them as

The non-local patch coherence loss for each semantic class $c$ aims to maximize the similarity between the patch couples:

We design appropriate supervised loss terms for learning the inpainting and segmentation tasks at each scale to obtain multi-scale predictions. We adopt the reconstruction loss, the adversarial loss and the proposed coherence losses to promote the fidelity of the inpainted images. The cross entropy loss is adopted for ensuring the accuracy of the predicted segmentation maps.

We evaluate our method on the Outdoor Scenes [31] and Cityscapes [45] datasets. Outdoor Scenes contains 9,900 training images and 300 test images. Cityscapes contains 5,000 street-view images in total. In order to enrich the training set of Cityscapes, we use 2,975 images from the training set and 1,525 images from the test set for training, and test on the 500 images from the validation set. Since the test set lacks human-labeled semantic annotations, we generate the annotations for training by using the state-of-the-art segmentation model Deeplab [3]. We resize each training image to ensure its minimal height/width to be 256 for Outdoor Scenes and 512 for Cityscapes, and then randomly crop sub-images of size $256\times 256$ as inputs to our model. The fine annotations of segmentation labels for both datasets are also provided for training, in which Outdoor Scenes and Cityscapes are annotated to 8 and 20 categories, respectively. Please note that the annotations can also be replaced by the extracted segmentation maps from the state-of-the-art segmentation models.

We compare our method with the following four learning-based inpainting methods: 1) GatedConv [40]: Contextual attention for leveraging the surrounding textures and structures. 2) EdgeConnect [19]: Two-stage inpainting framework with edges as low-level structural information. 3) PEN-Net [42]: Cross-layer attention transfer and pyramid filling in a multi-scale framework. 4) SPG-Net [26]: Two-step inpainting with a semantic segmentation map as high-level structural information. 5) SGE-Net [15]: Semantic guidance for inpainting based on spatially adaptive normalization [21].

Figure 4 shows the qualitative comparisons of our method with all the baselines. The corrupted area is simulated by sampling a central hole ($128\times 128$ for Outdoor Scenes and $96\times 96$ for Cityscapes) or randomly placing multiple irregular masks based on [40]. As shown in the figure, the baselines usually suffer from artifacts and unsatisfactory boundaries while completing complex holes. GatedConv and PEN-Net adopt contextual attention to bring in the features of the known region, but they usually distort the structures when referencing to incorrect semantic textures from the surrounding, especially in completing the complex holes. EdgeConnect and SGE-Net are able to recover correct structures owning to the use of structure priors. However, EdgeConnect may generate mixed edges, making it difficult to generate correct textures, whereas the textures of SGE-Net are often over-smoothed without texture refinement. In contrast, our method generates more realistic textures and better boundaries delineating semantic regions than all the baselines thanks to the coherence priors between semantics and textures.

Table 2 shows the quantitative comparisons on Outdoor Scenes and Cityscapes datasets based on three quality metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) and Fréchet Inception Distance (FID) [8]. In general, the proposed method achieves significantly better objective scores than the baselines, especially in PSNR and SSIM.

We randomly select 100 images from the two datasets (50 from Outdoor Scenes and 50 from Cityscapes) and invite 20 subjects with image processing expertise to rank the subjective visual qualities of images inpainted by the five inpainting methods (GatedConv, PEN-Net, EdgeConnect, SPG-Net, SGE-Net, and our method). They are not informed of any mask information. For each test image, its five inpainting results are presented in a random order, and each subject is asked to rank the five methods from the best to the worst. The result shows that our method receives 51.8 % favorite votes (i.e., the top-1 in 1,036 out of 2,000 comparisons), surpassing 19.3 % with SEG-Net, 11.3 % with EdgeConnect, 7.8 % with GatedConv, 5.6 % with PEN-Net, and 4.2 % with SPG-Net. Hence, our method outperforms the other methods.