Dense Feature Interaction Network for Image Inpainting Localization

Inpainting is a long-standing problem in image processing and computer vision [16]. The task is ill-posed because there is no unique solution based on the remaining part of the image, and various prior information is needed to hallucinate the missing region. Early inpainting methods using non-learning algorithms aim to recover lost pixels using pixel redundancy and local continuity priors, either propagating information from the surrounding areas [24] or using patch-based completion in a Markov random field model[25]. More recent effective inpainting methods are obtained with deep neural networks (DNNs) trained on large datasets [1], which allow for automatic feature learning and more context-aware inpainting results. Various types of DNN models have been explored, including the Generative Adversarial Network (GAN) [26, 27], Shift-Net [28] used U-Net architecture, coarse-to-fine network [29, 30], recent Fast Fourier Convolution models [16, 31, 32, 33], and decoupled probabilistic modeling [34]. The emergence of diffusion-based image generation models, such as DALL$\cdot$E 2[35] and Stable Diffusion[36] further significantly improve the performance of image inpainting [37, 38, 39, 40]. The diffusion-based image inpainting leads to more natural and realistic results that pose challenges to detection algorithms.

Detecting of image inpainting is often solved with machine learning based on convolutional neural networks (CNNs) [41]. These CNN-based approaches exploit hierarchical neural networks to automatically learn discriminative features for detecting inpainted regions. Zhu et al. [42] pioneered CNN-based image inpainting detection by employing a label matrix to guide automatic feature learning and the weighted cross-entropy as the loss function. Several studies [43, 44, 45] employed Long Short-Term Memory (LSTM) models to learn artifacts for manipulation localization. HP-FCN [17] was proposed to enhance inpainting traces via high-pass filtering and learn discriminative features from image residuals. Similarly, IID-Net [18] used various combined filters to enhance inpainting features, and introduced a Neural Architecture Search (NAS) algorithm to adjust feature extraction cells. These conventional encoder-decoder structures without pooling layers may hinder the model’s capability to detect intricate tampering and result in spatial information loss. Several methods [46, 11, 47, 14] introduces multi-branch models to combine features from noise/frequency domains or patches [48]. New feature extraction models, including Transformer-based architectures [19, 49] or DenseNet-based approaches [50, 20] have been used for inpainting detection. Furthermore, researchers have explored multi-scale feature learning techniques. For example, MVSS-Net [13, 14] employed features learned from different stages of the backbone network and incorporated a two-branch supervision method considering RGB domain and noise patterns. Liu et al. [10] introduced PSCC-Net, leveraging the multi-stage features learned from the HRNet [51] for progressive mask prediction. GCA-Net [21] designed a gated context attention network to capture fine image discrepancies in inpainted images. To improve global context representations and minimize feature loss, they refer to the UNet++ [52] architecture, which utilizes multi-scale bottom-up feature propagation in the decoding process. Recently, Yu et al. introduced DiffForensics [15], which employs a self-supervised denoising diffusion paradigm integrated with an encoder-decoder architecture to detect and localize tampering.

Unlike existing methods, this work highlights the complementary features of neural networks, and designs a dense feature interaction network tailored for capturing rich and discriminative patterns in image inpainting detection, as depicted in Fig. 2. For the additional task of inpainting edge detection, edge extraction operator [13, 14, 53] and ordinary convolutional structures [54] are susceptible to interference from noise (shapes and boundaries of other authentic objects). Therefore, a priori feature-guided edge attention module is developed in this work to help the model focus on subtle differences inside and outside the boundary and improve the localization accuracy at edges.

Different layers in a Convolutional Neural Network (CNN) model have receptive fields of different sizes and extract image features of different levels of detail. Generally, early or shadow layers mostly extract low-level and generic image cues (e.g., edges and corners), and the middle layers carry shapes and textural patterns. In contrast, deeper layers capture high-level semantics or specific image cues [55, 44, 23]. Artifacts of image inpainting may be reflected at all three levels of features, each contributing in a complementary manner. For example, pixel filling can cause distortions or blur at edges, detectable by low-level features. Middle-level features mainly capture texture inconsistencies in the filled areas, like structural mismatches or color discordance. Differently, high-level features identify changes in an image’s semantic content, such as missing object parts or altered scene interpretation, aiding the model in identifying inpainting areas.

Our goal is to design an effective extraction and fusion of these complementary features (as shown in Fig. 1) to identify image inpainting. We first use the backbone network to extract multi-stage features from the input image. Using a CNN (e.g.,  [56]) divided into five stages as an example, we define the shallow features from Stage 1 and Stage 2 as low-level features $f_{1}$ and $f_{2}$, the features extracted from Stage 3 as mid-level feature $f_{3}$, while the more complex deep features from Stage 4 as high-level features $f_{4}$. Inspired by the Feature Pyramid Network (FPN) [22] and Path Aggregation Network (PANet) [57], we design a multi-directional, multi-scale fusion method, as shown in Fig. 4. Differently, we treat multi-level features as independent entities, emphasizing the specificity of features at different stages. Our densely connected fusion approach improves the efficiency of information interaction between adjacent and cross layers, progressively enhancing context awareness.

Taking a single-scale feature as an example, our DFPL module considers the features of the same scale and all the neighboring features to fully exploit and enhance the useful context information for image inpainting localization. Specifically, to obtain the feature $f_{i}^{j}$ (where $i\in[1,4]$ denotes the stage of the backbone to which the current feature belongs, and $j\in[1,3]$ represents the state of the feature in DFPL, $j$ as 1 for the initial state involving channel reduction, 2 for the bottom-up path, and 3 for the top-down path), we concatenate its neighboring features in the channel dimension and use $1\times 1$ convolution to integrate and select features. This process can be expressed by the equation below, where we have removed the max pooling, upsampling operations and cases with $i$ in the boundary condition for simplicity.

$Cat(\cdot)$ refers to concatenating the required features in the channel dimension, and $Conv_{1}(\cdot)$ represents $1\times 1$ convolution layer.

Filling content to a specific area in an image can leave detectable traces between the altered region and its surroundings. Therefore, leveraging these edge artifacts is crucial for detecting inpainting, which the mid-lowe-level features can guide. Nevertheless, mid-low-level features not only contain inconsistencies caused by inpainting but also a large amount of detail and structural noise, such as texture, shape, and object boundaries. Using traditional edge extraction operators [14, 53] or convolutional structures [54] will weaken the representation of inpainting artifacts. Leveraging context-rich features that include the structural information of the restoration region as a prior can effectively suppress irrelevant semantic noise and guide boundary learning.

We develop a method for detecting and enhancing edge artifacts using reverse attention called REAE, which uses the structural details in the mid-level feature $f_{3}^{3}$ as a guide, directing the attention of low-level features toward subtle boundary changes of the inpainting area. This promotes learning distinctive features to differentiate between inpainting edges and natural boundaries. As shown in Fig. 3(a), the REAE module accepts mid-level features $f_{3}^{3}$ and low-level features $f_{1}^{3}$ and $f_{2}^{3}$ as input. Its upper branch aggregates structural information of suspicious regions through average pooling, max pooling, and $7\times 7$ convolution. Subsequently, the sigmoid output is inverted to obtain a reverse attention map emphasizing edges, expressed by Eq. (III-B). Meanwhile, the lower branch of the REAE progressively combines $f_{3}^{3}$, $f_{2}^{3}$, and $f_{1}^{3}$ for $f_{123}^{e}$ through $3\times 3$ convolution, Batch Normalization (BN), and bilinear interpolation upsampling. Finally, we use Eq. (3) to fuse the feature for edge supervision and output the edge mask $M_{e}$ through a prediction head composed of multiple convolutional layers.

Due to the nature of neural networks, deep features exhibit a significantly stronger response to the target region compared to shallow features.The difference in receptive field inevitably leads the model to bias towards the side with a more robust response during the fusion process, allocating more weights to the deep features, which can severely suppress the role of shallow features. Shallow features can provide detailed edge positioning and structural information, significantly impacting the detection accuracy.

To unify the feature representation of the inpainting region, we design a spatial adaptive feature fusion module (SA-FF) to fuse high-level features and mid-low-level features to correct the bias brought by response intensity. Normalization can limit the response of different features to a similar magnitude, effectively eliminating the interference of response intensity. After normalization, the model can treat deep and shallow features fairly, making it easier to learn effective fusion weights. Specifically, we reduce the channels of $f_{123}$ and $f_{4}$ to 2 through a $1\times 1$ convolution, followed by normalization. The process can be expressed as: Eq. (4) and (5).

For different spatial locations, there exists a disparity in the contribution of deep and shallow features. Deep features, rich in context, are semantically accurate but lose many spatial and geometric details. Shallow features better retain these aspects, and we hope that the model trusts shallow features more in the boundary area and uses the information provided by deep features to fill the target area. As shown in the Fig. 5, we designed a spatially adaptive weight learner to adjust the fusion weights according to positional features, fully utilizing both advantages. Specifically, we feed the features $f_{w}$ used for learning weights into alternating connection blocks composed of $1\times 1$ convolution, BN, and ReLU to generate position-aware fusion weights $\textbf{W}\in\mathbb{R}^{H\times W\times 4}$. Then, the final fusion output is obtained according to element-wise multiplication and element-wise addition, as shown in Eq. (8). Finally, a region mask $M_{r}$ is output through a prediction head composed of convolutional layers.

To train our DeFI-Net, we use an objective function that combines the region loss and the edge loss functions. The region loss enhances the model’s sensitivity to the inpainting region from a global perspective. It promotes the overlap between the predicted mask and the ground truth. In contrast, the edge loss guides the model to learn non-semantic detail differences from a local perspective, achieving more precise localization. According to [58], we define the region loss function as $L_{region}=L_{bce}^{w}+L_{iou}^{w}$. Compared with the traditional BCE loss, $L_{bce}^{w}$ assigns different weights to pixels in different positions, paying more attention to the target area and hard pixels in the area (locations such as boundaries prone to classification errors). The IoU loss $L_{iou}^{w}$ assigns greater weight to the target area, prompting the network to pay attention to the global structure. For the edge loss, we use the $L_{bce}$ similar to supervise the edge prediction, denoted as $L_{edge}$. The overall loss function is a convex combination of two losses: $L_{total}=\lambda_{r}L_{region}+\lambda_{e}L_{edge}$.

In Fig. 7, we demonstrate the qualitative results on several image inpainting datasets. Our method achieves outstanding accuracy in localizing image inpainting across diverse manipulation regions and models, significantly surpassing the baseline methods. The segmentation results produced by DeFI-Net are closest to the ground truth, with rare occurrences of false positives. Specifically, DeFI-Net exhibits excellent performance when dealing with random regions (first and third columns) and specific semantic objects (second, fourth, fifth, and sixth columns) for inpainting. In contrast, MVSS-Net, IID-Net, and PS-Net give unsatisfactory results, with many mis-segmented regions and even overlooking the correct targets. PSCC-Net achieves decent results on some datasets, but it is prone to misjudgment in some challenging cases (such as slender, small targets) due to the influence of semantic information. In response to this issue, DeFI-Net obtains precise predictions thanks to our proposed dense feature pyramid learning strategy. This strategy delves into the potential relationships among multi-level features, better learning the inconsistencies in image details, structure, and semantics, enabling the model to response inpainting at different scales better. Moreover, the reverse edge attention enhancement effectively enhances edge features and suppresses the impact of semantic noise on inpainting representation, resulting in clearer boundary localization. Overall, DeFI-Net achieves satisfactory results.

In addition, we found that DeFI-Net significantly improved the localization of small target inpainting compared to existing methods. Removing unnecessary people or objects in the background is the most commonly used application scenario for image inpainting. Tiny inpainting makes minor changes to the semantics of the image and leaves behind artifacts that are difficult to detect, which greatly increases the localization difficulty. As shown in Fig. 8, IID-Net and MVSS-Net have failed, and PS-Net and PSCC-Net have many false positives, while our method achieves accurate localization. This primarily benefits from our DFPL and SA-FF, which dig deep into the correlation between multi-scale information and skillfully combine them.

To comprehensively evaluate the proposed method, we present comparisons of time consumption and computational complexity in Table V. All the compared methods are evaluated on the same device (GeForce RTX 3090 GPUs) with $256\times 256$ images. We can see that the proposed DeFI-Net introduces little computation overhead, and achieves comparable performance to existing models in terms of inference time.

To further present the discriminative and complementary features learned from various levels by DeFI-Net, we provide the visualization results of the feature maps at the low, middle, and high levels, as shown in Fig. 9. We can observe that the feature maps progressively capture the required discriminative information from low-level to high-level, including details, local structures, and global abstract semantics. Low-level features capture discriminative patterns around edges, middle-level features focus on texture inconsistencies in the filled areas, while high-level features identify changes in an image’s semantic content, such as localizing the entire objects. By effectively integrating these features, our model achieves precise localization across various inpainting regions and generation models.

Essentially, DeFI-Net mainly benefits from three modules: DFPL, REAE, and SA-FF. Dense Feature Pyramid Learning (DFPL) highlights the specificity of features at different stages in the neural network, interacts quickly with neighboring layers in a densely connected manner, and improves the representation capability of features from other perspectives. Reverse Edge Attention Enhancement (REAE) enables the model to capture subtle differences at the boundaries by utilizing structural information provided by mid-level features as a priori guide. Spatial Adaptive Feature Fusion (SA-FF) adaptively learns how high-level features are fused with low and mid-layer features. To evaluate the effectiveness of DFPL, REAE, and SA-FF, we removed them separately from DeFI-Net and evaluated the inpainting detection performance on the IID-10K dataset with the ablation setup shown in Table VI. In addition, we performed ablation analysis for each module separately to demonstrate the effectiveness of their design.

Fig. 10 illustrates cases where our DeFI-Net does not yield accurate localization results, notably in scenarios featuring multiple tiny regions (the first two columns), images with in dark lighting conditions (the third column), and full-size synthesis (the last column). It is worth noting that even under these circumstances, our DeFI-Net still demonstrates highly competitive performance compared to existing methods.