Noise and Edge Based Dual Branch Image Manipulation Detection

Ordinary CNN tends to learn features representing semantic information of images rather than manipulation details. To solve this problem, some works use SRM, as shown in Fig. 2, to obtain the noise image corresponding to the image as the model input. SRM is essentially a set of high-pass filters that can get the image’s different high-frequency information through different weights.

Since the weights of SRM are predefined and cannot be learned, some works use constrained convolution instead of SRM. Constrained convolution imposes constraints after kernels update weights, making the weights distribution of kernels similar to the high-pass filter. The constraints are as follows:

where $\boldsymbol{w}_{k}$ represents the $k$-th convolution kernel, $(c,c)$ is the center position coordinate of $\boldsymbol{w}_{k}$, and $(m,n)$ is the non-center position coordinate. After the constraints, the center position weight of the convolution kernel is -1, and the sum of the weights of other positions is 1. The constraints can be achieved by the following stpdf:

1. 1.

   Calculate the sum of the weights of the non-central positions.

2. 2.

   Divide the weights of the non-center positions by the sum in 1).

3. 3.

   Set the center position weight to -1.

Although constrained convolution solves the problem that SRM cannot be trained, it is unstable during actual training, resulting in poor training results. As shown in Fig. 3, we assume that the $3\times 3$ matrix in Fig. 3LABEL:sub@fig_cc_problem_a is the weight of a convolution kernel updated by backpropagation. Fig. 3LABEL:sub@fig_cc_problem_b is the result after it is constrained by constrained convolution. It can be seen that since the weights before constraints may be positive or negative, their sum may be negative, and the absolute value of the sum may be relatively small. At this time, according to the constraint rules, dividing the weights of all non-central positions by this sum will cause three problems:

• •

  Dividing the weight by this relatively small sum causes the weight to be amplified a lot, but the center position weight is still -1. The weights of the non-central positions and the weight of the central position are much different in order of magnitude, which affects the effect of high-pass filtering.

• •

  If the sum is negative, the division operation will make the positive weights negative and the negative weights positive, causing the model’s input to change drastically.

• •

  There is an order of magnitude difference between the weights of the constrained convolution and the weights of the subsequent normal convolution, which will affect the weights already trained and cause training fluctuations.

The kernel weights shown in Fig. 4 are the result of training with constrained convolutions. It can be seen that the weights of the convolution are very different. To address the above issues, we set new constraints on convolution. The improved constrained convolution training process is shown in Algorithm 1.

Firstly, as shown in Fig. 5LABEL:sub@fig_laplace_operator, we learn from the traditional edge detection operator Laplacian to initialize the convolution kernel. Unlike the Laplacian operator, we set the weight of the center position to -1, and the weights of other positions equal to 1 divided by the number of non-center positions. The initialized weights are shown in Fig. 5LABEL:sub@fig_cc_initial. On this basis, the distance factor is further added. For example, for a $3\times 3$ convolution kernel, if the Euclidean distance is used to calculate the distance between the center position and other positions, there are only two distances of 1 and 1.414. The number of both distances is 4. Then the following equation can be constructed:

Solving the equation to get $x$ equal to 0.146, then the weight of the position with a distance of 1 is 0.416, and the weight of the position with a distance of 1.414 is $x\div 1.414=0.104$. The overall weights are shown in Fig. 5LABEL:sub@fig_cc_initial_distance. We initialize the constrained convolution in this way.

Secondly, calculating the sum of the absolute values of the weights can avoid the result being negative or very small in absolute value. Thirdly, after the division operation, the small positive values and all negative values are set to 0.001 so that their effect is small and the values are not particularly small. Finally, the weight of the center position is not strictly equal to -1 but is equal to the negative sum of the weights of all non-center positions. With these constraints, the convolution can act as a high-pass filter and slow down fluctuations during training.

For the image manipulation detection task, the traces of image manipulation are very subtle. In addition to traces of manipulation regions, as described in Section II, the subtle differences between the manipulation edge and the non-manipulation region around it are also very important. To better capture this subtle information, features in CNN need to be maintained at a relatively high resolution. However, due to the local characteristics of convolution, it isn’t easy to obtain sufficient contextual information from high-resolution features.

To solve this problem, we draw on BiSeNet and DDRNet to design a dual branch network with Resnet-34 as the backbone. As shown in Fig. 1, the high-resolution branch maintains the feature’s height and width at 1/8 of the input image for more detailed information. The context branch changes the feature’s height and width to 1/32 of the input image through multiple convolutions to obtain richer contextual information. Finally, the features of the two branches are fused to obtain information with sufficient context and details.

Since each layer in the CNN learns different feature contents, we extract the manipulation edge based on the features output by each layer of the network. To extract edges better, we propose a specially designed edge extraction block (EEB). The flow of EEB is shown in Fig. 6. To reduce the calculation and fully use the feature information, we first use $1\times 1$ convolution to reduce the number of features’ channels to 1/4 of the original and then construct residual learning. Finally, $1\times 1$ convolution is used to reduce the number of channels to 1.

The global correlations of pixels in features are essential for common CV tasks and image manipulation detection. Due to the locality limitation of convolution, it isn’t easy to obtain a complete global relationship. Therefore, similar to the self-attention mechanism in the NLP field, the non-local module is applied to CV tasks, which can break through the distance limitation of convolution and obtain the correlations between each pixel and other pixels.

However, ignoring the channel dimension, the image is 2-dimensional and contains much richer spatial information than 1-dimensional natural language. The influence of one pixel on other pixels in an image is closely related to distance. The closer two pixels are in the distance, the stronger their correlation is (pixels of the same class are positively correlated, and pixels of different classes are negatively correlated). As shown in Fig. 7, it is assumed that the yellow part in the figure is class A, and the rest is class B. Points a, b, and c belong to class A, so they are positively correlated. But the distance from point a to point b is closer than the distance from point a to point c, so the correlation between point a and point b is stronger. Point a and d belong to different classes, so they correlate negatively. But the distance from point a to point d is closer than the distance from point a to point b, so the correlation between point a and point d is stronger.

The non-local module calculates the correlations among pixels across distances while ignoring the effect of distance factors on the correlation. To reflect the distance relationship between pixels, we use Euclidean distance to construct the distance matrix of each pixel to other pixels. As shown in Fig. 8LABEL:sub@fig_distance_matrix_a, if we have an image with a height of 2 and a width of 2, then its corresponding distance matrix D is shown in Fig. 8LABEL:sub@fig_distance_matrix_b, with a size of $4\times 4$. The image has a total of 4 pixels. $D_{ij}$ in the distance matrix represents the distance from the i-th pixel to the j-th pixel, where i is the row, and j is the column. As shown in Fig. 9, the matrix Cor obtained by multiplying Q and K in the Non-local module represents the underlying correlations between each pixel and other pixels. We obtain the matrix Cor_D by element-wise division of the basic correlation matrix Cor and the distance matrix D. The matrix Cor_D represents the correlation after distance optimization. Then the final relationship among pixels is obtained through the Softmax operation. As shown in Fig. 8LABEL:sub@fig_distance_matrix_c, to avoid the divisor being 0, we add 1 to the value of the distance matrix D.

Since the features of the high-resolution branch and the context branch contain different granularities of information, we use BiSeNet’s attention refinement module (ARM) and feature fusion module (FFM) to fuse them to predict the manipulation mask. ARM and FFM are shown in Fig. 10LABEL:sub@fig_bisenet_arm and Fig. 10LABEL:sub@fig_bisenet_ffm, respectively.

First, The ARM is used to optimize the features output by layer 3l and layer 4l in the channel dimension:

where $f_{3l}$ and $f_{4l}$ are the features output by layer 3l and layer 4l. $f_{3l_{\_}A}$ and $f_{4l_{\_}A}$ are ARM-optimized features. Then use the non-local module to perform self-attention calculation in the spatial dimension:

where $f_{3l\_N}$ and $f_{4l\_N}$ are features optimized by the non-local module. Then we perform the element-wise sum of the feature $f_{4h}$ output by layer 4h with $f_{3l\_N}$ and $f_{4l\_N}$, respectively. Since they differ in resolution and dimensionality, they are processed using bilinear upsampling or $1\times 1$ convolution. Finally, use FFM to get the fused features:

where $Up$ represents spatial upsampling, $Conv_{1\times 1}$ represents a $1\times 1$ convolution, and $ff$ represents the fused feature. The number of $ff$’s channels is 256.

$3\times 3$ convolution is performed after upsampling ff by a factor of 2. After repeating the above operation three times, the final prediction of manipulation regions is obtained. Three upsampling changes the resolution of ff to 1/4, 1/2, and 1/1 of the original input, respectively. That is, the resolution of the prediction result is the same as the original input. Three convolutions change the number of channels to 1/4, 1/16, and 1/256 of the initial $ff$. That is, the final number of channels in the prediction result is 1. These operations prevent the channel from decreasing too drastically while upsampling, thereby increasing the accuracy of the manipulation prediction.

We use the EEB in Section III-B to separately predict the manipulation edges of the features output by the six layers in the model. Because the feature size in layer 1 production is 1/4 of the original input, we upsample the remaining five predicted manipulation edges by the same size. After concatenating the six prediction results in the channel dimension, a $3\times 3$ convolution is used to obtain the final predicted manipulation edge. The process is as follows:

where $Cat$ represents the connection of the channel dimension, $edge_{i}$ represents the six edge prediction results, and $Conv_{3\times 3}$ represents the convolution of $3\times 3$.

The manipulation regions of the image and the manipulation edges are small parts of the image. We use Dice Loss [28] as the loss function to alleviate the problem of an unbalanced number of pixels in the manipulation and non-manipulation regions. The ground-truth of manipulation regions is processed by the dilation operation and the erosion operation to obtain $GT_{D}$ and $GT_{E}$, respectively. $GT_{D}-GT_{E}$ obtains the ground-truth of manipulation edges. The overall loss consists of the manipulation region prediction loss and the manipulation edge prediction loss. The calculation process is as follows:

where $loss_{region}$ represents the loss of the manipulation regions, $loss_{edge}$ represents the loss of the manipulation edges, and $\alpha$ is the weight.

Implementation Details. We implement our model using the Pytorch framework. The high-resolution branch and the context branch of the model are initialized with the weights of ResNet-34 pre-trained on ImageNet [29]. The width and height of the input image are uniformly adjusted to 512. Image pixel values are divided by 255 for normalization. Standardize by subtracting the mean and dividing by the variance. The mean values of the three channels of BGR are 0.406, 0.456, and 0.485, respectively. The variances are 0.225, 0.224, and 0.229, respectively. Flipping and mirroring are used to perform simple data augment. The training batch size is set to 48. In the loss function, $\alpha$ is set to 0.3. We train the model for 12K stpdf. The learning rate is initially set to 0.01 and then reduced to 0.0075, 0.005, and 0.0025 after stpdf 5K, 7.5K, and 10K, respectively. All experiments are performed on a single NVIDIA Tesla V100 GPU with 32GB memory. The source code is available at: https://github.com/kakashiz/NEDB-Net.

Evaluation Criteria. We evaluate the prediction results using pixel-level precision, recall, F1, and AUC. Because in data without GT, the most appropriate threshold cannot be predicted, we use the median value of 0.5 as the threshold to determine the positive and negative classes.

Datasets. We select CASIAv1 [30], CASIAv2 [31], COVERAGE [32], COLUMBIA [33], NIST16 [34] these publicly available image manipulation datasets as the experimental datasets. The composition of the manipulation image types of datasets is shown in Table II. It is worth noting that some of the manipulation images lack the corresponding ground-truth or do not match the shape of the ground-truth. We only count the correct manipulation images here.

Evaluation method. When comparing effects in many studies, the model will be trained on other large datasets and then tested on the above datasets. Or select a portion of each dataset for fine-tuning and another portion for testing. We do not think this comparison is particularly plausible because it does not adequately demonstrate the model’s generality against unknown data. So, we adopt the same evaluation method as [22], let the model train only on the CASIAv2 dataset, and then directly test it on the remaining dataset. This approach directly reflects whether the model has learned how to detect manipulation rather than just fitting the dataset.

Baselines. The baseline models we chose are as follows: Fully Convolutional Networks (FCN) [35], High-Pass Fully Convolutional Network (HP-FCN) [36], Manipulation Tracing Network (Mantra-Net) [18], Constrained R-CNN (CR-CNN) [16], Generate, Segment, Refine Network (GSR-Net) [20], Multi-View Multi-Scale Supervision Network (MVSS-Net) [22], and Dense Fully Convolutional Network (D-FCN) [37]. Among them, HP-FCN and ManTra-Net directly use the models provided by the authors due to the lack of training codes and private datasets. The authors provide the weight trained on the private manipulation dataset and 10% of the NIST16 dataset. We train on the CASIAv2 dataset based on this pretrained weight and report two experimental results, where D-FCN (pre-train) means directly using the author’s pre-trained weights for testing, and D-FCN (re-train) means using our retrained model for testing. Other methods either follow the same evaluation protocol or retrain on the CASIAv2 dataset. The experimental results are shown in Table III (part of the experimental results are obtained from [22]).

Manipulation detection effect comparison. As can be seen from Table III, our model outperforms other models on other datasets, except that it is worse than D-FCN (pre-train) and MVSS-Net on the NIST16 dataset. Especially the results on COLUMBIA and CASIAv1 are much ahead of other models. The experimental results verify the detection effect of our model and show that our model can also achieve good results on unknown data sets.

Robustness evaluation. To compare the robustness of the models, we follow the same approach as [22], applying different levels of JPEG compression and Gaussian blur to CASIAv1, respectively. The experimental results are shown in Fig. 11. From this, we can see that our model is not ideal in terms of robustness performance. As the interference to the original image gradually increases, the detection effects of CR-CNN and our model decay more than the detection effects of other models. For this case, one possible reason is that both CR-CNN and our model only use the noise image as the model’s input. Compression and blurring damage the noise information more than semantic information. On the other hand, if the entire image is compressed or blurred, can the whole image be considered a manipulation region? If so, the above experimental results are not accurate. However, many studies’ robustness experiments use post-processing on the whole image. We think there should be a better way for ablation experiments.

To better illustrate the effect of each module in our model, we gradually add each component on ResNet-34 for training and compare the results. Since the number of images in the datasets mentioned in Section IV-B is small, to make the model fully trained, we use the COCO [38] dataset to generate a large number of simple manipulation images as the dataset for ablation experiments. The COCO dataset contains a large number of images and annotations of objects in the images. Based on these object annotations, manipulation images can be easily created. The generation process of the manipulation image is as follows:

1. 1.

   Randomly select one image (copy-move type) or two images (splicing type) from the dataset as the source and destination images. The source and destination images are the same if only one image is selected.

2. 2.

   Randomly select an object from the source image, and perform operations such as rotation and scaling on the object.

3. 3.

   Paste the object randomly to a certain position of the destination image.

4. 4.

   Perform post-processing operations such as blurring.

A total of 76K synthetic manipulation images were generated using the above method. We train models using these manipulation images and test them on the COLUMBIA dataset and the CASIAv1 dataset. The models for the ablation experiment are as follows:

• •

  SB: Use ResNet-34 as the base model.

• •

  DB: Use ResNet-34 to build a dual-branch network and add the ARM module and FFM module for features fusion.

• •

  DB+origin-CC: Add the original constrained convolution on the dual-branch network.

• •

  DB+CC: Add the improved constrained convolution on the dual-branch network.

• •

  DB+CC+Edge: Add the improved constrained convolution and edge detection on the dual-branch network.

• •

  DB+CC+Edge+NL: Add the improved constrained convolution, edge detection, and Non-local module on the dual branch network.

• •

  DB+CC+Edge+NL-D: Add the improved constrained convolution, edge detection, and Non-local module with distance factor on the dual-branch network, i.e. the final model.

The results of the ablation experiments are shown in Table IV. Comparing SB and DB shows that the dual-branch network can capture more detailed information that is beneficial for manipulation detection. The comparison of the results of DB, DB+origin-CC, and DB+CC confirms the problem of the original constrained convolution we mentioned in Section III-A and illustrates the effect of our improved constrained convolution. From the comparison between DB+CC and DB+CC+Edge, we can find that manipulation edge detection is very helpful to the overall detection effect. The results of DB+CC+Edge, DB+CC+Edge+NL, and DB+CC+Edge+NL-D show that the global correlations computed by the self-attention mechanism are beneficial to distinguishing manipulation and non-manipulation pixels, and the self-attention mechanism after adding the distance factor works better.

Since our model uses the noise image extracted by the constrained convolution instead of the original image as the manipulation detection cue, the initialization method of the constrained convolution is very important for the model network. The initialization methods for comparison are as follows:

• •

  Random: The weight of the center position is -1, and the weight of other positions is randomly initialized to a value between 0-1.

• •

  Random-Sum: The center position weight is -1. The weight of other positions is randomly initialized to a value between 0-1, and the sum of the weights of the non-central positions is 1.

• •

  Laplace-like: The initialization process is described in Section III-A and Fig. 5LABEL:sub@fig_cc_initial.

• •

  Laplace-like-D: The initialization process is described in Section III-A and Fig. 5LABEL:sub@fig_cc_initial_distance.

The training and test sets of the experiments are the same as in Section IV-C. Since Random and Random-Sum are initialized randomly, we retrain them three times. The experimental results are shown in Table V. It can be seen that the results of Random and Random-Sum are both unstable due to random initialization. But the initialization of Random-Sum is similar to a high-pass filter, so the overall effect is better than Random. Laplace-like and Laplace-like-D perform better than the two random initialization methods. And Laplace-like-D has a better overall effect because it can better reflect the correlation between pixels according to the distance in the initialization process.

Different sizes of convolution kernels greatly influence the effect of constrained convolution. If the convolution kernel is too small, its receptive field is too small to capture complete information. If the convolution kernel is too large, it will also capture a lot of irrelevant information while affecting the calculation speed. So, we gradually increase the convolution size of the constrained convolution kernel from $3\times 3$ to $11\times 11$ for experiments. The training and test sets of the experiments are the same as in Section IV-C. Experimental results are shown in Table VI. From the experimental results, we can see that when the convolution kernel size is $5\times 5$, the overall detection effect is better. Although the effect is also good when the convolution kernel size is $11\times 11$, a too-large convolution kernel will affect the calculation speed.

Since the number of channels of the input image is three, to keep the number of channels of the noise image consistent, the size of the constrained convolution in the batch dimension is three. Different sizes of convolution kernels capture different information, but it is questionable whether setting the convolution kernels of different batches to different sizes can get better results. To verify this question, we conduct experiments with the following settings:

• •

  3, 5, 7: The sizes of three batches of convolution kernels are set to $3\times 3$, $5\times 5$, and $7\times 7$, respectively.

• •

  5, 7, 9: The sizes of three batches of convolution kernels are set to $5\times 5$, $7\times 7$, and $9\times 9$, respectively.

• •

  5, 5, 5: The sizes of three batches of convolution kernels are all set to $5\times 5$.

The experimental results are shown in Table VII. Contrary to our assumption, combining convolution kernels of different sizes does not improve the detection effect but leads to poor detection results. One possible reason for the poor results is that the noise features extracted by different convolution kernels are quite different. These differences are not conducive to subsequent predictions.

As mentioned earlier, the manipulation edge is a vital clue. The ground-truth of manipulation regions is processed by OpenCV’s dilation and erosion operations to obtain $GT_{D}$ and $GT_{E}$. $GT_{D}-GT_{E}$ gets the ground-truth of manipulation edges. In OpenCV’s dilation and erosion operations, the shape and size of the kernel can be set. The shape of the kernel can be oval, rectangular, or cross. The width of manipulation edges in the ground-truth can be adjusted by setting the size of the kernel. If the edge is too narrow, it is tough for training and prediction. If the edge is too wide, it will cause too many irrelevant pixels to affect the prediction effect.

So, we use kernels of different shapes and sizes to generate the ground-truth of manipulation edges for experiments. The training and test sets of the experiments are the same as in Section IV-C. Experimental results are shown in Table VIII. From this table, we can see that the average effect of elliptical and rectangular kernels is better than that of cross-shaped kernels, and the elliptical kernels perform best. One possible reason is that the number of pixels involved in the cross-shaped kernel operation is too small, while the rectangular kernel operation involves more irrelevant pixels. In addition, the detection performance does not continually improve as the kernel size increases. That is, the width of the ground-truth of edges should be moderate. In our experiments, the edges produced by the $5\times 5$ kernel performed best for manipulation detection.