UVL2: A Unified Framework for Video Tampering Localization

Recent research has witnessed remarkable progress in video inpainting localization techniques. Zhang et al.[11] advanced the field by enhancing artifact modules through the concatenation of a feature extractor and forgery output module, enabling a deeper learning of inpainting features and resulting in more precise localization of inpainting regions. Zhou et al.[4] pioneered a dual-stream encoder-decoder architecture, incorporating attention modules and LSTM, to detect restored video regions. Their approach leverages error level analysis and temporal structure information to accurately identify inpainted areas.

Wei et al.[10] took a different approach, focusing on analyzing spatial-temporal traces left by inpainting operations. They enhanced these traces using intra-frame and inter-frame residuals, guided by optical flow, and then utilized a dual-stream encoder and bidirectional LSTM decoder to predict inpainted regions.

Lou et al. [18] introduced a 3D Uniformer encoder to learn video noise residuals, effectively capturing spatio-temporal forensic features. They further employed supervised contrastive learning to highlight local inconsistencies in repaired videos, improving localization performance. In a follow-up work [19], Lou et al. explored a multi-scale noise extraction module based on 3D high-pass layers to create noise modalities. A cross-modal attention fusion module was utilized to explore the correlation between these modalities, and an attentive noise decoding module selectively enhanced spatial details, further enhancing the network’s localization capabilities.

Given the diversity of object inpainting methods, enhancing the generalization capability of detecting various object-restored videos in a computer vision context is imperative. This capability is crucial for addressing the variations introduced by different inpainting techniques and ensuring accurate detection across various scenarios, thereby providing a more reliable safeguard for the authenticity and integrity of video content.

The field of video splicing detection, particularly at the object level, has witnessed significant advancements in recent years. Techniques such as PQMECNet[20] have emerged, leveraging local estimation of JPEG primary quantization matrices to distinguish spliced regions originating from different sources. MVSS-Net[2], on the other hand, learns semantically irrelevant yet generalizable features by analyzing noise distributions and boundary artifacts surrounding tampered regions. ComNet[13] customizes approximate JPEG compression operations to enhance detection performance against JPEG compression artifacts.

DCU-Net[21] further advances this field by effectively extracting tampered regions through dual-channel encoding fusion of deep features, the utilization of dilated convolutions, and the integration of decoders. Additionally, TransU2-Net[22] proposes a novel hybrid transformer architecture specifically designed to improve object-level forgery detection in images. MSA-Net[23], in particular, introduces a multi-scale attention network that integrates a multi-scale self-attention mechanism, enabling it to capture global dependencies and fine-grained details, resulting in precise localization of various types and sizes of spliced forged objects.

Yadav et al.[24] also contribute to this field with a visually attentive image splicing localization network that comprises a Multi-Domain Feature Extractor (VA-MDFE) and a Multi-Receptive Field Upsampler (VA-MRFU). The VA-MDFE extracts features from RGB, edge, and depth modalities, while the VA-MRFU upsamples these features using multi-receptive field convolutions. Several researchers have successfully applied semantic segmentation techniques to the task of splicing localization, achieving notable results[25, 26, 2].

However, the challenge in splicing localization lies in the common video manipulations often found on social media platforms, such as compression, cropping, detail enhancement, and blurring. These manipulations can disrupt the tampering traces of the original forged videos, significantly reducing the robustness of existing detection methods. Therefore, further research is needed to develop splicing detection techniques that are more resilient to such manipulations.

When two video segments are spliced together, they often exhibit distinct textural characteristics due to their potentially differing capture locations or physical environments. Detecting these textural features in spliced videos is paramount for analyzing video authenticity. By comparing and analyzing the textural differences between video segments, we can assist in determining whether a video has undergone splicing or tampering. To extract textural features, this section utilizes CNN convolution kernels to implement Gabor filters. Gabor filters, through multi-scale and multi-directional analysis, can efficiently capture various textural information in images. They exhibit excellent sensitivity to textural features of different frequencies, directions, and polarities, making them suitable for a wide range of textural analysis tasks. The mathematical expressions of Gabor filters are given as follows:

Here, $x$ and $y$ represent the spatial coordinates of the image, while $x^{\prime}$ and $y^{\prime}$ represent the coordinates after rotation and scaling transformations. $\sigma$ controls the bandwidth of the filter, $\lambda$ determines the central frequency of the filter, $\gamma$ is the decay factor, and $\phi$ represents the phase offset. By applying Gabor filters to input video frames, we can obtain textural responses at different scales and directions. These textural responses can be utilized to construct representations of textural features, which aid in detecting inconsistencies between objects and backgrounds at splicing junctions.

In order to gain profound insight and comprehension of edge detection characteristics, we have deliberately employed two classical edge detection operators: the Sobel operator and the Laplacian operator. Sobel and Laplacian operators are commonly used for first-order and second-order edge detection, respectively. Sobel filters seek brightness changes in the image by calculating the first-order discrete derivative values of the grayscale function, particularly at edges. This helps highlight areas of brightness and lighting variations. Laplacian filters more precisely locate brightness changes and edges by calculating the second-order gradient of the image, while also enhancing details, including textures and small brightness variations.

As demonstrated in Figure 3a and 3b, we utilize the x and y components of these operators as convolutional kernels within a Convolutional Neural Network (CNN) during implementation. Specifically, these kernels are independently applied to the red (R), green (G), and blue (B) channels of the original video frames to extract edge features. To facilitate the training of the neural network, we further normalize these features to a range of 0-1. The Sobel operator achieves edge detection by computing the image intensity gradient for each pixel, while the Laplacian operator, based on the divergence of the scalar function gradient, exhibits rotational invariance, thus enabling more robust edge detection. The unique advantages of these two operators, when used in tandem, allow for the extraction of more robust and accurate edge features.

Due to imaging principles, pixel and noise distribution from different regions can vary. Therefore, analyzing these distributions is crucial. Noise info in videos is stored in pixels, and the Spatial Rich Model (SRM) operator can analyze pixel features. This algorithm relies on pixel statistical features like inter-pixel correlation, skewness, and kurtosis, which reflect the relationships between pixels. Manipulated regions using deep learning often disrupt natural pixel correlations. The SRM operator uses neighboring pixels’ statistical features to represent local image structure.

By analyzing these statistical features, the SRM operator can better represent the destruction of multiple correlations between neighboring pixels in manipulated regions of the image, thereby improving the accuracy and generalization of image tampering detection. We apply the SRM operator to RGB frames, treating pixels in the red, green, and blue channels as three separate images. Each channel will yield a set of SRM features.

The extraction of frequency domain information is crucial for video quality analysis. The Discrete Cosine Transform (DCT) is a common method to convert video signals from the spatial domain to the frequency domain. It decomposes videos into different frequency components, and the amplitude and phase information of these components can be used to assess video quality. High amplitudes suggest strong signals, while low ones may indicate signal loss or noise. By analyzing this info, we can evaluate metrics like clarity and distortion levels, enhancing splicing tampering detection accuracy and robustness. This ensures video content’s credibility and integrity.

In our implemented frequency filter model, we utilize four distinct block sizes to process video information across various frequency ranges. These filters process: (1) low-frequency features, encompassing overall structure and larger features. (2) mid-frequency features, between low and high. (3) high-frequency features, capturing details and texture. (4) the entire spectrum, covering low, mid, and high frequencies, for transformations on the entire image. Finally, the outputs from these filters are concatenated to form the final output.

The objective of multi-view fusion learning of tampering traces is to generate pixel-level prediction results that align with ground truth. To achieve this, we leverage the strengths of CNN and ViT models. We use the ResNet[2] module for local feature extraction and the InterlacedFormer[17] module for global feature extraction. By combining CNN and ViT, we gain a deeper understanding of image details and content. CNN excels at extracting local features, focusing on edges and textures, aiding in more precise tampering region localization. Meanwhile, ViT’s strength lies in global feature extraction, enhancing overall video semantic understanding. By integrating global features, the model better understands the relationship between tampered objects and their environments, further improving localization accuracy. Ultimately, we produce pixel-level localization results for tampered regions in videos.

Table I shows the pixel-level detection performance of various models. Compared to the baseline method, our method achieves superior performance in most cases, but it is only slightly inferior to VIDNet-BN and VIFST on one sub-dataset. At the same time, our method performs well in detecting all unknown forged data, which is mainly due to the ability of multi-view features to enhance the generalized feature extraction ability of our method. The experimental results demonstrate that our method is not only efficient, but also has good generalization ability.

Table II presents comparative experiments with the latest methods and robustness testing results. Although related methods perform well, our method demonstrates superior performance in fine-grained pixel-level detection. This advantage may stem from our targeted analysis of video splicing forgery. Additionally, compared to other semantic segmentation methods such as HRFormer, PoolFormer, and MetaFormer, known video splicing detection methods exhibit significant advantages. This indicates that the analysis of forgery traces in these methods is effective. Meanwhile, the detection results for various processed videos, such as those that have undergone compression, blurring, cropping, etc., show that our method exhibits smaller variations in outcome ($<$ 0.03). Experimental results indicate that multi-view forgery trace features contribute to improving the performance and robustness of video splicing detection.

Figure 5 illustrates the localization results of video inpainting and video splicing alterations. Figure 5a present the results of video inpainting detection, including animals, cars, humans, and combined objects. Our detection for single-object alterations is excellent, even for complex objects. Figure 5b demonstrates video object splicing detection with military objects such as aircrafts, tanks, warships, and soldiers. We can easily detect these alterations through analysis of various forgery traces. This indicates that our method is effective for video alteration detection. Through the analysis of multi-view alteration signature characteristics, we have achieved good results in fine-grained detection such as edges, small components, and combined objects. However, there is still room for improvement in the localization of complex alteration objects. In the future, we will strive to enhance our ability to detect alterations of multiple combined objects.