Traceable and Authenticable Image Tagging for Fake News Detection

Generally, existing fake news detection methods try to capture unusual patterns by extracting text information [18, 19, 23, 32], visual features [24, 11, 31, 4], and multi-modal fusion of features  [33, 27, 40, 9] from news content. As the one main component of news, textual content [18, 32] is widely used as the main domain for detecting fake news. Ma et al. [18] adopted a 2-layer Gated Recurrent Unit model to extract the hidden representations, which can capture the variation of contextual information of relevant posts over a period. Besides, there are a lot of algorithms that utilize different NLP models [37, 30], autoencoder [20], and GAN [5] to learn more comprehensive representations of textual content to detect fake news. Moreover, several algorithms take stance classification [19, 32] and writing style [23] into account to detect fake news.

In addition, as another component of news, visual features such as news images or videos are more easily disseminated and maliciously manipulated. Some methods [11, 31, 36] analyze the basic statistical features from news image amount, image type, and image popularity, while some other methods extract forensic features for fake image detection [4, 3]. Besides, Qi et al. [24] proposed Multi-domain Visual Neural Network (MVNN) to capture the complex patterns of fake news images in the frequency domain and extract visual features in the pixel domain, which improves the performance of fake news image detection. Since the representation capability of one specific modal feature is limited, some multi-modal methods [33, 27, 40, 9] combine the visual features and textual features to explore their consistency to verify the fake news.

All the above content-based fake news detection methods detect fake news passively by analyzing textural features and/or visual features of news. Although these methods improve fake news detection from different levels, these passive detectors do not consider the news traceability to achieve reliable fake news detection. Hence, to provide a reliable forensic chain for fake news detection, we propose a proactive tagging scheme by embedding traceable and authenticable tags into original news images to detect fake images and trace the news source.

Digital image watermarking can achieve traceability and content integrity authentication. To achieve source traceability, robust watermarking [35, 41, 29, 8] is proposed to resist all kinds of attacks. Zhu et al. [41] proposed the end-to-end trainable framework based on autoencoder, i.e., HiDDeN, which inserts noise layers between encoding and decoding to simulate image distortion. Subsequently, Jia et al. [8] proposed Mini-Batch of Real and Simulated JPEG compression (MBRS) to enhance the JPEG robustness based on autoencoder. However, the two sub-networks in autoencoder-based methods have two sets of parameters, which cause color distortion and low invisibility. HiNet [12] and ISN [17] have been proposed to integrate the embedding and extraction process into an invertible model to improve image quality and security, which are based on INN. Besides, RIIS [35] introduced a conditional normalizing flow to model the distribution of high-frequency components as lost information and add a distortion model to simulate the distortion for robust watermarking.

Besides, to achieve content integrity authentication, fragile watermarks-based methods [22, 2, 38] are proposed. Neekhara et al. [22] proposed to proactive embed a semi-fragile neural watermark into real images for digital media authentication. Besides, Asnani et al. [2] proposed a template estimation scheme to proactively detect manipulated images by embedding orthogonal and learnable templates into real images. It achieves better image manipulation detection performance on unseen generative models.

Some dual-watermarking methods [21, 15, 26, 16, 6] have been proposed to embed different watermarks to achieve multi-purposes of image protection. Lu and Liao [16] embedded the robust and fragile watermarks in an image for copyright protection and image authentication. For the same purposes, Liu et al. [26] embedded one watermark in YCbCr color space by using DWT and another watermark in RGB components. Besides, TRLH [6] is proposed to embed fragile and blind dual-watermarks for image tamper detection and self-recovery based on lifting wavelet and halftoning techniques. However, these methods embed watermarks by extracting features manually and thus cannot flexibly handle different attacks.

For fake news detection, the existing methods cannot be directly used to achieve good performance on both news traceability and authentication simultaneously. Therefore, we propose to embed the dual-tags, i.e., traceable tags and authenticable tags, into each news image to achieve both authenticity identification and source tracing to build a complete forensic chain in fake news detection.

The architecture of our proposed approach is shown in Fig. 2. It contains three main structures: 1) The forward embedding process, 2) Simulate manipulations attack in practical, and 3) The reverse revealing process. Note that Fig. 2 shows the training process of our proposed scheme, and the inference process is the same as the training process.

First, traceable tags and authenticable tags are embedded into news images with three branches and then are extracted from the news images under a variety of manipulations. In the process, the lost information of dual-tags with the conditional features extracted from tagged images is encoded into two sets of latent variables under Feature Aware Projection Model (FAPM) to improve the extraction accuracy of the embedded tags.

With the consideration of news characteristics, we simulate the manipulations that news images may undergo, including moderate manipulation and malicious manipulation. The former will not affect the news authenticity, such as compression, adjusting contrast, blur, etc. While the latter will tamper with the content of news images to mislead the public, including splicing, copy-move, etc. Therefore, the images under moderate and malicious manipulations are defined as real images and false images, respectively.

In the revealing process, with the help of Distance Metric-Guided Module (DMGM), the pre-embedded traceable tags and authenticable tags are extracted with different accuracy from the manipulated news images for authenticity verification and source tracing.

To facilitate the embedding process, we normalize the tags, such as logos, news information, etc., as bitstreams. We adopt the normalization strategy similar to that in IWN [34].

Different from the existing INN-based methods [12, 17, 35, 34], we design to embed the dual-tags, i.e., traceable tag and authenticable tag, and they should show different robustness performances under image manipulations. In our design, authenticable tags should be robust to moderate manipulations, while fragile to malicious manipulations. And traceable tags should be robust to all manipulations.

The two branches of existing INNs share the same network parameters if using classical INN, and thus it is hard for those INNs to achieve different robustness performances for the two embedded tags.

To achieve our goals, we propose the Decoupled Invertible Neural Network (DINN) to decouple the embedding and revealing processes of traceable tag and authenticable tag, so that these dual-tags will not interfere each other. Before training the proposed network, each original news image is first decomposed by DWT with Haar wavelet kernel to obtain $I_{Ori}$, and traceable tag and authenticable tag are normalized to obtain $T_{Trace}$ and $T_{Auth}$. Then, $I_{Ori}$, $T_{Trace}$ and $T_{Auth}$ are served as the inputs of the forward embedding process to output the tagged news images $I_{Tag}$ and the corresponding lost information $L_{Trace}$ and $L_{Auth}$. Here, we encode $L_{Trace}$ and $L_{Auth}$ into a set of predefined distributed latent variables via conditional INNs for reserving more tag information. Meanwhile, the tagged news images would undergo moderate manipulations and malicious manipulations, resulting in real news images $I_{Real}$ and fake news images $I_{Fake}$, respectively. For the reverse revealing process of DINN, $I_{Real}$, $I_{Fake}$, and revealed lost information of $L_{Trace}$ and $L_{Auth}$ are used as inputs, and then the $T_{Trace}$ and $T_{Auth}$ are extracted from both $I_{Real}$ and $I_{Fake}$ . According to the extraction accuracy rates of tags to verify the news source and authenticity.

As shown in Fig. 2, DINN consists of several decoupled invertible blocks with the same structure. For the $n$-th decoupled invertible block, the input is $\left[T_{\text{Trace }}^{n},T_{\text{Auth }}^{n},I_{\text{Ori }}^{n}\right]$ and its output is $\left[T_{\text{Trace }}^{n+1},T_{\text{Auth }}^{n+1},I_{\text{Ori }}^{n+1}\right]$. Formally, the forward process is calculated as follows:

$$I_{\textit{Ori }}^{n+1}=I_{\textit{Ori }}^{n}+\phi_{1}\left(T_{\textit{Trace }}^{n}\right)+\phi_{2}\left(T_{\textit{Auth }}^{n}\right)\;\;\;\;\;\;\;\;\;\;\;$$

(1)

	$\displaystyle\;T_{\text{Auth }}^{n+1}$	$\displaystyle=T_{\text{Auth }}^{n}\odot\exp\left(\rho_{2}\left(I_{\text{Ori }}^{n}+\phi_{2}\left(T_{\text{Auth }}^{n}\right)\right)\right)\;\;\;$		(2)
		$\displaystyle\;\;\;+\eta_{2}\left(I_{\text{Ori }}^{n}+\phi_{2}\left(T_{\text{Auth }}^{n}\right)\right)$

$$\;\;\;T_{\text{Trace }}^{n+1}=T_{\text{Trace }}^{n}\odot\exp\left(\rho_{1}\left(I_{\text{Ori }}^{n+1}\right)\right)+\eta_{1}\left(I_{\text{Ori }}^{n+1}\right)$$

(3)

Where, $\phi_{1}(\cdot)$, $\phi_{2}(\cdot)$, $\rho_{1}(\cdot)$, $\rho_{2}(\cdot)$, $\eta_{1}(\cdot)$, and $\eta_{2}(\cdot)$ are sub-modules with convolution operations, $\exp(\cdot)$ is the Exponential function, and $\odot$ is the Hadamard product operation.

In the reverse revealing process of DINN, the concatenated images of $I_{\textit{Real}}$ and $I_{\textit{Fake}}$, and revealed lost information is fed in, and then four tags including $T_{\textit{Trace}}$ and $T_{\textit{Auth}}$ from both $I_{\textit{Real}}$ and $I_{\textit{Fake}}$ are extracted. Specifically, in the revealing process, the information inverse direction is from the $(n+1)$-th decoupled invertible block to $n$-th decoupled invertible block, as shown in Fig. 2. For the $(n+1)$-th triplet invertible block, the inputs are the revealed lost information $L_{\textit{Trace }}^{n+1}$ and $L_{\textit{Auth }}^{n+1}$, which are randomly sampled from distributions. $I_{\textit{Tag }}^{n+1}$ represents the manipulated news images, which is the concatenation of real news images $I_{\textit{Real}}$ and fake news images $I_{\textit{Fake}}$, as the input in the revealing process of DINN. Accordingly, the reverse process is calculated as follows:

	$\displaystyle T_{\textit{Trace }}^{n}$	$\displaystyle=\left(L_{\textit{Trace }}^{n+1}-\eta_{1}\left(I_{\textit{Tag }}^{n+1}\right)\right)\;\;\;\;\;\;\;\;\;\;\;\;$		(4)
		$\displaystyle\;\;\;\;\odot\exp\left(-\rho_{1}\left(I_{\textit{Tag }}^{n+1}\right)\right)$

	$\displaystyle\;\;\;T_{\text{Auth }}^{n}$	$\displaystyle=\left(L_{\textit{Auth }}^{n+1}-\eta_{2}\left(I_{\textit{Tag }}^{n+1}-\phi_{1}\left(T_{\textit{Trace }}^{n}\right)\right)\right)$		(5)
		$\displaystyle\;\;\;\;\;\odot\exp\left(-\rho_{2}\left(I_{\textit{Tag }}^{n+1}-\phi_{1}\left(T_{\textit{Trace }}^{n}\right)\right)\right)$

$$\begin{gathered}I_{\textit{Tag }}^{n}=I_{\textit{Tag }}^{n+1}-\phi_{1}\left(T_{\textit{Trace }}^{n}\right)-\phi_{2}\left(T_{\textit{Auth }}^{n}\right)\end{gathered}$$

(6)

After the last revealing block, both the embedded traceable tag and authenticable tag are extracted from each news image. It is noted that the extracted authenticable tag from fake news images should be quite different from the original one. And other tags should be extracted correctly from other news images.

Although the proposed DINN can decouple the embedding and extarcting of dual-tags, the desirable extraction accuracy performance still can not be guaranteed. To improve method usability, we design FAPM as the auxiliary to further support the decoupling process and guarantee extraction performance.

Inspired by the invertible image decolorization [39] and the model of content-aware noise projection in RIIS [35], we encode some lost information of $L_{\textit{Trace}}$ and $L_{\textit{Auth}}$ into two sets of pre-defined distributed latent variables via conditional INNs to improve the extraction performance. The embedded $T_{\textit{Trace}}$ and $T_{\textit{Auth}}$ can be efficiently revealed by randomly re-sampling new sets of pre-defined distributed variables.

In the embedding process, we expect to preserve more tag information in lost information $L_{\textit{Trace}}$ and $L_{\textit{Auth}}$, to perfectly reveal the embedded tags. Besides, to completely decouple the dual-tags and achieve different extraction goals, we employ conditional INN to encode the lost information of $L_{\textit{Trace}}$ and $L_{\textit{Auth}}$. With the conditional features extracted from tagged images $I_{Tag}$, the lost information is converted to two sets of latent variables $z_{Trace}$ and $z_{Auth}$, which follow a pre-defined uniform distribution $U(-1,1)$.

The specific scheme is shown in Fig. 3. Taking the traceable tag as an example, its corresponding lost information $L_{Trace}$ is also divided into two groups as input to the INN with the conditional feature from $I_{Tag}$. We employ the first two layers in Resnet18 [7] as the feature extractor $\mathcal{O}_{1}(\cdot)$ and $\mathcal{O}_{2}(\cdot)$ to extract features as the conditions on the mapping of $L_{Trace}$ and $L_{Auth}$, respectively. For the $K$-th conditional invertible block of FAPM, the forward process of lost information $L_{Trace}$ is calculated as shown in Eq. 7 and Eq. 8. The operation of authenticable tags is the same as the operation of traceable tags. The forward calculated process is shown in Eq. 9 and Eq. 10.

	$\displaystyle L_{\textit{Trace}_{1}}^{k+1}$	$\displaystyle=\exp\left(\alpha_{1}\left(L_{\textit{Trace}_{2}}^{k},\mathcal{O}_{1}\left(I_{\textit{Tag }}\right)\right)\right)\odot L_{\textit{Trace}_{1}}^{k}\;\;\;\;\;\;$		(7)
		$\displaystyle\;\;\;+\beta_{1}\left(L_{\textit{Trace}_{2}}^{k},\mathcal{O}_{1}\left(I_{\textit{Tag }}\right)\right)$

$\displaystyle L_{\textit{Trace}_{2}}^{k+1}=\exp\left(\gamma_{1}\left(L_{\textit{Trace}_{1}}^{k+1}\right)\right)\odot L_{\textit{Trace}_{1}}^{k}+\delta_{1}\left(L_{\textit{Trace}_{1}}^{k+1}\right)$

(8)

	$\displaystyle L_{\textit{Auth}_{1}}^{k+1}$	$\displaystyle=\exp\left(\alpha_{2}\left(L_{\textit{Auth}_{2}}^{k},\mathcal{O}_{2}\left(I_{\textit{Tag }}\right)\right)\right)\odot L_{\textit{Auth}_{1}}^{k}\;\;\;\;\;\;$		(9)
		$\displaystyle\;\;\;+\beta_{2}\left(L_{\textit{Auth}_{2}}^{k},\mathcal{O}_{2}\left(I_{\textit{Tag }}\right)\right)$

$\displaystyle L_{\textit{Auth}_{2}}^{k+1}=\exp\left(\gamma_{2}\left(L_{\textit{Auth}_{1}}^{k+1}\right)\right)\odot L_{\textit{Auth}_{1}}^{k}+\delta_{2}\left(L_{\textit{Auth}_{1}}^{k+1}\right)$

(10)

In the revealing process, the lost information is recovered through FAPM, and then the embedded tags are recovered from news images.

In the revealing process, we extract the traceable tag from all news images to verify the sources of news image. To effectively detect fake images, authenticable tag should be robust to moderate manipulation, while fragile to malicious manipulation, and the robustness can be evaluated by extraction error rates. Although extraction performance can be controlled by the designed loss function, the problem is more complex in a practical scenario since the types of manipulations are probably unseen unseen to the models. Thus, the generalization of the proposed model is still very important in fake news detection. Thus, we design a distance metric-guided modulation for tag extraction against the unseen types of manipulations.

The specific design is shown in Fig. 4. We define the extracted correct authenticable tag from real news images as $T_{Auth}^{ext_{-}r}$ and the extracted wrong authenticable tag from fake news images as $T_{Auth}^{ext_{-}f}$. The target feature spaces are that $T_{Auth}^{ext_{-}r}$ should be near the origin and converge to a hypersphere of radius $r$ to maintain intra-class compactness. On the contrary, $T_{Auth}^{ext_{-}f}$ is expected to be away from the smaller hypersphere by a predefined margin $m$ to ensure inter-class separation between $T_{Auth}^{ext_{-}r}$ and $T_{Auth}^{ext_{-}f}$. The designed loss function is as follows, as inspired by the attack detection of unseen face presentation [13].

	$\displaystyle\mathcal{L}_{\textit{margin }}$	$\displaystyle=\alpha_{\textit{Auth }}^{\textit{cor }}\times\max\left(d_{\text{Auth }}^{\textit{cor }}-r,0\right)$		(11)
		$\displaystyle\;\;\;+\alpha_{\textit{Auth }}^{\textit{wro }}\times\max\left((r+m)-d_{\textit{Auth }}^{\text{wro }},0\right)$

Where, $d_{Auth}^{cor}=\left\|T_{Auth}^{ext_{-}r}-T_{Auth}\right\|_{1}^{2}$, $d_{Auth}^{wro}=\left\|T_{Auth}^{ext_{-}f}-T_{Auth}\right\|_{1}^{2}$. The distance is calculated between extracted tag samples and the embedded tags by using the square of $L$1-norm. Besides, $\alpha_{Auth}^{cor}$ and $\alpha_{Auth}^{wro}$ are the parameters to control the weights of loss contributed by extracted tag samples of different types.

The total loss function is composed of four kinds of losses: the embedding and revealing losses of the main model, feature aware projection loss, distance metric-guided loss, and the low-frequency wavelet loss.

Since we expect to extract the wrong authenticable tag from fake news images, there is no loss between $T_{Auth}^{ext_{-}f}$ and $T_{Auth}$.

	$\displaystyle\mathcal{L}_{total}$	$\displaystyle=\omega_{1}\mathcal{L}^{embed}_{main}+\omega_{2}\mathcal{L}^{ext_{r}}_{Trace}+\omega_{3}\mathcal{L}^{ext_{f}}_{Trace}$		(19)
		$\displaystyle\;\;\;+\omega_{4}\mathcal{L}^{ext_{r}}_{Auth}+\omega_{5}\mathcal{L}_{z_{Trace}}+\omega_{6}\mathcal{L}_{z_{Auth}}$
		$\displaystyle\;\;\;+\omega_{7}\mathcal{L}_{margin}+\omega_{8}\mathcal{L}_{freq}$

To further evaluate the generalization of the proposed approach, we use COCO 2017 [14] validation dataset, and select 1,000 images from Weibo dataset [10], NewsStories dataset [28], and Flickr8k dataset  [25] separately to test the proposed method.

To evaluate the performance of the proposed approach, we conduct the following experiments.

The experimental results are shown in Tab. 4. It can be observed that the extraction performances of traceable tags and authenticable tags of proposed approach outperforms those of SoTA approaches under both moderate and malicious manipulations. Note that only the proposed approach shows distinguishable differences between the extraction of traceable tags and authenticable tags under malicious attacks. In summary, our approach achieves the SoTA performances for both news traceability and authenticity verification in fake news image detection. Note that the image quality of HiNet is inferior, because the solid color image-like hidden information is not considered in their training.

By comparing the experimental results of the proposed DINN, there are low extraction error rates for $T_{Trace}^{ext\_r}$, $T_{Trace}^{ext\_f}$, and $T_{Auth}^{ext\_r}$ while high extraction error rates for $T_{Auth}^{ext\_f}$. That indicates the proposed approach can achieve fake news image detection and source tracing. Besides, the image quality of the tagged news images is also improved.