TBFormer: Two-Branch Transformer for Image Forgery Localization

To exploit the potential forgery cues in different domains, we design two feature extraction branches to extract discriminative features from the RGB domain and the noise domain. The two branches have the same architecture, and their weights are not shared which makes them concentrate on their specific domains. We adopt BayarConv [33] for converting the RGB domain to the noise domain. Transformer can overcome the shortcomings of convolutional neural networks with only limited receptive fields and has the powerful ability to model contextual global dependencies [34]-[35]. The rich contextual information is also crucial for locating forged regions, so Transformer is adopted for our feature extraction.

The input color RGB image ${\bm{I}_{c}\in\mathbb{R}^{H\times W\times 3}}$ is first converted to the noise map ${\bm{I}_{n}\in\mathbb{R}^{H\times W\times 3}}$ by BayarConv, where $W$ and $H$ denote the width and height of the input image. We divide ${\bm{I}_{c}}$ into image patches of size ${16\times 16}$ to obtain the sequence ${\bm{X}_{c}=\left\{\bm{x}_{c}^{\left(1\right)},\bm{x}_{c}^{\left(2\right)},\cdots,\bm{x}_{c}^{\left(N\right)}\right\}}$, where ${\bm{x}_{c}^{\left(i\right)}\in\mathbb{R}^{16\times 16\times 3}}$ and ${N=H/16\times W/16}$ is the number of image patches. Each image patch ${\bm{x}_{c}^{\left(i\right)}}$ is reshaped into a one-dimensional vector, followed by a linear projection layer to obtain the image patch embedding sequence ${\bm{P}_{c}=\left\{\bm{p}_{c}^{\left(1\right)},\bm{p}_{c}^{\left(2\right)},\cdots,\bm{p}_{c}^{\left(N\right)}\right\}\in\mathbb{R}^{N\times L}}$, where $L$ denotes the feature dimension. The corresponding position embedding ${\mathbf{pos}_{c}^{\left(i\right)}}$ is added to the image patch embedding ${\bm{p}_{c}^{\left(i\right)}}$ to obtain the resulting input sequence ${\bm{E}_{c}=\left\{\bm{e}_{c}^{\left(1\right)},\bm{e}_{c}^{\left(2\right)},\cdots,\bm{e}_{c}^{\left(N\right)}\right\}\in\mathbb{R}^{N\times L}}$, where ${\bm{e}_{c}^{\left(i\right)}=\bm{p}_{c}^{\left(i\right)}+\mathbf{pos}_{c}^{\left(i\right)}}$. Then ${\bm{E}_{c}}$ is fed into the feature extractor which is constructed based on $12$ Transformer layers. The feature maps of the $4$th, $8$th, and $12$th layers (i.e., $\bm{T}_{c}^{\left(4\right)},\bm{T}_{c}^{\left(8\right)},\bm{T}_{c}^{\left(12\right)}$) are output for further investigation:

where ${f_{c}}$ denotes the feature extractor of the RGB branch. The Transformer layer consists of a Multi-Head Self-Attention (MSA) block and a Multi-Layer Perceptron (MLP) block, and the architecture of the $i$th layer can be represented as:

where ${\rm{LN}}$ represents layer norm. The ${{\rm{MSA}}_{c}^{\left(i\right)}}$ block is constituted by the Self-Attention (SA) operation:

where query, key, value are computed as ${{\bm{Q}}_{c}^{\left(i\right)}=\bm{T}_{c}^{\left(i-1\right)}{\bm{W}}_{\rm{cQ}}^{\left(i\right)}}$, ${{\bm{K}}_{c}^{\left(i\right)}=\bm{T}_{c}^{\left(i-1\right)}{\bm{W}}_{\rm{cK}}^{\left(i\right)}}$, ${{\bm{V}}_{c}^{\left(i\right)}=\bm{T}_{c}^{\left(i-1\right)}{\bm{W}}_{\rm{cV}}^{\left(i\right)}}$, and ${{\bm{W}}_{\rm{cQ}}^{\left(i\right)}}$, ${{\bm{W}}_{\rm{cK}}^{\left(i\right)}}$, ${{\bm{W}}_{\rm{cV}}^{\left(i\right)}}$ are the learnable parameters of three linear projection layers in self-attention [36].

The same processes are performed on the noise map ${\bm{I}_{n}}$ to obtain ${\bm{E}_{n}\in\mathbb{R}^{N\times L}}$. The noise features are obtained by feeding ${\bm{E}_{n}}$ into the feature extractor of the noise branch:

where $f_{n}$ denotes the feature extractor of the noise branch, ${\bm{T}_{n}^{\left(4\right)},\bm{T}_{n}^{\left(8\right)},\bm{T}_{n}^{\left(12\right)}\in\mathbb{R}^{N\times L}}$ denote the features output by the $4$th, $8$th, and $12$th Transformer layers.

The feature maps of the two branches have significant differences for that they are extracted from different domains. A carefully designed decoder is helpful for mask reconstruction from different domains, and a well-designed feature fusion module is also an indispensable part of a network to investigate multi-domain information. We design an Attention-aware Hierarchical-feature Fusion Module (AHFM) to effectively fuse hierarchical features from two different domains.

For the RGB features and noise features from the same layer, we construct a position attention block [37] to investigate their correlation and fuse them into unified feature maps. Taking the $4$th-layer features as examples, the matrixes ${\bm{T}_{c}^{\left(4\right)}\in\mathbb{R}^{N\times L}}$ and ${\bm{T}_{n}^{\left(4\right)}\in\mathbb{R}^{N\times L}}$ are transposed and reshaped to get the three-dimensional tensors ${\hat{\bm{T}}_{c}^{\left(4\right)}\in\mathbb{R}^{L\times h\times w}}$ and ${\hat{\bm{T}}_{n}^{\left(4\right)}\in\mathbb{R}^{L\times h\times w}}$, where ${N=h\times w}$, ${h=H/16}$, and ${w=W/16}$. Then, ${\hat{\bm{T}}_{c}^{\left(4\right)}}$ and ${\hat{\bm{T}}_{n}^{\left(4\right)}}$ are concatenated along the channel dimension to get ${\bar{\bm{T}}^{\left(4\right)}\in\mathbb{R}^{2L\times h\times w}}$. A convolution operation is performed on ${\bar{\bm{T}}^{\left(4\right)}}$ to get ${\hat{\bm{T}}^{\left(4\right)}\in\mathbb{R}^{L\times h\times w}}$, then three different convolutional layers are constructed for ${\hat{\bm{T}}^{\left(4\right)}}$ to obtain ${\hat{\bm{T}}^{\left(4\_1\right)}\in\mathbb{R}^{L/8\times h\times w}}$, ${\hat{\bm{T}}^{\left(4\_2\right)}\in\mathbb{R}^{L/8\times h\times w}}$, and ${\hat{\bm{T}}^{\left(4\_3\right)}\in\mathbb{R}^{L\times h\times w}}$. Then, they are reshaped to ${\bm{T}^{\left(4\_1\right)}\in\mathbb{R}^{L/8\times N}}$, ${\bm{T}^{\left(4\_2\right)}\in\mathbb{R}^{L/8\times N}}$, and ${\bm{T}^{\left(4\_3\right)}\in\mathbb{R}^{L\times N}}$. Position attention weights ${{\bm{A}}^{\left(4\right)}\in\mathbb{R}^{N\times N}}$ can be computed as:

Then, we conduct matrix multiplication between ${\bm{T}^{\left(4\_3\right)}}$ and ${{\bm{A}}^{\left(4\right)}}$, and the computed result is reshaped to ${\hat{\bm{Z}}^{\left(4\right)}\in\mathbb{R}^{L\times h\times w}}$. Then, $\hat{\bm{Z}}^{\left(4\right)}$ is multiplied with a learnable weight ${{\alpha}^{\left(4\right)}}$, and we perform element-wise addition between the weighted $\hat{\bm{Z}}^{\left(4\right)}$ and ${\hat{\bm{T}}^{\left(4\right)}}$. A convolution operation is conducted to get the fused feature map ${\bm{Z}^{\left(4\right)}}\in\mathbb{R}^{L\times h\times w}$ as follows:

where ${\oplus}$ denotes element-wise addition. The detailed computational procedure is shown in Fig. 2. Following the same computational procedure, we can also get the fused feature maps ${\bm{Z}^{\left(8\right)}}$ for the $8$th layer and ${\bm{Z}^{\left(12\right)}}$ for the $12$th layer.

In order to sufficiently integrate the hierarchical features, we conduct element-wise addition followed by a convolution operation to get the final fused feature map ${\bm{Z}\in\mathbb{R}^{L\times h\times w}}$:

The general framework of our AHFM module is shown in Fig. 1 (the bounding box of AHFM).

Image forgery localization classifies each pixel in an image into two classes, i.e., authentic class and forged class. It can essentially be considered as a special image segmentation task. We set two learnable category embeddings in the decoder to further learn the feature representations of authentic and forged classes [38], and they are interacted with the patch embeddings of the fused feature map to produce the predicted masks. Our decoder mainly contains $2$ Transformer layers.

Specifically, ${\bm{Z}\in\mathbb{R}^{L\times h\times w}}$ is sequentially reshaped, transposed, and linearly projected to obtain the embedding sequence ${\dot{\bm{Z}}\in\mathbb{R}^{N\times L}}$. Then $\dot{\bm{Z}}$ and the category embeddings ${\bm{S}\in\mathbb{R}^{2\times L}}$ are reconstructed by Transformer layers to obtain ${\ddot{\bm{Z}}\in\mathbb{R}^{N\times L}}$ and ${\ddot{\bm{S}}\in\mathbb{R}^{2\times L}}$. After performing linear projection and L2 normalization on ${\ddot{\bm{Z}}}$ and ${\ddot{\bm{S}}}$, respectively, the quantization value ${\ddot{\bm{Y}}\in\mathbb{R}^{N\times 2}}$ can be obtained by the scalar product operation:

where $L_{2}$ denotes L2 normalization and $f_{\rm{proj}}$ denotes linear projection. The transpose and reshape operations are performed sequentially on ${\ddot{\bm{Y}}}$ to obtain ${\bm{Y}\in\mathbb{R}^{2\times h\times w}}$, and the predicted mask ${\bm{M}}$ is computed as:

where $\rm{Upsample}$ denotes the upsampling operation which can resize $\bm{Y}$ to the same size as the input image. Our model is trained using a pixel-level binary cross-entropy loss function.

TBFormer is compared with six state-of-the-art methods, i.e., RGB-N [24], ManTra-Net [23], SPAN [15], MVSS-Net [20], PSCC-Net [19], and ObjectFormer [31]. Table III reports the compared results on four publicly available datasets, and Fig. 3 visualizes predicted masks of the methods with publicly available codes. On the NIST16 dataset, RGB-N, SPAN, PSCC-Net, and ObjectFormer are fine-tuned, and we follow the same training/testing splits for fine-tuning the model to make a fair comparison. On the CASIA dataset, RGB-N, SPAN, PSCC-Net, and ObjectFormer are fine-tuned on CASIA v2.0 and tested on CASIA v1.0. The training dataset of MVSS-Net and our synthesized dataset are generated from CASIA v2.0, so the results of MVSS-Net and TBFormer on CASIA v1.0 are generated by the models without fine-tuning. The results of MVSS-Net on all datasets, the scores of ManTra-Net and SPAN on the IMD20 dataset, and the results of compared methods on the Realistic dataset are obtained by the pre-trained models released by the authors, and the rest of scores are borrowed from their original papers. Table III shows that TBFormer achieves the best performance on each dataset, and it also can be seen in Fig. 3 that TBFormer can locate forged regions more accurately.