Camera Model Identification with SPAIR-Swin and Entropy based Non-Homogeneous Patches

In deep learning applications, before processing, images are typically resized to a fixed size that can be handled efficiently using the parallelism available in hardware. Although resizing preserves semantic content for vision-related tasks, it is sub-optimal for forensics, including SCMI, because it can suppress or eliminate subtle features indicative of the camera model that are unrelated to the image semantic content. Instead of scaling, SCMI methodologies therefore use fixed size patches cropped from the input image for use in deep learning pipelines. The selection of image regions for the extraction of patches plays a critical role in determining the performance of SCMI algorithms. A good selection strategy can better isolate regions carrying camera model information and enhance the generalization performance.

We propose a patch extraction strategy based on entropy, which is illustrated in Fig. 2. By partitioning the images into non-overlapping patches, the model can focus more on the regions with higher information content that contain sensor noise patterns like PRNU noise, chromatic aberrations, and demosaicing artifacts. We argue these high-entropy regions are likely to hold various textures and patterns well-suited for source camera identification.

Areas with higher entropy correlate to regions containing variations in pixel intensity and often such areas contain sensor-specific noise patterns relevant for identifying cameras. Entropy based selection preferably retains patches with high frequency noise that are critical for SCMI, in contrast with traditional discrete cosine transforms (DCT) based JPEG compression methods that tend to discard high-frequency noise. By concentrating on non-homogeneous patches that capture the most discriminative features, our method facilitates the extraction of robust camera fingerprints.

Specifically, our patch selection is based on the (first-order) entropy is computed for an $8$-bit grayscale image patch $X$ as

$$H(X)=-\sum_{x=0}^{255}p(x)\cdot\log(p(x))$$

(1)

where, $p(x)$ denotes the fraction of pixels in the patch $X$ having a grayscale intensity $x$. First, the source image is center-cropped to maintain the overall composition of the image. The purpose of center cropping is to ensure the image is evenly divisible by the patch size, preserving the central content while removing peripheral details from the boundary areas.Then, non-overlapping patches are extracted by using a $k\times k$ sliding window with a stride of $k$ pixels along horizontal and vertical directions. A patch size of $k=256$ is used as a balanced choice to capture global and local features. This size maintains crucial information and is suitable for the SWIN Transformer architecture. Following the patch extraction, we compute the entropy of each patch using Equation (1). Patches are then sorted in descending order of entropy and the $P$ patches with the highest entropy are chosen. Empirically, the parameter $P$ is determined to optimize the trade-off between information richness and computational requirements. In Section 5, we analyze the effect of different values of $P$ on model performance and determine the optimal configuration that maximizes the discriminative capability of our model without adding redundancy.

To extract features useful for SCMI from individual image patches, we proposed a SPAIR block that combines an Inverted Residual Block [11] with a modified spatial attention block [24]. The SPAIR block makes use of spatial attention to focus on distinctive features of interest by amplifying camera-specific features whereas the Inverted Residual Block utilizes its expansion layer and depthwise convolutions to retain information and to focus on unique channel-wise patterns.

The Inverted Residual Block [11] is commonly used in many deep convolutional neural networks (CNNs). Its lightweight nature helps to create feature maps efficiently as it utilizes point-wise convolutions for dimensionality reduction and expansion. This is followed by a depth-wise separable convolution that significantly reduces the computational cost compared to the standard convolution. This combination promotes efficient feature learning while maintaining spatial information. The initial input is added to the output of the second convolution layer before passing it to modified spatial attention module. This residual connection helps with the gradient flow during training.

Unlike traditional modified spatial attention module [24], which uses both Max Pooling and Average Pooling, we simplify the process by focusing only on Average Pooling to reduce the redundancy while still capturing global features. The softmax layer utilized at the end ensures that attention features are globally distributed across the entire spatial dimension, enabling our model to focus on the most discriminative features. This targeted attention mechanism helps to suppress irrelevant background content and focus on global noise patterns that differentiate the camera models. By selectively improving informative regions, SPAIR aims to extract more discriminative features that effectively distinguish images captured by different cameras.

The output of the SPAIR block is processed by a Swin Transformer [10]. The Swin Transformer is a hierarchical transformer model designed specifically for computer vision/image processing that is more efficient than the initial proposals for vision transformers. The embedded patches are converted to tokens that form the base of the input to the transformer. Positional encoding is added to the tokens to supply spatial information for the model to be aware of the position of patches relative to each other. Another unconventional characteristic of the Swin Transformer is the low-cost manner in which it accommodates multi-scale features. Tokens at lower levels represent fine-grained details, whereas tokens at higher levels capture more abstract and global features. The final feature maps are obtained by combining all tokens which gives an overall view of the input features. These features obtained are then entered into the classification head.

To assess the robustness and generalizability of the proposed SCMI method, we perform experiments on four large publicly available datasets: Dresden [25], Vision [26], Forchheim [27], and Socrates [28]. The selection of smartphone image datasets is significant as most images these days are acquired from smartphone cameras. Table 1 shows a summary of the datasets used. Before discussing the experiments, we outline the pertinent details of each of these datasets:

• •

  The Dresden dataset [25] is the largest with respect to the number of images used in our experiments. It comprises $14999$ images captured by $18$ camera models. As indicated in [29], Nikon_D70 and Nikon_D70s have been consolidated into a single camera model.

• •

  The Vision dataset [26] includes images captured with various smartphone cameras. It comprises $11732$ original images distributed over $35$ camera devices and contains $5$ identical camera models. We combine these classes for our experiments.

• •

  The Forchheim dataset [27] includes $143$ scenes taken with $9$ different brands of smartphones. It has $3851$ images taken under various conditions, which include changes in lighting parameters, ambient light, and imaging parameters. Two out of the $27$ camera devices are identical, indicating that two models have more than one device, which are merged for Source Camera Model Identification tests. Consequently, there are $25$ different camera devices. This study focuses on native camera images that are labeled original (orig.) in [27].

• •

  The Socrates [28] is the largest dataset in terms of number of camera devices and models, containing $9721$ images taken with $65$ different camera models. This dataset offers a rich variety of scenarios and authenticity, with images captured by individual smartphone users.

All the experiments are performed on a $3.38$ GHz AMD EPYC $7742$, $64$ core processor with $64$ GB RAM and an Nvidia A$100$ GPU with $40$ GB memory. Images in the dataset are split in an $80:20$ ratio. Splitting is performed before patching to ensure that the derived patches are present in either the training or the test set. The PyTorch [30] framework was used for implementation. A pre-trained Small Swin Transformer [10] is used that has a batch size of $32$. AdamW [31] optimizer is used that has an initial learning rate of $0.0001$. All models were trained for $100$ epochs, at which point, they exhibited convergence in training loss, except for our proposed model which showed convergence around $50$ epochs.

To evaluate the proposed method, we measured Image Level Accuracy (ILA) and Patch Level Accuracy (PLA). The evaluation was carried out with a total of $N$ images collected by $C$ camera devices. For the $i^{th}$ image, $X_{i}$, $Y_{i}$, and $\hat{Y}_{i}$ denote the specific image, the actual camera model, and the predicted camera model, respectively. Furthermore, the $\mathcal{P}_{i}$ number of patches has been extracted from the $i^{th}$ image. For the $j^{th}$ patch of the $i^{th}$ image, $y_{ij}$ and $\hat{y}_{ij}$ represent the actual and predicted camera models, respectively, for that patch image. PLA and ILA are computed as

$$PLA=\frac{\sum_{i=1}^{N}\sum_{j=1}^{\mathcal{P}_{i}}\mathds{1}(\hat{y}_{ij}=y_{ij})}{\sum_{i=1}^{N}\mathcal{P}_{i}}$$

$$ILA=\frac{\sum_{i=1}^{N}\mathds{1}(\hat{Y}_{i}=Y_{i})}{N}$$

where $\mathds{1}(.)$ is the indicator function and $\hat{Y}_{i}$ corresponds to the camera model with the maximum number of votes according to the patch-level predictions of the image and its mathematical formulation is $\hat{Y}_{i}=\operatorname*{argmax}{k}(\sum_{j=1}^{\mathcal{P}_{i}}\mathds{1}(\hat{y}_{ij}=k))$.

For comprehensive analysis across the datasets, we also used F1-score where unweighted mean was used to compute a single value for ease of understanding and visualization.

We compare the performance of the proposed method against several competing methods for Camera Model Identification (CMI), specifically those proposed by Chen et al. [17], Liu et al. [18], Rafi et al. [6], Bennabhaktula et al. [5], Sychandran et al. [21], and Huan et al. [22]. Chen et al. [17] employed a CNN-based method without pre-processing strategies while all other SCMI methods utilized varied pre-processing strategies to enhance image features. This variety of methodological frameworks enhances comparative analysis by highlighting the strengths and limitations of each method in the context of source camera model identification.

This section provides a comprehensive analysis in order to demonstrate the efficacy of the proposed patch selection strategy. The key motivation for selecting non-homogeneous patches, i.e., high-entropy patches, is their ability to capture camera-specific features that are unevenly distributed across the image. These features include sensor noise, demosaicing patterns, and compression signatures, which are prominently found in parts with complex textures, edges, and high-frequency component areas.

Table 3 presents the comparative results of various baseline methods experimented across four different datasets with their respective native patch selection strategy and the proposed patch selection strategy while keeping all the experimental settings consistent with the original studies, which include the model and training pipelines, their respective hyperparameters, the number of patches, and patch sizes. The results demonstrate the effectiveness of the proposed entropy-based patch extraction in enhancing camera model identification performance across various state-of-the-art methods. It can be observed, baseline methods with our proposed patch selection strategy achieve an average relative PLA improvement of $2.93\%$, $3.65\%$, $12.98\%$, and $4.48\%$ , and an average relative ILA improvement of $1.41\%$, $2.09\%$, $8.89\%$, and $2.26\%$ over their native patch selection strategy on the Dresden [25], Vision [26], Forchheim [27], and Socrates [28] datasets, respectively. Notably, all studies show improvements in both PLA and ILA across all the datasets, except for a marginal ILA loss of $0.34\%$ in Liu et al.[18]. The relative gains are massive especially in calibration using difficult datasets like Forchheim which contains a high amount of similar images, and Socrates which has fewer samples per camera model and the highest classes for identification. For example, the method developed by Huan et al., experiences a drastic change in PLA from $83.74\%$ to $98.01\%$ for the Forchheim [27] data set and from $86.35\%$ to $94.16\%$ of the Socrates [28] dataset. These improvements are statistically significant across all the methodologies and datasets considered, demonstrating the consistency and expansiveness of the proposed patch extraction strategy.

Table 4 shows the performance of the proposed CMI method by considering both scenarios: (1) using homogeneous patches and (2) using non-homogeneous patches. The higher patch-level accuracy is achieved by using the non-homogeneous patches, as their high-entropy regions contain rich textures, edges, and structural details, effectively captured by the shifted window multi-head self-attention module of Swin Transformers. The modified spatial attention mechanism assigns weights to emphasize distinctive regions, aiding camera-specific feature extraction. In contrast, homogeneous patches, being uniform, offer fewer distinguishing features, making artifact isolation for SCMI more challenging. Moreover, the improvement in image-level precision, except for a marginal decline on Dresden [25], highlights the advantage of high-entropy patches.

An ablation analysis was conducted to determine the optimal number of patches for maximizing the proposed model’s performance, while addressing underfitting and overfitting. These patches must also be non-redundant about the information they convey as well as the patches should be high in quality. Table 5 shows that increasing the number of patches from 15 to 20 consistently improved both patch-level accuracy (PLA) and image-level accuracy (ILA) across four datasets. Overall, the best results are achieved when the number of high-entropy patches selected per image is 20. Increasing the number of such patches to 30 or 60 results in performance drop in most of the cases.