On Addressing the Impact of ISO Speed upon PRNU and Forgery Detection

When a digital image is used in a forensic investigation or presented as evidence to the court, it is important to authenticate the image to ensure its content is free from manipulation. Thus, image forgery detection draws substantial attentions from researchers. Among different techniques developed for image forgery detection, Photo Response Non-Uniformity (PRNU) based methods have shown their unique strength. PRNU is a sensor pattern noise intrinsically embedded in images. It arises as a result of the manufacturing imperfections of silicon wafers in image sensors. As such, pixels on a sensor would have a non-uniform response to the incident light and introduce a unique pattern noise to the image, which can be treated as the fingerprint of a device. Many different algorithms have been proposed for PRNU-based source camera identification [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] and image forgery detection [12, 13, 14, 15, 16, 17]. In most of these works, PRNU is utilized by computing the image-wise or block-wise correlations between the source device's reference PRNU and the test image's PRNU. The corresponding image-wise (source camera identification) or pixel-wise decision (forgery detection) can be made by comparing the correlations with a decision threshold.

The PRNU is often estimated in the form of the noise residual of an image. The noise residual can be extracted from an image by simply subtracting the de-noised image from the original image. By nature, PRNU is a weak noise. The existence of camera artifacts and other PRNU-irrelevant noises (e.g. shot noise, thermal noise, etc.) in an image's noise residual can reduce the correlation between the noise residual and the device's reference PRNU. It becomes a non-trivial problem to separate the inter-class (images from different source devices) from the intra-class (images from the same source device) correlations. It becomes particularly problematic when the PRNU quality in the noise residual is poor such that these two types of correlations' distributions can have large overlaps.

Despite a large number of works that have been done to better extract, estimate and enhance the PRNU [4, 5, 6, 9, 13, 18, 3, 19, 20, 21, 22], the overlap between inter- and intra-class correlations cannot be completely avoided. Thus, many researchers have been working on refining the choice of the decision thresholds to better separate the two classes, especially for image forgery detection [13, 15, 17]. The decision thresholds are often set with reference to the expected intra-class correlations predicted by a correlation predictor. The correlation between an image's noise residual and the device's reference PRNU reflects the strength of the PRNU in the image. As the strength of the PRNU is multiplicative of the pixel intensity and some highly textured image content or post-processing may damage the PRNU's quality, correlation prediction should be performed in an adaptive manner. A content-dependent correlation predictor is proposed by Chen et al. in [13], which formulates the correlation predictor as a regressor model of four image features, namely the intensity, texture, signal-flattening and a texture-intensity combinative term. This correlation predictor has been adopted by many PRNU-based forgery detection algorithms (e.g. [13, 14, 15, 16, 17]). Due to the complex nature of the PRNU correlation, despite different attempts to re-engineer the correlation predictor over the past decade, we have not witnessed much success. Thus, the digital forensic community still relies greatly on the correlation predictor from [13] for PRNU-based forgery detection.

However, over the last decade, we have also witnessed great advancement in the digital camera industry, especially in sensor design. Such advancement also brings new challenges to PRNU-based digital forensics. Therefore, we have observed a few issues about the correlation predictor proposed in [13]. An important feature ignored by the correlation predictor is the camera sensitivity setting, which is commonly known by the name of ISO speed. The ISO speed together with the shutter speed and the aperture size are the three parameters, which control an image's exposure in digital photography. The shutter speed and aperture size control the number of photons arriving at the image sensor during the exposure process while the ISO speed determines the camera's signal gain. In real-life, photographers may face many physical restrictions on the aperture size and shutter speed. Such restrictions require more freedom of choice in ISO speeds to achieve the desired exposure. Thus, many camera manufacturers have been working on improving sensor performance and providing more and higher ISO speeds to digital cameras. While the improvements have been brought to sensor technology, it is also a known fact that high ISO speeds may introduce more noise to an image. As a result, the quality of the PRNU left in the noise residual will be reduced when a high ISO speed is used. A recent work presented in [23] empirically shows that different ISO speeds may affect the performance of PRNU-based source camera identification. With camera manufacturers increasingly supporting broader ranges of ISO speed settings on digital cameras and mobile devices, a proper analysis of the ISO speed's influence on PRNU-based image forensics, especially on the correlations, needs to be carried out.

As this work focuses on the correlation between an image's noise residual with its reference PRNU, for simplicity, we will call it the correlation. The contribution of this work can be summarized as follows:

• •

  We first analytically and empirically proved in Section II that the correlation between an image’s noise residual and its reference PRNU is not only content-dependent as previously known, but also dependent on the camera sensitivity setting (i.e. the ISO speed).

• •

  We then validate our postulate in Section III that, due to such ISO speed dependency, reliable predictions of the correlation between an image’s noise residual and its reference PRNU can only be accurately made when a correlation predictor is trained on images of similar ISO speeds to the image in question.

• •

  Base on the postulate, we propose an ISO specific correlation prediction process. Recognizing that in the real-world, information about the ISO speed may not be available to facilitate the implementation of our postulate in the correlation prediction process, we propose a method called Content-based Inference of ISO Speeds (CINFISOS, /’[email protected]@s/) in Section IV to infer the ISO speed from the image content.

In order to carry out this in-depth investigation into how the ISO speed can affect PRNU-based image forensics, we use the purposefully built Warwick Image Forensics Dataset [24]. The images in this dataset are taken with diverse exposure parameter settings. The dataset involves 14 cameras and images of various scenes. In particular, for 20 different scenes for each camera, multiple images of the same scene are shot with varying ISO speeds and exposure times. Thus, these images allow us to conduct studies on the ISO speed's influence on the correlation.

In this section, we demonstrate that an image's ISO speed can affect its correlation. As a general noise model can be complicated, to show the existence of such an ISO Speed-Correlation relationship in a concise manner, we use a special case to prove this relationship analytically and then empirically show it with more general cases. The special case considered is a single color channel of a flat-field RAW image, from which we expect the same value for every pixel if they are noise-free. To conduct PRNU-based pixel-wise forgery detection, the correlation between the noise-residual of a block centered at each pixel and the corresponding block of the reference PRNU is calculated. Let $\boldsymbol{z}$ be a noise residual within a block $N_{i}$ centered at pixel $i$ and $\boldsymbol{\omega}$ be the reference fingerprint within the corresponding block. Assume both $\boldsymbol{z}$ and $\boldsymbol{\omega}$ are standardized, which means they follow the normal distribution $\mathcal{N}(0,1)$. We can model both signals as the sum of a PRNU component and a PRNU-irrelevant part. At pixel $j\in N_{i}$:

where $\boldsymbol{x}$ and $\boldsymbol{y}$ are the PRNU components of $\boldsymbol{\omega}$ and $\boldsymbol{z}$ while $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are the PRNU-irrelevant noises. As for a flat-field image, we can approximate its PRNU component, $\boldsymbol{x}$ in this case, as a normal distribution $\mathcal{N}(0,{\sigma_{x}}^{2})$ and $\boldsymbol{\alpha}$ conforms to $\mathcal{N}(0,1-{\sigma_{x}}^{2})$. For intra-class pairs, $\boldsymbol{x}$ and $\boldsymbol{y}$ represent the same PRNU. As they may differ in strength, without losing generality, we can express $\boldsymbol{y}$ as $\mathcal{N}(0,{\sigma_{y}}^{2})$ with ${\sigma_{y}}=\sqrt{\lambda}\sigma_{x}$ and $\boldsymbol{y}=\sqrt{\lambda}\boldsymbol{x}$. $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are mutually independent. So when we compute the correlation $\rho_{i}$ of the block $N_{i}$, the correlation $\rho_{i}$ becomes:

with

From the above expression, we can see that the expected correlation value, $\mu_{i}$, is proportional to the standard deviation $\sigma_{y}$ of the PRNU component, $\boldsymbol{y}$, in the image's noise residual, $\boldsymbol{z}$. Based on the Poissonian-Gaussian noise model [25, 26, 27], we can see that the ISO speed would affect this standard deviation $\sigma_{y}$ and eventually exert influence on the PRNU correlations.

The relationship between the camera gain, $g$, which is directly determined by the camera's ISO speed, and the noisy raw pixel intensity, $I$, is analyzed in [26]. The raw pixel intensity is proportional to the number of electrons counted on the sensor. Photo-electron conversion is the main source of the electrons collected from the sensor. [26] considers the Poissonian statistics of the incident photon counting process as follows. At pixel $i$, the number of the counted electrons is the sum of the electrons generated from photo-electron conversion ${N_{p}}_{i}$ and dark electrons ${N_{t}}_{i}$ from the thermal noise. It is assumed that the variance of the thermal noise is uniform across the sensor and all other electronic noises can be modeled as a zero-mean Gaussian noise with variance $s^{2}$. So the raw pixel intensity, $I_{i}$, at pixel $i$, can be written as:

where $\mathcal{P}(\cdot)$ represents the Poisson distribution and $\eta_{i}$ is the photon-electron conversion rate at pixel $i$. $p_{0}$ is a base pedestal parameter introduced in the camera design to provide an offset-from-zero of the pixel's output intensity. For each pixel, as a large number of electrons are counted, the normal approximation of Poisson distribution can be exploited. Therefore, $I_{i}$ can be modeled as:

with

$\varphi$ can be viewed as the expected pixel intensity. Notice that this model from [26] has not yet considered the PRNU. To include the PRNU in this model, we write the photo-electron conversion rate $\eta_{i}$ as the following expression by considering the non-uniform response of each pixel to the photons:

where $\bar{\eta}$ is the average photo-electron conversion rate and $k_{i}$ is the PRNU factor at pixel $i$. $k$ follows normal distribution $\mathcal{N}(0,\sigma_{k}^{2})$. As we are considering the case of a flat-field image here so we can fix the number of photons, $N_{p_{i}}$, collected at every pixel. By expanding Equation (5), we have:

As in most cases, both the PRNU and the thermal noise are weak noises. We can ignore the terms involving $k_{i}{N_{t}}_{i}$. When we consider a block $N_{i}$, often it consists of thousands of pixels (e.g. 4096 pixels for a $64\times 64$ block). Such a large number of pixels allow us to approximate the overall distribution of the pixel values in this block by another normal distribution. By substituting $t$ of Equation (8) with the expression for $t$ in Equation (6), we approximate the distribution of the pixel values in block $N_{i}$ as:

We expect the de-noised version of this block to have pixels of uniform intensity, $\varphi$. Thus, we can approximate the variance of the noise residual of this block as:

The PRNU component in the noise residual has a variance of $\varphi^{2}\sigma_{k}^{2}$. By normalizing the noise residual, the standard deviation of the PRNU component in the normalized noise residual becomes:

Clearly, $\sigma_{y}$ is dependent on the camera gain $g$. By substituting this expression back to Equation (3), we can conclude that the correlation $\rho_{i}$ can be affected by the camera gain $g$ and thus affected by ISO speed.

Notice that when we introduce PRNU by considering different photo-electron conversion rate, $\eta_{i}$, at each pixel to the raw pixel intensity model from [26], the noise residual variance model described in Equation (10) becomes a quadratic function of the expected pixel intensity $\varphi$, which can be expressed as:

with

It differs from the linear model in [26]. We will empirically validate Equation (10) to show the physical importance of the PRNU term, $\varphi^{2}{\sigma_{k}}^{2}$, in the equation despite the approximations made.

We use four cameras for the test, namely a Nikon D7200, a Canon 6D MKII, a Canon 80D, and a Canon M6. Each of the four cameras can generate 14-bits RAW images, which means their pixel values can vary between the range of $[0,16383]$. To better show the physical meaning of the coefficients in Equation (10), we standardize the pixel values to the range of $[0,1]$. To validate Equation (10), we plot the variance of the noise in the flat-field images against different pixel values in Fig.1. We use the cameras to take images of a screen of flat color. Each camera's ISO speed is set to 100. The exposure time is varied to change the pixel intensity for different shots. As the cameras use Bayer-filter as their color filtering array (CFA), we subsample the RAW images with a stride of 2 in both vertical and horizontal directions to make sure the pixels we test are from the same color channel. Despite the set-up, the images are not completely flat due to other camera artifacts, e.g. vignetting. Thus, we use the method from [26] to estimate the expected pixel value and variance for multiple image blocks from each noisy RAW image. Fig.1 shows the fitting of Equation (12) to the experiment data, which is computed using ordinary least squares (OLS) [28]. A good agreement between the model and the data can be observed.

In addition to showing the good agreement of the derived model and the real data, we would like to show the physical meaning of the first order coefficient, $B=g$ in the model as well. We use the RAW images from the same Canon 6D MKII from the previous test for this test. We repeat the previous experiment four times but set the cameras' ISO speed to ISO 200, 400, 800, and 1600, respectively. Again, we fit Equation (12) to the data. As for the same camera, despite the change of ISO speed, we can assume that the PRNU factor on the sensor should remain the same and so does the variance of the PRNU factor, $\sigma_{k}^{2}$. Thus, it is reasonable for us to fix the second order coefficient $A=\sigma_{k}^{2}$ to $5.24\times 10^{-5}$, the value estimated from Fig.1, in Equation (12) for these fittings and the corresponding fittings generated using OLS are shown in Fig.2. Once again, good agreement between the fitted curve and the data can be observed. In addition, we show a $\log-\log$ plot of the estimated first order coefficients $B$ from Fig.1(b) and 2 against the ISO speed of their corresponding images in Fig.3. We fitted a straight line to the plot given slope close to $1$. As a camera's ISO speed is proportional to its camera gain, $g$, Fig.3 validates our noise model from Equation (10) with $B=g$. Therefore, it confirms that the correlation model is dependent on ISO speed.

The above conclusions are made for the special condition when we consider the images to be RAW flat-field image. When we take post-processings (e.g. color interpolation and JPEG compression) and the influence due to the image content into consideration, the noise model could become rather complicated. This is both because the PRNU is multiplicative of image content and image content may propagate into the noise residual due to imperfect denoising. And actually, higher ISO images are more likely to suffer from strong JPEG compression and imperfect denoising (see supplementary material). Thus, though Equation (10) cannot be translated directly to the general conditions, all the factors suggest a higher ISO speed can introduce more PRNU-irrelevant noise. As a result, this will reduce the proportion of signals corresponding to the PRNU in the noise residual and eventually reduce the correlation. We use Fig.4 to empirically show that the correlation is dependent on the image's ISO speed when post-processings like de-mosaicing, gamma correction, JPEG compression, etc., are applied to a non-flat RAW image.

The images shown in Fig.4 are from a Canon 6D MKII camera in the Warwick Image Forensics Dataset. All the images shown here are saved in the JPEG format by the camera's default setting. Images of two scenes are taken under different ISO speeds using different exposure times to ensure that every image can reach the same exposure level. Thus, there is nearly no difference in pixel intensity between the images of the same scene. As the PRNU is a multiplicative signal, having images of the same pixel intensity of the same image content allows us to make a fair comparison with ISO speed's impact on the correlation. The correlation heat maps in Fig.4 are computed by correlating the noise residuals from the images' green channel with the device's reference green channel PRNU. The reference PRNU is extracted from 50 flat-field images. The block size for the computation of the correlation at each pixel is $128\times 128$ pixels. We use yellow to show high correlation regions and blue to show the opposite. Apparently, as the ISO speed increases, the correlation map shows more regions with low correlation. It can be concluded that despite these images with complex image content have undergone post-processing, their correlation with the reference PRNU is still dependent on the image's ISO speed.

A correlation predictor is an important component of many PRNU-based tampering localization methods. Many PRNU-based tampering localization methods are applied by comparing the block-wise correlations with a decision threshold set according to the predicted correlation. As a result, the choice of the decision threshold and the performance of these methods can be greatly affected by the accuracy of the correlation prediction. As the correlation is content dependent, without considering the ISO speed, [13] models the correlation as a function of four image features, namely the intensity, texture, signal flattening and a texture-intensity combinative term. However, due to the correlation's dependency on the ISO speed, we postulate that: a correlation predictor can only produce accurate predictions for images with the same ISO speed as the training images. And we call such a correlation predictor as a matching ISO correlation predictor.

To show the ISO speed's influence on correlation predictor and validate our postulate, we first compare the performance of the correlation predictors trained with (a) images with mixed ISO speeds and (b) images with the same ISO speed as the test images. We did the test on 13 cameras from the Warwick Image Forensics Dataset (An Olympus EM10 MKII camera from the dataset doesn't show the existence of PRNU. Thus it is not included in this test). 50 flat-field images from each camera are used to extract the cameras' reference fingerprints. For each camera, we select images from three ISO speeds to form three test sets, namely ISO 100, 800, and 6400, apart from the two Panasonic LumixTZ90, which do not have ISO 6400. For these two cameras, we test on ISO 3200 images instead. Accordingly, we trained three matching ISO correlation predictors, each with 20 images of the corresponding ISO speed following the method from [13]. The correlations are computed between image blocks of $128\times 128$ pixels. To make the comparison, for each camera, we trained another correlation predictor with 20 images randomly selected from the 60 images used for the training of the camera's three matching ISO correlation predictors. We call this correlation predictor as a mixed ISO correlation predictor. Block-wise correlation predictions are made for the test sets. For each set, we computed the coefficient of determination ($r^{2}$) and the root mean square error (RMSE) for the matching ISO and mixed ISO correlation predictors as shown in Table I. We highlighted the better performance for each test set in terms of larger $r^{2}$ and smaller RMSE with bold font.

The matching ISO correlation predictors show superior performance over the mixed correlation predictors for all test sets except for the two Fujifilm XA-10 and the two Panasonic Lumix TZ90 at high ISO speeds. These two models of cameras are more prone to strong noise at high ISO speeds. As a result, the correlations with their reference PRNU become close to zero despite different image features. Due to the relatively large variance of the correlations introduced by the PRNU-irrelevant signal in the noise residuals, neither of the correlation predictors managed to produce large $r^{2}$ for the correlation predictions. However, by using the Matching ISO correlation predictor for these cameras, we notice small RMSE still can be observed. This is particularly important as the correlation predictors would not generate predictions that deviate too much from the actual correlation. False positives can be significantly reduced when we apply these correlation predictors for forgery detection.

In addition to the test on the Warwick Image Forensics Dataset, the experiments are extended to 9 cameras from the Dresden Image Dataset [29] as well. In the Dresden Image Dataset, about 150 images of natural scenes are produced by each camera. However, as the dataset was created without considering the ISO speed as an influential factor, the images' ISO speeds span over many different values. For most ISO speeds, the number of images available is not enough for us to train a matching ISO correlation predictor using the method mentioned above and to test it with the matching ISO images. So we test the matching ISO correlation predictor on the most popular ISO speed from each camera only, each with 20 test images. For each camera, we trained a matching ISO correlation predictor with 20 images of the same ISO speed as the test images and another 20 images are selected randomly from all the images available for the training of the mixed ISO correlation predictor. $r^{2}$ and RMSE of the predictions are shown in Table II. Again, the superior performance of the matching ISO correlation predictors can be observed in every case. Both the tests on images from Warwick Image Forensics Dataset and Dresden Image Dataset show that the performance of a correlation predictor may degenerate by completely ignoring the impact of ISO speed and trained images of mixed ISO speed.

Knowing that we cannot ignore the ISO speed in the correlation prediction training process, we also would like to investigate how mismatched ISO speeds of training and testing images would affect correlation prediction and subsequent forgery detection. In specific, we would like to investigate to what extent, a correlation predictor trained with images with a particular ISO speed can predict reliable correlation with images taken at other ISO speeds without significantly influencing the forgery detection results. We use Fig.5 to demonstrate the potential outcomes of forgery detection when the training image's ISO speed is significantly different from the test image's ISO speed.

Fig.5 shows the forgery detection results from tampered images with ISO speed 100, 800 and 6400 from a Canon M6. Images of the same scene taken at different ISO speeds are manipulated using Adobe Photoshop. For each image, the tampered region is replaced by using Photoshop's content-aware filling function, which leaves the tampered region at a similar noise level as its surrounding regions. We apply the Bayesian-MRF forgery detection algorithm from [15] to the images. For all the images, we set the same parameters for the forgery detection algorithm: with the interaction parameter $\beta$ set to $10$ and probability prior $p_{0}$ set to $0.01$. The detection results show that the forgery detection algorithm works the best in terms of false detections when it is equipped with the matching ISO correlation predictor. We also notice that when we use ISO 100 correlation predictor for the forgery detection of the ISO 6400 forgery, despite the tampered region is correctly identified, there are a lot of false positives in the result. And when ISO 6400 correlation predictor is used for the detection of forgery in ISO 100 forgery image, while the entire authentic region is regarded as tampered, there are parts of the tampered region still undetected.

To explain these observations, we have to consider the two potential outcomes of using images of different ISO speeds for the training of correlation predictors: the predicted correlation being either overestimated or underestimated.

Overestimation of the correlations (when correlation predictions are larger than the actual values) often occur when we use a correlation predictor trained with images of lower ISO speeds than the test image's ISO speed. As the actual intra-class correlations will be smaller than the predicted correlation, the corresponding pixels are more likely to be labeled as tampered, which results in an increased number of false detections as we have seen in Fig.5. This is particularly harmful to real-life forensics. For most forgery detection algorithms, the authenticity of a pixel is checked by comparing its actual correlation with a threshold set with reference to the predicted correlations and expected inter-class correlation, which is expected to be zero. Though the actual algorithms can be different with more complexity by considering the distribution of the correlations from both inter- and intra-class as well as neighboring pixels' correlations, the comparison of whether the actual correlation sits closer to the predicted correlation or inter-class correlation when the correlation is overestimated can be a good indicator of how likely false detections can be introduced by a correlation predictor. Thus we would like to compare the two values: $d_{1}=\rho-\bar{\rho}_{\text{inter}}$, which is the relative position from the inter-class correlation, $\bar{\rho}_{\text{inter}}$, to the actual computed correlation $\rho$ and $d_{2}=\bar{\rho}_{\text{intra}}-\rho$, which is the relative position of the actual correlation, $\rho$, to the predicted intra-class correlation, $\bar{\rho}_{\text{intra}}$. Instead of comparing the $L_{1}$ distances, we compare these two values to focus more on the situation when the correlation is overestimated, which causes the actual correlation to be a value between the expected inter-class correlation and predicted correlation. We estimate $\bar{\rho}_{\text{inter}}$ as zero and use the predicted correlation to estimate $\bar{\rho}_{\text{predict}}$, and it gives $d_{1}-d_{2}\approx 2\rho-\bar{\rho}_{\text{predict}}$. When $d_{1}-d_{2}$ is negative, it indicates that the correlation has a large chance of being misidentified as an inter-class correlation.

Again, use the camera Canon M6 as an example, we show the percentages of the image blocks with $d_{1}-d_{2}$ smaller than $0$ in Fig.6 when we use an ISO 100 and 800 correlation predictors to predict for test images with ISO speed number of stops above the training images. The plot shows that when the test images' ISO speeds are within the one-stop range of the training images' ISO speed, there is only a relatively small portion of blocks (i.e. less than $10\%$) with $d_{1}-d_{2}$ smaller than 0 for both ISO 100 and ISO 800 correlation predictors. As the deviation from the test images' ISO speed to the training images' ISO speed increases, we start to see a higher percentage from Fig.6, indicating an increased number of false detections could be introduced into forgery detection results. As we approximate $d_{1}-d_{2}$ as $2\rho-\bar{\rho}_{\text{predict}}$, it becomes an universal problem when $\rho<\frac{1}{2}\bar{\rho}_{\text{predict}}$.

Base on the correlation model derived from Equation (11) and observations from experiments, we found that for image blocks of the same scene from images taken at different ISO speeds, it is generally true that the block-wise correlation in an image taken with ISO speed $G_{1}$ is twice larger than the correlation of the corresponding block from an image taken at ISO speed $G_{2}=2G_{1}$. Thus, we claim that $G_{2}=2G_{1}$ is a safe choice to be set as the largest ISO speed a correlation predictor trained with images of ISO speed $G_{1}$ can reliably predict for. Similar behavior can be observed on other cameras as well and we show the receiver operating characteristic (ROC) curve for forgery detection in Fig.7 for further validation.

Each ROC curve in Fig.7 is plotted by running the Bayesian-Markov random field (MRF) based forgery detection algorithm from [15] on 80 synthetic forgery images at each of the 7 presented ISO speeds. Three correlation predictors, each trained with 20 natural images taken at ISO speed 100, 800 and 6400, respectively, are used to predict the correlations for the forged images. We vary the interaction parameter $\beta$ in the range of $[1,1200]$ and the probability prior $p_{0}$ between $[0,1]$ to set different combinations of the parameters for the algorithm. This allows us to generate the enveloping curves for the ROCs to show the best performance. The 80 synthetic forged images are generated from 20 full-sized authentic images. From each full-sized image, we select 4 regions of $1024\times 1024$ pixels. We replace the center of each $1024\times 1024$ pixel region's center with a tampered patch of $256\times 256$ pixels. The patch used to replace the center is cropped from the same original image but from a different position to ensure that it does not have the same PRNU. In fact, we fully facilitate the Warwick Image Forensics Dataset which provides images of the same content at different ISO speeds. This allows us to generate the synthetic forged images in the way that for one synthetic forged image at one ISO speed, we can find images of the same content at other ISO speeds as well. By doing this, Fig.7 not only allows us to compare the performance of different correlation predictors for forged images at one ISO speed but we can also systematically compare the performance of one correlation predictor for different ISO speeds.

We run the test on different cameras from Warwick Image Forensics Dataset. To save space, we only show the ROC curves of two most representative cameras, a Canon M6 and a Sigma SdQuattro in Fig.7. Canon M6 represents the cameras that can generate relatively less noisy images (with a large peak to noise ratio (PSNR)) for most ISO speeds from the camera while Sigma SdQuattro represents the cameras whose image quality is highly dependent on the selected ISO speed. The false positive rate (FPR) and true positive rate (TPR) are computed at the pixel-level. As for real-life tampering localization application, we usually require the method to produce a small FPR, thus we focus on the range of $[0,0.2]$ of FPR in the plots.

From Fig. 7, we first notice that for ISO 100, 800 and 6400 forgery images, the matching ISO correlation predictor works the best in both cameras in almost every case. The only exception is for Sigma SdQuattro ISO 6400 forgery images. In this case, despite the ISO 6400 correlation predictor can make predictions accurately as we have seen from Table I, none of the three correlation predictors can produce accurate detections. This is because, for high ISO images from this camera, the images' intra-class correlations are generally very close to zero and hard to be separated from inter-class correlations. For such images, PRNU-based methods may not be the best tool to perform forgery localization. However, the ISO specific correlation predictor can still be helpful in such a scenario as it will be able to accurately predict the correlations close to zero. Thus, the users can be warned that the PRNU based methods may not be suitable under such a scenario. Overall, the results show the benefit of using a matching ISO correlation predictor for forgery detection.

For both cameras, we observe that the detection results of using the ISO 100 correlation predictors (i.e. predictors trained with images taken at ISO speed 100) are better when the forged image's ISO speed is smaller than 400. While the Canon M6's relatively good high PSNR at higher ISO speeds allows the ISO 100 correlation predictor to perform reasonably well for a forged image with ISO speed up to 1600, it is not the case for the Sigma SdQuattro camera. From ISO 400 and above, the ISO 100 correlation predictor for the Sigma SdQuattro starts to struggle. And the similar effect can be observed for ISO 800 correlation predictors when they are used to predict for images with ISO speed much higher than 800. Thus, it conforms to our argument that a predictor trained with images taken at ISO speed $G_{1}$ can perform reliably on the images taken at an ISO speed $G_{2}$ that is lower than or equal to $2G_{1}$. While depending on the camera, some correlation predictors may perform when the test image's ISO speed is above the range, the above argument provides a safe range for the choice of correlation predictor's training ISO speed without risking too many false detections.

Fig.7 also shows the situation when the correlation predictors underestimate the test image's correlations. Underestimation often occurs when we use a correlation predictor trained with images of a much higher ISO speed than the test image's ISO speed. In the plots, we noticed that the ISO 6400 correlation predictors, especially for the Canon M6 camera, appear to have difficulty in correctly localizing the forgery for images with low ISO speed. This is because when the correlation predictor underestimates the correlations, it eventually reduces the forgery detection algorithm's capability of correctly identifying tampered pixels. Thus, to avoid the underestimation but still provides a practical range from which a training ISO speed can be conveniently selected, we empirically set the lower bound of the ISO speed a correlation predictor can be used for to half of the ISO speed of its training images. From the plots, we see by using this range, the corresponding detection results either outperform other correlation predictors or are on par with the best performance. Altogether, we conclude that for a test image taken at ISO speed $G_{1}$, using correlation predictors trained with images of ISO speed, $G_{2}$, which is in the one-stop range of $G_{1}$ ($G_{2}\in[G_{1}/2,2G_{1}]$) can produce forgery detection result without risking false detections being excessively introduced due to the correlation predictor.

Observing the ISO speed's impact on correlation prediction, we concluded that reliable correlation predictions should be made in an ISO specific way. Thus, we propose an ISO specific correlation prediction process. To predict correlations for an image of ISO speed $G_{1}$, we have to use a correlation predictor, preferably trained with images of the same ISO speed at $G_{1}$, or similar to $G_{1}$. An ISO speed $G_{2}$ is considered as similar to $G_{1}$ if $G_{2}$ is in the one-stop range of $G_{1}$. The images used for the training of the correlation predictor should cover diverse image feature settings: including both bright and dark scenes, highly textured and flat patterns, etc. To cover such a diverse set of image features, it usually requires a large number of images. Thus, a good correlation predictor should be trained with no less than 20 full-sized images. With a relatively large collection of images of good feature diversity taken at an ISO speed similar to the test image, the weight for each defined feature can be learned following the process presented in [13] for the correlation predictor.

In order to complete the correlation prediction process, we need to have the knowledge of the ISO speed $G_{1}$ to find images of the same or similar ISO speeds to form the training set. However, as the image in question may have undergone some unknown manipulations, either on its image content or metadata, the ISO speed information presented in the metadata can be unreliable or even unavailable. Thus, we can often face the problem when we have an image of unknown ISO speed and we would like to select images with the closest ISO speed to the image to train a correlation predictor.

As a known factor, for the same camera, the higher the ISO speed is, the higher the level of noise is introduced to the content of images. Thus, it is intuitive to infer an image's ISO speed by exploiting its noise characteristics in the content. Based on the Poissonian-Gaussian noise model [25], methods are proposed in [26, 27, 30] to infer the camera gain, $g$, from a RAW image, which then can be directly related to the camera's ISO speed. Despite these methods showing promising performance on RAW images, as the noise model generally cannot be applied directly to non-RAW image formats, their performance is suboptimal and cannot be practically used to infer a JPEG image's ISO speed. Furthermore, for similar reasons, though many noise level estimation algorithms [31, 32, 33, 34] may work well on RAW images to give clues about an image's ISO speed, JPEG images still pose challenges. As JPEG is one of the most common image formats, being able to identify a JPEG image's ISO speed is a prerequisite for ISO specific correlation prediction.

Though finding an accurate noise model for a JPEG image can be of great complexity, we can simplify this problem by making the following assumption: image patches from the same camera with similar content and JPEG quality factor should show similar noise characteristics if they are of the same ISO speed, and vice versa as shown in Fig.8. Thus, we propose a method called Content-based Inference of ISO Speed (CINFISOS, pronounced as /’[email protected]@s/) to determine an image's ISO speed by doing patch-wise noise comparison with patches of similar content from images taken with the same camera at different ISO speeds.

Consider the case when we have a query image, $\mathcal{Q}$, and $t$ candidate training sets, $\mathcal{S}=\{S_{1},...,S_{t}\}$, each consists of multiple images and the sets are with different ISO speeds. We would like to find the set with the ISO speed closest to the query image $\mathcal{Q}$. The query image is first partitioned into a set of non-overlapping patches, $\mathcal{P}=\{p_{i}\}$, each patch of size $d\times d$ pixels. As we would like to use the patches to best represent the image's noise characteristics, patches with too many dark and saturated pixels in any color channel should be removed. We consider the patches in the RGB color space. For each pixel $q$ in the $j$th channel of the patch, $p_{i}^{j}$, the pixel is considered as dark or saturated if its pixel value $I(q)$ is not in the range $[\lambda_{1},\lambda_{2}]$:

The $i$th patch is excluded from $\hat{\mathcal{P}}$ if $\forall j(\sum\limits_{q\in p_{i}^{j}}U(q)>\lambda_{\tau}d^{2})$, when the ratio of the dark or saturated pixels in every channel of the patch is over a limit $\lambda_{\tau}$. In addition to removing the dark and saturated pixels, the image's noise characteristics can be better revealed by including only the less textured patches. Thus, we only keep $m$ least textured patches in $\mathcal{P}_{\mathcal{Q}}$, the set of patches that we believe can best represent the query image's noise characteristics. To evaluate how textured a patch is, we use the texture feature definition from [13] but extends its definition to patches of three color channels by a simple summation:

where $\textbf{F}()$ is the high-pass filter and $\text{var}_{5}()$ measures the variance of $5\times 5$ neighbourhood. The feature $f_{T}$ is defined in the range $[0,1]$ with lower values for more texture patches. We select $m$ least textured patches from $\hat{\mathcal{P}}$ to form the set of qualified query image patches $\mathcal{P}_{\mathcal{Q}}$:

$f_{T}{{}_{m+1}}$ is the texture feature of the $m+1$th least textured patch from $\hat{\mathcal{P}}$. As $\mathcal{P}_{\mathcal{Q}}$ only contains patches with relatively smooth texture, we can approximate their image content by applying a low pass filter. We implement the method of finding patches with similar content using a block-matching method similar to [35]. The distance between two patches in each color channel is measured as the Euclidean distance between the discrete cosine transforms (DCT) of the two with hard thresholding applied. And the overall distance between two patches is the summation of the distances in the three color channels:

where $\Gamma(x,\lambda_{\text{DCT}})$ is the hard thresholding operation:

For each patch $p_{i}$ in $\mathcal{P}_{\mathcal{Q}}$, from each candidate training set $S_{k}$, $n$ patches with the least distance to $p_{i}$ will be selected. Though the exhaustive search for the patches with the shortest distance is computationally expensive, this step can be easily parallelized. We call this set of selected patches as $\mathcal{P}_{k}^{i}$. We define the distance, which measures the sum of the absolute differences in noise characteristics in all three color channels from each patch $p_{i}$ in $\mathcal{P}_{\mathcal{Q}}$ to each candidate training image set $S_{k}$, as:

where $\tilde{p}_{l}^{j}$ is the low-pass filtered version of the patch $p_{l}$ of the $j$th channel:

For each patch $p_{i}$ in $\mathcal{P}_{\mathcal{Q}}$, it will have a vote for a candidate training set, $S_{k}$, who has the smallest $D(p_{i},S_{k})$. The candidate training set with the closest ISO speed to the query image will be determined by a simple majority vote from all the patches in $\mathcal{P}_{\mathcal{Q}}$. The ISO speed that receives the majority votes will be deemed as the ISO speed of the query image and the correlation predictor can be trained with the corresponding images.

In this work, we did both analytical and empirical studies on the impact of different camera sensitivity (ISO speed) settings on PRNU-based digital forensics. First, we show how the correlation between an image's noise residual with the device's reference PRNU can be dependent on the image's ISO speed. With this dependency in mind, we empirically show how mismatched ISO speeds may influence the correlation prediction process. Thus, we proposed an ISO-specific correlation prediction process to be used in PRNU-based forgery detection. To address the problem that the information about the ISO speed of an image may not be available, a method called Content-based Inference of ISO Speed (CINFISOS) is proposed to infer the image's ISO speed from its content. Clear improvements are observed in correlation predictions and forgery detection results by applying our proposed ISO specific correlation prediction process with CINFISOS. By pointing out the influence of camera sensitivity setting on PRNU-based forensic methods, the provided solutions from this work can make the forensic analysis more reliable and trustworthy.

This work is supported by the EU Horizon 2020 Marie Sklodowska-Curie Actions through the project entitled Computer Vision Enabled Multimedia Forensics and People Identification (Project No. 690907, Acronym: IDENTITY)