Shaking the Fake: Detecting Deepfake Videos in Real Time via Active Probes

Deepfake attack model. The scenarios of deepfake attacks mainly involve video communication where the participants use digital video and audio to communicate. Generally speaking, deepfake attacks focuses on replacing the attacker’s faces with that of a legitimate participant, aiming to deceive other participants during video communication.

Defense goals. SFake should detect the deepfake-generate videos in different scenarios as accurately as possible. Besides that, The detection process should be resource-saving as it is designed for the mobile device.

Defender’s capability. Like other facial recognition functionality, SFake requires the user to remain as still as possible for a short period (e.g., 4 s) for detection. SFake requires cameras with resolutions of 1920*1080 pixels or higher, and they must support at least 2x zoom capabilities. If a phone’s front camera does not meet these requirements, the rear camera can be used as an alternative. SFake should to some extent be able to control the hardware, such as zooming out the camera and playing the “vibration” sound effect during the detection process.

In this part, we briefly illustrate the basic principle of the face swap algorithm (FSA) and the limitations of existing detection methods. The main steps of the FSA are summarized in Figure 2. The FSA detects the face area of the target face and extracts several tens of facial landmarks. Based on the facial landmarks, the FSA generates the fake face based on the source face and blends it with the the background of the real image, thus get the result face. To generate a fake video, FSA applies the above steps to each frame of the video and merge the generated fake images into a fake video.

However, due to the imperfection of the current deepfake algorithms, the forged video contains unnatural traces caused by face swap, which can serve as a feature for distinguishing between real and fake videos. The general idea of existing detection methods is to extract these features passively. The typical features include the unregular artifacts on the fake face caused by the face synthesis step[36, 64, 17, 34, 10, 13], the blending traces at the splices of the fake face and the head at the blending step [40, 59, 27, 61, 44], the fingerprints of the corresponding generative network that is inherently contained in the fake video [74, 43], the temporal mismatch between the physiological signal and footage [79, 73, 7, 14, 32], and so on. After selecting the specific features that can represent the differences between real and fake videos, a network is trained based on the well-established datasets [78, 24, 29, 18, 58] to detect the subtle differences mentioned above. However, the existing detection methods lack robustness due to the diversity of deepfake algorithms, their rapid development, and the variety of usage scenarios. G. Pei et al. [53] compiles statistics from 14 papers on deepfake detection published in recent years, showcasing the results of cross-validation performed on datasets like DFDC [18], Celeb-DF, [39], Celeb-DFv2 [38], and DeeperForensics-1.0 [29]. On the DFDC dataset, which has a larger size and more deepfake algorithms incorporated, the best accuracy among the 14 detection methods achieved 90.3$\%$, while the lowest accuracy was 67.44$\%$, which shows there is still room for improvement in the generalizability of current work.

To further illustrate the unstable performance of the existing detection methods, we show four cases where the detection systems fail to determine the authenticity of the face in Figure 3. We input four pairs of images/videos into four models: SBI [59], Face artifacts [37], CNN detection [65] and LRNet [62]. In these four cases, the models incorrectly assign a higher fakeness score to the real images/videos than to the fake ones. This might be because we use an online face swap tool [1] to generate fake content not contained in the mainstream deepfake datasets and unseen to the detection model. The above examples show that the passively extracted features are uncontrollable and change with different deepfake algorithms, which are readily manipulable to attackers.

As passively extracting features depends much on the attacker’s strategies and has limited robustness, we think about how to actively introduce the features that are controllable to the defender and insensitive to alteration of the attacker’s strategies. The very straightforward idea is to shake the attacker’s smartphone intensively, which blurs the captured video. As the source face remains unchanged, the different sharpness of the source face and target face is likely to cause inconsistency on the result face, no matter what algorithm the attacker uses. As shown in Figure 4, the circled part of the fake picture on the right is sharper than other parts, which can even be observed by raw eyes, not to mention the well-designed detection algorithm. Therefore, the shake-caused blur can act as a feature to detect the fake faces. However, the defender cannot shake the attacker’s smartphone as they are typically not in the same space. Therefore, we conduct a comprehensive study of the smartphone’s structure to find methods that can make smartphones move controllably even far away from it. Finally, we find the “vibration” sound effect meets our requirement, because it not only generates mechanical vibrations sufficiently large but also is remotely controllable.

“Vibration” sound effect. “Vibration”, a widely used sound effect of the smartphone, is triggered by a built-in motor. Traditionally, with an off-centered weight attached, the center of gravity moves as the motor spins. Recently, linear motors which utilizes the electromagnetic effect as drive source are also commonly used in smartphone applications. Regardless of their running principles, the motor inside the smartphone can be controlled by the software, so the period and duty cycle of the vibration can be customized, enabling different vibration patterns for various applications.

Refocusing of the camera. In order to keep the captured image clear, the camera needs to adjust the position of the optical system in real time, so that the imaging position of the object falls on the sensor pixel array. This process is called focusing. When the relative position between the camera and the object changes, the camera needs to refocus to make the object clear. The image will become blurred until the refocusing is completed, which is potential to serve as the criteria for determining whether the smartphone has moved.

To remotely vibrate the smartphone, we are initially inspired by the idea in SideEye attack²²2In the SideEye attack, the attacker exploits the smartphone’s camera to infer the sound played by a speaker located on the same surface as the device.[41] of using the smartphone’s speaker to generate a sound wave that shakes the camera. However, based on our experiments, the phone’s volume is too low to cause sufficient vibration. We collect the acceleration data of the smartphone by its inertial measurement unit (IMU) while it is playing a piece of music at its maximum volume, and the result in Figure 5(a) shows that the sound wave hardly introduces movement to the smartphone. Therefore, we conduct a comprehensive study and attempt to identify methods that can substantially vibrate the smartphone. We finally turn to the “vibration” sound effect. Unlike other sound effects where the smartphone’s movement is merely a byproduct, the “vibration” sound effect is generated by a motor specially designed for mechanical movement. As a result, the “vibration” sound effect causes a much greater amplitude of movement, as shown in Figure 5(b). To more quantitatively describe the motion of the smartphone, we calculate the displacement of the smartphone in the z-axis (as the movement primarily occurs along the z-axis) according to Eq. 1

where $V_{i}$ and $D_{i}$ represent the speed and displacement of the smartphone along the z-axis at the $i^{th}$ moment respectively. Figure 5(c) shows the results. The displacement curve has a similar trend to the acceleration curve and reaches its maximum of about 1.5 $\mu$m.

In addition, vibration-induced movements are easier to control, thus enriching the feature patterns and increasing the randomness of detection. For example, the detector can randomize the period and duty cycle of vibration before each detection. As a result, even if an attacker knows the detection method in advance, it would be challenging to determine the specific detection parameters, making it difficult to bypass the defense.

The structure of the camera and the blurriness under ideal conditions. In this part, we briefly introduce the fundamental structure of the imaging system and illustrate the theoretically minimum blurriness. An imaging system is simplified to consist of an object, a lens, and a photosensitive sensor array, as shown in Figure 6. All light rays originating from one point of the object passing through the lens converge to the image point. To get a sharp image, the sensor array should adjust its position along the optical axis until it contains the image point. Under ideal conditions, one object point maps precisely to one image point, shown as the black point on the right side of Figure 6. However, when the object point vibrates, the image point also vibrates, making light rays not focus ideally on the sensor array. The imaging result is no longer an ideal geometric point but becomes a circle with a physical size in space, termed the “circle of confusion” (CoC), shown as the blue and green circles in Figure 6. The size of the CoC represents the degree of the image’s blurriness.

Even with the smartphone being stationary and focused, an ideal camera exhibits inherent blurriness due to several factors. First, the sensor array’s pixel size limits the imaging resolution. Even if the light rays focus ideally on the sensor array, the image is not a perfect point but a rectangle with the size of a pixel. Second, due to the diffraction of light, an object point is naturally imaged to a circle known as the Airy disk [52], whose size theoretically limits the upper bound of imaging resolution with a particular aperture [69]. We take the camera on Xiaomi Redmi 10X [20] as an example to calculate the ideal resolution. First, the size of each pixel is 0.8 $\mu$m, as referred to in the specifications. Second, according to Rayleigh criterion[69], the diameter of the Airy disk is 0.87 $\mu$m with the aperture of f/1.8. Therefore, the smallest CoC diameter of the camera on Xiaomi Redmi 10X is limited by its sensor arrays, which is 0.8 $\mu$m.

The theoretical analysis of blurriness induced by vibration. As illustrated in Section 2, the movement along the z-axis leads the camera to refocus and introduce blur on the video. In this part, We further explore the quantitative relationship between the movement of the smartphone and the degree of blurriness represented by the CoC’s radius. We assume the object distance as $u$, the image distance as $w$, the focus length as $f$, and the aperture as $F$ (then the size of the aperture equals $\frac{f}{F}$ by definition) and the diameter of the CoC as $r$. According to the thin lens equation

We first consider the situation in which the object point moves towards the lens, represented as the blue lines and blue circle in Figure 6. According to the geometric properties of similar triangles

We substitute Eq. 2 to Eq. 3, and obtain the equation set

Solving the equation set, we get the expression of $L_{2}$, the radius of CoC caused by the object point moving towards the lens

By analogy, when the object point moves away from the lens, the radius of CoC can be represented by the expression

To illustrate the impact of vibration on the degree of image blurriness, we take the Xiaomi Redmi 10X smartphone as an example where $f$ is 26 mm, and $F$ is $\frac{f}{1.8}$. According to Eq. 1, a typic value of the maximum vibration amplitude is 1.5 $\mu$m. We assume the distance $u$ between the lens and the object is 10 cm, and then $u_{1}$ and $u_{2}$ are respectively $10cm+1.5\mu m$ and $10cm-1.5\mu m$. According to Eq. 6 and Eq. 5, $L1$ and $L2$ are approximately equal to 0.095 $\mu$m, which represents the blurriness caused by vibration. Compared to the inherent error of 0.8um in an ideal camera, the error caused by vibration is about one-tenth of it. That means the light rays initially contained within a single pixel’s sensor now partially affect the surrounding pixel, leading to mutual interference between the pixels. Since the edges of color blocks experience the most significant color changes, the blurriness caused by vibration is primarily manifested in the slowing of color transitions at the edges and the widening of the edges. When the camera vibrates rapidly along with the smartphone, the camera does not have enough time to complete the focusing process, resulting in a continuous defocus state, leading to sustained blurriness in the video.

The experimental verification of blurriness induced by vibration. To validate the above theoretical hypothesis, We record an 8-second video with the vibration period of 2 seconds and the duty cycle of 0.3. We observe the vibration of the subject’s collar in the video as an example. The result is shown in Figure 7. When the smartphone does not vibrate, the collar area is sharp and the edge between the skin and the clothing is distinct. When the smartphone vibrates, the skin and clothing are blended along the edge of the two parts. We calculate the gradient of two images, which can better represent the distribution of edges. The result shows that gradients of the vibrating image are more widely distributed in space, whereas they are more concentrated in the non-vibrating images.

Therefore, when the camera defocuses, the edges of the imaging will be less sharp, resulting in a more gradual variation in pixel values. We can use variance to measure the degree of variation of an image, which is calculated by Eq. 7

Here, $\sigma^{2}$ represents the variance, N is the number of pixels, $x_{i}$ represents the value of each pixel and $\mu$ is the mean of pixel values. We calculate the variance of the gradient of the collar area in Figure 7 frame by frame in the 8-second video and collect acceleration data from the IMU during video recording. The relationship between IMU readings and video variance is shown in Figure 8. When the smartphone vibrates, the variance of the images decreases, which means the footage gets blurred. The experiment shows that vibration blurs the video, which can be detected by variance sequence.

How deepfake algorithm deals with blurriness. FSA exhibits a certain adaptability for blurry images. For example, when it performs a face swap on a blurry target image, the result image is also quite blurry, as shown in Figure 9(a). However, the vibration caused blurriness is too small to be recognized by FSA. To prove this conclusion, we select a facial area of the first frame of the 8-second video and use Gaussian filtering to blur it, simulating the effect of vibration on the video. To choose the proper size of the Gaussian kernel, we compare the variance of the video with and without vibration. we find that when the kernel size is between 1 and 3, the blurriness effect of Gaussian filtering is similar to the vibration. Therefore, we Gaussian filter the same sample image 100 times with random size of kernel and apply the FSA algorithm to generate 100 fake faces. The variance of real and fake faces are respectively shown as blue and green curves in Figure 9(b). The trends of the green and blue curves are not consistent, which indicates that FSA cannot recognize the slight variation of blurriness caused by vibration.

The above analysis leads to a conclusion: for the real face, artificially controlled vibrations cause a regular blurriness which can be reflected by the variance. However, the FSA can not precisely determine the vibration-caused blurriness of the target image and the fake face does not exhibit regular blurriness. Therefore, vibration-caused blurriness can serve as the feature that differentiates the real and fake videos.

Selection of vibration patterns. SFake can set the period and duty cycle of the vibration to configure its pattern. Theoretically, the variance sequence should be directly proportional to the vibration pattern. However, the sensitivity of the variance sequence to vibration is limited. For example, if the duty cycle of vibration changes from 0.50 to 0.51, the variance sequence may not change so much. To quantitatively explore the relationship between the vibration patterns and variance sequence, we set the vibration period from 1 to 5 seconds with a step size 1 second and the duty cycle from 0 to 1 with a step size 0.05. For each vibration pattern, we record a video, calculate the variance sequence frame by frame, and utilize the middle value of the variance sequence as a threshold to calculate its duty cycle. The experiment result is shown in Figure 11. In general, the variance sequence can reflect the pattern of vibration. As for period, Figure 11(b) shows that under different duty cycles, the period of variance sequence equals that of vibration. As for the duty cycle, the overall trends of variance sequence reflect the duty cycle of the vibration, but the details are not entirely accurate. For example, when the period of vibration is 1s or 2s, the duty cycle of variance decreases with that of vibration increasing from 0.45 to 0.55, as shown in Figure 11(a). Therefore, we coarsely set three options for the duty cycle: 0.2, 0.5, and 0.8. The blurriness of the videos corresponding to these three settings exhibits significant differences, ensuring that the variance sequence can reflect the vibration patterns. Additionally, since the variance sequence almost aligns with the vibration periods, we do not set specific options for the period to increase the diversity of vibration patterns.

From an implementation perspective, the existing mobile framework provides feasibility for controllable vibration. Manual vibrations can be triggered using the Vibrator class’s VibrationEffect for more nuanced patterns.

Camera related preparation As analyzed in Section 3.2, inducing blur through the smartphone’s vibration relies on the camera being in focus. Therefore, before initiating the vibration process, we need to make the camera focus on the face. We use a face recognition algorithm provided by Dlib [31], a widely used machine learning toolkit, to detect the facial area in the captured video. After that, we operate the camera and focus on the face. According to Eq. 6 and Eq. 5, the blurriness caused by vibration is related to the camera’s focal length. As the focal length increases, the radius of the CoC also increases, resulting in more severe blur. However, as the focal length increases, the field of view (FOV) decreases, which means the camera may not be able to capture all areas of the face. Therefore, we need to adjust the focus length to make it as large as possible while still being able to capture the entire facial area. From experience, the focus length can be set around 50mm, effectively doubling the images, which can induce sufficient vibration while capturing the entire facial area.

From an implement perspective, in Android environment, the focus distance can be adjusted by setting LensFocusDistance in the CaptureRequest.Builder. After that, the application can choose the focus area by setting the value of MeteringRectangle and applying it to the class CaptureRequest.Builder.

The fundamental idea of SFake is to determine the authenticity of a video by comparing the changes in video blurriness and vibration patterns. To accurately measure the blurriness, we calculate the gradient of the image. However, to conserve computational resources, we only select several representive areas to perform gradient calculation.

Gradient calculation. As previously analyzed, the blurriness caused by vibration primarily manifests at the edges of color blocks in the footage. In previous discussions, we directly use the variance of the image as a measure to assess the level of blurriness for further analysis. This method is applicable under good shooting conditions because the distribution of pixel values inside the color blocks is relatively uniform with slight variation in such cases. However, when the shooting conditions are poor, noise may occur within the color blocks due to lighting fluctuations, sensor noise, etc. The noise is widespread throughout the entire image and constantly changes over time, thereby affecting the value of the variance, making the variance not solely affected by the vibration.

To eliminate the influence of noise, we apply gradient processing to the image and remove areas with minimal gradient values to prevent the noise within the color blocks from interfering the variance calculation. we calculate the gradient of the image by Eq. 8

where $f[i,j]$ refers to the pixel value at $i^{t}h$ row and $j^{t}h$ column, while $g[i,j]$ denotes the grayscale value at the same coordinate. After that, we zero out the gradients smaller than one-tenth of the maximum gradient value. Figure 12(a) shows the raw and gradient image. To demonstrate the effect of the gradient calculation, we record a 4-second video with the vibration pattern of 1 second period and 0.5 duty cycle. We calculate the variance sequence frame by frame, either without gradient processing or after applying it, which is shown in Figure 12. The variance sequence obtained after gradient processing has less noise and can better reflect the vibration patterns.

Landmarks detection. Although gradient calculation helps extract the blurriness information, we do not perform it accross the entire image for two reasons. First, computing the gradient requires a lot of computational resources. Second, the entire image may contain some vibration-unrelated changes such as variations of the video’s background or alterations of facial expression like blinking and frowning. Therefore, we need to select some areas representing the whole video for subsequent processing. In the previous section, we identify these areas through manual or random selection. In this part, we describe the method for automatically selecting these areas.

Vibration caused blurriness are more likely to occur in edges which are more likely to appear in the face features. Therefore, we use landmark detection algorithm to extract these areas. We make use the Dlib [31] for facial landmark detection. The detection algorithm marks 68 landmarks on the face, which are distributed in a fixed pattern; for instance, points 49 to 68 outline the contour of the mouse, while points 18 to 28 locate the eyebrows. The example of the landmarks is shown in Figure 13(a). As the grayscale value represents the number of edges in this area, which indicates the extent to which vibration affects blurriness, we then calculate the average gradient value within the scope of these rectangles. To avoid the influence of expressions such as blinking and frowning on variance calculation, we exclude the landmarks of eyes and eyebrows from the selection range, and the top n areas with the highest average gradient values in the remaining rectangles are selected to calculate the blurriness degree, where n is a configurable parameter. As the smartphone stays still during detection process, we use the video’s first frame to select the areas and keeps their postion in the whole video.

We demonstrate the effect of selecting areas by landmarks with an experiment. First, in the 4-second video, we randomly pick a 50x50 area for variance sequence calculation. Then, following the algorithm described above, we select one area for variance sequence calculation. The result is shown in Figure 13. Our selected area reflects the vibration pattern better.

Although at most times, the variance sequence of the selected area can reflect the vibration patterns, it can also be affected by other factors, such as lighting changes and subtle body movements. We capture a 4-second video indoors, deliberately opening the curtains slowly during the recording to introduce subtle lighting variations. The vibration pattern is set with a period of 1 second and a duty cycle of 0.5. The variance sequence of the selected area is shown as the blue waveform in Figure 14(a). The sequence does not reflect the vibration pattern clearly because the lighting variations (and other environmental factors) also influence the variance calculation.

To extract vibration-related components from the variance sequence, we analyze the frequency domain of the ideal variance sequence, which is a square wave with a period of 1 second and a duty cycle of 0.5. In theory, the actual variance sequence can be considered as the cumulative effect of various factors influencing the blurriness of the video, whereas the ideal variance sequence can be viewed as the isolated impact of vibration on the blurriness. Therefore, to extract the feature reflecting the vibration patterns, we combine the information of both ideal and actual variance sequences. We perform a Fourier transform on the ideal variance sequence to obtain its frequency spectrum and sort the top 80$\%$ of its non-zero frequency components as the effective ones. We filter out all other frequency components of the actual variance sequence, and the filtered sequence is shown as the orange waveform in Figure 14(a). The sequence drops periodically with a cycle of 30 frames and a duty cycle of about 0.5, which is consistent with the vibration pattern. This way, we have completed the feature extraction, representing the feature as a vector of length 120.

To further demonstrate the effect of the extracted features, we generate a fake video based on the recorded 4-second video. We then apply the same processing to the fake video, with the results illustrated in Figure 14(b). Neither the raw variance sequence nor the filtered one has reflected the vibration pattern. This indicates that our features can differentiate between real and fake videos.

Dataset. Considering the absence of available deepfake datasets including physical probe mechanisms, we use 8 different brands of smartphones to record 15 participants of varying genders and ages to build our own dataset. We place the smartphone on the phone holder 20 cm away from the participant and zoom in twice, aiming at the participant’s face to encompass all his facial features while vibrating the smartphone in different patterns. For phones whose front cameras cannot zoom, we use the rear cameras as a substitute. We record 150 long videos, each 20 seconds in duration. By default, we assume the detection period lasts 4 seconds. We trim 10 clips of 4 seconds long from one long video by randomizing the start time. Therefore, we get a total of 1500 real clips, each 4 seconds long.

Based on the real videos, we use five different deepfake algorithms to generate fake videos: Hififace [67] and FSGANV2 [51] which represent the new deepfake algorithm proposed in the academic community, DeepFaceLive [28] and RemakerAI [1] which represent the widely used online face swap applications, and MobileFaceSwap [71] which represents the lightweight deepfake algorithm specifically designed for the mobile devices. For each deepfake algorithm, we generate 1500 fake videos, each corresponding to a real video.

Ethical Considerations. We prioritize societal security and ethical concerns. All participants comply with approved IRB protocols, ensuring participant awareness of the usage of their images. Additionally, the deepfake samples created for the study are not utilized beyond its scope and securely discarded post-research.

Object of comparison. We test the performance of 5 existing detection methods from the academic community (SBI [59], FaceAF [37], CnnDetect [65], LRNet [62], DFHob [11]), including two of them specifically designed to be lightweight networks [11] [62]. Additionally, we test Deepaware [16], an online deepfake website, considering its widespread use due to its simplicity and being free of charge.

Classifier design. We employ a simple two-layer neural network as our classifier with dimensions 120x30 for the hidden layer and 30x1 for the output layer. We utilize ReLU as the activation function and set the learning rate as 0.01. We randomly select 1000 real videos and 1000 fake videos generated solely by DeepFaceLive. For each video, we select 3 areas as the training data. The network converges after 40 epochs. When testing, we also select three areas for every video and determine the authenticity by the majority classfication results among the three areas. It is worth noting that there are possibly better choices for the classifier design, and we will explore the impact of different classifiers on the detection task later.

Evaluation metrics and devices. We use NVIDIA RTX 3060 to build the dataset and implement our detection method. The detection methods run in Ubuntu 22.04.2 LTS. We record the videos using the Xiaomi Redmi 10x, Xiaomi Redmi K50, OPPO Find x6, Huawei Nova9, Xiaomi 14 Ultra, Honor 20, Google Pixel 6a, and Huawei P60. We use Pytorch to reproduce existing detection methods. We utilize standard metrics for assessing the detection methods: the Area Under the Receiver Operating Characteristic Curve (AUC) and Accuracy (ACC).

In this part, we compare SFake with other detection models on the dataset to evaluate its capability to classify real and fake videos. Additionally, we measured the computational speed and overhead of various detection methods, demonstrating the efficiency advantages of SFake.

Considering the various scenarios of video communications in real life, we explore the relationship between environmental factors and the distinctness of the feature. We launch several experiments to illustrate the relationship between the performance of SFake and several factors such as lighting conditions, camera resolution, shooting distance, time consumed during the detection process, zoom factor, and classifiers.

Metrics and setup. To measure the distinctness of features, we define the Proportion of Sequence (POS), which means the ratio of the ideal frequency components to all the frequency components in the actual variance sequence. The ideal frequency components are defined as the non-zero frequency bands of the ideal variance sequence. The mathematical expression for POS is

Where $f_{ivs}$ and $f_{avs}$ represent the discrete frequency spectrum of the ideal and the actual variance sequence, respectively. A higher POS value suggests that the vibration more significantly influences the variance sequence. We record a 40-second video for each environmental circumstance and generate the corresponding fake video by RemarkerAI as it presents the most significant challenge for detection models according to our experiment results. Subsequently, we trim 100 video clips of four seconds each from the real and fake videos, choosing the start times randomly. We use POS of real and fake videos and the detection accuracy of the 200 clips as the metrics to determine the distinctness of the feature and the performance of the SFake. Unless otherwise specified, all other experimental conditions remain unchanged as illustrated in Section 5.1.

In our previous experiments, we place the phone on a phone holder to prevent hand movement and affect the footage. However, in most cases, users hold the phone to complete the detection process. With the natural tremor of the hands, the smartphone slightly but rapidly moves parallel to the plane of the phone, thereby impacting the recognition by SFake. We record ten videos of 40 seconds with a hand holding the smartphone. The vibration period is 1 second, and the duty cycle is 0.5. We slice it into one hundred 4-second clips. We randomly select one clip to calculate the raw and filtered variance sequence without other processes. The waveform reflects the vibration patterns but not so clearly, as shown in Figure 18(a). The POS of the real video decreases to 0.25, and the accuracy decreases to 58$\%$, significantly impacting performance. To solve this problem and maintain the performance, we provide three potential solutions.

Expand size of selected area. We set the default selected area size to 50 in our previous experiments, which is very small and can accurately capture the blurriness caused by vibrations when the footage is still. However, due to its small size, even slight movements can cause significant changes in the pixel values within the selected area. When the footage moves naturally with the person’s hands, the variance is affected not only by the phone’s vibration but also by the variations of image content in the selected area. To address this issue, we increase the selected area size to 300. This reduces the proportion of content within the selected area that changes due to hand movement. Experimental results indicate that by expanding the selected area, we have increased the POS value to 0.39 and achieved an accuracy rate of 87$\%$.

Move the selected area by IMU data. The relative movement between the camera and the face impacts the variance sequence calculations. If we can calculate how much distance in pixels the whole image has shifted due to the camera movement, we can then move the selected area in the same direction and distance to frame the same part of the footage. Using the principle of similar triangles, we estimate the number of pixels by which the image moved as Eq. 10

Where $p$ is the number of pixels the image moves, $d$ is the distance the camera moves, $D$ is the object distance, and $f$ is the focus length. We assume the object distance is 20 cm. We measure the distance of smartphone movement by IMU data according to Eq. 1, check the datasheet for focus length, and calculate the number of pixels the whole footage moved. Subsequently, we move the selected area with the same number of pixels in the same direction. The result shows that we marginally improve the performance of SFake. We increase the POS to 0.32 and the accuracy of detection to 65$\%$. We believe that this is because of inaccuracy in estimating the distance from the face to the camera, and the IMU errors also limit performance improvement.

Move the selected area by centroid of landmarks. We perform landmark detection frame by frame on the video. As the positions of facial features may change due to expressions such as blinking or frowning, whereas the contour of a person’s face tends to be more fixed, we characterize the position of the scene by calculating the centroid of landmarks along the facial contour. We use the offset of the centroid position to represent the overall shift of the smartphone within the frame. We accordingly adjust the position of the selected area so that each frame’s selected area gets the same content. The result shows we improve the POS to 0.45 and the accuracy to 92$\%$.