Deepfake Representation with Multilinear Regression

A data tensor ${\boldsymbol{\mathcal{D}}}\in{\rm I\!R}^{I_{1}\times I_{2}\times\dots\times I_{M}}$ is a multi-way array. In fact, when an array has three or more than three dimensions, we call it a tensor. The dimensions of a tensor are usually referred to as modes.

In linear algebra, we factorize a matrix ${\bf D}\in{\rm I\!R}^{I_{1}\times I_{2}}$ using Singular Value Decomposition (SVD) as follows:

where the columns of ${\bf U}\in{\rm I\!R}^{I_{1}\times r}$ and ${\bf V}\in{\rm I\!R}^{I_{2}\times r}$ are orthonormal and ${\bf\Sigma\in{\rm I\!R}^{r\times r}}$ is a diagonal matrix with positive real entries know as singular values. The rank $R$, SVD decomposition represents a matrix as following equation:

Rewriting equation 1 in conventional linear algebra, the Principal Components Analysis (PCA) is:

TMode-$m$ matrixizing of tensor ${\boldsymbol{\mathcal{D}}}\in{\rm I\!R}^{I_{1}\times I_{2}\times\dots\times I_{M}}$ is defined as the matrix ${{\bf D}}_{\mbox{\tiny\rm$[m]$}}\in{\rm I\!R}^{I_{m}\times(I_{1}\dots I_{m-1}I_{m+1}\dots I_{M})}$ where the parenthetical ordering indicates that column vectors are ordered by sweeping indices of all other modes through their ranges (Vasilescu09, ). Therefore:

A $3$-mode tensor may be metricized in three different ways by stacking first, second and third mode slices which are illustrated in Figure.2.

The mode-$m$ product (Vasilescu09, ; Carroll70, ; Delathauwer00a, ) of a tensor ${\boldsymbol{\mathcal{D}}}\in{\rm I\!R}^{I_{1}\times I_{2}\times\dots I_{m}\times\dots\times I_{M}}$ and matrix ${\bf A}\in{\rm I\!R}^{J_{m}\times I_{m}}$ denoted by ${\boldsymbol{\mathcal{D}}}\times_{\mbox{\tiny\rm m}}{\bf A}$ is a tensor of size ${\rm I\!R}^{I_{1}\times I_{2}\times\dots J_{m}\times\dots\times I_{M}}$ where the entries are calculated as

The mode-$M$ product is interchangeably denoted by matrix multiplication and tensor multiplication as follows:

We can define SVD decomposition in terms of n-mode product as follows:

In multilinear algebra there is a generalization of SVD know as multilinear SVD (Delathauwer00a, ; Delathauwer00b, ) or $M$-mode SVD (Vasilescu02, ; Vasilescu05, ) which decomposes an $M$-mode tensor ${\boldsymbol{\mathcal{D}}}$ into the $M$-mode product of orthonormal spaces:

where ${\boldsymbol{\mathcal{Z}}}$ is the core tensor that governs the interaction between the orthonormal mode matrices, ${\bf U}_{\mbox{\tiny\rm m}}$. The core tensor is analogues to the singular value matrix $\Sigma$ but unlike the $\Sigma$ the core tensor is not always diagonal (KoBa09, ; Papalexakis:2016, ; Tensor, ).

The $M$-mode SVD of a 3-mode tensor is demonstrated in Figure 3. ${\bf U}_{i}$ is approximated by left singular vectors of truncated SVD decomposition of ${{\bf D}}_{\mbox{\tiny\rm$[i]$}}$. Meanwhile, since ${\bf U}_{i}$ is orthonoramal, we have ${\bf U}_{m}^{-1}={\bf U}_{m}^{T}$ and the core tensor ${\boldsymbol{\mathcal{Z}}}$ is estimated as follows:

Vasilescu (Vasilescu09, , Appndix A) argues that in most cases, it is preferable to vectorize an image and treat it as a single observation rather than a collection of independent column/row observations. By vectorizing an image, we treat an image as a point in high dimensional pixel space and calculate all possible combinations of pixel statistics, both near and faraway statistics. On the other hand, when we consider an image as a matrix, every image column (row) is treated as an independent observation, and column (row) covariances are computed. Having this in mind, we also follow the same strategy and vectorize the frames and create a vector for each one of the video frames in the dataset.

Eigenfaces are eigenvectors when the images are human face. The eigenfaces are derived from the covariance matrix of the pixel distribution over the high dimensional face space. The eigenfaces represent a basis set of all faces used to construct the covariance matrix. So far, the eigenfaces have been successfully leveraged for many facial image related tasks  (Turk91b, ). Leveraging eigenfaces allows for dimensionality reduction such that a smaller set of basis vectors represent the original training faces. Classification could be achieved by comparing how different faces are represented by the basis set of the corresponding class.

Based on principle component terminology, the eigenfaces are equal to basis vectors of PCA decomposition. Therefore, by staking the vectorized frames of each class, we create two separate matrices and decompose them using SVD to capture the eigenfaces of the corresponding class as follows:

Where ${\bf B}_{\mbox{\tiny\rm real}}$ and ${\bf B}_{\mbox{\tiny\rm fake}}$ are basis matrices and ${\bf V}_{\mbox{\tiny\rm real}}$ and ${\bf V}_{\mbox{\tiny\rm fake}}$ are the normalized coefficient matrices of the corresponding classes.

As seen in previously mentioned tensor is an effective framework for decomposing a set of observation into underlying factors. After reducing the dimentionality of observations using eigenface representation of the classes, we propose leveraging a three-mode tensor where the first mode i.e., measurement mode represents the pixels of an eigenface, the second mode corresponds to the eigenfaces and the third mode is the class mode i.e., DeepFake vs. real. We propose using an $M$-mode SVD which as we discussed earlier decomposes a tensor into $M$ orthonormal matrices ($M=3$), and a core tensor which governs the interaction between these spaces. Since the first mode is the measurement mode, We only calculate the $M$-mode SVD of the tensor by flattening the second and the third modes as follows:

where the ${\bf U}_{\mbox{\tiny\rm c}}$ comprises underlying vector representation of original and fake classes.Moreover, the core tensor ${\boldsymbol{\mathcal{T}}}$ is the signature of this dataset and shows interactions of orthonormal subspaces. Later on, we leverage this signature to project the test frames into the subspaces we derive here.

Applying $M$-mode SVD results in a mode matrix ${\bf U}_{\mbox{\tiny\rm c}}\in{\rm I\!R}^{2\times 2}$ that spans the class representations. We embed the vector class representations into a higher dimensional space to increase the class separability of the test data. We embed the row vectors of ${\bf U}_{\mbox{\tiny\rm c}}$ into $\mathbb{R}^{3}$, setting the third coordinate of the real and fake class to $+1$ and $-1$ respectively, and normalizing the vector length to $1$.

As mentioned above, the core tensor of each decomposition is the signature of the decomposed space which governs the interaction of constituent factors. we leverage the core tensor and perform a multilinear projection of the incoming frame into the subspaces we derived in the previous step. Let say we have the vectorized frame ${\bf d}$. If ${\bf d}$ is supposed to be in the same subspaces we derived, then

where the vectors $\vec{f}$ and $\vec{c}$ are the coefficient vector representations of a video frame ${\bf d}$ in the orthonormal subspaces that are governed by the extended core tensor ${\boldsymbol{\mathcal{T}}}$. The goal is to find out weather the class coefficient vector $\vec{c}$ is more similar to the vector representation of real class or Deepfake class. To this end, we estimate $\vec{c}$ representation vector by employing the multilinear projection algorithm (Vasilescu11, ; Multilinear_Projection2007, ) that decomposes a vectorized observation, ${\bf d}$ into a set of latent vector representation, ${\bf r}_{\mbox{\tiny\rm n}}$ that corresponds to the constituent factors of data formation. The basic multilinear projection is the $M$-mode SVD/CP decomposition of ${\boldsymbol{\mathcal{T}}}^{{\dagger}\lower 2.0pt\hbox{\hskip-1.0pt\hbox{\tiny${}1$}}}\times_{\mbox{\tiny\rm 1}}{\bf d}^{\mathrm{T}}$ which can be expressed mathematically as

where ${\boldsymbol{\mathcal{T}}}^{{\dagger}\lower 2.0pt\hbox{\hskip-1.0pt\hbox{\tiny${}1$}}}$ is mode-1 pseudo-inverse of ${\boldsymbol{\mathcal{T}}}$ that in matrix notation is expressed as ${{\bf T}}_{\mbox{\tiny\rm$[1]$}}^{{\dagger}\lower 2.0pt\hbox{\hskip-1.0pt\hbox{\tiny${}$}}}$, and ${\bf r}_{\mbox{\tiny\rm c}},{\bf r}_{\mbox{\tiny\rm f}}$ are estimates of vectors ${\bf c}$ and ${\bf f}$ from eq.(15), respectively.

Up to this step, we have the vector representation of each classes in addition to class coefficients of the incoming frame. We use a linear Support Vector Machine (SVM) and estimate the decision boundaries using validation frames and then leverage the defined boundaries for classification of test frames. An overview of the proposed approach is demonstrated in Algorithm 1.

As mentioned earlier, factor matrix ${\bf U}_{f}$ comprises underlying structures of basis vector continent. Despite the fact that we construct our predictive model by approximating discriminating regions, still there are many shared components which getting rid of them make the model more distinguishable. Since we are interested in noisy regions i.e., artifacts, we propose truncating components of the core tensor ${\boldsymbol{\mathcal{T}}}$ which correspond to top values of ${\bf U}_{f}$ and keeping lower value components as representatives of noisy parts. In the next section, we will show how this truncation boosts the classification performance of the proposed framework. An example of truncating components corresponding to the second mode of a 3-mode tensor is depicted in Figure. 3.

One of the most popular and widely used databases for image or video forgeries detection is FaceForensics++⁶⁶6https://github.com/ondyari/FaceForensics which first was introduced in 2018  (roessler2018faceforensics, ). FaceForensics++ comprises more than 500,000 frames from 1000 youtube videos that contain mostly frontal faces (roessler2019faceforensicspp, ). This dataset also includes 1000 videos which are the manipulated version of the original onesand have been manipulated by four automated face manipulation methods: Deepfakes, Face2Face, FaceSwap and NeuralTextures. All original and manipulated videos have constant frame rate of 30 fps and have been compressed lossless with H.264. Moreover, the videos are split up into train set of size 720, validation set of size 140 and test of 140 videos. binary classification scenario on this dataset. A summarized benchmark of existing techniques on videos manipulated by DeepFakes method is demonstrated in table 2. The state-of-the-art benchmark on FaceForensics++ is available in GitHub⁷⁷7http://kaldir.vc.in.tum.de/faceforensics-benchmark/. In this work, we experiment on images manipulated by DeepFake technique.

Our work was implemented in MATLAB partially using Tensor Toolbox version 2.6.  (TTB_Sparse, ; TTB_Software, ). Since all videos have constant frame rate 30 fps, we extracted up to 7 frames for each video by snapping almost one frame per each 30 seconds using OpenCV library in Python. Moreover, for detecting facial landmarks, we used pretrained dlib face detector⁸⁸8http://dlib.net/face_landmark_detection.py.html which is created using the classic Histogram of Oriented Gradients (HOG) feature combined with a linear classifier, an image pyramid, and sliding window detection scheme (landmark, ). For the second step, we calculated the SVD rank r where the r is equal to ”$\text{number of train videos}\times 7=720\times 7=5040$” for all experiments. The intuition behind this estimation is to have an individual component for each frame. Moreover, in contrast to many deep learning approaches for Deepfake detection, our approach does not require GPU base configuration and both train and test steps can be executed on an ordinary CPU based configuration. The description of the CPU based configuration we experimented on is as follows: Intel(R) Core (TM) i5-8600K CPU @3.60GHz,CentOS Linux 7 (Core) operating system and 40GB RAM memory.