Privacy-Preserving Face Recognition Using Random Frequency Components

The current method of choice for face recognition is CNN-based embedding. The service provider trains a CNN with a softmax-based loss to map face images into one-dimensional embedding features which achieve large inter-identity and small intra-identity discrepancies. While the state-of-the-art (SOTA) FR methods [6, 38, 20] achieve impressive task utility in real-world applications, their attention to privacy protection could be deficient.

The past decade witnessed significant advances in privacy-preserving face recognition [41, 24, 25]. We roughly categorize the related arts into two branches:

Converting spatial or temporal data into the frequency domain provides a powerful way to extract interesting signals. For instance, the Fourier-based discrete cosine transform [5] is used by the JPEG standard [37] to enable image compression. Researches in deep learning [9, 45] suggest models trained on images’ frequency components can perform as well as trained on the original images. Advance [39] further reveals humans and models perceive low- and high-frequency components differently. In the realm of PPFR, three recent methods [42, 26, 15] are closely related to ours as we all conceal visual information by exploiting the split in human and model perceptions. However, these methods bear inadequacy in defending recovery, according to our previous discussion of channel redundancy and later testified in experiments.

Recall our privacy goals are to conceal the face images’ visual information and to prevent recovery attacks on them. They respectively target the adversarial capability of humans and models. We naturally concretize them into two technical aims, on how to eliminate human perception, and, on how to reduce exploitable information for attack models.

To eliminate human perception, we leverage the finding that models’ utility can be maintained almost in full on the images’ high-frequency channels [39]. Contrarily, humans mostly perceive the images’ low-frequency channels, as only they carry signals of conspicuous amplitude for human eyes to discern. Whereby spatial-frequency transforms, these channels can be easily located and pruned from raw images. We concretely opt for DCT as our transform as it facilitates the calculation of energy, to serve as the quantification for human perceivable information. Experimentally, we can prune a very small portion of channels to eliminate 95% of total energy, hence satisfying our aim.

Reducing the attacker’s exploitable information requires further pruning of high-frequency channels, as previously discussed. However, it is equally crucial to maintain the recognition model’s accuracy, which presents a seeming dilemma, as the training features utilizable by recognition and attack models are tightly interwoven in these channels. We offer a viable way as our major contribution: We notice different channel subsets each carrying certain local facial features. Hence, we feed the model with a random subset from each face image, with the hope to let the model learn the entire face’s impression from different images’ complementary local feature partitions. This approach is proved feasible by recent theoretical studies [51] and demonstrated by our experiments. Therefore, information exposure is minimized as only each image’s chosen subset is exposed, and the model still performs surprisingly well.

Despite being well justified by theory, we find sampling channels under complete randomness could under-perform in actual training due to two technical limitations: the inadequacy in training samples and the biased sampling within mini-batches. We propose two targeted fixes to reconcile the constraints, by augmenting samples and seeking a moderate level of randomness. We find the modified approach satisfactorily addresses the privacy-accuracy equilibrium.

We first set up some basic notions: $\langle X,y\rangle$ denotes a data sample of a face image and its corresponding label. $\mathbf{x}$ denotes the frequency composition of $X$ and $x_{i}$ denotes its individual frequency channels. $f(\cdot;\theta)$ denotes the recognition model parameterized by $\theta$. $l(\cdot,\cdot)$ denotes a generic loss function (e.g., ArcFace). $\mathcal{T}(\cdot)$ denotes the discrete cosine transform and $\mathcal{T}^{-1}(\cdot)$ denotes its inverse transform.

To conceal visual information, recall humans and models mainly perceive low- and high-frequency components, respectively. Therefore, we need to find a frequency decomposition of $X$=$\{\mathbf{x}_{l},\mathbf{x}_{h}\}$, where $\mathbf{x}_{l},\mathbf{x}_{h}$ are the respective low- and high-frequency channels, then prune $\mathbf{x}_{l}$.

We presume $X$ is with the shape of $(H,W)$. We employ DCT to transform $X$’s spatial channels into a frequency spectrum. While an RGB image typically has 3 spatial channels, we pick one for simplicity. We perform an 8-fold up-sampling ahead to turn $X$ into $(8H,8W)$. As DCT later divides $H$ and $W$ by 8, this makes sure the resulting frequency channels can be fed into the model as usual. Figure 3 illustrates the process of DCT, where $\mathbf{x}$=$\mathcal{T}(X)$. Concretely, $X$ is divided into $(8,8)$-pixel blocks. DCT turns each block into a 1D array of 64 frequency coefficients and reorganizes all coefficients from the same frequency across blocks into an $(H,W)$ frequency channel (there are 64 of them), that is spatially correlated to the original $X$. As a result, $X$ is turned into $\mathbf{x}$ of $(64,H,W)$.

We then decouple $\mathbf{x}$ into $\{\mathbf{x}_{l},\mathbf{x}_{h}\}$. Notice that human-perceivable low-frequency channels should meanwhile be those with higher amplitude, as humans preferentially identify signals with conspicuous value changes. We measure a channel’s amplitude by its channel energy $e(\cdot)$, which is the mean of amplitudes of all its elements:

We choose $\sigma$=10 highest-energy channels as $\mathbf{x}_{l}$ to be pruned and the rest as $\mathbf{x}_{h}$ to be available to models. We experimentally find $\sum_{x\in\mathbf{x}_{l}}{e(x)}\geq 0.95\sum_{x\in\mathbf{x}}{e(x)}$, and a model trained with $\operatorname*{arg\,min}_{\theta}{l(f(\mathbf{x}_{h},\theta),y)}$ obtain close accuracy to one trained with $\operatorname*{arg\,min}_{\theta}{l(f(\mathbf{x},\theta),y)}$. For privacy, example of a channel-pruned image in Fig. 4(a) show that most visual information is concealed. While its recovery is still carried out quite successfully, we are to address the issue in the following.

To elucidate the seeming dilemma between reducing model-available channels and retaining high recognition accuracy, we first introduce an intuitive protection, referred to as the vanilla method. It trains models on a fixed small subset of channels straightforwardly: Concretely for every $X$, the model pick $s$<$d$ channels $\mathbf{x}_{s}$=$(x_{a_{1}},\dots,x_{a_{s}})$ from its high-frequency $\mathbf{x}_{h}$=$(x_{1},\dots,x_{d})$, where $a_{1}$<$a_{2}$<$\cdots$<$a_{s}$ are fixed indices. We benchmark the trained model on IJB-B/C with $s$=9,18,36 and $d$=54, and verify its privacy.

The model indeed effectively prevents recovery, as in Fig. 4(b) its recovered image is very blurred. However, the benchmark in Fig. 5(a-b) suggests its accuracy dropped by at most 9%. To find out why is the model’s performance impaired, we inspect its attention via Grad-CAM [34]. Results are exemplified in Fig. 5(c-d), which we note show similar patterns among different face images. We derive two observations: (1) Unlike high-utility models that typically have attention to the full face, this model only gains attention to certain local features, which suggests some vital face-describing information is missing in $\mathbf{x}_{s}$; (2) By training vanilla models on different $\mathbf{x}_{s}$, we find it can acquire distinct information from different channels, suggesting its attention is correlated to the specific choices of $\mathbf{x}_{s}$.

The first observation suggests the cause of the accuracy downgrade. Meanwhile, the second pursues us to consider a viable bypass: We can gather complementary local features from different images and initiate mixed training, to let the model learn a holistic impression of the entirety while keeping the individual information exposure minimized. Our idea is corroborated by [51], which proves in time-series, training models on random frequency components can preserve more information compared to training on fixed ones.

We concrete our theory into a randomized strategy, by advocating training and inferring the recognition model on image-wise randomly chosen channels. Formally, we construct matrix $S\in\{0,1\}^{d\times s}$, with $s_{ij}$=1 if $i$=$a_{j}$ and $s_{ij}$=0 if otherwise. We also construct permutation matrix $P\in\{0,1\}^{s\times s}$. For each $X$, we draw $(a_{1},\dots,a_{s})$ (therefore $S$) and $P$ uniformly at random, and calculate

Here, $S$ randomly pick $s$ channels out of $\mathbf{x}_{h}$, then their order is permuted by $P$. We introduce $P$ to further impede recovery, as later discussed in Sec. 3.5. The model is trained in the same way as standard FR using $\langle\mathbf{x}_{s},y\rangle$. However, it does not require to know the sample-wise specific $\{S,P\}$, which is favorable for privacy. The approach sounds a bit counter-intuitive since receiving random inputs seems to mess with the model’s understanding. However, recall that the DCT frequency channels are spatially correlated to $X$ and each other. The randomness only pertains to the frequency spectrum while the face’s structural information is preserved and unaltered. The model, therefore, can associate different $\mathbf{x}_{s}$ to the same $X$ quite naturally.

Equation 2 distills our core idea to sampling $\mathbf{x}_{s}$ at complete random. However, for practical training, there need to further reconcile two technical constraints to achieve satisfying model performance: First to note that $\langle\mathbf{x}_{s},y\rangle$ are more varied in feature representations than the unprotected $\langle X,y\rangle$, owing to the randomized frequency. It hence could plausibly take the model with more training samples and a longer training time to approach a well-generalized convergence. Second, ideally, channels at random should be sampled with equal probability to ensure balanced learning of different local features. However, when the training data is partitioned into small mini-batches, sampling is often less unbiased batch-wise so the occurrence of different channel combinations may vary greatly, which we find could undermine the training stability.

Targeting the constraints, we slightly adjust our established approach in two ways. We first augment the training dataset by picking multiple $\mathbf{x}_{s}$ each time from a face image $X$: From $X$ we sample $\{\mathbf{x}_{s}^{1},\dots,\mathbf{x}_{s}^{r}\}$, where $r$ determines the degree of augmentation. Each $\mathbf{x}_{s}^{i}$ is independently drawn and appended to the training dataset as individual training samples. The dataset is then fully shuffled so that $\mathbf{x}_{s}^{i}$ corresponding to the same face image cannot be easily associated.

We also control the randomness to a moderate level by specifying the possible combinations of $S,P$ in advance. Concretely, we opt for choosing one $S,P$ from $\mathbf{S}$=$\{S_{1},\dots,S_{m}\}$ and $\mathbf{P}$=$\{P_{1},\dots,P_{n}\}$ respectively, where $\mathbf{S},\mathbf{P}$ are determined by the service provider. We specifically require $\{S_{1},\dots,S_{n}\}$ to be a non-overlapping partition of channels from $\mathbf{x}_{h}$ (i.e., divide $\mathbf{x}_{h}$ into equal-length subsets) to maximize the use of $\mathbf{x}_{h}$ and reduce model bias. Therefore, each $\mathbf{x}_{s}^{i}$ is picked from one of $m$$\times$$n$ fixed combinations of channels, called ranks. This allows us to facilitate the training and overcome sampling biases. The service provider shares $\{\mathbf{S},\mathbf{P}\}$ with all local devices, so the latter can generate their query $\mathbf{x}_{s}$ accordingly. During inference, the model is expected to provide consistent results of the same query $X$ regardless of the choice of rank, since it learned about a mapping from local features to the face’s entirety. We later testify to it in Sec. 4.3. Meanwhile, the model is still unaware of the sample-wise specific choice of $\{S,P\}$ as the recognition relies on nothing else but $\mathbf{x}_{s}$ alone. Hence privacy is maintained.

To conclude, we present PartialFace that train and infer the recognition model with $\operatorname*{arg\,min}_{\theta}{l(f(\mathbf{x}_{s}^{i},\theta),y)}$, where $X$=$\{\mathbf{x}_{l},\mathbf{x}_{h}\}$ and $\mathbf{x}_{s}^{i}$=$\mathbf{x}_{h}\cdot S\cdot{P}$ in random, parameterized by $\{\mathbf{S},\mathbf{P}\}$ and $(\sigma,s,r,m,n)$. Later analyses in Secs. 4.2 and 4.3 show PartialFace overcomes the drawback of the vanilla method, plus outperforms most prior arts, to achieve satisfactory recognition accuracy.

The benefit of randomness is multi-fold. We have discussed it for now on helping address the accuracy and privacy balance and enhance recognition performance. In retrospect, we briefly elaborate on how randomness further safeguards privacy to a large extent.

Recall we remove visual information by pruning $\mathbf{x}_{l}$ and impede recovery by choosing a subset of $\mathbf{x}_{s}$. After that, randomness further obstructs recovery: To impose recovery, the attacker exploits not only the channels’ information but also their relative orders and positions in the frequency spectrum. We introduce $P$ to distort the order of channels to this end. As the recognition model does not require sample-wise $\{S,P\}$, they won’t be exposed to the attacker. Figure 4(c) shows the improvement against recovery if the attacker doesn’t know the specific choice of subsets.

We compare PartialFace with the unprotected baseline and prior PPFR methods on three criteria: recognition performance, privacy protection of visual information, and that against recovery attack. We further study the computation and cost of PartialFace. We mainly employ an IR-50 [11] trained on the MS1Mv2 [10] dataset as our FR model, while also using a smaller combination of IR-18 and the BUPT [40] dataset on some resource-consuming experiments. We set $(\sigma,s,r,m,n)$=$(10,9,18,6,6)$ and use fixed $\{\mathbf{S},\mathbf{P}\}$, if not else specified. Benchmarks are carried out on 5 widely used, regular-size datasets, LFW [13], CFP-FP [35], AgeDB [31], CPLFW [49] and CALFW [50], and 2 large-scale datasets, IJB-B [43] and IJB-C [23].

We elaborate that randomized PartialFace outperforms the vanilla method that fixes channels in not only recognition accuracy but also robustness. As PartialFace employs data augmentation, for a fair comparison regarding the volume of training data, we consider two experimental settings: the standard PartialFace and one without augmentation ($r$=1). We also set $n$=1 to rule out permutation: $P$ is introduced for privacy and is out of our interest here. There are therefore $m$$\times$$n$=6 combinations of channels, or ranks. As channels from different ranks carry distinct information that may affect the performance, we train one vanilla model on each rank and average their test accuracy, with the range marked in parentheses. Though PartialFace is able to infer from arbitrary rank, we evaluate its robustness by inferring from each fixed rank separately. The model is robust if it performs on different ranks consistently (small range). Results are reported on IJB-B/C by TPR@FPR(1e-4) in Tab. 2.

We investigate PartialFace’s protection of privacy. We reiterate that the first privacy goal is to conceal the visual information of face images. We compare ParitalFace with 3 PPFR SOTAs using the frequency domain: PPFR-FD [42], DCTDP [15], and DuetFace [26]. These prior arts are closely related to ours since we all leverage the perceptual difference between humans and models as means of protection. However, they differ in the processing of frequency components: Both PPFR-FD and DCTDP remove the component with the highest energy (the DC component). To obfuscate the remaining components, PPFR-FD employs mix-and-shuffle and DCTDP applies a noise mechanism. DuetFace is the most related method in the pruning of low-frequency components. We note in special that they retain 36, 63, and 54 high-frequency channels, respectively, while PartialFace retains 9. We now demonstrate the methodological differences and varied number of channels result in contrasting privacy protection capacities.

Figure 7 exemplifies face images processed (therefore are supposed to be protective) by each compared method. As all methods carry out protection in the frequency domain, we convert images back by padding any removed frequency components with zero and applying an inverse transform. PartialFace, e.g., has $X^{\prime}$=$\mathcal{T}^{-1}(\{\mathbf{0},\mathbf{x}_{h}\})$. We visualize the training and inference phases separately, as the compared methods’ processing on them may vary: During inference, we argue PPFR-FD and DuetFace (Fig. 7(b)(d)) provide inadequate protection, as one can still discern the face quite clearly in their processed images. DCTDP effectively conceals visual information after obfuscating the channels with its proposed noise mask (Fig. 7(c)). However, its protection during training is impaired, as it requires access to abundant non-obfuscated components to learn the mask. DuetFace manifests a similar weakness as it relies on original images to rectify the model’s attention (Fig. 7(d)).

PartialFace first discards most visual information by pruning $\mathbf{x}_{l}$. The rest is further partitioned when sampling $\mathbf{x}_{s}$ from $\mathbf{x}_{h}$. By energy, each $\mathbf{x}_{s}$ carries less than 1% of visual information compared to the original $X$. As a result, Fig. 7(e) shows PartialFace provides better protection on visual information than related SOTAs in both phases.

We here provide an in-depth analysis of how PartialFace impedes recovery. Our primary aiming is to show how the removal of model-exploitable information and the randomness of channels contribute to the defense. Assume the recovery attacker possesses a collection of face images $\{X\}$ and is aware of the protection mechanism. It can locally generate protected representations $X^{\prime}$ ($\mathbf{x}_{h}$ in our case) from $X$, then train a malicious model $g(\cdot)$ with $\operatorname*{arg\,min}_{\delta}l^{\prime}(g(X^{\prime},\delta),X)$, with the aiming to inversely fit $X$ from $X^{\prime}$. Having intercepted a recognition query, the attacker leverages $g(\cdot)$ to recover the concealed visual information. Upon the knowledge it possesses, a black-box attacker is one unknowing of necessary parameters ($\{\mathbf{S},\mathbf{P}\}$ in our case), whilst a white-box attacker possesses such knowledge. We also study the capability of PPFR-FD, DCTDP, and DuetFace against the white-box attacker. We further investigate an enhanced white-box attacker, who imposes threats dedicated to the randomness of PartialFace.

We plus compare with PPFR-FD, DCTDP and DuetFace under the white-box settings. Among them, DuetFace shows almost no resistance to recovery (Fig. 9(c)). PPFR-FD and DCTDP provide inadequate protection, as images recovered from them (Fig. 9(a-b)) are blurred yet still clearly identifiable. The protection is impaired as their processed images retain most high-frequency components, and the excessive perceivable information is learned by the recovery model. In comparison, PartialFace (Fig. 9(d)) offers significantly outperformed protection.

By the principle of FR, all $\mathbf{x}_{s}$ of face image $X$ are later extracted to very similar identity templates, so the recognition on them produces aligned results. Here, one would arise concern that recovery can also be carried out on the extracted templates, which theoretically contain the faces’ full identity information, to realize better inversion of images. We demonstrate that such a proposed attack is also ineffective to PartialFace.

Although there could be dedicated attacks on templates, in practice, high-quality recovery from them is difficult as templates are far more compact representations than images and channel subsets. In Fig. 10, the recovered images are highly blurred, and can hardly be inferred by humans or models, hence won’t impose an effective threat.

PartialFace is parameterized by $(\sigma,s,r,m,n)$. Among them, we note $\sigma$ is chosen according to channel energy, and $s$ is determined by $(\sigma,m)$. Table 4 analyzes the choice of the remaining parameters. Results show augmentation ($r$) enhances the model’s performance, at the cost of taking longer time to train. $m$ affects the performance mainly by its influence on $s$, as larger $m$ leading to fewer channels in each $\mathbf{x}_{s}$. We introduce $P$ solely for privacy purposes. Results on $n$ indicate a trade-off between accuracy and privacy. Generally, PartialFace is robust to the choice of parameters.

Though PartialFace is proposed by studying the model’s behavior, privacy protection is solely realized by processing the face images. The decoupling with model architecture and training tactics benefits resources and compatibility.