Knowledge Cross-Distillation for Membership Privacy

Machine learning (ML) has been extensively used in various aspects of society [dlsurvey]. We have seen great improvements in areas such as image recognition and natural language processing. ⁰⁰footnotetext: The authors are alphabetically ordered.

However, in the recent years, it has been reported that the privacy of the training data can be significantly undermined by analyzing ML models. Since, in most applications, privacy-sensitive data are used as the training data for the models, protecting the privacy of the training data is crucial for getting approval from data providers or essentially society.

Following the growing concern for privacy in society worldwide, many countries and regions are introducing regulations for data protection, e.g., the General Data Protection Regulation (GDPR) [GDPR], California Consumer Privacy Act (CCPA) [CCPA], and Health Insurance Portability and Accountability Act (HIPAA) [HIPAA]. Moreover, guidelines and regulations designed specifically for trustworthiness in artificial intelligence (AI) and ML are under discussion [EU-Guideline].

MIAs are dangerous because they reveal the information of individual pieces of data rather than the trend of the whole population of training data. For instance, consider an ML model for inferring a reaction to some drug from a cancer patient’s morphological data. An MIA attacker who knows the victim’s data and has access rights to the ML model can know whether the victim has cancer or not, although the victim’s data itself do not directly contain this information.

Another reason that MIAs are dangerous is that they can be executed through legitimate access to ML models only, meaning that they cannot be prevented by the conventional security methods such as data encryption and access control [shokri2017membership].

However, in many domains of ML applications, public data are scarce due to the sensitive nature of the data, e.g., financial and medical data. To overcome this, utilization of synthetic data is proposed [shejwalkar2021membership] as well. However, this method decreases accuracy [shejwalkar2021membership] due to the decrease in data quality.

In this paper, we propose a novel knowledge distillation-based defense that uses only private data for model training.

Our contributions are as follows.

• –

  We propose a novel MIA defense called knowledge cross-distillation (KCD)¹¹1After we submitted our work to PETS 2022 Issue 2, Tang et al. [tang2021mitigating] published a concurrent and independent work similar to ours in arXiv.. Unlike the state-of-the-art defense, DMP, it does not require any public or synthetic reference data to protect ML models. Hence, KCD allows us to protect the privacy of ML models in areas where public reference data are scarce.

• –

  For the benchmark tabular datasets used in MIA research, Purchase100 and Texas100, we empirically show that the privacy protection and accuracy of KCD are comparable to those of DMP even though KCD does not require public or synthetic data, unlike DMP.

• –

  For the image dataset CIFAR10, we empirically show that the accuracy of KCD is comparable to that of DMP, and KCD provides a much better privacy-utility trade-off than those of other defenses that do not require public or synthetic reference data.

We focus only on related works that are directly related to our contributions. See Hu et al. [DBLP:journals/corr/abs-2103-07853] for a comprehensive survey of MIAs.

Choo et al. [choo2020label] and Li et al. [DBLP:journals/corr/abs-2007-15528] independently and concurrently succeeded in attacking neural networks in a label-only setting, where an attacker can get only labels as outputs of a target neural network, while the attackers of other known papers require confidence scores as the outputs of it. Nasr et al. [DBLP:conf/sp/NasrSH19] proposed an MIA attack in a white-box setting, where an attacker can obtain the structure and parameters of the target neural network.

An important technique for protecting MLs against MIAs is knowledge transfer [hinton2015distil]. Using this technique, PATE by Papernot et al. [DBLP:conf/iclr/PapernotAEGT17, DBLP:conf/iclr/PapernotSMRTE18] achieved DP, Cronos [chang2019cronus] by Chang et al. protected ML from an MIA in a federated learning setting, and DMP [shejwalkar2021membership] achieved a higher privacy-utility trade-off by removing public data with low entropy.

Currently, DMP is the best defense in the sense of the privacy-utility trade-off. However, it requires public data. Other known defenses, AdvReg and MemGuard, have an advantage in that they do not require public reference data.

An ML model for a classification task is a function $F$ parameterized by internal model parameters. It takes a $d$-dimensional real-valued vector $x\in\mathbb{R}^{d}$ as input and outputs a $c$-dimensional real-valued vector $\hat{y}=(\hat{y}_{1},\ldots,\hat{y}_{c})$. The output $\hat{y}$ has to satisfy $\hat{y}_{i}\in[0,1]$ and $\sum_{i}\hat{y}_{i}=1$. Each $\hat{y}_{i}$ is called a confidence score. Its intuitive meaning is the likelihood of $x$ belonging to class $i$. The $\mathop{\rm argmax}\limits_{i}\hat{y}_{i}$ is called a predicted label (or predicted class).

An ML model $F$ is trained using a training dataset $D\subset\{(x,y)\mid x\in\mathbb{R}^{d},\ y\in\{0,1\}^{c}\}$, where $x$ is a data point, and $y$ is a one-hot vector reflecting the true class label of $x$. In the training procedure, the model parameters of $F$ are iteratively updated to reduce the predetermined loss $\sum_{(x,y)\in D}L(F(x),y)$, which is the sum of errors between the prediction $F(x)$ and true label $y$. For inference, $F$ takes input $x$ and outputs $\hat{y}=F(x)$ as a prediction.

The accuracy of $F$ for dataset $D$ is the ratio between the number of elements $(x,y)\in D$ satisfying $\mathop{\rm argmax}\limits_{i}F(x)_{i}=\mathop{\rm argmax}\limits_{i}y_{i}$. Here, $F(x)_{i}$ and $y_{i}$ are the $i$-th component of $\hat{y}=F(x)$ and $y$, respectively. The training accuracy and testing accuracy of $F$ are for the training and testing datasets, respectively. Here, testing dataset is a dataset that does not overlap with the training dataset. The generalization gap of $F$ is the difference between training and testing accuracies.

MIA is an attack in which an attacker attempts to determine whether given data (called target data) are used for training a given ML model (called a target model). In the discussion of MIAs, the training data of the target model are called member data, and non-training data are called non-member data.

There are two types of MIAs, white-box and black-box [shokri2017membership, DBLP:conf/sp/NasrSH19]. Attackers of the former can take as input the model structure and model parameters of the target model. Attackers of the latter do not take them as input but are allowed to make queries to the target model and obtain answers any number of times. A black-box MIA can be divided into the two sub-types, MIA with confidence scores and label-only MIA [choo2020label]. Attackers of the former can obtain confidence scores as answers from the target model but attackers of the latter can obtain only predicted labels as answers.

In all types of MIAs, the attackers can take the target data and prior knowledge as inputs. Intuitively, the prior knowledge is what attackers know in advance. What type of prior knowledge an adversary can obtain depends on the assumed threat model. An example of prior knowledge is a dataset sampled from the same distribution as the training data of the target model, not overlapping with the training data. Another example is a portion of the training data. The prior knowledge we focused in this study is described in Section 4.3. The attack accuracy of an attacker for an MIA is the probability that they will succeed in inferring whether target data are member data. As in the all previous papers, the target data are taken from member data with a probability of $50\%$.

One of the main factor causing MIA risks is overffiting of an ML model on the training ($=$member) data. The member data can be distinguished from non-member data [yeom2018privacy, salem2019ml] depending on whether it is overfitted to the target model, e.g. by checking whether the highest confidence score of output of the target model is more than a given threshold.

DMP [shejwalkar2021membership] is a state-of-the-art defense method against MIAs that leverages knowledge distillation [hinton2015distil]. Distillation was originally introduced as a model compression technique that transfers the knowledge of a large teacher model to a small student model by using the output of a teacher model obtained on unlabeled reference dataset. DMP needs public reference dataset $R$ disjoint from the training dataset $D$ to train ML models with membership privacy.

The training algorithm of DMP is given in Algorithm 1. Here, $L$ is the loss function. First, DMP trains a teacher model $F$ using a private training dataset $D$ (Step 1). $F$ is overfitted to $D$ and therefore vulnerable to MIA. Next, DMP computes the soft labels $F(x)$ of each peice of data $x$ of public reference dataset $R$ and lets $\bar{R}$ be the set of $(x,F(x))$ (Step 2). Finally, to obtain a protected model, DMP trains a student model $H$ using the dataset $\bar{R}$ (Step 3). $H$ has MIA resistance because it is trained without direct access to the private $D$. Note that DMP uses $H$ with the same architecture as $F$.

The authors of DMP [shejwalkar2021membership] proposed three different ways of achieving the desired privacy-utility tradeoffs:

• –

  increasing the temperature of the softmax layer of $F$,

• –

  removing reference data with high entropy predictions from $F$,

• –

  decreasing the size of the reference dataset.

All of the above changes reduce MIA risks but also the accuracy of $H$ and vice versa. When we use the second or third way to tune DMP, we select samples from the reference dataset and use them as $R$ in Step 2.

The starting point with our approach is DMP [shejwalkar2021membership]. That is, we train a teacher model $F$ using a training dataset $D$, compute soft labels $F(x)$ to $x$ of public reference dataset $R$, train a student model $H$ using $(x,F(x))$, and, finally, use $H$ for inference. DMP can mitigate MIAs as described in Section 2.3.

The problem with DMP is that it requires a public reference dataset, which may be difficult to collect in privacy-sensitive domains [shejwalkar2021membership]. A naïve idea to solve this problem is to use the original $D$ as a reference dataset. However, our experiment shows that this approach does not sufficiently mitigate the MIA risk (see Section LABEL:subsec:Discussions). The main problem of the naïve idea is that data $x$ of the reference dataset $R=D$ is member data of $F$. Therefore, $F$ results in overfitting on $x$ and the confidence score $\hat{y}=F(x)$ is close to the one-hot vector $y$ of the true label. Hence, $H$ trained on $(x,\hat{y})$ results again in overfitting on $x$, which can be exploited by an MIA.

Our proposed defense, denoted by knowledge cross-distillation (KCD) is designed to overcome the above problem. We divide the training dataset into $n$ parts, leave one part as a reference dataset, and train a teacher $F_{1}$ using the remaining parts. To increase the accuracy of KCD, we prepare teachers $F_{2},\ldots,F_{n}$ as well and repeat the above procedure for each teacher by changing the reference part. Finally, we use each reference part to distill the knowledge of each corresponding teacher into a single $H$. Our defense solves the problem of the naïve idea because none of the remaining parts of the training dataset are used to train the teacher model.

The training algorithm of the our proposed defense, KCD, is given in Algorithm 2 and is overviewed in Figure 1. Here, $F_{1},\ldots,F_{n}$, and $H$ are models with the same structure as that of the model $F$ that we want to protect²²2Although we use the term “distillation,” we use teacher and student models with the same structure as in DMP [shejwalkar2021membership]. This is because we are not concerned about the size of the resulting model.. $L$ is the loss function.

In Algorithm 2, we divide training dataset $D$ into $n$ disjoint subsets $D_{1},\ldots,D_{n}$ with almost the same size, such that $D=\bigsqcup^{n}_{i=1}D_{i}$ holds³³3$\bigsqcup^{n}_{i=1}D_{i}$ denotes a disjoint union of sets (Step 1). Then, for $i=1,\ldots,n$, we train the teacher model $F_{i}$ using the dataset $D$ but exclude $D_{i}$ (Step 2-4). Let $\bar{D}_{i}$ be the dataset that is obtained by adding soft labels $F_{i}(x)$ to $(x,y)\in D_{i}$ (Step 5). Finally, we train a student model $H$ using the dataset $\bigcup_{i}\bar{D}_{i}$ to minimize the combined loss function with hyperparameter $\alpha$ (Step 6).

Our loss function comprises two terms; the first term is the loss for soft labels $y^{\prime}$, and the second is the loss for the true label $y$. The hyperparameter $\alpha$ can tune the privacy-utility trade-off of KCD. In fact, if $\alpha=1$, our defense protects the privacy of the training data due to the reason mentioned in Section 3.1. If $\alpha=0$, KCD becomes the same as the unprotected ML.

Note that our privacy-utility trade-off based on $\alpha$ cannot be directly applied to the known knowledge distillation-based defenses, DMP [shejwalkar2021membership] and Cronus [chang2019cronus], because the public reference datasets for these defenses do not have the true labels and loss for the predicted scores and true labels cannot be computed.

CIFAR 10: This is a typical benchmark dataset used for evaluating the performance of image-classification algorithms [Krizhevsky09]. It contains $60,000$ RGB images. Each image is composed of $32\times 32$ pixels and labeled in one of $10$ classes.

As in the previous studies of MIAs [nasr2018machine, DBLP:conf/sp/NasrSH19], we consider a strong setting where the attackers know the non-member dataset and a subset of the member dataset of the target model as prior knowledge. (This subset of the member dataset does not contain the target data, of course). This setting is called supervised inference [DBLP:conf/sp/NasrSH19].

One may think that the supervised inference setting seems too strong as a real setting. However, the shadow model technique [shokri2017membership] allows an attacker to achieve supervised inference virtually [DBLP:conf/sp/NasrSH19]. A shadow model is an ML model that is trained by an attacker to mimic a target ML model. The attacker then knows the training data of the shadow model as in the supervised inference setting since the attacker trains it.

We conducted comprehensive experiments for three types of MIA: black-box MIA with confidence score, black-box MIA with only labels, and white-box MIA.