A Comprehensive Analysis of Information Leakage in Deep Transfer Learning

In this work, we focus on deep transfer learning (Tan et al., 2018; Zhuang et al., 2019), where the models discussed are neural network based. Without losing generality, we consider a transfer learning setting with two domain tasks $\mathcal{T}\in\{S,T\}$, where $S$ and $T$ refer to the source domain and target domain respectively, both containing private sensitive information. We aim to improve the target domain learning task performance by utilizing its own data $\mathcal{D}^{T}$ and source domain data $\mathcal{D}^{S}$. Each dataset contains a set of labeled examples $z_{i}=(x_{i},y_{i})$, where $x$ denotes the inputs and $y$ denotes the corresponding labels. The size of source domain data $\mathcal{D}^{S}$ is usually much larger than the target domain data $\mathcal{D}^{T}$, i.e., $N^{S}>>N^{T}$. The goal of a domain task is to learning a transformation function $f_{w}(\mathbf{x},\mathbf{y}):\mathbb{R}^{d}\rightarrow\mathbb{R}$, parameterized by model weights $\mathbf{w}$.

The basic idea of the inference attack is to exploit the leakages when a model is being trained or released to reveal some unintended information from the training data. In this section, we briefly present two types of inference attacks, i.e., membership and property attacks, for machine learning models.

The above-mentioned two types of attacks are closely related. Most of existing works perform those attacks against general machine learning models, while a few focus on the federated learning and collaborative learning scenarios (Nasr et al., 2019b; Melis et al., 2019a). None of the studies systematically explore the inference attacks explicitly for the context of deep transfer learning.

Different types of transfer learning models have been proposed over the years, depending on how the knowledge is shared. Although in transfer learning, there may exist several existing categorizations in the literature (Pan and Yang, 2010; Tan et al., 2018; Nasr et al., 2019a), we categorize the different deep transfer learning models based on how the two domains interact, their training strategy (e.g., co-training or self-training), and the potential leakages. Broadly speaking, those transfer learning models can be categorized into three types, i.e., model-based, mapping-based, and parameter-based, as illustrated in Figure 1.

• •

  Model-based (in Figure 1(a)) is a simple but effective transfer learning paradigm, where the pre-trained source domain model is used as the initialization for continued training on the target domain data. Model-based fine-tuning has been broadly used to reduce the number of labeled data needed for learning new tasks/tasks of new domains (He et al., 2016; Howard and Ruder, 2018; Devlin et al., 2019).

• •

  Mapping-based (in Figure 1(b)) methods aim to align the hidden representations by explicitly reducing the marginal or conditional distribution divergence between source and target domains. More specifically, alignment losses, i.e., usually in forms of distribution discrepancies/distance metrics such as MMD and JMMD, between domains are measured by such feature mapping approaches and minimized in the loss functions (Long et al., 2015, 2017; Rozantsev et al., 2019).

• •

  Parameter-based (in Figure 1(c)) methods transfer knowledge by jointly updating a shared partial network to learn transferable feature representations across domains in a multi-task fashion. This type of methods is mainly achieved by parameter sharing. Regardless of the design differences, they all utilize a shared network structure to transform the input data into domain-invariant hidden representations (Yosinski et al., 2014; Liu et al., 2017; Yang et al., 2017).

Based on the general categorization, we will discuss the threat model, information leakage and the customized attacks against each transfer learning paradigm in detail.

In this work, we assume all the parties , i.e., the domain-specific data owners, are semi-honest, where they follow exactly the computation protocols but may try to infer as much information as possible when interacting with the other parties. More specifically, we work on the threat model under a deep transfer learning setting with two domains. Without loss of generality, we assume the owner of the target dataset to be the attacker, who intentionally attempts to infer additional information from source domain data beyond what is explicitly revealed. Depending on the different deep transfer learning categorizations, the attacker may have different access to the source domain information. Note that we can naturally extend to the case where the attacker is the owner of the source dataset or the transfer learning setting with more than two data sources, however, it is not the focus of this paper.

As presented in Table 1, three types of model categorization require different information interactions and training strategies, thus posing different potential leakage profiles. More specifically:

• •

  Model-based methods rely on the pre-trained source domain model solely. Despite the necessity for a pre-trained model to be disclosed to the target domain, both the source and target training processes can be entirely separately, thus the potential leakage is only the final source domain model $\left\langle\mathbb{M}^{S}\right\rangle$. In this case, the attacker has full access to the source domain model including both the model structure and parameters.

• •

  Mapping-based methods are optimized in a co-training fashion and hidden presentations of both domains have to interact with each other to measure the alignment losses or domain regularizers. We denote $h_{ik}^{S}$ and $h_{ik}^{T}$ as the hidden representation of layer $i$ at a training iteration $k\in[1,2,...,K]$ for source and target domains, respectively. Specifically, such a feature matching process demands the hidden presentations, i.e., $h_{ik}^{S}$ and $h_{ik}^{T}$, of both domains to be exposed and aligned to reduce the marginal or conditional distribution divergence between domains. As a result, $\left\langle h^{S},h^{T}\right\rangle$ can potentially leak information from the training data.

• •

  Parameter-based methods jointly updating the shared partial network. The interactions between the source and target happen when exchanging the gradients to sync the shared network parameters. Let $w_{k}^{c}$ denote the model parameters for the shared network structure. At each training iteration $k$, $w_{k}^{c}$ is updated by averaging the gradients $\bigtriangledown_{W}$ learned by a mini-batch of training examples $z_{i}$ sampled from a domain dataset (either source or target). In such an alternating process, $w_{k}^{c}$ has to be revealed across domains at each iteration $k\in[1,2,...,K]$. Thus, the potential leakage profile $\left\langle w_{k}^{c}\right\rangle$ contains all the intermediate $w_{k}^{c}$ during the training process.

As a summary, in this paper, we focus on inferring information that is unintentionally disclosed by the data transmission between the source and target domains, such as membership information of an individual sample and property information of a specific sample or a subset of samples (see more details in the next subsection.).

Inference attacks against deep learning models generally exploit implicit memorization in the models to recover sensitive information from the training data. Previously studies have proven that information can be inferred from the leakage profile, such as membership information that decides whether a sample is used for training (Li et al., 2013; Shokri et al., 2017; Long et al., 2018; Melis et al., 2019a; Wu et al., 2019a), properties which reveals certain data characteristics, or reconstructed feature values of the private training records.

In this part, to empirically evaluate the privacy leakage in different transfer learning settings (shown in Figure 1), we present concrete attack methods for each setting. Note, we assume the attacker to be the owner of the target dataset for all the three transfer learning paradigms discussed above.

In this paper, we employ a widely used technique named shadow model training for building the membership predictor. Specifically, we assume the attacker can have extra knowledge about the source dataset $\mathcal{D}^{S}$. The extra prior knowledge in our setting is the shadow training dataset $\mathcal{D}_{shadow}$ that comes from the same underlying distribution as $\mathcal{D}^{S}$ for training the source model. The core idea of the shadow training dataset is to first train multiple shadow models that mimic the behavior of the source model. Then, the attacker can extract useful features from the shadow models/datasets and build a machine learning model that characterizes the relationship between the extracted features and the membership information.

To be specific, the attacker first evenly divides the shadow training dataset $\mathcal{D}_{shadow}$ into two disjoint datasets $\mathcal{D}_{shadow}^{train}$ and $\mathcal{D}_{shadow}^{out}$. Then the attacker trains the shadow model that has the same architecture as the source model using $\mathcal{D}_{shadow}$. Subsequently, features of samples from both $\mathcal{D}_{shadow}^{train}$ and $\mathcal{D}_{shadow}^{out}$ can be extracted. For each sample in $\mathcal{D}_{shadow}$, the attacker can use the output prediction vector of the shadow model as the feature following the prior work (Shokri et al., 2017). And each feature vector is labeled with 1 (member, if the sample is in $\mathcal{D}_{shadow}^{train}$) or 0 (non-member, if the sample is in $\mathcal{D}_{shadow}^{out}$). At last, all the feature-label pairs are used for training the membership predictor.

Once the membership predictor is obtained, we can use it to predict the membership label of a sample in $\mathcal{D}^{S}$, i.e., feeding the output vector of the source domain model to the predictor.

The core idea of the mapping-based method is to align the hidden features extracted from the source and target domains, i.e., reducing the discrepancy between feature distributions of these two domains. This can be done by minimizing some pre-defined alignment loss function (e.g., maximum mean discrepancy(Long et al., 2015, 2017; Rozantsev et al., 2019; Nasr et al., 2019a)). Thus, this method will lead to the hidden features of the source domain share a similar distribution to the features of the target domain, which comes with a strong privacy implication, i.e., the attacker can leverage the feature similarity to build the attack model to detect sensitive information contained in the source domain.

We consider the property attack (Ganju et al., 2018) in this setting. At a high level, the property attack aims to infer whether a feature comes from the source domain has a specific property or not. Here, we take the case of the $k$-th iteration as an example. Given a specific property, the attacker can first collect an auxiliary dataset for assisting the attack. Specifically, the attacker divides the target domain dataset $\mathcal{D}^{T}$ into two subsets, namely, $\mathcal{D}^{T}_{prop}$ which contains samples with the property and $\mathcal{D}^{T}_{nonprop}$ which consists of samples without the property. Subsequently, $\mathcal{D}^{T}_{prop}$ and $\mathcal{D}^{T}_{nonprop}$ are used as the auxiliary datasets for building the attack model. At the $k$-th iteration, given the current parameter $w_{k}$ of the model trained using the target dataset, the attacker calculates the hidden features $h_{k}^{T}$ of the alignment layer $k$ with respect to the samples in the auxiliary dataset. Hidden features are labeled with 1 if the corresponding sample is in $\mathcal{D}^{T}_{prop}$ and 0 if the sample is in $\mathcal{D}^{T}_{nonprop}$. Once the attacker collects all the feature-label pairs, she can train a property predictor $\mathcal{A}_{prop}$ using these pairs. The whole procedure is demonstrated in Algorithm 1.

Based on the property predictor obtained above, the attacker can conduct an online attack. Specifically, in the joint training process, at the beginning of the $k$-th iteration, the attacker receives a batch of hidden features $h^{S}_{k}$ from the source domain. Then, the attacker can employ $\mathcal{A}_{prop}$ to predict the property information contained in the source domain dataset.

We consider the batch property attack in this setting. The intuition behind this attack is that the attacker can observe the updates of the share layers calculated using a mini-batch of samples from the source domain. Thus the attacker can train a batch property predictor $\mathcal{A}_{bprop}$ to infer whether the update based on the mini-batch with a given property or not. We take the $k$-th iteration as an example to demonstrate how the attacker conducts the attack. We assume that the attacker has auxiliary dataset $D^{aux}$ consisting of samples from the distribution that is similar to the source domain distribution. Note that, this assumption is commonly used in various previous works (Sharma et al., 2019; Melis et al., 2019b). Given a specific property, the attacker can further divide $D^{aux}$ into two sub-datasets, namely, the dataset which has the property ($D^{aux}_{prop}$) and the dataset without the property ($D^{aux}_{nonprop}$).

Based on the above setting, at the $k$-th iteration, the attacker can receive the fresh parameter $w_{k}$ of the shared layer which is updated based on samples from the source dataset. Then, the attacker can calculate gradients $g_{prop}$ using the mini-batch sampled from $D^{aux}_{prop}$ and $g_{nonprop}$ based on samples from $D^{aux}_{nonprop}$. The batch gradients based on $D^{aux}_{prop}$ are labeled with 1 and others are labeled with 0. Based on these gradient-label pairs, we can train the batch property predictor $\mathcal{A}_{bprop}$. The whole procedure is shown in Algorithm 2.

Once $\mathcal{A}_{bprop}$ is obtained, the attacker can use it to predict the btach property information of the source domain. Specifically, the attacker first generates the gradient $g_{online}$ based on the parameters of the current and last iteration (i.e., $w_{k}$ and $w_{k-1}$). Then she can feed $g_{online}$ to $\mathcal{A}_{bprop}$ to obtain the prediction result.

We conduct experiments on two public and one industry datasets, i.e., Review, UCI-Adult, and Marketing. The Review dataset is used to examine membership attack as fine-tuning methods are often used in such NLP task and typically suffer from membership inference. While the UCI-Adult dataset has some sensitive properties, thus is used for conducting property based attacks. Besides, the Marketing dataset is sampled from real-world marketing campaigns to examine the defense solution. Data statistics are in Table 2.

Following the literature on this task (Chen et al., 2019), we adopt the TextCNN (Kim, 2014) as the base model for textual representation in the transfer learning model. For TextCNN, we set the filter windows as 2,3,4,5 with 128 feature maps each. The max sequence length is set as 60 for the task. We initialize the embedding lookup table with pre-train word embedding from GloVe with embedding dimension set as 300. The batch sizes for training the source domain model, the shadow model, and the attack model are set as 64.

All the above neural network based transfer learning models are implemented with TensorFlow and trained with NVIDIA Tesla P100 GPU using Adam optimizer, if not specified. The learning rate is set as 0.001. The activation function is set as ReLU.

For all the settings, we use Area Under Curve (AUC) to evaluate the overall attack performance. For membership attack, we also adopt precision for evaluation, as membership of the training dataset is the concern. Precision is defined as the fraction of samples predicted as members are indeed members of the training data. For property attacks, we further include accuracy for evaluating the overall attack performance.

In the model-based setting, the attacker has the full white-box access to the source model, thus the attacker is able to train shadow models to mimic the behavior of the source domain mode. In this setting, we explore the possibility of performing membership inference on the source model. Despite the context differences, membership attacks for model-based transfer learning models can be conducted the same way as for any trained stand-alone deep learning models(Shokri et al., 2017; Salem et al., 2019; Melis et al., 2019b).

To build the membership predictor introduced in Section 3, we split the original source dataset into three parts, namely, training dataset (248,011 samples), test dataset (70,860 samples), and shadow dataset (70,860 samples). We set the number of shadow models to 3. Table 3 shows the training and testing accuracy for the attack classifier on the shadow dataset and the attack performance, i.e., AUC and precision, on the source domain model. Smaller gaps between training and testing accuracy suggests better generalization and predictive power of the attack classifier. Overall, we find that even in the model-based setting without direct interactions between the source and target domains during the training process, the source domain model does leak a considerable amount of membership information. We also examine the effect of over-fitting by adjusting the number of epoch trained for both the source domain and shadow models. We observe the more over-fitted a model, the more information can be potentially leaked. However, it decreases when the number of epochs furthers increases. This echos the findings in (Shokri et al., 2017) that over-fitting is an important factor but not the only factor that contributes to the leakage of membership information.

We then investigate property attacks under the mapping-based transfer learning setting to infer the properties of the source domain data during the training process. The properties are not the same as the prediction label, or even independent of the prediction task. To examine this, we construct two more datasets based on the UCI adult dataset: Prop-race and Prop-sex, discussed as follows.

• •

  Prop-sex: we filter out the property “sex” feature from the input features in UCI-Adult data and use it as the attack task label. The attack label is set as 1 if the property “sex” is male and 0 otherwise. This is to examine whether a attacker can infer training instance’s gender during the training process.

• •

  Prop-race: similarly we filter out the “race” feature to study whether this property appears in training instances. Attack label is set as 1 if property “race” is white and 0 otherwise.

Table 4 shows the results of the property attack on the Prop-sex dataset. The attack has a good AUC of 0.77 in the property inference task, which shows the transfer learning setting does have information leakage. The attack precision is high on the property “sex:male”, and less satisfactory on “sex:female”. The female property has much less instances and thus may be harder to be predicted.

We have also conducted experiments on another property “race” in Table 5, the attack AUC is lower than on Prop-sex, 0.5885 vs. 0.7766. These results demonstrate that Prop-sex leaks more property information than Prop-race in terms of attack AUC. At the first glance, “race: white” has much higher attack precision than “race:non-white”, 0.8901 vs. 0.2179. Close examination shows “race: white” dominates the training data with around 80% label coverage.

We further examine the correlation between the chosen properties and the underlying Census Income prediction task, and compare it with the property attack results in Table 6. In general, both Prop-race and Prop-sex have information leakage issues, and the property with a higher negative/positive correlation with the main prediction task tends to cause higher potential information leakage.

For the parameter-based setting, we further process the UCI adult data to perform the batch property attack. Again we use these two properties “race” and “sex” and form two datasets namely BProp-race and BProp-sex respectively. For both datasets, we set the batch size as 8. For each batch in BProp-race, we set the label as 1 if a batch has at least one data instance containing the “non-white” property and 0 otherwise. For BProp-sex, we set the label as 1 if a batch has at least one instance containing the “female” property and 0 otherwise. The data statistics are shown in Table 7. We find for BProp-race, the positive instance ratio in the source domain is drastically different from the target, this may bring negative transfer. While for the BProp-sex, the positive instance ratio is similar for both domains.

As shown in Algorithm 2, the selection of $\mathcal{D}^{aux}$ is the key to building the batch property predictor. In this paper, we directly use the target dataset as the auxiliary dataset and find it works in practice, as both domains commonly are related for the transferring purpose. Generally, we can obtain better attack performance if we can use part of source domain samples to form the auxiliary dataset. The results are presented in Table 8. First, we find the attack AUC for the batch property attack in the parameter-based setting are generally not as high as those from the property attack in the mapping-based setting. The AUC is around 0.56 in batch property attack, while up to 0.77 for property attack. This shows the parameter-based setting is generally less vulnerable from attacks, which can possibly be justified by the fact that the gradients exchanged have been averaged at the batch-level. Second, the results for BProp-sex are better than BProp-race, as AUC results are 0.5654 and 0.5545 respectively. This is intuitive as observed from Table 7, the domain difference in BProp-race is larger than the BProp-sex. The model performance in BProp-race may suffer from the negative transfer learning due to the domain gap, thus may lead to the decrease in attack performance. Overall, the parameter-based method is possible to suffer from batch property attacks, however, the information leakage problem is not that severe.

A standard way to prevent statistical information leakage is differential privacy. However, the privacy guarantee provided by differential privacy comes with the decreasing of the model utility. Some recent works proposed to relax the requirement of differential privacy to prevent membership/property attacks while provide better model utility. For example, some regularization techniques such as dropout are helpful to prevent the information leakage of machine learning models (Wu et al., 2019a; Melis et al., 2019b). Recent studies consider to use Stochastic gradient Langevin dynamics (SGLD) (Wu et al., 2019a; Mou et al., 2018). In this paper, to prevent the information leakage in deep transfer learning, we employ SGLD as our optimizer to optimize deep models and show its effectiveness in reducing information leakage. Prior work (Wu et al., 2019a) demonstrates that SGLD is effective in preventing membership information leakage and provides theoretical bounds for membership privacy loss. Empirically, in this paper, SGLD is also effective for preventing the leakage of the property information while provides comparable model performance compared with those non-private training methods.

To examine the effectiveness of the defense solutions, we conduct experiments under the mapping-based setting. Specifically, we replace the non-private optimizer (i.e., SGD) with SGLD and train the source and target models from scratch. The overall result is shown in Table 9. We observe that the use of SGLD can prevent the information leakage of the training dataset, based on the above attack metrics for evaluating the information leakage. In contrast to the original optimizer, SGLD significantly reduces the attack AUC score of the inference attack from 0.7766 to 0.6862 in Prop-sex, and 0.5885 to 0.5442 in Prop-race, while achieves better model utility in terms of the task AUC score especially in Prop-race (0.7750 to 0.8012). In both datasets, SGLD achieves the best task AUC and slightly outperforms the original task results. We infer the boost of the task AUC score is due to the regularization effect of the noises injected in SGLD, which can prevent model overfitting as pointed out in the previous works (Wu et al., 2019a; Mou et al., 2018).

We also conduct experiments of DP-SGD introduced in the prior work (Abadi et al., 2016). As the result shows, although DP-SGD can achieve better anti-attack ability than SGLD, it comes with considerably model utility decrease (the task AUC score decreases from 0.8466 to 0.7662 in Prop-sex, and 0.7750 to 0.7192 in Prop-race ). Furthermore, experiments on Dropout are also performed. We can observe that Dropout can also prevent privacy leakage. We also note that when the drop ratio increases, the anti-attack ability of Dropout increases but model utility decreases drastically. As a conclusion, through experiments, we found that SGLD can be a good alternative to differentially private optimization methods and it can achieve a better trade-off between model utility and anti-attack ability.