Privacy-preserving Federated Adversarial Domain Adaptation over Feature Groups for Interpretability

Traditional domain adaptation (DA) approaches assume the data are centralized on one server, thus limiting their applicability to decentralized real-world scenarios. Federated domain adaptation aims to conduct domain adaptation modeling among independent parties of different domains without violating privacy. [24] applies a mixture of experts (MoE) strategy that each participant combines a collaboratively-learned general model and a domain-tuned private model to reconcile distribution differences among participants. [22] leverages federated adversarial domain alignment with a dynamic attention mechanism to enhance knowledge transfer. [16] applies methods proposed by [24, 22] to functional magnetic resonance imaging (fMRI) analysis. [20] proposes agnostic federated learning aiming to optimize the global model for any target distribution formed by a mixture of client distributions without overfitting data of any particular client. One major limitation of those (both traditional and federated ) DA approaches is that they almost use computer vision datasets, and only a few of them (e.g., [20]) are evaluated on tabular data.

Protecting privacy is a crucial element of federated learning. Homomorphic encryption (HE) is one of the major solutions to address the privacy issue. Although HE is a promising solution that allows computation to be performed on encrypted data, its expensive computational cost makes it impractical to be applied in training an entire DNN model. To address this issue, GELU-NET [32] adopts a client-server architecture in which the client encrypts the data while the server performs most computation on encrypted data. ACML [33] focuses on a more enterprise scenario where data and labels are distributed among two independent parties. It adopts a SplitNN [29] approach that each party is only responsible for updating its own portion of the whole DNN model. The novelty of ACML is that the costly encryption-decryption operations are only performed on the two partial models’ boundaries, leaving the rest of the computation in plaintext.

A variety of research works have been proposed for interpreting deep neural networks [21]. These methods focus on post-hoc interpretability that analyzes the relationships between input and output of the trained model rather than elucidating models’ internal structures. Other methods [3, 13, 1] construct prototypes or general concepts that shed light on the decision-making process. e.g., [3] propose ProtoPNet that learns a set of prototypes each can be considered as the latent representation of a small prototypical part of training images. Then, the label prediction can be calculated based on a weighted combination of the similarity scores between parts of the image and the learned prototypes. [18] calculates SHAP values of every feature for every sample based on model prediction. Complex models, such as ensemble methods or deep networks, can be explained through these SHAP values.

The essential idea of adversarial domain adaptation is to train feature extractors that are able to learn features that are both discriminative to the task and invariant to the change of domains. Thus, we have two optimization goals. (1) In order to obtain domain-invariant features, we perform adversarial domain adaptation to optimize the feature extractors that maximizes the domain classification losses, while simultaneously optimize the domain discriminators that minimizes the domain classification loss. (2) To obtain task-specific discriminative feature representations, we perform vertical federated learning (VFL) to optimize the feature extractors and label predictor that minimize the label prediction loss.

In our federated learning setting, party C leverages $g$ feature extractors $\mathscr{F}=\{F_{i}\}_{i=1}^{g}$ and their corresponding $g$ domain discriminators $\mathscr{D}=\{D_{i}\}_{i=1}^{g}$ to learn domain-invariant feature representations from $g$ feature groups. More specifically, the $i$th feature extractor $F_{i}$ learns feature representation from the $i$th feature group and then the $i$th domain discriminator $D_{i}$ maps this feature representation to a domain label $d\in\{0,1\}$. The overall domain classification loss is the sum of domain classification losses for all domain discriminators in $\mathscr{D}$. We give the formula as follows:

To make feature extractors obtain task-specific discriminative features, we optimize label prediction loss to train both label predictor and feature extractors to classify the source samples correctly. We define the label prediction loss as:

where $R^{B}$ is the label predictor curated at party B, $\boldsymbol{\mu}^{B^{c}}$ is the high-order feature vectors passed from party C and $\mathbf{x}^{B}$ is the feature vectors possessed by party B.

The pre-training stage optimizes the two losses presented in (2) and (3). We have the complete loss function for pre-training stage as follows:

where $\lambda$ is a hyperparameter that controls the trade-off between the two losses that shape the feature representations during training.

In our federated setting, $L_{adv}(\mathscr{F},\mathscr{D})$ is optimized locally at party C since it only involves data of party C, while $L_{ce}(\mathscr{F},R^{B})$ is optimized collaboratively by party B and party C in a federated manner since it involves data from the two parties. To this end, we train parameters $\{\theta_{f_{i}}\}_{i=1}^{g}$ of feature extractors $\mathscr{F}$, $\{\theta_{d_{i}}\}_{i=1}^{g}$ of domain discriminators $\mathscr{D}$, and $\theta_{R^{B}}$ of label predictor $R^{B}$ by solving following two optimization problems.

$\{\theta_{d_{i}}\}_{i=1}^{g}$ are trained by minimizing the domain classification loss, $\theta_{R^{B}}$ is trained by minimizing the label prediction loss, and $\{\theta_{f_{i}}\}_{i=1}^{g}$ are trained by minimizing the label prediction loss while at the same time maximizing the domain classification loss.

Figure 2 illustrates the overall workflow of the pre-training stage and Algorithm 1 describes the procedure of optimizing (5) and (6). We assume that the entity alignment procedure has been run and the indices of $\mathbf{D}^{s}$ have been shuffled and synchronized between party B and party C before training. For each inner iteration, party C and party B fetch the same mini-batch of aligned samples from $\mathbf{D}^{s}$ but each holds its own portion of private data: party C holds $\mathbf{x}^{B^{c}}$ while party B holds $\mathbf{x}^{B}$ and $\mathbf{y}^{B}$. In addition, Party C samples a mini-batch $\mathbf{x}^{A^{c}}$ from its target data $\mathbf{X}^{A^{c}}$. Based on $\mathbf{x}^{B^{c}}$ and $\mathbf{x}^{A^{c}}$, party C optimizes (5) locally. Based on $\mathbf{\mu}^{B^{c}}$, $\mathbf{x}^{B}$ and $y^{B}$, party B and party C collaboratively optimize (6) through Algorithm 3.

The fine-tuning stage aims to train the label predictor $R^{A}$ possessed by target party A using target labeled data $\mathbf{D}^{t}_{l}$. Prior to fine-tuning, party C initializes its feature extractors with pre-trained parameters. Note that since party A and party B are two independent parties, the trained label predictor $R^{B}$ of of source party B cannot be fine-tuned by $R^{A}$ of target party A. Thus, $R^{A}$ has to be trained from the scratch.

For each iteration, party C applies (1) to compute feature vectors $\boldsymbol{\mu}^{A^{c}}$ and then send $\boldsymbol{\mu}^{A^{c}}$ to party A for computing the label prediction loss:

Algorithm 2 describes the fine-tuning procedure and it is quite similar to Algorithm 1 except that it does not require party C to optimize (5).

Algorithm 4 aims to compute the label prediction loss in (3) without compromising the data privacy of participating parties. To achieve this, the party C encrypts $\mu^{C}$ with PHE and sends encrypted $[[\mu^{C}]]$ to party $p$ to prevent privacy leakage. When receiving $[[\mu^{C}]]$, party $p$ can compute logit $z$ according to (9). However, directly applying (9) yields $[[z]]$ that is not compatible with logistic function. The workaround is that party $p$ first computes $[[\tilde{z}^{C}]]$ (Algo 4, line 5) and sends it to party C with random noise $\epsilon^{p}$ (Algo 4, line 8). The party C then decrypts $[[\tilde{z}^{C}+\epsilon^{p}]]$ and adds it with $\boldsymbol{\mu}^{C}\varepsilon^{C}_{t}$, which is for cancelling out the accumulated random noise, denoted as $\varepsilon^{C}_{t}$, that was embedded in $\widetilde{\mathbf{W}}_{t}^{C}$ during the backpropagation of the previous iteration. For now, we assume $\widetilde{\mathbf{W}}_{t}^{C}=\mathbf{W}_{t}^{C}-\varepsilon^{C}_{t}$, which we will prove in section 7.2. Here, we prove the logit $z^{C}$ is calculated correctly (Algo 4, line 11):

As a result, party C has $z^{C}+\epsilon^{p}$. The noise $\epsilon^{p}$ is for preventing party C from accessing the plaintext $z^{C}$ and further collecting $\mathbf{W}_{t}^{C}$ = $z^{C}/\boldsymbol{\mu}^{C}$. Finally, the party $p$ computes the loss $\ell_{ce}(\sigma(z),\mathbf{y}^{p})$.

During the privacy-preserving backward propagation as described in Algorithm 5, the active party $p$ securely updates logistic regression model $R^{p}$ and backpropagate gradients to party C. As shown in (9), we partitioned weights of $R^{p}$ into $\mathbf{W}^{p}$ and $\mathbf{W}^{C}$. On the one hand, party $p$ can compute gradients $\Delta\mathbf{W}^{p}_{t}$ and $\Delta b^{p}_{t}$, and update $\mathbf{W}^{p}_{t}$ and $b^{p}_{t}$ (Algo 5, line 20-21) in plaintext since party $p$ owns these parameters. On the other hand, party A can not directly update $[[\mathbf{W}^{C}_{t+1}]]\xleftarrow{}\mathbf{W}^{C}_{t}-\eta[[\Delta\mathbf{W}^{C}_{t}]]$ since this leads to incompatibility with PHE for computing $[[\tilde{z}^{C}]]\xleftarrow{}[[\boldsymbol{\mu}^{C}]]\otimes[[\mathbf{W}^{C}_{t+1}]]$ in the next iteration of forward propagation. To work around this issue, party $p$ may send encrypted gradients $[[\Delta\mathbf{W}^{C}_{t}]]$ to party C and get the decrypted $\Delta\mathbf{W}^{C}_{t}$ back. However, this leads to privacy leakage for both parties. Because based on $\Delta\mathbf{W}^{C}_{t}=\boldsymbol{\mu}^{C}\otimes\delta_{l}$, knowing $\Delta\mathbf{W}^{C}_{t}$ party $p$ can infer the value of $\boldsymbol{\mu}^{C}$, while party C can infer gradient $\delta_{l}$ during training. Therefore, to conceal the real value of $\Delta\mathbf{W}^{C}_{t}$ from both parties, the two parties mask $\Delta\mathbf{W}^{C}_{t}$ by adding corresponding random noises. Specifically, party $p$ adds noise $\epsilon^{p}$ to $[[\Delta\mathbf{W}^{C}_{t}]]$ and sends $[[\Delta\mathbf{W}^{C}_{t}+\epsilon^{p}]]$ to party C (Algo 5, line 9-10). Party C in turn decrypts $[[\Delta\mathbf{W}^{C}_{t}+\epsilon^{p}]]$ and sends $\Delta\widetilde{\mathbf{W}}^{C}_{t}+\epsilon^{p}$ back to party $p$, where $\Delta\widetilde{\mathbf{W}}^{C}_{t}=\Delta\mathbf{W}^{C}_{t}+\frac{\epsilon^{C}}{\eta}$ (Algo 5, line 13), $\epsilon^{C}$ is the random noise generated by party C and $\eta$ is the learning rate. Then, the party $p$ updates $\widetilde{\mathbf{W}}^{C}_{t+1}$ based on gradient $\Delta\widetilde{\mathbf{W}}^{C}_{t}$ after removing noise $\epsilon^{p}$ (Algo 5, line 19). Note that while the noise $\epsilon^{p}$ can be removed by party $p$, the noise $\epsilon^{C}$ added by party C is accumulated in weight $\widetilde{\mathbf{W}}^{C}_{t}$ through $\Delta\widetilde{\mathbf{W}}^{C}_{t}$ at each iteration. Intuitively, the real value of $\mathbf{W}^{C}_{t+1}$ can be seem as shared by party C and party $p$. This concept is similar to secret sharing.

For party $p$ to correctly calculate the intermediate gradient $\delta^{C}$, party $p$ needs to cancel out the accumulated noise embedded in $\widetilde{\mathbf{W}}^{C}_{t+1}$. To this end, party C seeds the encrypted accumulated noise $[[\varepsilon^{C}_{t+1}]]$ to party $p$, which then calculates the gradient $[[\delta^{C}]]$ of loss $\ell_{ce}$ with respect to $\mu^{C}$ (Algo 5, line 22) using $[[\varepsilon^{C}_{t+1}]]$. To prove the value of gradient $\delta^{C}$ is calculated correctly, we prove that $\widetilde{\mathbf{W}}^{C}_{t+1}=\mathbf{W}^{C}_{t+1}-\varepsilon^{C}_{t+1}$ by mathematical induction, assuming $\widetilde{\mathbf{W}}^{C}_{t}=\mathbf{W}^{C}_{t}-\varepsilon^{C}_{t}$ and Initializing $\varepsilon^{C}_{0}=0$:

Finally, party $p$ sends $[[\delta^{C}]]$ back to party C, which decrypts $[[\delta^{C}]]$ and backpropagates $\delta^{C}$ locally to optimize local models.

In this section, we discuss the privacy-preserving capability of our PP-VFL, its possible privacy leakage and trade-off.

Proof. There are three ways through which party $p$ can leverage to recover the true value feature vectors $\boldsymbol{\mu}^{C}$ during training. The first way is to decrypt $[[\boldsymbol{\mu}^{C}]]$ directly. However, it is impossible to decrypt $[[\boldsymbol{\mu}^{C}]]$ without knowing the private key. The second way is to derive $\boldsymbol{\mu}^{C}$ from $z^{C}/\widetilde{\mathbf{W}}_{t}^{C}$ according to (10). However, this requires party $p$ to remove the noise $\varepsilon^{C}_{t}$ from $\widetilde{\mathbf{W}}_{t}^{C}$. Suppose $\varepsilon^{C}_{t}$ is the random noise accumulated by party C at iteration $t$ and $\hat{\varepsilon}^{C}_{t}$ is an attempt from party A. The probability that $\varepsilon^{C}_{t}=\hat{\varepsilon}^{C}_{t}$ is $Pr(\varepsilon^{C}_{t}=\hat{\varepsilon}^{C}_{t})\leq(1-e^{-2/\mathbb{|Z|}})$ [26]. Because $\mathbb{|Z|}$ is typically a very large number, the $Pr(\varepsilon^{C}_{t}=\hat{\varepsilon}^{C}_{t})$ is very close to zero. Third, $\boldsymbol{\mu}^{C}$ can also be derived from $\Delta\mathbf{W}^{C}_{t}/\delta^{l}$ after the noise $\epsilon^{C}$ being removed from $\Delta\widetilde{\mathbf{W}}^{C}_{t}$. However, the probability $Pr(\epsilon^{C}=\hat{\epsilon}^{C})$ that the attempt $\hat{\epsilon}^{C}$ made at party $p$ equals $\epsilon^{C}$ approximates zero.

Proof. There two ways that party $p$ can infer $\mathbf{W}^{C}$. One is via $z^{C}/\boldsymbol{\mu}^{C}$ according to (10), and another is removing accumulated noise $\varepsilon^{C}$ from $\widetilde{\mathbf{W}}^{C}$. According to Proposition 1 the true value of $\boldsymbol{\mu}^{C}$ is concealed from the party $p$ during training and inference, and the probability $Pr(\varepsilon^{C}=\hat{\varepsilon}^{C})\leq(1-e^{-2/\mathbb{|Z|}})$ that the party C can generate noise $\hat{\varepsilon}^{C}$ to cancel out $\varepsilon^{C}$ is close to zero. Therefore, the party $p$ cannot infer the true value of $\mathbf{W}^{C}$ during training and inference.

The active party $p$ cannot infer the data of passive party C during training because party $p$ has no access to $\boldsymbol{\mu}^{C}$, $\mathbf{W}^{C}$ and party C’s local model. [12] proposes model inversion (MI) that enables the attacker (i.e., party $p$) to recover the private data of the victim (i.e., party C) during inference. To recover data of reasonably high quality, [12] makes strong assumptions that the attacker knows the network structure of the victim and has access to the training data that follows the same distribution as those of the victim, aiming to approximate the victim’s local model. However, these assumptions typically do not hold in scenarios like finance. For one thing, financial data generally are not publicly available because they are sensitive, and thus their publication is regulated. For another, participating parties provide heterogeneous features in VFL, and thereby they typically adopt different model structures. Besides, [12] demonstrates that a local model with a full-connected layer on top can significantly drop the quality of the recovered data. The $\mathbf{W}^{C}$ automatically provides such a layer of protection.

Proof. The weights $\mathbf{W}$ is composed of $\mathbf{W^{C}}$ and $\mathbf{W}^{p}$. Party C receives no information on $\mathbf{W}^{p}$ of party $p$. Therefore, party C can learn nothing on $\mathbf{W}^{p}$. There are two ways that party C can infer $\mathbf{W}^{C}$. The first one is via $(z^{C}+\epsilon^{p})/\boldsymbol{\mu}^{C}$ and the another is via $\delta^{C}/\delta^{l}$. The noise $\epsilon^{p}$ in the former one prevents party C from revealing $\mathbf{W^{C}}$ because the probability $Pr(\epsilon^{p}=\hat{\epsilon}^{p})\leq(1-e^{-2/\mathbb{|Z|}})$ that the party C can generate noise $\hat{\epsilon}^{p}$ to cancel out $\epsilon^{p}$ is close to zero, while $\delta^{l}$ in the latter one resides only in party $p$. Therefore, the party C cannot reveal the true value of $\mathbf{W^{C}}$.

Recent research works propose that the attacker (i.e., passive party C) can leverage gradient inversion (GI) [35], model completion (MC) [8], and properties of cut-layer gradient (PCG) [15] to recover labels of the victim (i.e., active party $p$). Our PP-VFL can prevent GI attack because the attacker has no access to both the weights (i.e., $\mathbf{W}$) and the gradient of the label predictor model [14, 10]. While PP-VFL itself cannot prevent the MC and PCG attacks in that the cut-layer gradient $\delta^{C}$ is passed to the attacker in plain text and without any protection. The current form of PP-VFL trades a certain degree of increased label privacy leakage with enhanced model performance and training efficiency. In applications where the labels are important assets, the PP-VFL can be equipped with other privacy protection mechanisms (e.g., MARVELL [15] for PCG and CoAE [17] for MC) to trade the protection of label privacy with a certain degree of degraded utility.

We evaluate our proposed PrADA based on two datasets: one is Census Income dataset, and another is a real-world financial dataset called Loan Default. For each dataset, we run experiments under following four settings:

1. 1.

   $\mathbf{A}$-$\mathbf{Local}$: Target party A only uses its local data to train models without leveraging VFL and DA.

2. 2.

   $\mathbf{A}$-$\mathbf{VFL}$: Target Party A uses target domain data $\mathbf{D}^{t}_{l}$ to train models via VFL with party C. This setting serves as the conventional VFL to improve the model performance of party A with additional features from party C.

3. 3.

   $\mathbf{AB}$-$\mathbf{VFL}$: Assuming party A and B are from the same organization and privacy is not a concern. Thus, Party A uses $\mathbf{D}^{t}_{l}$ and $\mathbf{D}^{s}$ of both domains to train models via VFL with party C (with no DA). Models in this setting serve as strong baselines because they use all data together.

4. 4.

   $\mathbf{B}\rightarrow\mathbf{A}$: We conduct PrADA elaborated in sections 6 to perform federated adversarial domain adaptation from party B to party A.

In settings 2 and 3, we adopt SecureLR and SecureBoost implemented in FATE¹¹1https://github.com/FederatedAI/FATE, an industrial grade federated learning framework, as comparing models. These two models are VFL version of tree-boosting model and logistic regression model respectively, and they are using PHE to protect data privacy. To explore the effectiveness of different components of PrADA, we propose three different ablations:

• •

  $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ without domain adaptation (DA), feature grouping (FG) and feature group interaction (IR);

• •

  $\text{PrADA}_{\text{w/o FG\&IR}}$ applies domain adaptation, but without feature grouping and interaction;

• •

  $\text{PrADA}_{\text{w/o IR}}$ applies domain adaptation based on feature grouping, but without interaction.

In this paper, we focus on the binary classification problem. Because imbalanced class label is one of the major motivations for applying domain adaptation in real-world financial applications, we also investigate the effectiveness of our PrADA approach for different positive label ratios. Specifically, we investigate scenarios when the target training data has a positive label ratio {0.01,0.02,0.04}.

For SecureLR, we use default hyperparameters, while for SecureBoost, we sweep over all combination of max depth {2,4,6,8} and number of trees {100,200,300,400}, leaving other hyperparameters default. For PrADA, we use batch size 128 for Census Income data and 64 for Loan Default data, and use learning rate 0.0005 for pre-training and 0.0008 for fine-tuning for both datasets. Our PrADA is implemented with PyTorch. We repeat every experiment 5 times on each dataset, reporting the mean and standard derivation of AUC and KS (Kolmogorov-Smirnov test) [2] of all trained models on test data of the target party A.

After data preprocessing, the census income dataset contains 36 features, 31 of which are categorical. We put 5 numerical features on active parties (i.e., A and B) while the 31 categorical features on passive party C. We split 31 features on party C into 4 feature groups (FG) including employment(emp), demographics(demo), household(house), and migration(migr). Thus, we have $C^{4}_{2}$(i.e., 6) interactive feature groups, which are emp-demo, emp-house, emp-migr, demo-house, demo-migr, house-migr. We embed all categorical features into dense vectors. Table II shows the architecture of feature extractor for each of the 10 feature groups and the one (i.e., all_feat) for all features when feature grouping is not applied.

The experimental results are shown in Table I. From these results, we observe that: (1) SecureBoost and SecureLR in $\mathbf{A}$-$\mathbf{VFL}$ outperform their counterparts in $\mathbf{A}$-$\mathbf{Local}$ demonstrating that leveraging additional features improve the model performance. (2) The performance of models in $\mathbf{B}\rightarrow\mathbf{A}$ significantly outperforms that of models in $\mathbf{A}$-$\mathbf{VFL}$ across all settings of different positive labels. This is expected because a considerable amount of source data is involved in training. More specifically, when the number of positive labels is small (i.e., 40), the performance gain is the most significant. For example, PrADA outperforms $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ in AUC by 5.45% and in KS by 9.56%, and outperforms SecureBoost in AUC by 7.2% and in KS by 10.19% when the positive label is 40. (3) $\text{PrADA}_{\text{w/o FG\&IR}}$ in $\mathbf{B}\rightarrow\mathbf{A}$ outperforms $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ in $\mathbf{AB}$-$\mathbf{VFL}$ in AUC by 1.02% and in KS by 0.60% on average, and outperforms SecureBoost in AUC by 0.60% and in KS by 0.94% on average, demonstrating the effectiveness of PrADA on bridging the divergence between source and target domains. (4) In $\mathbf{B}\rightarrow\mathbf{A}$ setting, $\text{PrADA}_{\text{w/o IR}}$ outperforms $\text{PrADA}_{\text{w/o FG\&IR}}$ in AUC by 0.17% and KS by 0.58% on average, demonstrating the effectiveness of FG-based domain adversarial training on improving the transferability of feature extractors. In addition to boosting model performance, feature grouping also enhances the interpretability of target model $R^{A}$, which we discuss in section 8.4. (5) In $\mathbf{B}\rightarrow\mathbf{A}$ setting, PrADA outperforms $\text{PrADA}_{\text{w/o IR}}$ in AUC by 0.31% and KS by 0.70% on average, demonstrating the interaction on feature groups help improve model performance.

To dive deeper into the effect of FG-based domain adaptation on learned feature representations, we visualize the t-SNE embeddings [4] of the feature representations in Figure 3. Figure 3(a)-(d) and Figure 3(e)-(h) show the the t-SNE embeddings of the feature representations learned by $\text{PrADA}_{\text{w/o DA\&IR}}$ and $\text{PrADA}_{\text{w/o IR}}$, respectively, on feature groups of Census Income data. We observe that the adaptation in our method makes the two distributions of learned feature representations much closer in all feature groups.

After data preprocessing, the Load Default dataset has 162 features, 27 of which are categorical. For protecting privacy, user and feature names are anonymized. We put 6 demographics features and labels on active parties, while the rest 156 features on passive party C. We split features of party C into 5 groups including user location(loc), third-party period(period), education(edu), social network (soc), and micro-blog(mblog). Thus, we have $C^{5}_{2}$(i.e., 10) interactive feature groups, which are loc-period, loc-edu, loc-soc, loc-mblog, period-edu, period-soc, period-mblog, edu-soc, edu-mblog, soc-mblog. We embed all categorical features into dense vectors. Table IV shows the architecture of feature extractor for each of the 15 feature groups and the one (i.e., all_feat) for all features when feature grouping is not applied.

The experimental results are reported in Table III. From these, we observe that: (1) Table III reports a similar trend as Table I does that the performance of models improves from $\mathbf{A}$-$\mathbf{Local}$ setting to $\mathbf{A}$-$\mathbf{VFL}$ and then to $\mathbf{B}\rightarrow\mathbf{A}$ as more data is involved in training. The performance gains the most when the number of positive labels is small. Specifically, when the number of positive labels is 40, PrADA in $\mathbf{B}\rightarrow\mathbf{A}$ setting outperforms $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ in $\mathbf{A}$-$\mathbf{VFL}$ setting in AUC and KS by 12.49% and 20.55% respectively, and outperforms SecureBoost in AUC and KS by 17.87% and 28.83% respectively. As the number of positive labels increases, the performance gain narrows. (2) $\text{PrADA}_{\text{w/o FG\&IR}}$ in $\mathbf{B}\rightarrow\mathbf{A}$ setting outperforms $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ in $\mathbf{AB}$-$\mathbf{VFL}$ in AUC by 0.25% and in KS by 0.65% on average, demonstrating the effectiveness of PrADA on mitigating domain divergence. (3) In $\mathbf{B}\rightarrow\mathbf{A}$ setting, $\text{PrADA}_{\text{w/o IR}}$ outperforms $\text{PrADA}_{\text{w/o FG\&IR}}$, demonstrating the superiority of FG-based DA over conventional DA, and PrADA outperforms $\text{PrADA}_{\text{w/o IR}}$, demonstrating the interaction on feature groups help enhance model performance.

We demonstrate model interpretability by visualizing the impact of features on target model $R^{A}$ using SHAP [18], a tool widely used to explain black-box models. As discussed in section 7, the real values of model parameters of $R^{A}$ are not accessible by either party C or party A. This means that party A cannot interpret the model by simply looking at model parameters. SHAP provides party A with a way to interpret the model without accessing the model parameters. We select the Census Income dataset for this purpose since the semantic meaning of features in the Loan Default dataset is anonymized.

Figure 4 lists the most influential features of model $R^{A}$ in descending order. The top features have higher predictive power because they contribute more to the model than the bottom ones. For example, emp-demo, gender, capital_gain, migr-house and demo-migr are the top-5 most influential features.

The SHAP can further show the positive and negative relationships of features with the prediction target. Figure 5 plots the SHAP values of every feature for all samples to illustrate the impact of those features on the prediction output. Features are ranked in descending order. The color represents the feature value (red high, blue low). The horizontal location shows whether the effect of a feature value is associated with a higher or lower prediction. Specifically, emp-demo, captial_gain and demo-migr are positively correlated with the prediction, while gender and migr-house are negatively correlated with the prediction.

We compare the training time among SecureLR, SecureBoost, and PrADA using FATE 1.6. The experiments are conducted on a machine with 72 Intel Xeon Gold 6140 CPUs and 320 GB RAM. All experiments are simulated in standalone deployment mode. Note that the privacy-preserving VFL framework (PP-VFL) discussed in section 7 has been integrated into FATE, while the federated adversarial domain adaptation (FADA) discussed in section 6 has not because some of the core functionalities of FADA are not satisfied by FATE for now. Thus, we estimate the training time of PrADA by simulation using PP-VFL on FATE.

Table V reports the training time of $\text{PrADA}_{\text{w/o DA\&FG\&IR}}$ is roughly 4.10 hours, which is approximately twice the training time spent by SecureBoost. In $\mathbf{B}\rightarrow\mathbf{A}$ setting, $\text{PrADA}_{\text{w/o IR}}$ takes 5.47 hours to train because FG-based domain adaptation is involved. However, once pre-training is completed, $\text{PrADA}_{\text{w/o IR}}$ only takes half an hour to perform fine-tuning. PrADA takes 8.24 hours to train because it spends extra time on feature group interaction, additional feature extractor training and feature representations encryption. As reported in Table I and Table III, PrADA exceeds $\text{PrADA}_{\text{w/o IR}}$ by only a small margin. Therefore, if efficiency is a major concern, $\text{PrADA}_{\text{w/o IR}}$ is a better choice.