Hyperparameter Optimization for SecureBoost via Constrained Multi-Objective Federated Learning

Tree-based models are a common type of machine learning algorithm, and their applications in federated learning require the design of appropriate privacy protection methods. SecureBoost [4] is an XGBoost-based model that protects data privacy by applying homomorphic encryption to intermediate gradients. SecureBoost+ [10] extends the SecureBoost algorithm to multi-class tasks and improves SecureBoost’s training efficiency. VF²Boost [7] is a system based on SecureBoost that reduces model training costs through parallel computation. HEP-XGB [11] employs a customized secret sharing method to achieve efficient two-party XGBoost model construction. Pivot [12] utilizes a combination of homomorphic encryption and secure multi-party computation to train XGBoost models, ensuring that no intermediate results are leaked during the training process.

Label leakage in vertical federated learning refers to the situation where the passive party is able to obtain the label information of the active party during the training process. Fu et al. [5] systematically classified the label leakage in vertical federated learning into three categories: active attack, passive attack, and direct attack, and provided an attack method for each category. Zou et al. [13] proposed a label attack method on the black-boxed VFL and presented a privacy protection method. Xie et al. [14] demonstrated that split learning remains vulnerable to label inference attacks. Qiu et al. [15] found that the training of GNNs in VFL may also lead to the leakage of sample relationships. Tan et al. [16] proposed a gradient inversion attack that utilizes the gradients of local models to reconstruct label data. Pan et al. [17] implements an efficient data federation system using secure multi-party computation to avoid label leakage.

Multi-objective federated learning is a collaborative optimization approach in which participants of FL aim to optimize multiple conflicting objectives simultaneously and find Pareto optimal solutions. Personalized Federated Learning aims to optimize the federated model structure for each participating party’s data distribution[18, 19]. Cui et al. [20] considered the utility of participating parties as the optimization objective, model disparity as the constraint, and optimized the objectives of all participants to calculate the Pareto front. This approach ensures the fairness of federated learning and enables obtaining the optimal performance model. Zhu et al. [21] formulated the accuracy and communication cost of federated learning as the optimization objectives and adjusted the model sparsity to minimize both the communication cost and test errors. The algorithm proposed in [8] is a multi-objective federated learning algorithm with constraints, which adds constraints to the optimization objectives and finds the Pareto front more efficiently within the feasible range.

Vertical Federated Learning [2, 3] is one of the scenarios in Federated Learning. In this setting, feature data is vertically partitioned among multiple parties, with one party possessing the data labels (Fig. 1). Specifically, the active party holds both features and labels, while the passive parties hold only features.

All participating parties in VFL first align their instances by private set intersection (PSI) [22], and then perform federated model training and inference. During the training process, each party is not allowed to reveal its training data to others [23]. After the federated training is completed, the federated model $M$ is obtained, which is jointly held by all participating parties, i.e., model $M$ is split into $M_{1},M_{2},\dots,M_{K}$. In the inference phase, all participating parties collaboratively make the prediction, but only the active party can access the final predicted result.

SecureBoost [4] is a widely used gradient boosting tree algorithm designed for the vertical federated learning scenario. The core idea of SecureBoost is to use $n$ federated decision trees to fit instance labels.

For each iteration, SecureBoost constructs a new tree from the root node based on a node split finding algorithm, which is summarized in Algo. 1. In this process, the active party sends the homomorphically encrypted gradients $\left\langle g\right\rangle$, $\left\langle h\right\rangle$, and instance space $I$ of the current node to passive parties. The passive parties then calculate the gradient statistics and send them back to the active party. The active party finds the optimal split of the current node with the maximal splitting score and informs the corresponding parties to split the instance space into child nodes. The split-finding process continues until the maximum depth is reached. For a more detailed description, please refer to [4].

The instance distributions of leaf nodes in SecureBoost are not protected, which may lead to privacy leakage. Some methods [12, 11] use a combination of Homomorphic Encryption (HE) and Secure Multi-Party Computation (MPC) to address this issue and enhance privacy protection. However, these methods may result in significant training cost, making it challenging to apply them in practical and trustworthy federated learning scenarios.

Below, we discuss the threat model, which includes the attacker’s objective, capability, and knowledge.

Gradient boost decision tree (GBDT) construction involves computing information gain based on instance features owned by different parties. To ensure that private features are not disclosed to other parties during the process of calculating information gain, the active party needs to share instance gradients and distributions with passive parties in a privacy-preserving manner. While SecureBoost and its variant [10] utilize homomorphic encryption to protect gradient information passed between the active party and passive parties, they do not protect the instance distribution of each leaf node owned by the passive party. This indicates that the passive party (i.e., the attacker) can exploit this information to infer the active party’s labels through clustering.

We propose a label inference attack named Instance Clustering Attack (ICA) to infer the label information owned by the active party and leverage ICA to measure the privacy leakage of the SecureBoost algorithm. The procedure of ICA is illustrated in Fig. 2 and described in Algo. 2.

The idea behind the attack is straightforward: samples with the same label are highly likely to be grouped into the same leaf node. The attacker first constructs the similarity matrix based on the instance distribution of each leaf node using Eq. (3) (line 1 of Algo. 2).

where $n$ denotes the number of decision trees, and $j$ denotes the index of the leaf node.

Then, the attacker utilizes this similarity matrix to categorize instances into $C$ clusters (line 2). We employ spectral clustering [25], a straightforward yet effective method for clustering instances based on the similarity matrix. This stage essentially involves learning a mapping from cluster IDs to the actual labels. If the attacker possesses more labeled data, it can establish a better mapping, which improves the accuracy of ICA. We assume the attacker knows the label of one instance in each cluster. Hence, the attacker assigns instances in each cluster with the known labels (lines 3-5).

We use the accuracy of the instance clustering attack to measure the privacy leakage $\epsilon_{p}$ (see Sec. VI-A).

Due to the high communication and computational overhead associated with directly preserving the privacy of the instance distribution, Secure Multi-Party Computation (MPC) is often challenging to apply in real-world scenarios. In this work, we propose two defense methods that can mitigate privacy leakage caused by the instance clustering attack (ICA).

The performance of ICA depends on the purity of leaf nodes owned by the attacker (i.e., the passive party). We define the purity of a node as the ratio of instances belonging to the majority class to all instances in that node. A higher purity implies a more accurate similarity matrix, which can lead to more precise clustering results and, consequently, more privacy leakage. Therefore, our proposed defense methods mitigate privacy leakage by reducing node purity, and we name the two defense methods: (1) local trees ¹¹1 Our defense method has been implemented in FATE v1.11.2. ; (2) purity threshold.

The mechanism of ICA (see Sec. V-B) indicates that the stronger the correlation between the instance distribution and true labels, the higher the likelihood of label privacy leakage.

We utilize mutual information [26] to measure the correlation between labels and the instance distribution on each tree during the training of SecureBoost, to investigate the occurrence of label leakage. Specifically, we trained 10 SecureBoost models, each consisting of 50 trees. Fig. 3 illustrates the average variation of mutual information with respect to iterations (each representing a tree). It indicates that the mutual information is relatively high in the first few iterations of training, suggesting a higher likelihood of privacy leakage during the initial stages of SecureBoost training.

To mitigate the label leakage, we propose a local training stage preceding SecureBoost, where the active party trains $n_{l}$ decision trees locally. The objective is to reduce the mutual information between the instance distribution and instance labels in the federated SecureBoost model.

The decision trees obtained from the local training stage are stored exclusively on the active party’s side, ensuring no information leakage while still participating in the federated prediction process. Once the local training stage is completed, the federated learning stage begins, where all participants collectively train $n_{f}$ decision trees according to the SecureBoost protocol. These two stages together establish the SecureBoost model with privacy protection.

The purity threshold mitigates privacy leakage by preventing the sharing of instance distributions of high-purity nodes with passive parties. The active party sends the instance distributions to passive parties, calculates the information gain based on the returned statistical data, and identifies the optimal split point of a node $j$. Subsequently, the active party calculates its purity $p_{j}$, and if the purity $p_{j}$ exceeds the pre-specified threshold $\theta_{p}$, passive parties are excluded from the federated training of the subtree rooted at node $j$, and the active party proceeds to train this subtree locally. Otherwise, all participants continue with the SecureBoost algorithm.

Although the two defense methods can effectively thwart the instance clustering attack, they may introduce utility loss. Therefore, trade-offs need to be made between utility and privacy. We consider their hyperparameters as decision variables when optimizing the Constrained Multi-Objective SecureBoost problem (Sec. VI).

Prior to delving into the algorithmic aspects, we first establish the measurements used to evaluate the three objectives we aim to optimize: utility loss, training cost, and privacy leakage.

In this section, we propose the Constrained Multi-Objective SecureBoost algorithm (CMOSB), which aims to obtain approximate Pareto optimal solutions and front. It achieves a balance between the three optimization objectives and provides guidance for hyperparameter selection.

CMOSB is based on NSGA-II and described in Algo. 3. Offspring solutions of the current generation are generated by performing crossover and mutation using solutions found during the previous generation (line 3). All newly generated candidate solutions are evaluated using Algo. 4 (line 5). The algorithm adds a constraint to limit the search space of the solutions (lines 6-7). Lines 8-9 sort the candidate solutions and select the top $N$ for the next generation.

Algo. 4 aims to measure the utility loss, training cost, and privacy leakage while training a SecureBoost model. The variable $x$ represents a set of hyperparameters for SecureBoost, and Table III provides a detailed description of these hyperparameters. Lines 3-5 correspond to the local training stage mentioned in Sec. V-C, while lines 6-19 constitute the federated learning stage, forming the SecureBoost framework. In line 10, the active party calculates the purity of node $j$, and if it is below the threshold $\theta_{p}$, all participants will jointly calculate the split point; otherwise, only the active party will calculate it locally. Training cost and privacy leakage will be calculated during the iteration process, while utility will be calculated after the training is completed.

To create datasets for the federated scenario, each dataset was vertically partitioned into two sub-datasets. We used 2/3 of the data as the training set and the remaining as the test set. Table I summarizes these datasets.

We implement all the methods with Python 3.8. The experiments were conducted on two Intel(R) Xeon(R) Platinum 8269CY 3.10GHz CPUs each with 4 cores.

• •

  Empirical Selection (ES): Empirical Selection represents the default hyperparameters that are typically determined empirically.

• •

  Grid Search (GS): Grid Search is a traditional hyperparameter optimization method that exhaustively evaluates different combinations for the optimal hyperparameter combination [27].

• •

  Bayesian Optimization (BO): Bayesian Optimization [28] is a classic hyperparameter optimization method for machine learning algorithms. It utilizes Gaussian processes to improve search efficiency. However, BO is only suitable for single-objective optimization. For multi-objective optimization, we leverage BO to optimize each objective separately.

To make a fair comparison, the sampling rate is set to $0.8$, and complete secure is set to true.

The complete secure corresponds to the local tree defense method, where $n_{l}$ is set to $1$, indicating that local training of a single tree is performed before federated learning.

The specific hyperparameter settings are shown in Table II.

In this section, we investigate the efficacy of our proposed defense methods in mitigating privacy leakage. The experiments are conducted on Synthetic1 dataset, with the default hyperparameter set as follows: $n=20$, $d=7$, $\eta=0.1$, $r=0.8$.

Fig. 5 shows that both methods can trade-off model performance for privacy protection. The lack of protection on instance distribution resulted in up to 84% of instance label leakage. As shown in Fig. 5 (a), training 10 decision trees locally reduced privacy leakage risk by 25.1% while sacrificing only 1.6% of utility. The impact of the purity threshold is illustrated in Fig. 5(b), where we observe a decrease in privacy leakage as the active party constructs more nodes locally.

In this section, we use CMOSB (Algo. 3) without constraints to find the Pareto optimal solutions of hyperparameters for SecureBoost. Each solution is a set of hyperparameters that achieves an optimal tradeoff between utility loss, training cost, and privacy leakage.

We summarize the variables and their descriptions involved in multi-objective optimization in Table III. Among the variables, the definitions of $n_{f}$, $d$, $r$, and $\eta$ are the same as those used in SecureBoost [4]. $n_{l}$ refers to the number of iterations for boosting locally, and $\theta_{p}$ refers to the threshold of purity. For binary classification problems, we further constrain the range of $\theta_{p}$ to $[0.7,1.0]$ to improve search efficiency.

We first present the comparison between CMOSB and the baselines through 3D plots illustrated in Fig. 6 and Fig. 7 on the four datasets (see Table I).

Fig. 6 illustrates the comparison between CMOSB and Grid Search. The red, green, and blue dots represent solutions obtained from Empirical Selection (ES), Grid Search (GS), and CMOSB, respectively. It can be observed that the solutions found by CMOSB dominate those found by ES and GS, indicating that CMOSB achieves a better Pareto front (closer to the origin).

Fig. 7 illustrates the comparison between CMOSB and Bayesian optimization. The red, orange, and blue dots represent solutions obtained from Empirical Selection (ES), Bayesian Optimization (BO), and CMOSB, respectively. Similar to Fig. 6, CMOSB outperforms BO and ES overall, achieving a better Pareto front.

To provide a more intuitive observation of the experimental results, we project each 3D plot onto two 2D plots: one for training cost vs. utility loss and another for privacy leakage vs. utility loss, as shown in Fig. 8 and Fig. 9, respectively. In Fig. 8 and Fig. 9, the blue, green, orange, and red dots represent CMOSB, GS (Grid Search), BO (Bayesian Optimization), and ES (Empirical Selection), respectively. Each dot represents a solution, and each line represents a Pareto front.

For the trade-off between training cost and utility loss (see Fig. 8), all four datasets demonstrate that the training cost increases as the utility loss decreases. CMOSB (in blue) can find a better Pareto front than GS (in green), BO (in orange), and ES (in red). More specifically, BO (in orange) can find a set of decent solutions but is slightly inferior to CMOSB. This is because BO optimizes one objective at a time and does not consider all three objectives as a whole for optimization. The solutions found by GS are noticeably inferior to CMOSB, as GS only heuristically searches for solutions. The solutions achieved by ES are farther away from the lower-left corner, indicating that ES performs poorly in simultaneously optimizing utility loss, training cost, and privacy leakage. This suggests that the default hyperparameters set by FATE [6] and VF²Boost [7] were not suitable for multi-objective SecureBoost.

For the trade-off between privacy leakage and utility loss (see Fig. 9), all four datasets demonstrate that as the utility loss decreases, the privacy leakage increases. CMOSB outperforms all baseline methods in terms of the Pareto front. BO performs slightly worse than CMOSB because it is a single-objective optimization algorithm and thus cannot simultaneously optimize privacy leakage, utility loss, and efficiency. GS demonstrates inferior search results, as applying grid search to optimize continuous hyperparameters is often challenging. More specifically, the privacy protection threshold $p$, which significantly impacts privacy leakage, is a continuous variable, and the performance of grid search is limited in this case. ES overall performs quite badly in preserving label privacy (the worst among all methods) because FATE and VF²Boost were not well-prepared for defending against ICA.

We also compare the hypervolume of CMOSB with those of baseline methods (see Table IV) for a clear performance comparison between the four hyperparameter tuning methods. We normalize the three objectives for each dataset to ensure a fair comparison. The performance of ES varies widely across the four datasets due to its empirical selection of default hyperparameters for SecureBoost. While obtaining relatively stable results by searching a large number of hyperparameters, GS suffers from a heuristically pre-specified discrete hyperparameter search space, which is not well-suited for our proposed two defense methods, as they are sensitive to their defense parameters (i.e., $n_{l}$ and $p$). BO performs better than ES and GS but worse than CMOSB because BO optimizes each objective separately and does not consider all objectives as a whole for optimization.

In real-world VFL scenarios, participants typically have specific requirements or constraints on objectives. Hence, we apply the CMOSB algorithm (with constraints) to ensure that the Pareto optimal solutions found satisfy the constraints as much as possible. Adding constraints focuses the search space of the CMOSB algorithm on the feasible region, which improves the efficiency of finding better Pareto optimal solutions.

We conduct this experiment on Synthetic1 and constrain the training cost and privacy leakage to be below 100 seconds and 0.6, respectively. We set the penalty coefficient to 20 (see lines 6-7 of Algorithm 3).

Fig. 10 illustrates the hypervolume comparison between CMOSB and MOSB when applying constraints to training cost and privacy leakage, respectively. For the privacy leakage constraint, CMOSB grows more rapidly in the first few generations and surpasses MOSB in the final generation, indicating that it can effectively find better Pareto solutions. For the training cost constraint, CMOSB also surpasses MOSB after the 10th generation and maintains a lead until the last generation.

We further compare the Pareto fronts (at the 40th generation) obtained by the CMOSB and MOSB algorithms. Fig. 11(a) demonstrates the effect of adding a privacy leakage constraint, showing a reduction of approximately 3% in privacy leakage of the Pareto solutions at the same level of utility loss. The Pareto front found by CMOSB in Fig. 11(b) is also superior to that found by MOSB, especially for solutions with lower utility loss.