SecureBoost Hyperparameter Tuning via Multi-Objective Federated Learning

Label leakage in vertical federated learning refers to the situation where the passive party is able to obtain the label data of the active party during the training process. Fu et al. (2022) systematically classified the label leakage in vertical federated learning into three categories: active attack, passive attack, and direct attack, and provided an attack method for each category. Zou et al. (2022) proposed a label attack method on the black-boxed VFL and presented a privacy protection method. Xie et al. (2023) demonstrated that split learning remains vulnerable to label inference attacks. Qiu et al. (2022) finds that the training of GNNs in VFL may also lead to the leakage of sample relationships. Tan et al. (2022) proposed a gradient inversion attack that utilizes the gradients of local models to reconstruct label data.

Multi-objective federated learning is a collaborative optimization approach in which participants of FL aim to optimize multiple conflicting objectives simultaneously and find Pareto optimal solutions. Cui et al. (2021) considered the utility of participating parties as the optimization objective, model disparity as the constraint, and optimized the objectives of all participants to calculate the Pareto front. This approach can ensure the fairness of federated learning and obtain the optimal performance model. Zhu and Jin (2020) formulated the accuracy and communication cost of federated learning as the optimization objectives and adjusted the model sparsity to minimize both the communication cost and test errors. The algorithm proposed in Kang et al. (2023) is a multi-objective federated learning algorithm with constraints, which add constraints to the optimization objectives and find the Pareto front faster within the feasible range.

Vertical Federated Learning Yang et al. (2019) is one of the scenarios in Federated Learning, which refers to the situation where feature data is vertically partitioned among multiple parties, with one party possessing the data labels (Fig. 1): The active party holds features and labels while passive parties hold only features.

All participating parties in VFL first align their instances by private set intersection (PSI) Freedman et al. (2004), and then perform federated model training and inferencing. During the training process, each party is not allowed to reveal its training data to others Tong et al. (2022). After the federated training is completed, the federated model $M$ is obtained, which is jointly held by all participating parties, i.e., model $M$ is split into $M_{1},M_{2},\dots,M_{K}$. In the inferencing phase, all participating parties collaboratively make the prediction but only the active party can access the final predicted result.

SecureBoost Cheng et al. (2021) is a widely used gradient boosting tree algorithm designed for the vertical federated learning scenario. The core idea of SecureBoost is to use $n$ federated decision trees to fit instance labels.

For each iteration, SecureBoost constructs a new tree from the root node based on a node split finding algorithm, which is summarized in Algo. 1: the active party sends the homomorphically encrypted gradients $\left\langle g\right\rangle$, $\left\langle h\right\rangle$, and instance space $I$ of the current node to passive parties, who, in turn, calculate the gradient statistics and send them back to the active party; the active party finds the optimal split of the current node with the maximal splitting score and informs the corresponding parties to split the instance space into child nodes. The split finding process continues until the maximum depth is reached. We refer readers to Cheng et al. (2021) for a more detailed algorithm description.

The instance distributions of leaf nodes in SecureBoost are not protected by HE, which may lead to privacy leakage. Some methods Wu et al. (2020); Fang et al. (2021) use a combination of HE and MPC to protect privacy to address this issue. However, these methods may result in significant training cost, making it challenging to be applied in trustworthy federated learning.

The Constrained Multi-Objective Federated Learning (CMOFL) problem aims to find Pareto optimal solutions that simultaneously minimize multiple FL objectives under constraints. We adapt the definition of CMOFL proposed in the work Kang et al. (2023) to the VFL setting.

The FL participants aim to find Pareto optimal solutions and front for the Constrained Multi-Objective VFL problem. We provide the definitions of Pareto dominance, Pareto optimal solution, Pareto set, and Pareto front as follows.

A Pareto optimal solution refers to a solution that achieves an optimal tradeoff among different conflicting objectives. The collection of all Pareto optimal solutions forms the Pareto set, while the corresponding objective values for the Pareto optimal solutions form the Pareto front. The definitions of the Pareto set and Pareto front are formally given as follows.

We use hypervolume (HV) indicator Zitzler and Künzli (2004) as a metric to measure the Pareto front. The definition of hypervolume indicator is provided below.

Below we discuss the threat model, including the attacker’s objective, capability, and knowledge.

While SecureBoost utilizes homomorphic encryption to protect gradient information passed between the active party and passive parties, it does not protect the instance distribution of each leaf node owned by the passive party, indicating that the passive party (i.e., the attacker) can exploit this information to infer the active party’s labels through clustering. We name this label inference attack Instance Clustering Attack, and its procedure is illustrated in Fig. 2 and described in Algo. 2.

The attacker first constructs the similarity matrix based on the instance distribution of each leaf node using Eq. (3) (line 1 of Algo. 2).

where $n$ denotes the number of decision trees, $j$ denotes the index of leaf node.

Then, the attacker leverages this similarity matrix to categorize instances into $C$ clusters (line 2). We assume the attacker knows the label of one instance in each cluster. Hence, the attacker assigns instances in each cluster with the known labels (lines 3-5).

The performance of the instance clustering attack depends on the purity of leaf nodes owned by the attacker (i.e., passive party). We define the purity of a node as the ratio of instances belonging to the majority class to all instances in that node. A higher purity implies a more accurate similarity matrix, which can lead to more precise clustering results. We provide two defense methods to suppress this attack by reducing the purity: (1) local trees; (2) purity threshold.

Although the two defense methods can effectively thwart the instance clustering attack, they may introduce utility loss. Therefore, tradeoffs need to be made between utility and privacy. We consider their hyperparameters as decision variables when we optimize the Constrained Multi-Objective SecureBoost problem (Sec. 5).

We provide measurements of the three objectives we aim to minimize: utility loss, training cost, and label leakage.

In this section, we propose a Constrained Multi-Objective SecureBoost algorithm (CMOSB), through which approximate Pareto optimal solutions and front can be obtained to balance the three optimization objectives and provide guidance for hyperparameter selection.

CMOSB is based on NSGA-II, and it is described in Algo. 3: offspring solutions of the current generation are generated by performing crossover and mutation using the previous solution (line 3). All newly generated individuals are evaluated using Algo. 4 (line 5). The algorithm adds constraints on the solutions (line 6) to limit the search space of the solutions. Lines 7-8 sort the solutions and select the top $N$ individuals for the next iteration.

Algo 4 aims to measure utility loss, training cost, and privacy leakage for training a SecureBoost model. The solution $x$ represents a set of hyperparameters for SecureBoost, where Table 2 provides a detailed description of the hyperparameters. Lines 3-5 correspond to the local training stage mentioned in Sec. 4.3, while lines 6-19 correspond to the federated learning stage, which together constitutes the SecureBoost framework. In line 10, the active party calculates the purity of node $j$, and if it is below the threshold, all participants will jointly calculate the split point, otherwise, only the active party will calculate it locally. Training cost and privacy leakage will be calculated during the iteration process, while utility will be calculated after the training is completed.

Dataset and setting. We conduct experiments on two synthetic datasets and two real-world datasets. The synthetic datasets, generated using the sklearn library²²2https://scikit-learn.org/stable/, consist of Synthetic1 for binary classification and Synthetic2 for multi-class classification. The DefaultCredit ³³3https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset contains the task of predicting whether users can repay their loans on time, while the Sensorless is for sensorless drive diagnosis.

To create datasets for the federated scenario, each dataset was vertically partitioned into two sub-datasets. We use 2/3 of the data as the train set and the remaining as the test set. Table 1 summarizes these datasets.

CMOSB Setup. The proposed CMOSB algorithm is based on NSGA-II. Thus, we follow the setup proposed in literature Zhu and Jin (2020). We apply a single-point crossover for binary chromosomes with a probability of 0.9 and a bit-flip mutation with a probability of 0.1. We apply a simulated binary crossover (SBX) Deb and Kumar (1995) for real-valued chromosome with a probability of 0.9 and $n_{c}$ = 2, and a polynomial mutation with a probability of 0.1 and $n_{m}$ = 20, where $n_{c}$ and $n_{m}$ denote spread factor distribution indices for crossover and mutation, respectively. The number of generations for CMOSB is set to 40.

In this section, we investigate the efficacy of our proposed defense methods in mitigating privacy leakage. The experiments are conducted on Synthetic1, with the default hyperparameter set as follows: $n=20$, $d=7$, $\eta=0.1$, $r=0.8$.

Fig. 3 shows that both methods can trade-off model performance for privacy protection. As shown in Fig. 3(a), training local trees can reduce the correlation between sign of gradients and labels, thereby reducing privacy leakage of active party. Training 10 decision trees locally reduced label leakage risk by 25.1% while sacrificing only 1.6% of utility. The impact of purity threshold is illustrated in Fig. 3(b), where we observe a decrease in privacy leakage as more nodes are constructed by active party locally.

In this section, we use CMOSB (Algo. 3) without using constraints to find the Pareto optimal solutions of hyperparameters for SecureBoost. Each solution is a set of hyperparameters that achieve an optimal tradeoff between utility loss, training cost, and privacy leakage.

We summarize the variables and their descriptions involved in multi-objective optimization in Table 2. Among the variables, the definitions of $n_{f}$, $d$, $r$, and $\eta$ are the same as those in SecureBoost. $n_{l}$ refers to the number of rounds for boosting locally, and $p$ refers to the threshold of purity. For binary classification problems, we further constrain the range of $p$ to $[0.7,1.0]$ to improve the search efficiency.

We use the default parameters of Fate Liu et al. (2021) and VF²Boost Fu et al. (2021), as well as a parameter empirically selected by us, as baselines to evaluate the effectiveness of our algorithm. To make a fair comparison, the sampling rate is set to $0.8$, and $complete\_secure$ is set to true. The specific parameter settings are shown in Table 3.

In the binary classification task, the 3D Pareto front is shown in Fig. 4(a-b), where the blue points represent the Pareto front that is closer to the origin than the red baseline, indicating that the solutions on the Pareto front are better than those obtained by the baseline. It is more intuitive to observe the trade-off between utility loss, training cost, and privacy leakage in the 2D Pareto front illustrated in Fig. 5(a-d). For the tradeoff between training cost and utility loss (Fig. 5(a)(c)), both datasets show that as the utility loss decreases, the training cost increases continuously. This trend is more pronounced in the credit2 dataset due to its complexity. For the tradeoff between privacy leakage and utility loss (Fig. 5(b)(d)), decreasing one would increase in the other objective. The default hyperparameters always have relatively high privacy leakage due to the privacy leakage issue mentioned in Sec. 4.2.

In multi-class classification tasks, the tradeoff between the three optimization objectives also exists, as shown in the 3D Pareto front in Fig. 4(c-d). However, the probability of privacy leakage is lower than that in binary classification tasks (Fig. 5(f)(h)), as the model becomes more complex with increasing task difficulty, making the attack more difficult. In Fig. 5(g)(h), when the utility loss is small, both the training cost and privacy leakage increase dramatically. This is because the task is too simple, and a good model can be trained without passive parties.

In real-world FL scenarios, FL participants typically have requirements on objectives of federated learning. Hence, we apply the CMOSB algorithm (with constraints) to ensure that the Pareto optimal solutions found satisfy the constraints as much as possible. Adding constraints focuses the search space of the CMOSB algorithm on the feasible region, which helps find better Pareto optimal solutions more efficiently.

We conduct this experiment on Synthetic1 and constrain the training cost and privacy leakage below 100 seconds and 0.6, respectively. We set the penalty coefficient to 20.

Fig. 6 illustrates the hypervolume comparison between CMOSB and MOSB when applying constraints to training cost and privacy leakage, respectively. For the privacy leakage constraint, CMOSB grows more rapidly in the first few generations and surpasses MOSB in the final generation, indicating that it can effectively find better Pareto solutions. For the training cost constraint, CMOSB also surpasses MOSB after 10th generation and maintains a lead to the last generation.

We further compare the Pareto fronts (at the 40th generation) obtained by the CMOSB and MOSB algorithms. Fig. 7(a) demonstrates the effect of adding a privacy leakage constraint, showing a reduction of approximately 3% in privacy leakage of the Pareto solutions at the same level of utility loss. The Pareto front found by CMOSB in Fig. 7(b) is also superior to that found by MOSB, especially for solutions with lower utility loss.