Stochastic Alternating Direction Method of Multipliers
for Byzantine-Robust Distributed Learning

Most of the traditional machine learning algorithms require to collect training data from their owners to a single computer or data center, which is not only communication-inefficient but also vulnerable to privacy leakage [1, 2, 3]. With the explosive growth of the big data, federated learning has been proposed as a novel privacy-preserving distributed machine learning scheme, and received extensive research interest recently [4, 5, 6]. In federated learning, the training data are stored at distributed workers, and the workers compute their local variables using local training data, under the coordination of a master. This scheme effectively reduces the risk of data leakage and protects privacy.

However, federated learning still faces significant security challenges. Some of the distributed workers, whose identities are unknown, could be unreliable and send wrong or even malicious messages to the master, due to data corruptions, computation errors or malicious attacks. To characterize the worse-case scenario, we adopt the Byzantine failure model, in which the Byzantine workers are aware of all information of the other workers, and able to send arbitrary messages to the master [7, 8, 9, 10]. In this paper, we aim at solving the distributed learning problem under Byzantine attacks that potentially threat federated learning applications.

Related works. With the rapid popularization of federated learning, Byzantine-robust distributed learning has become an attractive research topic in recent years. Most of the existing algorithms modify distributed stochastic gradient descent (SGD) to the Byzantine-robust variants. In the standard distributed SGD, at every iteration, all the workers send their local stochastic gradients to the master, while the master averages all the received stochastic gradients and updates the optimization variable. When Byzantine workers are present, they can send faulty values other than true stochastic gradients to the master so as to bias the learning process. It is shown that the standard distributed SGD with mean aggregation is vulnerable to Byzantine attacks [11].

When the training data are independently and identically distributed (i.i.d.) at the workers, the stochastic gradients of the regular workers are i.i.d. too. This fact motivates two mainstream methods to deal with Byzantine attacks: attack detection and robust aggregation. For attack detection, [12] and [13] propose to offline train an autoencoder, which is used to online calculate credit scores of the workers. The messages sent by the workers with lower credit scores will be discarded in the mean aggregation. The robust subgradient push algorithm in [14] operates over a decentralized network. Each worker calculates a score for each of its neighbors, and isolates those with lower scores. The works of [15, 16] detect the Byzantine workers with historic gradients so as to ensure robustness. The work of [17] uses redundant gradients for attack detection. However, it requires overlapped data samples on multiple workers, and does not fit for the federated learning setting. For robust aggregation, the master can use geometric median, instead of mean, to aggregate the received messages [18, 19, 20]. When the number of Byzantine workers is less than the number of regular workers, geometric median provides a reliable approximation to the average of regular workers’ stochastic gradients. Other similar robust aggregation rules include marginal trimmed mean and dimensional median [21, 22, 23]. Some aggregation rules select a representative stochastic gradient from all the received ones to update the global variable, e.g., Medoid [19] and Krum [11]. Medoid selects the stochastic gradient with the smallest distance from all the others, while Krum selects the one with the smallest squared distance to a fixed number of nearest stochastic gradients. An extension of Krum, termed as $h$-Krum, selects $h$ stochastic gradients with Krum and uses their average. Bulyan [24] first selects a number of stochastic gradients with Krum or other robust selection/aggregation rules, and then uses their trimmed dimensional median.

When the training data and the stochastic gradients are non-i.i.d. at the workers, which is common in federated learning applications [25], naive robust aggregation of stochastic gradients no longer works. The works of [26, 27] adopt a resampling strategy to alleviate the effect caused by non-i.i.d. training data. With a larger resampling parameter, the algorithms can handle higher data heterogeneity, at the cost of tolerating less Byzantine workers. Robust stochastic aggregation (RSA) aggregates local variables, instead of stochastic gradients [28]. To be specific, it considers a total variation (TV) norm-penalized approximation formulation to handle Byzantine attacks, where the TV norm penalty forces the regular workers’ local variables to be close, and meanwhile, tolerates the outliers sent by the Byzantine workers. Although the stochastic subgradient method proposed in [28] is able to solve the TV norm-penalized approximation formulation, it ignores the separable problem structure.

Other related works include [29, 30, 31, 32], which shows that the stochastic gradient noise affects the effectiveness of robust aggregation rules. Thus, the robustness of the Byzantine-resilience methods can be improved by reducing the variance of stochastic gradients. The asynchronous Byzantine-robust SGD is considered in [33, 34, 35]. The work of [36] addresses the saddle-point attacks in the non-convex setting, and [37, 38, 39, 40] consider Byzantine robustness in decentralized learning.

Our contributions. Our contributions are three-fold.

(i) We propose a Byzantine-robust stochastic alternating direction method of multipliers (ADMM) that utilizes the separable problem structure of the TV norm-penalized approximation formulation. The stochastic ADMM updates are further simplified, such that the iteration-wise communication and computation costs are the same as those of the stochastic subgradient method.

(ii) We theoretically prove that the proposed stochastic ADMM converges to a bounded neighborhood of the optimal solution at a rate of $O(1/k)$ under mild assumptions, where $k$ is the number of iterations and the size of neighborhood is determined by the number of Byzantine workers.

(iii) We conduct numerical experiments on the MNIST and COVERTYPE datasets to demonstrate the effectiveness of the proposed stochastic ADMM to various Byzantine attacks.

Let us consider a distributed network with a master and $m$ workers, among which $q$ workers are Byzantine and the other $r=m-q$ workers are regular. The exact value of $q$ and the identities of the Byzantine workers are all unknown. We are interested in solving a stochastic optimization problem in the form of

where $\tilde{x}\in\mathbb{R}^{d}$ is the optimization variable, $f_{0}(\tilde{x})$ is the regularization term known to the master, and $F(\tilde{x},\xi_{i})$ is the loss function of worker $i$ with respect to a random variable $\xi_{i}\sim\mathcal{D}_{i}$. Here we assume that the data distributions $\mathcal{D}_{i}$ on the workers can be different, which is common in federated learning applications.

Define $\mathcal{R}$ and $\mathcal{B}$ as the sets of regular workers and Byzantine workers, respectively. We have $|\mathcal{B}|=q$ and $|\mathcal{R}|=r$. Because of the existence of Byzantine workers, directly solving (1) without distinguishing between regular and Byzantine workers is meaningless. A less ambitious alternative is to minimize the summation of the regular workers’ local expected cost functions plus the regularization term, in the form of

Our proposed algorithm and RSA [28] both aggregate optimization variables, instead of stochastic gradients. To do so, denote $x_{i}$ as the local copy of $\tilde{x}$ at a regular worker $i\in\mathcal{R}$, and $x_{0}$ as the local copy at the master. Collecting the local copies in a vector $x=[x_{0};\cdots;x_{i};\cdots]\in\mathbb{R}^{(r+1)d}$, we know that (2) is equivalent to

where $x_{i}-x_{0}=0$, $\forall i\in\mathcal{R}$ are the consensus constraints to force the local copies to be the same.

RSA [28] considers a TV norm-penalized approximation formulation of (3), in the form of

where $\lambda$ is a positive constant and $\sum_{i\in\mathcal{R}}\|x_{i}-x_{0}\|_{1}$ is the TV norm penalty for the constraints in (3). The TV norm penalty forces the regular workers’ local optimization variables to be close to the master’s, and meanwhile, tolerates the outliers when the Byzantine attackers are present. Due to the existence of the nonsmooth TV norm term, RSA solves (4) with the stochastic subgradient method. The updates of RSA, at the existence of Byzantine workers, are as follows. At time $k$, the master sends $x_{0}^{k}$ to the workers, every regular worker $i\in\mathcal{R}$ sends $x_{i}^{k}$ to the master, while every Byzantine worker $j\in\mathcal{B}$ sends an arbitrary malicious vector $u_{j}^{k}\in\mathbb{R}^{d}$ to the master. Then, the updates of $x^{k+1}_{i}$ for every regular worker $i$ and $x^{k+1}_{0}$ for the master are given by

where $F^{\prime}(x_{i}^{k},\xi^{k}_{i})$ is a stochastic gradient at $x_{i}^{k}$ respect to a random sample $\xi_{i}^{k}$ for regular worker $i$, $sgn(\cdot)$ is the element-wise sign function ($sgn(a)=1$ if $a>0$, $sgn(a)=-1$ if $a<0$, and $sgn(a)\in[-1,1]$ if $a=0$), and $\alpha^{k}$ is the diminishing learning rate at time $k$.

Although RSA has been proven as a robust algorithm under Byzantine attacks [28], the sign functions therein enable the Byzantine workers to send slightly modified messages that remarkably biases the learning process. In addition, RSA fully ignores the special separable structure of the TV norm penalty. In this paper, we also consider the TV norm-penalized approximation formulation (4), propose a stochastic ADMM that utilizes the problem structure, and develop a novel Byzantine-robust algorithm.

In this section, we utilize the separable problem structure of (4) and propose a robust stochastic ADMM to solve it. The challenge is that the unknown Byzantine workers can send faulty messages during the optimization process. At this stage, we simply ignore the existence of Byzantine workers and develop an algorithm to solve (4). Then, we will consider the influence of Byzantine workers on the algorithm. We begin with applying the stochastic ADMM to solve (4), and then simplify the updates such that the iteration-wise communication and computation costs are the same as those of the stochastic subgradient method in [28].

Stochastic ADMM. Suppose that all the workers are regular such that $m=r$. To apply the stochastic ADMM, for every worker $i$, introduce auxiliary variables $z(0,i),z(i,0)\in\mathbb{R}^{d}$ on the directed edges $(0,i),(i,0)$, respectively. By introducing consensus constraints $z(i,0)=x_{0}$ and $z(0,i)=x_{i}$, (4) is equivalent to

For the ease of presentation, we stack these auxiliary variables in a new variable $z\in\mathbb{R}^{2rd}$. As we will see below, the introduction of $z$ is to split the expectation term $\sum_{i\in\mathcal{R}}\mathbb{E}[F(x_{i},\xi_{i})]$ and the TV norm penalty term $\sum_{i\in\mathcal{R}}\|x_{i}-x_{0}\|_{1}$ so as to utilize the separable problem structure.

The augmented Lagrangian function of (6) is

where $\beta$ is a positive constant, while $\eta(i,0)\in\mathbb{R}^{d}$ and $\eta(0,i)\in\mathbb{R}^{d}$ are the Lagrange multipliers attached to the consensus constraints $z(i,0)-x_{0}=0$ and $z(0,i)-x_{i}=0$, respectively. For convenience, we also collect all the Lagrange multipliers in a new variable $\eta\in\mathbb{R}^{2rd}$.

Given the augmented Lagrangian function (3), the vanilla ADMM works as follows. At time $k$, it first updates $x^{k+1}$ through minimizing the augmented Lagrangian function at $z=z^{k}$ and $\eta=\eta^{k}$, then updates $z^{k+1}$ through minimizing the Lagrangian function at $x=x^{k+1}$ and $\eta=\eta^{k}$, and finally updates $\eta^{k+1}$ through dual gradient ascent. The updates are given by

However, the $x$-update in (8a) is an expectation minimization problem and hence nontrivial. To address this issue, [41] proposes to replace the augmented Lagrangian function $\mathcal{L}_{\beta}(x,z^{k},\eta^{k})$ with its stochastic counterpart, given by

where $\xi_{i}^{k}$ is the random variable of worker $i$ at time $k$ and $\sigma^{k}$ is the positive stepsize. Observe that (3) is a stochastic first-order approximation to (3), in the sense that

at the points $x_{i}=x_{i}^{k}$ and $x_{0}=x_{0}^{k}$, respectively.

With the stochastic approximation, the explicit solutions of $x_{i}^{k+1}$ and $x_{0}^{k+1}$ are

For simplicity, we replace the parameter $\frac{1}{\sigma^{k}+\beta}$ by $\alpha_{i}^{k}$ and $\frac{1}{\sigma^{k}+\sum_{i\in\mathcal{R}}\beta}$ by $\alpha_{0}^{k}$. Thus, (10) is equivalent to

Simplification. Observe that the $z$-update in (8b) is also challenging as the variables $z(0,i)$ and $z(i,0)$ are coupled by the TV norm penalty term. Next, we will simplify the three-variable updates in (11), (8b) and (8c) to eliminate the $z$-update and obtain a more compact algorithm. Note that the decentralized deterministic ADMM can also be simplified to eliminate auxiliary variables [42]. However, we are considering the distributed stochastic ADMM, and the TV norm penalty term makes the simplification much more challenging.

Presence of Byzantine workers. Now we start to consider how the stochastic ADMM updates (12), (13) and (14) are implemented when the Byzantine workers are present. At time $k$, every regular worker $i\in\mathcal{R}$ updates $x_{i}^{k+1}$ with (12) and $\eta^{k+1}_{i}$ with (14), and then sends $\eta_{i}^{k+1}$ to the master. Meanwhile, every Byzantine worker $j\in\mathcal{B}$ can cheat the master by sending $\eta_{j}^{k+1}\mathbb{R}^{d}$ where the elements are arbitrary within $[-\lambda,\lambda]^{d}$. Otherwise, the Byzantine worker $j$ can be directly detected and eliminated by the master. This amounts to that every Byzantine worker $j\in\mathcal{B}$ follows an update rule similar to (14), as

where $u^{k+1}_{j}\in\mathbb{R}^{d}$ is an arbitrary vector. After receiving all messages $\eta_{i}^{k+1}$ from the regular workers $i\in\mathcal{R}$ and $\eta_{j}^{k+1}$ from the Byzantine workers $j\in\mathcal{B}$, the master updates $x_{0}^{k+1}$ via

The Byzantine-robust stochastic ADMM for distributed learning is outlined in Algorithm 1 and illustrated in Figure 1. Observe that the communication and computation costs are the same as those in the stochastic subgradient method in [28]. The only extra cost is that every worker $i$ must store the dual variable $\eta^{k}_{i}$.

Comparing the stochastic subgradient updates (5) with the stochastic ADMM updates (12), (13) and (14), we can observe a primal-dual connection. In the stochastic subgradient method, the workers upload primal variables $x_{i}^{k}$, while in the stochastic ADMM, the workers upload dual variables $\eta_{i}^{k+1}$. The stochastic subgradient method controls the influence of a malicious message by the sign function. No matter what the malicious message is, its modification on each dimension is among $-\lambda$, $\lambda$, and a value within $[-\lambda,\lambda]$ if the values of the malicious worker and the master are identical. The stochastic ADMM controls the influence of a malicious message by the projection function. The modification of the malicious message on each dimension is within $[-\lambda,\lambda]$.

In this section, we analyze the convergence of the proposed Byzantine-robust stochastic ADMM. We make the following assumptions, which are common in analyzing distributed stochastic optimization algorithms.

In this section, we evaluate the robustness of the proposed algorithm to various Byzantine attacks. We compare the proposed Byzantine-robust Stochastic ADMM with the following benchmark algorithms: (i) Ideal SGD without Byzantine attacks; (ii) SGD subject to Byzantine attacks; (iii) Geometric median stochastic gradient aggregation [18, 19]; (iv) Median stochastic gradient aggregation [18, 19]; (v) RSA [28]. All the parameters of the benchmark algorithms are hand-tuned to the best. Although the stochastic ADMM and RSA are rooted in the same problem formulation (4), they perform differently for the same value of $\lambda$ due to Byzantine attacks, as we have observed in Example 1. Therefore, we hand-tune the best $\lambda$ for the stochastic ADMM and RSA, respectively. In the numerical experiments, we use two datasets, MNIST¹¹1http://yann.lecun.com/exdb/mnist and COVERTYPE²²2https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets. The statistics of these datasets are shown in Table 1. We launch one master and 20 workers. In the i.i.d case, we conduct experiments on both datasets by randomly and evenly splitting the data samples to the workers, while in the non-i.i.d. case we only use the MNIST dataset. Each regular worker uses a mini-batch of 32 samples to estimate the local gradient at each iteration. The loss functions $f_{i}(\tilde{x})$ of workers are softmax regressions, and the regularization term is given by $f_{0}(\tilde{x})=\frac{0.01}{2}\|\tilde{x}\|^{2}$. Performance is evaluated by the top-1 accuracy.

Gaussian attack. Under Gaussian attack, at every iteration every Byzantine worker sends to the master a random vector, whose elements follow the Gaussian distribution with standard deviation $100$. Here we set the number of Byzantine workers $q=8$. For Stochastic ADMM on the MNIST dataset, we set $\lambda$ = 0.5, $\beta=0.5$, $\alpha_{0}^{k}=\frac{1}{10+10\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{0.5+10\sqrt{k}}$. As shown in Figure 3(a), SGD fails, Stochastic ADMM, RSA and Geometric median perform very similarly and are close to Ideal SGD, while Median is a little worse than the others. On the COVERTYPE dataset, we set $\lambda$ = 0.5, $\beta=0.1$, $\alpha_{0}^{k}=\frac{1}{2+50\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{0.1+50\sqrt{k}}$ for Stochastic ADMM. As shown in Figure 3(b), SGD performs the worst, while Stochastic ADMM, Geometric median, and median are close to Ideal SGD. Among all the Byzantine-robust algorithms, Stochastic ADMM has the fastest converge speed.

Sign-flipping attack. Under sign-flipping attack, at every iteration every Byzantine worker calculates its local variable, but flips the sign by multiplying with a constant $\varepsilon<0$, and sends to the master. Here we set $\varepsilon=-3$ and the number of Byzantine workers $q=8$. On the MNIST dataset, we set the parameters of Stochastic ADMM as $\lambda=0.05$, $\beta=0.1$, $\alpha_{0}^{k}=\frac{1}{2+5\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{0.1+5\sqrt{k}}$. Figure 4(a) shows that SGD also fails in this situation. Stochastic ADMM, RSA, and Geometric median are close to Ideal SGD, and achieve better accuracy than median. In Figure 4(b), shows the performance on the COVERTYPE dataset. The parameters of Stochastic ADMM are $\lambda=0.5$, $\beta=0.3$, $\alpha_{0}^{k}=\frac{1}{6+100\sqrt{k}}$ ,and $\alpha_{i}^{k}=\frac{1}{0.3+100\sqrt{k}}$. Stochastic ADMM and RSA are close to Ideal SGD, while outperform Geometric median and Median.

Without Byzantine attack. We also investigate the case without Byzantine attack in both MNIST and COVERTYPE datasets, as shown in Figure 5. In Figure 5(a), Stochastic ADMM on MNIST dataset chooses the parameters $\lambda=0.5$, $\beta=0.5$, $\alpha_{0}=\frac{1}{10+10\sqrt{k}}$, and $\alpha_{i}=\frac{1}{0.5+10\sqrt{k}}$. Without Byzantine attack, performance of Stochastic ADMM, RSA, and Geometric median is very similar to Ideal SGD, while Median is worse than the other Byzantine-robust algorithms. On the COVERTYPE dataset, we set the parameters of Stochastic ADMM as $\lambda=0.5$, $\beta=0.3$, $\alpha_{0}^{k}=\frac{1}{6+10\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{0.3+10\sqrt{k}}$. As shown in Figure 5(b), Stochastic ADMM is the best among all the algorithms, and RSA outperforms Geometric median and Median. We conclude that although Stochastic ADMM introduces bias to the updates, it still works well in the attack-free case.

Impact of $\lambda$. Here we show how the performance of the proposed algorithm on the two datasets are affected by the choice of the penalty parameter $\lambda$. We use sign-flipping attack with $\varepsilon=-3$ in the numerical experiments, and the number of Byzantine workers is $q=4$. The parameters $\beta$, $\alpha_{0}^{k}$ and $\alpha_{i}^{k}$ are hand-tuned to the best. As depicted in Figure 6, on both datasets, the performance of Stochastic ADMM degrades when $\lambda$ is too small. The reason is that, when $\lambda$ is too small the regular workers are rely more on their local data, leading to deficiency of the distributed learning system [28]. Meanwhile, $\lambda$ being too large also leads to worse performance.

Non-i.i.d. data. To demonstrate the robustness of the proposed algorithm against Byzantine attacks on non-i.i.d. data, we redistribute the MNIST dataset by letting every two workers share one digit. All Byzantine workers $j$ choose one regular worker indexed by $p$, and send $u_{j}^{k+1}=x_{p}^{k+1}$ to the master at every iteration $k$. When the number of Byzantine workers is $q=4$, the best reachable accuracy is around 0.8, because of the absence of two handwriting digits’ data. Similarly, when the number of Byzantine workers is $q=8$, the best reachable accuracy is around 0.6. Here, we set the parameters of Stochastic ADMM as $\lambda=0.8$, $\beta=0.2$, $\alpha_{0}^{k}=\frac{1}{4+10\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{0.2+10\sqrt{k}}.$ when $q=4$. As shown in Figure 7(a), Median fails, Stochastic ADMM are close to Ideal SGD and outperforms all the other Byzantine-robust algorithms. When the number of Byzantine worker increases to $q=8$, in Stochastic ADMM, we set $\lambda=0.5$, $\beta=3$, $\alpha_{0}^{k}=\frac{1}{60+500\sqrt{k}}$, and $\alpha_{i}^{k}=\frac{1}{3+500\sqrt{k}}$. As depicted in Figure 7(b), Geometric median and Median fail because the stochastic gradients of regular worker $p$ dominate, such that only one digit can be recognized. Stochastic ADMM and RSA both work well, but Stochastic ADMM converges faster than RSA.

We proposed a stochastic ADMM to deal with the distributed learning problem under Byzantine attacks. We considered a TV norm-penalized approximation formulation to handle Byzantine attacks. Theoretically, we proved that the stochastic ADMM converges in expectation to a bounded neighborhood of the optimum at an $O(1/k)$-rate under mild assumptions. Numerically, we compared the proposed algorithm with other Byzantine-robust algorithms on two real datasets, showing the competitive performance of the Byzantine-robust stochastic ADMM.