Federated Learning with Erroneous Communication Links

Internet of things (IoT) applications and services have become popular in recent years due to major technological advancements in sensing, communications, and computation [1]. In many IoT applications, devices operate with extremely low power due to the size, cost, and technological limitations. Traditionally, the devices upload their data to a central node (CN), where all data gathered will be analyzed together. However, such an approach is not feasible in many IoT settings, due to privacy issues, limited power, and usually insufficient communication bandwidth [2]. An alternative solution is to use federated learning (FL) [3], where each user is responsible for computing the updates to the current global model based on its own local training data and transmitting them to the CN. Using the updates from the devices, CN improves the global model and redistributes the updated global model to the users. This process is repeated until the convergence is achieved [4].

In FL, communication between devices and CN can be significantly slower than local computing [5, 6]. Since devices need to communicate with CN repeatedly until the convergence is acquired, there exists a trade-off between local computational power and communication overhead. As we increase the number of local iterations, the required communication between users and CN reduces, and vice versa. The effect of large communication overhead on the scalability of FL and trade-off between local updates and global aggregation was investigated in [7, 8]. Another challenge of FL is the system heterogeneity in which different devices have different computational capabilities, power levels, and storage capacities. In [9], wight-based federated averaging was proposed to tackle this issue.

The majority of works on FL examined a simplified model for the communication channel, where transmissions are error-free but rate-limited [10, 11, 12]. However, since many IoT devices are operating with low power, have limited computational capabilities, and send short packets, the wireless channel between IoT devices and CN is usually erroneous. Recently, a few studies considered FL over a fading wireless channel [13]. In [14], to enhance the learning accuracy over a fading channel, a new scheme was proposed, in which at each iteration, based on the channel state, only one device is selected for transmission using a capacity-achieving channel code. However, there is still a lack of full understanding of the effect of communication error on the accuracy and convergence of FL approaches. This is particularly important for IoT applications, since the channel between devices and CN is usually weak. Due to the large number of devices, the local updates from some of the devices may not reach the CN, which deteriorates the FL performance. In this paper, we shed some light on the effect of communication errors on the convergence and accuracy of FL algorithms.

We consider a distributed learning problem, where the links between the devices and the CN is modeled by packet erasure channels. In particular, we assume the local updates sent by devices face communication errors; that is, the update is erased or received successfully with a certain probability. We consider two scenarios to calculate the global parameter, where the CN only uses the received fresh local updates and discard missing updates or reuse past updates for devices with missing updates. We further analyse the convergence of these approaches, and prove that by using old local updates in case of errors, the FL algorithm employing gradient descent (GD) converges, and the global parameter will converge to the optimal global parameter. This means that the CN does not necessarily needs fresh updates in every communication round of FL to calculate the global parameter. Instead it can reuse past updates in case of error and continue the FL without jeopardizing the global accuracy. We provide simulation results to verify our analysis and further discuss the convergence behaviour for various datasets with different statistical properties.

The rest of the paper is organized as follows. In Section II, we explain the system model and describe different FL approaches in the presence of communication errors. In Section III, we analyze the convergence and accuracy of the FL approaches in the presence of communication error. Simulation results are provided in Section IV. Finally, Section V concludes the paper.

We consider a general federated learning problem, where $N$ device with local datasets, $\mathcal{D}_{1}$, $\mathcal{D}_{2}$, $\cdots$, $\mathcal{D}_{N}$, communicate with the central node (CN). The channel between the $i^{th}$ device and CN is modeled by a packet erasure channel with erasure probability $\epsilon_{i}$; that is a packet sent from the device to the CN is erased with probability $\epsilon_{i}$ and received correctly with probability $1-\epsilon_{i}$. The erasure events for all channels are independent of each other. This model is particularly important for massive IoT scenarios, where many IoT devices communicate with a CN using short packets and due to limited power at devices, the communication channel is mostly erroneous. In particular, let us assume that the local updates from each device is a packet of length $k$ bits, which is encoded using a channel code of rate $R=k/n$, where $n$ is the codeword length. By using the normal approximation bound [15], the packet error rate at the receiver, when the signal-to-noise ratio is $\gamma$, is given by:

where $Q(.)$ is the standard $Q$-function and $V(\gamma)=\left(1-(1+\gamma)^{-2}\right)\log_{2}^{2}(e)$ is the channel dispersion [15]. The short packet communication can then be modeled by the packet erasure channel with erasure probability $\epsilon$, when $k$, $n$, and $\gamma$ is known. Using this simplified model, the channel quality can be characterized by a single parameter $\epsilon$.

We also assume that the downlink channel, i.e., the channel from the CN to devices, is error-free since the CN can transmit with high power; therefore, the error in the downlink channel can be appropriately mitigated.

For the simplicity of the analysis, we assume that the size of all local datasets are the same, i.e., $D_{i}=D/N$, for $i=1,\cdots,N$. When the CN does not have memory, all updates which have not been received will be discarded from the global aggregation step. Let us, for a moment, assume that the number of local iterations at the nodes is sufficiently large. That is, when each node performs a local update, the GD at that device converges. In this case, we define $w^{(t)}_{i}=w_{i}$ for $i=1,\cdots,N$. By using (4), the global update at time instant $t$ is given by:

Since the link between the $i^{th}$ device and CN is an erasure channel with erasure probability $\epsilon_{i}$, the probability mass function $(\mathrm{pmf})$ of $w^{(t)}$ can be calculated as follows:

where $I_{i}\in\{0,1\}$ denote the erasure event for the link between the $i^{th}$ device and CN, i.e., $\mathrm{Prob}\{I_{i}=1\}=1-\epsilon_{i}$ and $\mathrm{Prob}\{I_{i}=0\}=\epsilon_{i}$. From (7) it can be easily observed that the global parameter always fluctuates due to the random nature of the erasure events. This can be clearly seen in Fig. 1(a).

In this section, we focus on the FL algorithm, where the CN uses old local updates received from devices when their fresh updates are not available due to communication error. Fig. 2 shows the overall MSE of the FL algorithm at various erasure probabilities, when $N=3$. We use the same dataset as in Fig. 1(b). As can be seen in Fig. 2, when the erasure rate is small, i.e., $\epsilon=0.1$, the performance of the FL with reuse of old updates closely approaches that of the FL without communication errors. However, when the erasure probability increases, the MSE in the early iterations has a significant gap to the ideal FL case with no error. In all scenarios, even when the erasure rate is large, i.e., $\epsilon=0.5$, the FL algorithm with reuse of old updates in the presence of error converges to that of the ideal FL without error.

Fig. 3(a) shows the performance of FL algorithm with reuse of old updates at various erasure probabilities, when the number of devices is $N=10$. The datasets are shown in Fig. 3(b). The data is created by using a non-linear model $y=x^{2}+z$, where $z\sim\mathcal{N}(0,\sigma^{2})$, is additive white Gaussian noise. As can be seen in Fig. 3(b), the linear regression curve for each local dataset has a different slope; therefore, losing any dataset due to error will result in a significant change in the global parameter. This model is of particular importance for IoT scenarios, where devices are distributed in a field at various locations; therefore, their measurements are location/time dependent and accordingly their datasets are non-iid. As can be seen in Fig. 3(a), when the CN reuses past local updates in case of error, the MSE performance of the FL algorithm converges to that of the ideal FL algorithm without error.

It is important to note that when the dataset is uniformly distributed among devices and the local parameters are not significantly different, even without reusing old local updates, the FL algorithm converges. An example is provided in Fig. 4, where the dataset is uniformly distributed between 3 devices. In this case, the local parameter from all devices are relatively close to each other. Therefore, missing some devices’ updates does not affect the overall performance. As can be seen in Fig. 4(a), the FL algorithm without memory at the CN outperforms the FL algorithm with reusing local updates, when the dataset is uniformly distributed among devices. This can be explained for an extreme case, when $D$ is relatively large, no error occurs up until the $i$th communication round, and all local updates are the same in each round. In this case, by using (4), we will have $w^{(t)}=w^{(t)}_{i}=w^{(t-1)}-\eta\nabla F_{i}(w^{(t-1)})$. Since the datasets are uniformly distributed, $w^{(t)}=w^{(t-1)}-\eta\nabla F(w^{(t-1)})$, even if an error occurs. However, if in the case of error we reuse old updates, by using (5) the global parameter will be $w^{(t)}=w^{(t)}_{i}-\epsilon\left(w^{(t)}_{i}-w^{(t-1)}_{i}\right)$. This is equivalent to $w^{(t)}=w^{(t-1)}-\eta\nabla F(w^{(t-1)})-\epsilon\left(w^{(t)}_{i}-w^{(t-1)}_{i}\right)$, which means that the global parameter will have a gap, proportional to the erasure rate, to that of the error-free case. Therefore, for uniformly distributed datasets, by reusing old updates in case of communication error, the FL algorithm takes longer to converge.

We further consider an image classification task using a real dataset from MNIST [17], which consists of 4000 handwritten images of the numbers $0$ to $3$. The example runs in parallel using 4 workers (i.e., devices), each processing images of a single digit. In particular, worker $i$ has 700 handwritten images of number $i-1$ as its training set. The validation and test set at the CN each has 150 images of each number $0$ to $3$. Similar to [17], at each worker, we use a CNN with two $5\times 5$ convolution layers (the first with 32 channels, the second with 64, each followed with $2\times 2$ max pooling), a fully connected layer with 512 units and ReLu activation, and a final softmax output layer. We also use the stochastic GD optimizer with learning rate $0.001$. Fig. 5 shows the accuracy of the FL algorithm after each communication round in the presence of communication error. As can be seen in this figure, when the CN uses the old local updates if the fresh updates are erased due to communication error, the FL algorithm converges to the same level of accuracy as that for the FL algorithm without any communication error. However, when the CN does not store past local updates and only uses fresh updates and discard missing updates in the aggregation stage, the accuracy changes significantly with the communication round.

In this paper, we studied the federated learning algorithm in the presence of communication errors. We modelled the channel between the devices and the central node (CN) by packet erasure channels. We presented two approaches to deal with error events. That is when an error occurs, and local updates are not received at CN, the CN can either calculate the global parameter by using only the fresh local updates received correctly or reuse the old updates for missing local updates. We proved that the FL algorithm with reusing local updates in case of error converges to the same global parameters as that with FL without any communication error. This means that the CN does not need to wait for the retransmission of missing updates. Instead, it can reuse the past local updates to calculate the global parameters and continue the FL algorithm. This approach converges to the same result as FL achieves with no communication error. This is of practical importance as IoT devices can significantly save energy by relaxing the communication reliability requirements without jeopardizing the overall learning accuracy. We further provided simulation results to verify our theoretical results. We also highlighted that when the dataset is uniformly distributed among devices, the FL algorithm without memory at the CN converges faster than that with reusing local updates.