Mode Connectivity and Data Heterogeneity of Federated Learning

Federated Learning (FL) [1] is a collaborative paradigm that enables multiple clients to train a model while preserving data privacy. A primary challenge in FL is data heterogeneity across clients [2]. Previous works like [3] found that data heterogeneity induces a misalignment between clients’ local objectives and the FL’s global objective (hereinafter referred to as client objective and FL objective, respectively) and drifts client updates. Nevertheless, despite client-update drift, the FL objective can typically be optimized [4] effectively. This implies that the destinations of these drifting updates (i.e., the solutions to the client and FL objectives) may overlap. However, a systematic investigation of this relationship is currently lacking.

A straightforward method to examine this overlapping relationship is to investigate mode connectivity [5], where mode refers to a suit of solutions to a specific objective, i.e., a suit of neuron parameters, under permutation invariance. Mode connectivity measures the connectivity of two modes via a parametric path and reveals the geometric relationship in the solution space. Specifically, it evaluates the performance change (e.g., taking loss/error as metrics) along a given path with endpoints corresponding to two different modes. Maintaining a minor change along the path indicates better connectivity (i.e., the geometric landscape becomes increasingly connected between the two modes). Some works [5, 6] have discovered that two modes can be connected well in centralized training, but research on FL remains scarce.

In this work, we conduct empirical and theoretical studies on mode connectivity in FL under data heterogeneity to unravel the relationship between client modes (i.e., the solutions to client objectives) and global modes (i.e., the solutions to the FL objective). Empirically, we consider linear and non-linear paths to evaluate the connectivity between client modes and global modes to show their overlap relationships, as illustrated in Figure 1. We also investigate the connectivity of global modes obtained under varying levels of data heterogeneity. Theoretically, we leverage mean-field theory [7] and dropout stability to establish a quantitative bound on the connectivity errors of global modes. In addition, we conduct a comprehensive numerical analysis to verify our analytical findings.

This study takes the first step to explore mode connectivity in FL. Our main contributions are summarized as follows:

• •

  Our empirical and theoretical investigations demonstrate that data heterogeneity deteriorates the connectivity between client and global modes, making it challenging to find solutions that meet both client and FL objectives.

• •

  It is revealed that FL solutions from different data heterogeneity belong to distinct global modes within a common region of the solution space. They can be connected on a polygonal chain while keeping error low but face an error barrier when considering a linear path.

• •

  We establish quantitative bounds on dropout and connectivity errors of global modes using mean-field theory, showing that both errors decrease with reduced data heterogeneity and widening trained models.

Data heterogeneity in FL. According to [3], data heterogeneity across clients significantly degrades the FL performance. Subsequent works have observed that data heterogeneity induces weight divergence and feature inconsistency. For weight divergence, studies such as [8, 9] show that client-update drift slows the FL convergence due to inconsistent client and FL objectives. Moreover, some studies, e.g., [10, 11], discover that the classifier divergence across clients is the main culprit of feature inconsistency, negatively affecting FL generalization. In contrast to previous works, we offer a novel perspective on data heterogeneity’s impact within the FL solution space.

Mode connectivity. The connectivity between neural network (NN) solutions due to the NN non-convexity, called mode connectivity, provides a new view into the geometric landscape of NNs in recent years. Despite the high dimension of NNs, the different solutions follow a clear connectivity pattern [12, 13, 6]. Specifically, it is possible to connect the local minima through specific paths that make it easier to find high-performance solutions when using gradient-based optimizers. The paths are often non-linear curves and are discovered by specific tasks. This suggests that the local minima are not isolated but rather interconnected within a manifold. Moreover, the geometric landscape of an NN becomes approximately linear-connected when considering the infinite neuron numbers as per [12]. Previous studies mainly focus on mode connectivity in centralized training with homogeneous data. Several pilot studies [14, 15] visualize the loss landscape of FL while not mode connectivity, and then our work tries to fill this gap of mode connectivity in FL under data heterogeneity.

Mean-field theory. A mean-field theory for the NN dynamics analysis is proposed by [7, 16] based on the gradient-flow approximation. The theory shows that when the neuron number is sufficiently large, the dynamics of stochastic gradient descent (SGD) to optimize NNs can be well approximated by a Wasserstein gradient flow. This approximation has been studied both in the two-layer case [7, 17] and the multi-layer case [18, 19]. In particular, the approximation is utilized to analyze the dropout stability of SGD solutions by [19], and the dropout stability implies mode connectivity of NNs according to [20]. However, these results are solely observed in centralized training. To the best of our knowledge, this paper is the first to expand upon the theoretical results into FL.

Mode connectivity. The forward function of an NN parameterized by $\bm{\theta}\in\Theta$ is represented as $f_{\bm{\theta}}:\mathcal{X}\rightarrow\mathcal{Y}$, where $\Theta$ denotes the parameter space of an NN architecture, and $\mathcal{X}\in\mathbb{R}^{d}$ and $\mathcal{Y}\in\mathbb{R}$ are input and output spaces, respectively. When models ${\bm{\theta}_{1}}$ and ${\bm{\theta}_{2}}$ have the same neurons but different neuron permutations, they belong to the same mode. According to [20, 19], given an NN architecture $\Theta$, a dataset $\mathcal{D}$ and a loss function $\mathcal{L}(\cdot)$, mode connectivity is defined as follows:

This definition measures the connectivity of two modes by comparing the loss change along a path to its endpoints. Moreover, we consider a path-finding task proposed by [5], which minimizes the expected loss over a uniform distribution on the path as:

where the path $\pi_{\bm{\theta}}(\nu)$ can be characterized by a Polygonal chain (PolyChain) with one or more bends [5]. We consider a PolyChain case with one bend as:

where $\bm{\theta}_{\rm b}$ is the model of the PolyChain-bend point and can be found by $\min_{\bm{\theta}_{\rm b}}\mathcal{L}_{\rm curve}(\bm{\theta}_{\rm b})$ in (1). We refer to mode connectivity as linear (or non-linear) mode connectivity (LMC) when using linear interpolation (or PolyChain). Compared with the non-linear counterpart, LMC is a stronger constraint since LMC requires the two modes to stay in the same basin.

Federated leaning (FL). We consider an FL framework with $K$ clients, each with its own dataset $\mathcal{D}_{k}=\{({\bm{x}}_{i_{k}},y_{i_{k}})\}$, where data samples $({\bm{x}}_{i_{k}},y_{i_{k}})\sim\mathbb{P}_{k}:\mathbb{R}^{d}\times\mathbb{R}$ are indexed by $i_{k}\in[n_{k}]$. The global dataset is the union of all client datasets and denoted by $\mathcal{D}=\cup_{k=1}^{K}\mathcal{D}_{k}\sim\mathbb{P}$ on $\mathbb{R}^{d}\times\mathbb{R}$, which includes $n=\sum_{k=1}^{K}n_{k}$ data samples. The FL objective is to minimize the expected global loss $\mathcal{L}(\bm{\theta})$ on $\mathcal{D}$, and is formulated as:

where $l(\cdot)$ and $l_{k}(\cdot)$ denote the global and client loss functions, respectively. The $k$-th client objective is $\min_{\bm{\theta}_{k}}\mathcal{L}_{k}(\bm{\theta}_{k})=\mathbb{E}_{({\bm{x}},y)\in\mathcal{D}_{k}}[l_{k}(\bm{\theta}_{k};({\bm{x}},y))]$. When $\mathbb{P}_{k}\neq\mathbb{P}$, FedAvg [1] takes client models $\{\bm{\theta}_{k}\}_{k=1}^{K}$ to solve (3) by minimizing client objectives locally and obtaining the global model $\bm{\theta}=\sum_{k=1}^{K}\frac{n_{k}}{n}\bm{\theta}_{k}$ round by round.

Experimental setups. We explore mode connectivity in FL on classification tasks, including MNIST [21], CIFAR-10 [22], and PACS [23]. The experimental setups used throughout this paper are as follows, unless stated otherwise. For NN architectures, we consider a two-layer NN, a shallow convolutional neural network (CNN) with two convolutional layers and a deep CNN, i.e., VGG architecture [24]. For the FL setup, we consider ten clients participating in FL with a total of 200 rounds. Client optimizers are SGD with a learning rate of 0.01 and momentum of 0.9, mini-batches of size 50, the local epoch of 1, and dropout of 0.5 for training VGG. For data heterogeneity, we follow [2] and examine label distribution skew by using Dirichlet distribution $Dir(\alpha)$ [25] to create client label datasets. A larger $\alpha$ suggests more homogenous distributions across clients. On this basis, we also make use of the built-in domain shift of PACS to introduce feature distribution skew across clients.

In this section, we explore mode connectivity among client modes trained on client datasets, where $\mathbb{P}_{k}\neq\mathbb{P}_{k^{\prime}}$ when $k\neq k^{\prime}$. We first investigate the geometric landscape of client modes on their corresponding test sets. Specifically, we create a hyperplane by using the global mode $\bm{\theta}$ and client modes $\bm{\theta}_{k}$, $\bm{\theta}_{k^{\prime}}$, which is represented as $\{(a,b)|\bm{\theta}+a(\bm{\theta}_{k}-\bm{\theta})+b(\bm{\theta}_{k^{\prime}}-\bm{\theta})\}$. We then use the test sets of clients $k$ and $k^{\prime}$ to evaluate the performance of all points on the hyperplane. Figure 2 displays the loss landscapes obtained on different test sets and distinguishes them by different colors. We also depict the loss landscape of the global mode (i.e., the initialization of client modes) on the global test set in Figure 2. For comparison, we set the legend scale of the client-mode landscape to be the same in each sub-figure of Figure 2 except the lowest value, and keep the scale of the global-mode landscape consistent among all the sub-figures.

Visualizing mode connectivity on client modes. As shown in Figure 2, the comparison between the cases of $\alpha=0.1$ and $\alpha=0.5$ indicates that when data heterogeneity increases, low-loss overlaps between client and global modes decrease, and the number of overlapping solutions to both client and FL objectives reduces. Similar results are also observed by comparing the cases of CIFAR-10 and PACS. Meanwhile, although client modes achieve a lower test loss than the global mode, they stray from the low-loss landscapes of the global mode. This indicates that client modes are prone to overfitting their own objectives when FedAvg solves (3) even if the global mode provides adequate initialization for them in the final round. On the other hand, we can linearly connect client models to the global model in all cases, as shown in the LMC of Figure 2. As the loss changes along these linear paths, it may eventually reach the low-loss landscapes of the global mode while still remaining in the low-loss landscapes of client modes. The above observations reveal that there exist some solutions that meet both client and FL objectives, but they may be overlooked by FedAvg.

Traversing loss along the paths connecting client modes. To delve deeper into the loss landscapes of Figure 2, we consider two paths to connect two client modes. One path is a linear connection between them (i.e., $\bm{\theta}_{k}\rightarrow\bm{\theta}_{k}^{\prime}$), while the other is a PolyChain with the global mode as its bend (i.e., $\bm{\theta}_{k}\rightarrow\bm{\theta}\rightarrow\bm{\theta}_{k}^{\prime}$) as per (2). We follow the paths to traverse the landscape from $\bm{\theta}_{k}$ to $\bm{\theta}_{k}^{\prime}$ on their test sets, and show the traversing losses along the paths in Figure 3. For the PolyChain in the case of $\alpha=0.1$, the intersection point of the two curves deviates from the global mode (i.e., $a=0.5$), where the intersection represents a solution that works effectively on both client datasets. Moreover, the traversing-loss gap between the two paths in the case of $\alpha=0.1$ is greater than that of $\alpha=0.5$. This means that when heterogeneous data exist at clients, it becomes more difficult to find solutions that align with both the client and FL objectives.

Client-mode trajectory. The trajectory of client modes during optimization in the case of $\alpha=0.1$ is depicted in Figure 4. This trajectory is based on t-SNE [26] and covers both training rounds and local iterations within each round. The figure illustrates that throughout the training process, the distance between client modes is controlled even under serious data heterogeneity, such that they closely surround the global mode. During the same round, data heterogeneity pushes client modes with the same initialization to be optimized locally along different directions. Therefore, the solutions found by client objectives vary when facing data heterogeneity even though they are close to each other in the parameter space. Figure 12 shows that the shallow CNN has similar results.

In this section, we explore the mode connectivity of global modes obtained by different data heterogeneity of the same training task. We consider two paths, linear interpolation and PolyChain, to investigate the mode connectivity among global modes on CIFAR-10 under $\alpha=0.1,0.2$ and $0.5$, as shown in Figure 5. Note that the initialization of global modes remains the same across these three cases.

Visualizing mode connectivity on global modes. In Figure 5, the linear-interpolation landscape reveals that global modes obtained from varying data heterogeneity are situated in distinct basins with an error barrier separating them. The landscape also shows that when sharing the same initialization, the global-mode basins are within a common region surrounded by high errors. With more heterogeneous data, like those involving $\alpha=0.1,0.2$, both training and test error barriers become higher, compared with that of $\alpha=0.1,0.5$. Furthermore, we also use a PolyChain found by (1) to connect the global modes of $\alpha=0.1,0.2$. Along the PolyChain, all the solutions keep no barrier in both training and test landscapes. The above results indicate that global modes can be connected without any barrier by some paths and they are situated within a manifold, as illustrated in Figure 1.

Barrier among global modes. To further quantify the mode connectivity of two global modes ${\bm{\theta}}$ and ${\bm{\theta}^{\prime}}$, we define a barrier metric as:

where $\pi_{\bm{\theta}}(a)$ is equal to $(1-a){\bm{\theta}}+a{\bm{\theta}}^{\prime}$ and (2) when considering linear interpolation and PolyChain, respectively. Here, $B({\bm{\theta}},{\bm{\theta}^{\prime}})$ measures the largest performance gap along $\pi_{\bm{\theta}}(a)$ comparing with its endpoints ${\bm{\theta}}$ and ${\bm{\theta}^{\prime}}$. We say that ${\bm{\theta}}$ and ${\bm{\theta}^{\prime}}$ are well-connected if the barrier $B({\bm{\theta}},{\bm{\theta}^{\prime}})$ is close to $0$.

We then use $B({\bm{\theta}},{\bm{\theta}^{\prime}})$ to explore mode connectivity during the whole training round. In Figure 6, the sub-figure on the left shows that more severe data heterogeneity causes greater barriers by comparing the cases of $\alpha=0.1,0.2$ and $\alpha=0.1,0.5$, which persist throughout training. Meanwhile, as the rounds progress, the barrier shows an increasing trend after an initial oscillation. This causes various global modes to reach their own basins, as found in Figure 5. We also examine the difficulty of finding the global-mode PolyChain when facing varying data heterogeneity. Here, the difficulty is quantified by the relationship between barrier $B({\bm{\theta}},{\bm{\theta}^{\prime}})$ and data sample to optimize (1) and is depicted in the right sub-figure of Figure 6. The sub-figure illustrates that more samples are needed to find the specific PolyChain as data heterogeneity increases, resulting in more incredible difficulty in connecting global modes with no error barrier.

Function dissimilarity and distance of global modes. According to [27], permutation invariance may contribute to a linear-interpolation barrier between NN solutions in centralized training. When considering FL, data heterogeneity may disturb the permutation of FL solutions, leading to the barriers found in Figure 5. To verify whether the FL solutions obtained from different data heterogeneity belong to the same global mode, we measure their function dissimilarity on the left-hand side of Figure 7. The figure shows that function dissimilarity remains $>0.2$ among these FL solutions even when the data heterogeneity is weakened. This means that the FL solutions belong to different global modes. Furthermore, Figure 7 presents that the distance between these FL solutions is smaller than the distance between them and their initial points. This means that different global modes with the same initialization are close to each other. Combined with the above results, it can be inferred that these FL solutions belonging to various global modes are adjacent and in a common region, but they cannot be linearly connected.

As indicated by our experimental results, a sufficient condition to find the solutions meeting both client and FL objectives (3) is that the distributions represented by client modes $\bm{\theta}_{k}$ are the same as the global mode $\bm{\theta}$. However, satisfying this condition is generally challenging since data heterogeneity induces client-gradient drifts [3] and then implicit output bias among client modes [15]. Building upon the drifts and bias, we consider $K$ client modes $\{\bm{\theta}_{k}\}_{k=1}^{K}$ and formulate data heterogeneity as:

In this section, we will utilize dropout stability to analyze mode connectivity of global modes. First, let us define it as:

Then, we adopt the mean-field theory developed by [7] to analyze dropout stability under data heterogeneity. Mean-field theory shows that the trajectory of SGD can be approximated by a partial differential equation (PDE) called distributional dynamic (DD). Namely, under the assumption of enough neurons (i.e., $N$ is large enough) and one-pass data processing (i.e., for all $i\in[n]$, samples $({\bm{x}}_{i},y_{i})\in\mathcal{D}\sim\mathbb{P}$ are independent and identically distributed), the SGD trajectory of $N$ neurons are close to the movement of $N$ i.i.d particles following the description of DD. Here, we consider a two-layer NN with $N$ hidden neurons, denoted as ${\bm{\theta}}=({\bm{\theta}_{1},\dots,{\bm{\theta}_{N}}})$, where ${\bm{\theta}_{i}}\in\mathbb{R}^{D}$ for all $i\in[N]$. The forward function of ${\bm{\theta}}$ is represented as:

where $\sigma_{*}:\mathbb{R}^{d}\times\mathbb{R}^{D}\rightarrow\mathbb{R}$ is the activation function. For simplicity, we focus on the ReLU activation and the mean-square loss, i.e., $\sigma_{*}({\bm{x}};{\bm{\theta}_{i}})=\sigma(\langle{\bm{x}},{\bm{\theta}_{i}}\rangle)$ and $l(\bm{\theta};({\bm{x}},y))=(y-f_{\bm{\theta}}({\bm{x}}))^{2}$, which can be extended to other activation functions and losses in [7, 16].

Meanwhile, we consider FedAvg with each client undergoing $p$ local iterations in the $m$-th round, and represent the global-mode update on the $i$-th neuron $\bm{\theta}_{i}$ as $\Delta\bm{\theta}_{i}^{(m)}=\bm{\theta}_{i}^{m+1,0}-\bm{\theta}_{i}^{m,0}$ to get:

where $\tau\in[p]$ denotes the $\tau$-th local iteration, ${s}^{m,\tau}$ denotes the step size at the $\tau$-th iteration, $\bar{\bm{\theta}_{i}}^{m,\tau}=\sum_{k=1}^{K}\frac{n_{k}}{n}\bm{\theta}_{i,k}^{m,\tau}$ denotes the weighted average of all the $i$-th client neurons, $\bar{y}^{m,\tau}=1/N\sum_{i=1}^{N}\sigma({\bm{x}^{m,\tau}};{\bar{\bm{\theta}_{i}}^{m,\tau}})$ denotes the output of the weighted-average model $\bar{\bm{\theta}}^{m,\tau}$ based on one-pass data $({\bm{x}^{m,\tau}},{\bm{y}^{m,\tau}})\in\mathbb{P}$, and $\bm{n}_{i}^{m,\tau}$ denote the noise induced by data heterogeneity. Specifically, we follow Definition 2 and represent $\bm{n}_{i}^{m,\tau}$ (the superscript is omitted for brevity) as:

where the gradient drift and output bias depend on data heterogeneity, i.e., $(x_{k},y_{k})\in\mathbb{P}_{k}$ and $\in\mathbb{P}$, but $\mathbb{P}_{k}\neq\mathbb{P}$.

Furthermore, the trajectory of $\bar{\bm{\theta}_{i}}$ in (6) subsumes the trajectory of the global model ${\bm{\theta}_{i}}$ since FedAvg performs weighted averaging on the global model at each round. Therefore, we represent the global-mode trajectory as follows:

where $\tau\in[Mp]$ when given a total round $M$ and $\bm{n}_{i}^{\tau}=0$ when $\tau\mod p\in[M]$. See the appendix for detailed proof.

We make the following assumptions as per [7]:

Assumptions 1-4 denote the requirements of mean-field theory on the learning rate $s_{k}$, data distribution $(\bm{x},y)\sim\mathbb{P}$, activation function $\sigma$, and initialization $\rho_{0}$. Assumption 5 indicates that the noise $\bm{n}_{i}$ in (7) depends on data heterogeneity. Specifically, a generalized function $h(\alpha)$ is taken to measure the impact of data heterogeneity on $\bm{n}_{i}$, including the gradient drift and output bias, where $\beta h(\alpha)\in[\beta,\infty)$ and $1/\alpha$ denotes the degree of data heterogeneity across clients (i.e., smaller $\alpha$, larger $h(\alpha))$.

We take the mean-field theory to quantify the difference between the global-mode trajectory loss and the DD-solution loss and have:

Lemma 1 shows when the number of neurons $N$ is sufficiently large and the step size $\varepsilon$ is sufficiently small, the global-mode neurons $\bm{\theta}_{i}^{\tau}$ obtained by running $\tau$ steps as (8) can be approximated as $N$ i.i.d. particles that evolve according to the DD at time $\tau\varepsilon$. Then, we utilize Lemma 1 to show that the global mode remains dropout-stable even when faced with data heterogeneity.

Following [20], we demonstrate that dropout-stable NNs in FL have mode connectivity as follows.

Theorem 2 shows that the global modes obtained from FL under different data heterogeneity can be connected, as shown in Figure 5, and then mode-connectivity error depends on the data heterogeneity $h(\alpha)$ and neuron numbers $N$. See the appendix for the detailed proof of Theorems 1 and 2.

To verify our analysis results, we undertake two classification tasks: i) using a two-layer NN on MNIST; ii) using a VGG11 network from scratch on CIFAR-10. The NNs take ReLU activations for both tasks and are optimized based on the cross-entropy loss. For the FL setup, we consider ten clients with $50/200$ training rounds for MNIST/CIFAR-10. Client optimizers are SGD with a learning rate of 0.02/0.01 for MNIST/CIFAR-10, where the mini-batch size is 50 and the number of local iterations is 10. Meanwhile, within the same task, the NN initialization is consistent under varying data heterogeneity and independent of neuron number $N$, which aligns with the theoretical assumptions. We perform 20 independent trials to measure mean value and standard deviation in Figures 8 and 9.

Dropout stability under varying data heterogeneity.

Figure 8 compares the dropout error $\varepsilon_{\mathrm{D}}$ under varying data heterogeneity as per Definition 3 on the training dataset (blue curve) and test dataset (orange curve). For MNIST, the neuron number of the two-layer NN is $N=1600$, and for CIFAR-10, VGG11 follows the standard setup in [24]. As expected from Theorem 1, the dropout error shows a downward trend as client datasets become more homogeneous. For example, the case of $\alpha=0.1$ has a more significant training/test dropout error than the i.i.d case at the different training stages of both tasks. When the same heterogeneity setting (e.g., $\alpha=0.1$) is adopted, the effect of data heterogeneity on CIFAR-10 is stronger than that on MNIST. This leads to a larger dropout error in CIFAR-10, where the dropout error is $\varepsilon_{\mathrm{D}}<0.04$ for MNIST and $\varepsilon_{\mathrm{D}}<0.15$ for CIFAR-10. Moreover, we also consider two training stages in Figure 8, including the middle stage (dashed curve) and the ending stage (solid curve). For both tasks, the dropout-error gap between the two stages keeps small, i.e., $\varepsilon_{\mathrm{D}}<0.01$ for MNIST and $\varepsilon_{\mathrm{D}}<0.05$ for CIFAR-10. As expected, this indicates that the mean-field theory can view the dropout-out dynamics as long as the training time is sufficient.

Mode connectivity under varying neuron numbers and data heterogeneity.

Figure 9 takes dropout stability (left sub-figure) and linear-interpolation barrier (right sub-figure) to validate our analysis on mode connectivity of global modes in FL. Theorem 2 shows that dropout stability implies mode connectivity, and the connectivity error $\varepsilon_{\mathrm{C}}$ depends on the neuron number and data heterogeneity. For the MNIST under $\alpha=0.1$, we consider a two-layer NN and plot the relationship between its dropout error $\varepsilon_{\mathrm{D}}$ and neuron number $N$. As expected, the training/test dropout error rapidly decreases as the width of the network grows. The dropout error for $N=6400$ is less than $0.01$, while for $N=200$, it is up to $0.06$. Then, we take the linear-interpolation barrier to explore the connectivity error $\varepsilon_{\mathrm{C}}$, and connect the global mode obtained in the case $\alpha=0.1$ to that of the cases $\alpha=0.2$ (solid curve) and $\alpha=0.5$ (dashed curve). With the expansion of the network, the barrier (i.e., connectivity error $\varepsilon_{\mathrm{C}}$) diminishes rapidly, which is consistent with dropout error $\varepsilon_{\mathrm{D}}$ v.s. neuron number $N$. Furthermore, the barrier of $\alpha=0.1$ to $\alpha=0.2$ is higher than that of $\alpha=0.1$ to $\alpha=0.5$ when $N=200$. This is expected because when $N$ is small, the connectivity error $\varepsilon_{\mathrm{C}}$ is amplified by the effect of data heterogeneity $h(\alpha)$, i.e., the effect of data heterogeneity amortized to neurons would be even greater.

Neuron noise $\bm{n}_{i}$ under varying data heterogeneity.

Figure 10 illustrates the L2 norm of the global-mode update noise under varying data heterogeneity in (7) along all iterations in the right sub-figure, where the mean, maximal, and minimal noise norms are reported in the left sub-figure. As shown in the left sub-figure, the noise norm decrease as data heterogeneity alleviates. Meanwhile, the maximal noise norm is limited, i.e., $<0.012$, but the noise norm does not decrease to zero even in the i.i.d case due to the randomness of SGD. Moreover, according to the right sub-figure, the noise is not strongly correlated with the iterations when the data heterogeneity is mild (i.e., $\alpha\geq 0.4$), and slightly decreases along the training iterations. This verifies Assumption 5, where the neuron noise arises from data heterogeneity and remains independent of the number of iterations.

Through experimental and theoretical analysis, we unravel the relationship between client and global modes in FL under varying data heterogeneity. As data become more heterogeneous, finding solutions that optimize both the client and FL objectives becomes increasingly challenging due to the reduction of overlapping solutions between client and global modes. Meanwhile, the FL solutions obtained from different setups of data heterogeneity belong to distinct global modes, and are located in distinct basins of a joint region. These solutions can be connected by a specific PolyChain while showing an error barrier along the linear interpolation. This suggests that global modes in FL are not isolated but rather interconnected within a manifold. Our analysis demonstrates that the connectivity error between global modes decreases with weakened data heterogeneity and wider trained models.

The findings suggest potential directions for future work for the design of FL: i) exploring the solutions to the FL objective that maintain low error for both client and global test sets, i.e., the solutions to both client and FL objectives; ii) investigating the effect of model depth and architecture in FL based on mean-field theory; iii) understanding FL training dynamics by the gap between random and pre-trained initialization (see a preliminary investigation in Appendix).