Privacy-Preserving Collaborative Split Learning Framework for Smart Grid Load Forecasting

This section reviews the recent ML methods proposed for load forecasting, followed by studies that use FL and SL for distributive load forecasting.

Consider a scenario where an energy provider company distributes power to multiple neighbourhoods/districts of a city. Each community is served by a single Grid Station (GS). The Service Provider (SP) is interested in training a load forecasting model for medium (few hours) to long-term (few days - weeks) forecasting to better manage their generation capacity and reduce energy waste. To do so, they can employ different strategies, e.g., the SP might want to learn a single prediction model for all districts, which is easier but not optimal as households across neighbourhoods may have different load profile distributions. Thus, training a single model to cover all distributions may lead to significant prediction errors. Conversely, training a single model for every client is cumbersome and infeasible if the number of clients is substantial. Instead of either of these extremes, we propose to learn a single model for each neighbourhood as one would expect clients from the same neighbourhood to have similar load profiles, facilitating the training of an accurate prediction model.

Next, we need to decide on a training framework, i.e., central or decentralized. As data privacy is our top priority, decentralized learning strategies like FL and SL should be employed where the private data never leaves the client’s premises. However, as the client-side training has to be performed by a smart meter, the training process’s computational and data transfer requirements have to be modest so as not to hinder their main functionalities. The SL framework is selected to ensure this due to its low computational and communications requirements and privacy-preserving nature.

The major contributions of this paper are as follows:

• •

  We propose a novel SL framework with a dual split strategy, i.e., the network’s first split (Split-1) resides at the GS covering a single neighbourhood, and the second split (Split-2) stays at the SPs’ end. To reduce computational load on smart meters, each client is only responsible for performing a forward pass on its private data using their GSs’ Split-1 network, computing the loss, and initiating back-propagation. GS is responsible for carrying out back-propagation through the Split-1 model and updating its weights. The models, as a whole, are trained using two alternative strategies: SplitGlobal, which trains unique Split-1 models for each neighbourhood and a global Split-2 model shared by all neighbourhoods, and SplitPersonal, which trains personalized split models (Split-1 and Split-2) for each neighbourhood. Once the training is complete, the SP will have access to both network splits and can perform individual-level (requiring respective clients’ involvement) and neighbourhood-level predictions using the cumulative load trend from the respective neighbourhood GS.

• •

  As our base learner, we utilize the transformer [12] based architecture, called FEDformer [16]. Compared with widely used LSTM, transformers enjoy better performance and can fully utilize the acceleration offered by discrete graphics processing units. Based on our literature review, ours is the first work that uses a transformer architecture to implement split learning for electricity load forecasting. Extensive experiments are conducted to assess the performance of the trained split model against a centrally trained model under multiple scenarios.

• •

  We present a detailed quantitative assessment of the extent of information leakage between clients’ private data and their respective Split-1 activations using mutual information-based neural estimation (MINE) [31]. Based on the estimated mutual information (MI) between input and activations, a vigilant client can decide whether the current activations batch is secure enough to be forwarded to the GS or not. For additional privacy, we incorporate differential privacy [29, 32] to further obfuscate the client’s Split-1 activations into the model framework and analyze its effects on information leakage and model performance.

The rest of the paper is organized as follows: Section II briefly describes FEDformer, the transformer variant used in this work. We further discuss the concepts of split learning, differential privacy, and mutual information neural estimation. Section III outlines the proposed system model, the FEDformer model split and the SL training framework. Section IV presents experimental evaluations of the proposed framework under different testing scenarios, followed by privacy leakage analysis using mutual information and differential privacy. We conclude the paper in Section V.

FEDformer follows the deep decomposition architecture proposed in Autoformer [14], where the input time series is analyzed by decomposing it into a seasonal and a trend-cyclical part. The seasonal part is expected to capture seasonality, whereas the trend-cyclical part is expected to capture the long-term temporal progression of the input. FEDformer uses a series decomposition block with a single or a set of moving average filters (of different sizes) to perform such decomposition. FEDformer implements self-attention mechanisms in the frequency domain using two distinct blocks, a Frequency Enhanced Block (FEB) and a Frequency Enhanced Attention (FEA) Block. Their working is briefly discussed next.

In split learning (SL) [7], a deep neural network (DNN) is split into two halves; clients maintain the first half and the remaining layers are maintained by a server. Consequently, a group of clients are able to train a DNN collaboratively using (but not sharing) their collective data. Additionally, the server performs most of the computational work, reducing the clients’ computational requirements. However, this comes at the cost of privacy trade-off, i.e., the output of earlier layers leaks more information about the inputs [33]. Here, choosing a suitable split size is important for expecting data privacy as it has been shown that for a relatively small client model, an honest-but-curious [34] server can extract the clients’ private data accurately just by knowing the client-side model architecture [28]. Thus, it is recommended that in SL, clients should compute more layers, increasing computational load but incurring stronger privacy [28].

During training, clients perform forward passes using their own data up to the final split layer of DNN. These activations are then shared with the server, which continues the forward pass on its DNN split. If the label sharing between clients and servers is enabled, the server can compute the loss itself. Otherwise, it has to send the activations of its final layer back to the client for loss computation. In this case, the gradient backpropagation begins on the client side, and the client feeds the gradient back to the server, which continues backpropagation through its DNN split. Finally, the server shares the gradients at its first layer with the client, who finishes the backpropagation through its DNN split. When more than one client is participating in training, SL adds all clients into a circular queue, whereby each client takes turns using their private data to train with the server. At the end of a training round, the client shares its updated model weights with other clients either through a central server, directly with each other, or via a P2P network.

One of the most widely used privacy-preserving technologies is Differential Privacy (DP) [29]. Its effectiveness in safeguarding user data privacy has been extensively demonstrated by adding carefully computed noise to the data. The machine learning community has widely used DP to ensure data privacy [35, 32]. Let $\mathcal{X}$ be the input space, $\mathcal{Y}$ the output space, $\epsilon$ the privacy budget parameter, $\delta\in o(\frac{1}{n})$ be a non-negative heuristic parameter, $n$ be the number of samples in the dataset, and $\mathcal{M}$ a randomization mechanism. We say that the mechanism $\mathcal{M}:\mathcal{X}\rightarrow\mathcal{Y}$ is $(\epsilon,\delta)-$differentially private ($(\epsilon,\delta)$-DP) if, for any neighbouring datasets $\textbf{X}_{1}$ and $\textbf{X}_{2}$ (differing by a single element) in $\mathcal{X}$, and any output $S\subseteq\mathcal{Y}$, as long as the following probabilities are well-defined, there holds

Intuitively, (3) provides an upper bound ($e^{\epsilon}$) on the difference between outputs of the mechanism $\mathcal{M}$ when applied to two neighbouring datasets, where the value of $\epsilon$ controls the overall strength of the privacy mechanism and $\delta$ accounts for the probability that privacy might be violated [32]. Thus, to ensure stronger privacy protection, both $\epsilon$ and $\delta$ should be kept low. With $\delta=0$, the pure $\epsilon-$DP is shown to be much stronger than the $(\epsilon,\delta)-$DP (with $\delta>0$) in terms of mutual information [36]. Nevertheless, the $(\epsilon,\delta)-$DP offers the advantage of advanced composition theorems, enabling a substantially greater number of training iterations compared to pure $\epsilon-$DP with the same $\epsilon$. As a result, most recent works in differentially private machine learning have shifted away from $\epsilon-$DP.

Let $f:\mathcal{X}\rightarrow\mathbb{R}$ be a deterministic real-valued function. Then, in order to approximate this function with an $(\epsilon,\delta)-$DP mechanism, noise calibrated with f’s sensitivity ($s$) is added to its output. Here, sensitivity is defined as $s=\max_{\textbf{X}_{1},\textbf{X}_{2}}\|f(\textbf{X}_{1})-f(\textbf{X}_{2})\|$. Intuitively, for high sensitivity, it is much easier for an adversary to extract information about the input [29]. The general mechanism that satisfies the DP is defined by

where $\eta$ is a random variable from distribution $\mathcal{N}(0,\frac{2s^{2}}{\epsilon^{2}}\log(\frac{2}{\delta}))$ under $(\epsilon,\delta)-$DP or $p(\eta)\propto e^{-\epsilon\|\eta\|/s}$ under $\epsilon-$DP [35]. In our case, $f$ is the split model, X is a clients’ private input dataset, sensitivity $s$ is computed across the batch axis of the Split-1 activations tensor, and the noise is added to the clients’ batch activations at Split-1 model output. In this way, the noise level is controlled by the sensitivity $s$, which comes from the data, the probability $\delta$, and the privacy budget $\epsilon$, which can be set according to the privacy requirements, e.g., $\epsilon=10$ results in a weak privacy guarantee as compared to $\epsilon\approx 0$, which gives the strongest privacy guarantee but makes the data useless. The privacy guarantee of a DP mechanism (4) is that the likelihood of revealing sensitive information about any individual in the input dataset through the algorithm’s output is significantly reduced [35]. In our study, we analyze both DP mechanisms in terms of the mutual information leakage between input and output of the clients’ split network.

The clients’ split layer activations in SL frameworks are shared with the server. However, these activations may carry enough information about the input that an adversary might be able to precisely reconstruct the original data [37]. Various researchers have utilized noise addition mechanisms offered by $\epsilon$-Differential Privacy as a security measure [28] to mitigate this issue. However, it is difficult to quantify the relationship between the added noise level and the information leakage risk. Mutual information (MI) is commonly used in information theory to assess how much information can be inferred from one random variable (RV) about another. Compared to correlations, MI can capture non-linear statistical dependencies between RVs [31]; however, it is difficult to compute, especially for high-dimensional RVs. In [31], the authors have proposed to compute MI using neural networks using the fact that MI between two IID RVs X and Y is equivalent to the Kullback-Leibler Divergence (KLD) between their joint ($\mathbb{P}_{\textbf{XY}}$) and product of their marginal ($\mathbb{P}_{\textbf{X}}\otimes\mathbb{P}_{\textbf{Y}}$) distributions. According to the Donsker-Varadhan representation of KLD [38], the MI between X and Y is lower bound by

Here $\mathcal{T}$ is any class of functions $T:(\textbf{x}_{i},\textbf{y}_{i})\rightarrow\mathbb{R}$ that satisfies the integrability constraints of Donsker-Varadhan theorem. Under this setting, the authors in [31] use a neural network to model $\mathcal{T}$, which converts the MI problem to a network optimization one, leveraging neural networks’ ability to approximate arbitrary complex functions.

Consider an SL framework where x and y are the batched inputs and outputs of the split network, and we are interested in finding how much information about x can be inferred from y. To do so, we need to estimate the MI between them. Ref. [31] says that this MI is lower bounded by (5). Let $T$ be a neural network. Then, the expectations in (5) are empirically estimated by sampling from joint distribution as $(\textbf{x},\textbf{y})\sim\mathbb{P}_{\textbf{XY}}$ and from marginals by shuffling y across the batch axis to get $(\textbf{x},\overline{\textbf{y}})$. In other words, in $(\textbf{x},\textbf{y})$ the input-output relationship is intact, whereas, in $(\textbf{x},\overline{\textbf{y}})$, this relationship has been broken. The network $T$ is trained by maximizing (5). Thus, if MI between a batched x and y is large, (5) computed using a trained network $T$ will be high, and vice versa. We demonstrate this effect in detail in Section IV-E1.

We consider a three-tier system model with four major entities, as shown in Fig. 2. On top, we have the electricity Service Provider; in the middle, we have Grid Stations responsible for distributing electricity to the individual districts/neighbourhoods. The lowest tier comprises smart meter clients (industrial, commercial, or residential) lumped into neighbourhoods. Under this model, the SP is an organization which procures electricity from various providers and is responsible for meeting the energy requirements of all connected GSs. Moreover, the smart meters can connect to their GS using a secure communications protocol, e.g., cellular network or power-line communications. The GSs are connected to SP via a private network or the Internet. The SP can not connect directly with any client and has to go through the GS to communicate with a client. The objective is to learn a DL time series prediction model, using a split learning framework, on all clients’ data without compromising the individual clients’ privacy. Once trained, the entire model will be accessible to the SP, and the SP can effectively perform medium to long-term load forecasting for any client from any neighbourhood.

In this paper, we assume a modest security environment, i.e., the GS, SP, and the clients are honest-but-curious [34, 28]. Thus, the participants may not try to poison the training process; however, both the GS and SP may collude to infer information about clients’ private data as they have access to the entire model and the client-side split layer activations (details in Section III-D). Furthermore, external adversaries or some malicious clients may try to intercept the split layer activations sent by the other clients to GS to extract their private data. Thus, the attack model considered here is the model inversion attack, which aims to extract clients’ sensitive data given only their activations. Under this attack, the objective of the adversary is to find a function $\mathcal{G}$ which can infer the client’s private data X from its split activations A as $\textbf{X}=\mathcal{G}(\textbf{A})$. However, in practice, this inference need not be exact, as a close approximation of the client’s data is usually enough.

We have selected FEDformer architecture [16] (with some modifications) as our backbone prediction model, where the entire model consists of two encoders and a single decoder block. The FEDformer model is split into two halves, termed Split-1 and Split-2. The Split-1 model contains the first two inner blocks of the FEDformer encoder and FEDformer decoder, whereas the Split-2 model contains the remaining inner blocks of the first FEDformer encoder followed by a complete FEDformer encoder block and remaining inner blocks of the FEDformer decoder. Each GS gets its own copy of the Split-1 model, and the SP maintains Split-2. The resulting split FEDformer is shown in Fig. 3. In Section II-A we have discussed the series decomposition, FEB and FEA blocks, and the rest are briefly discussed next.

The data embedding layer included in model Split-1, as shown in Fig. 3, consists of a series decomposition layer, a 1D convolutional layer, and two linear layers. It takes three inputs, an input time-series of length L $\textbf{X}\in\mathbb{R}^{L\times Z}$, where $Z$ is the dimension of each time point (for univariate case, $Z=1$), the timestamp encoded information of the input time-series as $\textbf{X}_{t}\in\mathbb{R}^{L\times U}$, and output time-series $\textbf{Y}_{t}\in\mathbb{R}^{(L/2+O)\times U}$, where $U$ depends upon the temporal granularity, and $O$ denotes the prediction time horizon, for instance, for the scale of Year-Month-Day-Hour, $U=4$. First, X is passed through the series decomposition block to get a seasonal component $\textbf{X}_{S}$ and a trend component $\textbf{X}_{trend}$. Then, the embedded inputs to the encoder and decoder blocks of Split-1 are generated as:

where $\textbf{X}_{0},\textbf{X}_{mean}\in\mathbb{R}^{O\times Z}$ denote the placeholders filled with zeros and mean of X, respectively, $\textbf{X}_{emb}\in\mathbb{R}^{L\times D}$, $\textbf{S}_{emb}\in\mathbb{R}^{(L/2+O)\times D}$, and $trend_{ini}\in\mathbb{R}^{(L/2+O)\times Z}$. Here, $D$ is the inner dimension of the model. In this way, the decoder takes guidance from the later half of the input time series to fill in the remaining placeholder data points during training. Since the series-wise connection will inherently keep the sequential information, we do not need to perform position embedding, which differs from vanilla Transformers. This specific setting for the Split-1 network was chosen to keep the computational requirements low while ensuring a complex non-linear relationship between the input and outputs of Split-1 to ensure low information leakage between them (see details in Section IV-E1).

The feed-forward network, seen in the Split-2 encoder and decoder, is a two-layer fully connected neural network (FCNN) with input and output dimensions $D$, and inner dimension $D_{ff}$. The output of its first layer is passed through GELU activation function before passing to layer two. The output (seasonal and trend) tensors generated by series decomposition blocks found in Split-1 and Split-2 decoders have dimensions $D$, whereas the incoming $trend_{ini}$ and $trend_{dec}^{S_{1}}$ containing temporal trend information are $Z$ dimensional (dimension of the input time series). Thus, the trend output tensor of a series decomposition block is first projected back into $Z$ dimensional tensors using trainable projection matrices $\textbf{W}_{i}\in\mathbb{R}^{D\times Z}$, where $i=\left[1,2,3\right]$, before adding to the incoming trend tensors. Similarly, the seasonal tensor of the final series decomposition block of the Split-2 decoder is also projected down to $Z$ before adding to the incoming trend tensor to generate the final prediction output.

Figure 2 depicts the SL training process for load forecasting. Note that the clients do not transfer prediction targets (labels) to the GS or SP. As discussed in Section III-C, the FEDformer is split into two halves, and each GS maintains a personalized copy of model Split-1. The model Split-2 resides at the SP. Unlike a traditional Split Learning framework where clients are responsible for the forward pass, backpropagation through their network split, and parameter updates, our framework proposes a different approach. In our proposed framework, smart meters perform only the forward pass through their data, compute loss (after receiving predictions from SP through GS), and initiate backpropagation (gradient computation of loss with respect to the predictions only). The GS performs backpropagation through Split-1 and parameter updates, significantly reducing the computational load on the smart meters. As a byproduct, we do not need to over-simplify the Split-1 model; thus, non-linear relationships between clients’ inputs and activations are maintained, ensuring stronger privacy.

Two strategies are employed when training at SP, i.e., SP can learn a single Split-2 model for all GSs (neighbourhoods) or one personalized Split-2 model per GS. In the case of the former, the overall learned model is referred to as SplitGlobal, whereas the latter model is called SplitPersonal. The goal is to analyze the generalization capabilities of the model when performing predictions for clients from the same as well as those coming from different neighbourhoods, as clients from different neighbourhoods are expected to have different load patterns and distributions. Nevertheless, the training process is relatively similar for both models. Algorithm 1 outlines the pseudo-code for training the SplitGlobal variant of the proposed SL model.

We start with weight initialization for all SP and GS models. At the start of each training epoch, A GS selects all or a subset $C_{t}$ of $K$ neighbourhood clients (chosen randomly or the same as the previous epoch). Following this, the training begins in parallel across all GSs, where the participating clients use a single private training batch to perform the model update in 7 distinct sequential steps (see Fig. 2). These steps are summarized in relation to Algorithm 1 next.
Steps 1-2: Each GS sends its Split-1 model weights to all selected neighbourhood clients $C_{t}$. In parallel, each receiving client $k$ performs a forward pass through the received Split-1 model using one of its private training batches $b$. The activations of the final split layer, denoted by $\textbf{A}^{GS}_{k,b}$ of Split-1 model are forwarded to the GS (lines 8-11 of Algorithm 1).
Step 3: Once a GS has received batched activations from all of its training clients, it concatenates their activations, denoted by $\textbf{A}_{b}^{GS}$, and forwards them to the SP to continue the forward pass (lines 12-13 of Algorithm 1).
Step 4: Depending upon the model being trained, i.e., SplitGlobal or SplitPersonal, this step is performed little differently. In the case of SplitGlobal, the SP performs the forward pass through a single Split-2 model using a batch size of $K\times(clients\,batch\,size)$, where $K$ denotes the number of clients (lines 14-15 of Algorithm 1). For SplitPersonal, the SP performs a forward pass through the individual personalized Split-2 models using the activations received from their respective GSs.
Step 5: As the training data is never allowed to leave the client’s premises, the final Split-2 layer outputs are forwarded to their respective clients through GSs for loss computation (lines 17-19 of Algorithm 1).
Step 6: Each client computes loss, initiates back-propagation and shares gradients w.r.t. outputs $\textbf{O}_{k,b}^{GS}$ with their GS, that forwards these gradients to SP (lines 20-22 of Algorithm 1).
Step 7: Once gradients from all GSs are received, SP continues the gradient back-propagation through its Split-2 model(s). The gradients w.r.t. each $\textbf{A}_{b}^{GS}$ are then forwarded back to their respective GSs. Each GS averages the received gradients across clients’ dimensions and continues back-propagating through their respective Split-1 models (lines 24-25 of Algorithm 1).
Step 8: Once all gradients have been populated, SP and each GS update their model weights (line 26 of Algorithm 1).

Following the model updates, the GSs pass their updated models back to their respective clients for another round of batch training. This is repeated until all batches have been iterated over, thus completing a single training epoch. For the next epoch, GSs can continue training with the same clients or select new ones. Once training is finished, each GS will have a personalized Split-1 model.

In our first experiment, we compare the performance of models trained under the proposed SL framework with a centrally trained FEDformer model. To do so, we train SplitGlobal and SplitPersonal models for 3 GSs and 10 clients per GS. The clients are kept fixed during all training epochs. For the centralized model, we choose the same 10 clients from each GS (30 clients total), as used by our proposed models to train a univariate FEDformer model in a centralized manner. The objective is not only to validate our proposed SL training framework but also to assess the efficacy of training multiple models w.r.t. a single central one. The resulting MAE, MSE, and $R^{2}$ scores on the test sets of clients belonging to different neighbourhoods are summarized in Table II. The MAE, MSE, and $R^{2}$ metric scores for GS 2 and 3 for all 3 models are relatively similar, with the Centrally trained model edging the proposed models slightly in terms of MAE, whereas, in terms of MSE and $R^{2}$, the SplitPersonal model performing better. These results show that all 3 models could generalize the load patterns for GS 2 and 3’s neighbourhood clients. The load profiles of these clients were not too erratic compared to clients belonging to neighbourhood 1, for whom the centrally trained model performed objectively worse.

As discussed in Section IV-1, the load profiles for clients belonging to cluster 1 (GS 1) show high diversity and are the most challenging out of the three. Observing the superior performance of the proposed models as compared to the central model shows that expecting a single central model to perform well for a diverse range of load profiles is not viable. Moreover, learning multiple models for data from similar distributions should work comparatively well. Thus, the proposed framework essentially offers a balanced approach between learning a single model for all clients and learning a single model for each client.

When comparing scores of SplitGlobal and SplitPersonal, we see that the latter performs well both in terms of individual GS scores and average ones. This, however, is expected as for SplitPersonal, the SP trains personalized Split-2 networks for each GS. Thus it should be able to generalize well for the respective neighbourhoods. To visualize the model convergence, we present the MSE scores for train, validation, and test sets for the 3 models in Fig. 5. Here, we see that although the central model achieved the lowest training error, its validation and test set errors are the highest, owing to over-fitting to the training data. We can say this because most, if not all, of the difference between central and proposed methods test set scores, is coming from its prediction scores for the GS 1s’ load profiles (see MSE scores given in Table II). As otherwise, the test scores for the central model over GS 1 and 2 were very close to the proposed model’s scores. The run-times of a single epoch for SplitGlobal, SplitPersonal, and Central models are 20, 20, and 40 minutes respectively.

In our next experiment, we analyze the trained models’ prediction ability when the tested data comes from another GSs’ neighbourhood, i.e., from a different distribution. The MSE scores for both models are reported in Table III, where rows GS 1-3 denote the trained split models and columns denote the neighbourhoods. Thus, the table cell for row GS 1 and column 1 represent the MSE score for neighbourhood 1’s test data when GS 1’s model is used for prediction. Consider GS 1 models’ scores for all neighbourhoods (top row) under both SP training strategies. We see that, apart from testing scores for their own neighbourhood data, the SplitGlobal models’ prediction errors for neighbourhoods 2 and 3 are well below those reported by SplitPersonal model (see Fig. 6 for individual scores). This is attributed to the fact that under SplitGlobal, the SP learns a single global Split-2 model, which is jointly trained to learn from all neighbourhood clients. Whereas, under SplitPersonal training strategy, the SP learns 3 personalized Split-2 models, which have never seen the data coming from different neighbourhoods. Similar trends can be observed for models GS 2 and 3 where cross-neighbourhood scores reported by SplitGlobal are better. Similarly, the mean scores for a single neighbourhood overall GS 1-3 models also show that SplitGlobal has better generalization capabilities as compared to SplitPersonal.

To visualize individual test scores for each neighbourhood client under different learned models, we present their MAE scores in Fig. 6. Here, the first 10 clients are from neighbourhood 1 and so on. For all 3 models, we see that the most MAE variation is found in neighbourhood 1’s clients, whereas for 2nd and 3rd neighbourhoods, the variations are small. Moreover, for each neighbourhood, their respective GS models perform best. We further present example batch predictions for 4 clients in Fig. 7. Looking at predictions from both models, we see that for clients 15 and 23, all three GS models do follow the true trajectory very closely, with SplitGlobal’s cross neighbourhood predictions being slightly better. However, for clients 3 and 5, belonging to neighbourhood 1, discrepancies are large, especially for cross-neighbourhood predictions of client 5. Moreover, SplitPersonal’s GS 1 model followed the true trajectory much more closely, especially near the prediction endpoints.

In this experiment, we test the trained models’ prediction abilities for clients that are completely new to them. Under usual circumstances, we might not be able to use every client for training; thus, the trained models’ generalization capabilities need to be tested. To do so, we start with the models trained on the same 10 clients per neighbourhood and test them on 10 randomly selected clients from each neighbourhood. The results of this test are given in Table IV, where we see that clients coming from 2nd and 3rd neighbourhoods receive slightly worse prediction errors, whereas for neighbourhood 1, the error increases by almost 66%. This performance loss, again, can be attributed to the data diversity found in clients from Neighbourhood 1. To investigate whether further training using these clients brings improvement, we train the model for 5 additional epochs using the selected random clients, whose testing results are summarized in the last two columns of Table IV. We see that additional training not only reduced the error for neighbourhood 1 from 66% to 35%, but also improved scores for the rest of the neighbourhood clients.

In this experiment, we compare the prediction performance of models when trained using the same clients (10 per GS) compared to randomly selected clients at every training epoch. The models were trained for 10 epochs and tested against randomly chosen clients, and their results are shown in Table V. Comparing SplitPersonals’ GS 1 performance for both training schemes; we see that scores for the model trained using random clients are significantly lower than those trained using the same clients. This could be attributed to the fact that in the former case, the model sees several unique clients during the training stage and is thus able to generalize well for unseen data. Apart from this case, the remaining scores are very similar across the two training schemes. From this observation, we can conclude that training using randomly chosen clients every epoch should enable the models to perform well when presented with unseen data. However, as discussed in the previous section, performing a few training iterations using the unseen data should be performed to get improved results.

Under the proposed framework, both network splits are trained outside the clients’ premises, and the training is fully controlled by GS and SP entities. Under the honest but curious security assumption, GS and SP shall not deviate from the training process. However, they may try to infer clients’ private data from the received Split-1 activations. In order to secure clients’ private data, we propose to use $\epsilon$-DP [29, 28] to safeguard against such inference attacks. However, before we do this, we first analyze information leakage between the input and output of the proposed Split-1 FEDformer model. To this end, based on the discussion presented in Section II-D, we perform MI estimation using a fully connected neural network (FCNN). A similar strategy has also been used in [42] to infer MI between inputs and gradients of an NN under the FL framework.

In Section IV, we used two SL strategies, SplitPersonal where we train neighbourhood-level personalized split networks, and SplitGlobal where a single global Split-2 is trained at SP for all personalized Split-1 models at GSs. Networks trained under both strategies achieved better or comparable performance compared to a centrally trained network, as seen in Table II. When predicting across neighbourhood clients, SplitGlobal model was able to get lower errors as compared to SplitPersonal, as shown in Table III, which is expected.

In Section IV-C, we tested the trained models on data from clients not used during the training stage. The results in Table IV show that the trained models performed well in this scenario; however, additional training using the new client’s data improved performance. This is essential as new clients are constantly added to the system, and we might have very little data on them. Additionally, in Section IV-D, we compare the performance of models trained on the same clients vs training on random clients every epoch against unseen data and found that the models trained on random clients performed better. As with the availability of large amounts of smart meter data, training using all of it is often not feasible. Instead, training using a random subset of clients every epoch can lead to a model with good generalization capabilities. Furthermore, this model can be refined using unseen clients’ data for added performance.

In Section IV-E, we analyzed the extent of privacy leakage arising from the sharing of clients’ activations using MINE. In Fig. 8, we showed that even without the added noise, the Split-1 activations have significantly low MI w.r.t. the inputs due to their non-linear and complex relationship induced by the Split-1 model. Moreover, based on the estimated MI, a client can decide whether to forward the current batch activations to GS or not to mitigate privacy leakage. We further analyze the effects of introducing differential privacy as an additional layer of security on the model’s performance. In Fig. 10, we see that with a moderate privacy budget of $\epsilon=5.0$, the models performed similarly to the non-DP case and saw performance degradation only when $\epsilon\leq 2.5$, leading to strong privacy. With a trained model, the electricity service provider can perform individual-level predictions (requiring respective clients’ involvement) and neighbourhood-level predictions using the cumulative load trend from the grid station servicing the neighbourhood. The LSTM-based FL models [22, 19] proposed so far were able to make 1h to 24h ahead predictions; our transformer-based model can perform accurate predictions from 24h - 720h [16], although our experiments were limited to 96h ahead predictions only.