OCTOPUS: Overcoming Performance and Privatization Bottlenecks in Distributed Learning

This work aims to develop a practical distributed learning scenario in which a variety of downstream tasks can be learned from dynamically-updated and non-iid distributed data sources. Intuitively, we hope to learn a global feature dictionary, and distributed local encoder to map a high-dimensional sample to the dictionary indexes, then transmit the index matrix and learn multiple downstream tasks on the gathered compressed features. The global dictionary is expected to be updated easily to address local data bias and dynamic updates. Further, plug-in privatization of local data could be achieved by forcing the global dictionary to carry only those features that are not sensitive.

There are many practical scenarios, for example, speech-to-text recognition. Speech consists of content (phonemes) and style (speaker identification). It has demonstrated that the global dictionary learned for speech can be highly related to phonemes with little speaker identification information. Hence, the same sentence spoken by different speakers would be projected to a similar dictionary item. As a result, the local encoder could map speech to the index of the dictionary to transmit instead of the raw speech. Furthermore, the dynamic dictionary update could consider the spatial and temporal bias associated with local data. Multiple downstream tasks could be conducted on the gathered compressed features via a simple model with less computation than raw data.

Formally, we consider the problem setting where a stream of gathered samples $x\sim D_{i},~{}x\in\mathbb{R}^{n}$ is continuously and independently collected from $M$ different distributed devices $D_{i},~{}i=1,\cdots,M$, to a server for model training. The variety and quantity of the training data collected from distributed devices at a high frequency determine the robustness of the final model. Using this technology, it is expected to be possible to collect the distributed information without having to incur large communication overheads and without being concerned with the possibility that sensitive information $x^{s}$ could be derived from the transmitted data $\hat{x}$. Further, the compressed representations that reveal most of the important features of the data could address the computational and storage concerns.

Consequently, we propose the OCTOPUS distributed learning scheme, which is designed to possess the following properties:

(a) Encoding: transforms raw data to low dimensional latent space compressed representation features (latent codes for short) for transmission and downstream tasks, $E:X\rightarrow Z$ and $Z\in\mathbb{R}^{k},~{}k\ll n$, to address the communication constraints. This encoding is characterized by the conditional probability distribution $P_{Z|X}$;

(b) Disentanglement: the encoder is incorporated with an additional disentanglement strategy to facilitate the decomposition of the latent representation, $Z$, into a public component $Z_{\bullet}$ irrelevant to PII, and a private component $Z_{\circ}$ relevant to PII, i.e. $DT(Z)=Z_{\circ}+Z_{\bullet}$, to address privatization on local data. Specifically, we isolate sensitive and non-sensitive attributes into separate subspaces while ensuring that the latent space factorizes these subspaces independently.

(c) Flexible and Stabilized Training: all collected latent codes $Z$ are stored and used for training of downstream models $T_{i}$, e.g. classifier, at the cloud. The compressed version of global data could address the computing and storage overheads while taking advantage of the global data. Training the encoding mechanism is expected to require a minimal allocation of communication and computation resources for each distributed node while permitting continuous updates.

As shown in Figure 1, the workflow of the OCTOPUS is:
Step 1. Learn an initial global DVQ-AE on a relevant public-available dataset at the server. This involves learning a global feature dictionary (which is composed of K items with dimension M), encoding an input into a H*W feature vector with dimension M, and finding the index of the most similar dictionary item for each feature vector, resulting in H*W index matrix as the compressed features.
Step 2. One-shot locally fine-tuning with distributed encoders and a joint decoder for the local DVQ-AE at each participant in a distributed manner based on the initial global DVQ-AE and the global dictionary.
Step 3. Based on the disentanglement strategy of the local fine-tuned DVQ-AE, each distributed participant can release only the latent codes as public components to the server, without concerns of releasing private information or incurring the additional cost of encryption or perturbation.
Step 4. Local encoder transmits low-dimensional compressed latent features of collected samples at a higher frequency.
Step 5. When new data comes in, the distributed autoencoder will be continuously updated by updating the local codebook via a simple exponential moving average, instead of retraining the local encoder and decoder, and then sent to the global dictionary at a lower frequency.
Step 6. A variety of downstream tasks could be learned from the gathered compressed features via simple models at the server. The simple inference model could be returned to the node for real-time inference if required.

OCTOPUS consists of the basic Distributed Vector Quantized Autoencoder (for general communication, detailed in Section 2.3) with three performance enhancement strategies: Group and Sliced Vector Quantization (for accuracy performance, detailed in Section 2.4), Disentanglement (for privatization performance, detailed in Section 2.5), and Flexible and Stabilized Training (for computing and storage performance and continuously updating, detailed in Section 2.6) strategies. The basic Distributed Vector Quantized Autoencoder consists of an encoder $E$, a vector quantization operation $VQ$, and a decoder $D$. $E$ maps the sample into latent representation. $VQ$ finds the nearest embedding from a learned codebook for each latent representation vector to generate a discrete latent code using the embedding index. $D$ decodes latent embedding back to input space. Data collected from each participant will be processed locally to reduce the dimension using the fine-tuned encoder to take advantage of the data locality at each node. The most critical features in the data are extracted via the local encoder, and low dimensional latent codes reduce the dimension of the input data. We learn and transmit the expressive encoded latent codes of the data associated with local privatization through the disentanglement strategy during the client stage. During the server stage, various downstream tasks, e.g., classifications, are performed on the collected latent codes, alleviating computing and storage overheads. If required, the privacy-preserving representation of the raw data can be reconstructed at the server using the gathered public components and replaced with private components via the fine-tuned decoder.

The basic encoding function is introduced in this section. Generally, the VQ-based autoencoder consists of three components, encoder $E$, decoder $D$, and codebook $C$. For the DVQ-AE, the encoder maps the sample $x$ into latent representation $E(x)=Z_{e}(x)\in\mathbb{R}^{H\times W\times M}$, where $H\times W$ is the number of the embedded feature vectors, and $M$ is the dimension of the feature vector. To further restrict the embedding space and compress the $Z_{e}(x)$ to a more low-dimensional latent representation, we apply the vector quantization operation to transfer the $Z_{e}(x)\in\mathbb{R}^{H\times W\times M}$ to $Z(x)\in\mathbb{R}^{H\times W}$ using a codebook $C$.

In particular, the codebook $C$ is regarded as the common feature dictionary that is shared between encoder and decoder to restrict the embedding space, consisting of K items of M-dimensional embedding vector (called atoms), i.e., $e\in\mathbb{R}^{K\times M}$. The vector quantization operation finds the nearest embedding $e_{i}\in\mathbb{R}^{M}$ from the learned codebook for each $M$-dimensional vector of $z_{e}(x)\in Z_{e}(x)$, using the index $[i]$ of $e_{i}$ to construct the discretized latent codes $z(x)\in[K]^{H\times W}$. Finally, the decoder $D$ retrieves the index matrix $z(x)$ back to corresponding embeddings $z_{q}(x)\in\mathbb{R}^{H\times W\times M}$ according to the codebook, and decodes $z_{q}(x)$ back to input pixel space. Parameters of the encoder, decoder, and codebook are the trainable parameters. And the learning objective of DVQ-AE is given as:

The first part of the loss function is a reconstruction loss $-logp(x|z_{q}(x))$, which represents the negative log-likelihood between the outputs of the encoder and the decoder after quantization respectively. The second part is codebook loss that determines VQ embedding vectors $e$ to figure out encoder results $z_{e}(x)$. The third term is the commitment term, which makes the encoding commit to a VQ embedding vector $e$ and constrains how the VQ space is used. Here, $sg[\cdot]$ is the stop gradient operator, maintaining its argument during the forward pass and returning zero gradients during the backward pass. The gradient of this non-differentiable step is approximated using the straight-through estimator.

As the first enhancement of DVQ-AE, this strategy is used to increase the accuracy performance affected by local data’s spatial and temporal heterogeneity. For general VQ-VAE, no constraints are placed on the atom distribution in the embedding dictionary, enabling atoms adjacent to one another to represent entirely different features. To mitigate the distortions caused by the mismatch of atoms, a Group Vector Quantization (GVQ) is adopted by GSVQ to decrease the probability of quantizing the encoder’s output to a mismatched atom, similar to group sparse coding algorithms [13]. GVQ aims to bring similar atoms closer to each other, and different atoms further away. GVQ divides codebook $e\in\mathbb{R}^{K\times M}$ into $G$ groups along K dimension, and $e^{(i)}\in\mathbb{R}^{N_{g}\times M},~{}N_{g}=\frac{K}{G}$ denotes the $i-$th group. During forward-propagation, each output of the encoder $z_{e}(x)$ is mapped to the nearest atom group, based on the average distance over all the atoms in the group.

The corresponding $i^{th}$ index of latent code for $z(x)$ is then computed as the weighted average of the atoms in the $e^{*}$.

All atoms will be updated in the back-propagation, based on the loss in Equation (1), by replacing $e^{*}$ with $\frac{\sum_{k=1}^{M}w_{k}e^{*}_{k}}{\sum_{k=1}^{M}w_{k}}$.

In addition, Sliced Vector Quantization (SVQ) [14, 15] is adopted by GSVQ to improve the efficiency of nearest neighbor embedding further. Each atom of the codebook is separated into $n_{c}$ parts along the M dimension, i.e. $e=\{e^{j}\in\mathbb{R}^{K\times M/n_{c}}\}_{1}^{n_{c}}$. Accordingly, the output of the encoder $z_{e}(x)$ is divided along the M dimension into $n_{c}$ components, and the VQ procedure is conducted in terms of the separated codebook at the corresponding spatial location. Figure 2 presents the GSVQ diagram.

We aim to disentangle the latent representation into two components. One reflects the semantics behind a specific sensitive attribute, and the other is irrelevant to this attribute as much as possible. Namely, we assume that the latent codes of instances consist of a public component (e.g., speech-to-text recognition) and a private component (speaker identification).

Generally, two separate encoders and sensitive classifiers are required to extract style embeddings (private component) and content embeddings (public component), respectively, e.g., [16]. However, it has been demonstrated that the style encoder is redundant [17, 18, 19]. Therefore, we adopt the single encoder scheme to reduce the size of the parameters. Besides, supervision over the latent factors is always limited. Therefore, we organize the training samples for the disentangled GSVQ model in groups as well, where the samples share a common specific attribute within a group. For example, the group can be a set of face images with the same facial expression among different identities or the same phonemes voice belonging to the different persons. Group supervision enables the alignment of the semantics of the data (identity and other attributes) into the learned latent representation, as a form of weak supervision that is inexpensive to collect. Note that the independent and identically distributed (iid) is not necessary in the case of grouped samples. The only supervision at training is the organization of the data into groups [20].

OCTOPUS’ privatization service relies on disentanglement strategies only. To address the privatization of the local data, we first apply two disentangled strategies to divide the latent representation into public components (e.g., phonemes of speech) and private components (e.g., speaker identification). The disentanglement strategies employed by OCTOPUS are codebook quantization and instance normalization, without the use of identifiable information classifiers or adversarial training. The first disentangled strategy is codebook quantization. A well-trained vector quantized model can learn some commonly-shared features as a similar series of atoms of the codebook. For example, the codebook learned for speech can be highly related to content such as phonemes [21, 17, 22, 23]. The same sentence spoken by different speakers would thus be projected to a similar codebook series. We force the codebook as the carrier for the commonly-shared features only (public component), while learning to represent private components such as speaker identification using the information discarded by the codebook quantization, i.e., the difference between continuous space and the discrete codes. The second disentangle strategy is Instance Normalization [24]. To further reduce the information of the public component (content) relevant to the private component (style), the Instance Normalization is adopted as the style normalization strategy for the codebook learning. IN can normalize the style of each individual input to the standard style.

Given any input $x\in R^{C,H,W}$, normalized representation $IN(x)$ is computed based on the channel-wise mean $\mu$ and the channel-wise standard deviation $\sigma$ across spatial dimensions independently for each channel and each input.

As the style information $\mu$ and $\sigma$ are temporally invariant, they can be considered as private (style) representations. Consequently, an IN layer is added in the encoder block (before the VQ step) to filter more information about the private component from the encoded latent representation. Figure 3 shows the framework of the disentanglement strategies.

Formally, let $X^{c}=\{x_{1},x_{2},\cdots,x_{T}\}^{c}$ as a group of samples with the same sensitive class $c$, such as a sequence of acoustic features or a set of face images belonging to a specific person. Given $X^{c}$, the public component $Z_{\bullet}$ and the private component $Z_{\circ}$ is given as

Here, $Z_{\circ}$ can be considered as the expectation of difference between $Z_{e}(x)$ and its vector quantized codes $Z_{\bullet}$. Then, $Z_{\bullet}$ is added back to $Z_{\circ}$, fed into the decoder for the reconstruction.

Further, the latent loss $L_{l}$ is added to the reconstruction loss, aiming to minimize the distance between the $Z_{\bullet}$ and the $Z_{\bullet}$ via the IN layer. Accordingly, the total loss is updated to be

A disentanglement strategy can be included in the initial DVQ-AE training at the server, followed by the deployment of the trained DVQ-AE to the distributed nodes. It is up to the data owner to decide whether to incorporate the sensitive components or not in the local data sharing.

This strategy improves computing and storage performance and continuously updates local and global encoder/decoder/codebooks. The distributed DVQ-AE is fine-tuned at each node by alternately updating the distributed encoder, joint decoder, and codebook on local data. Initially, the codebook is frozen for local fine-tuning and then updated at a lower frequency when local distribution changes. Local encoders and joint decoders have a fixed update frequency (e.g., one-shot fine-tuning), using local data to fine-tune. Updating the codebook can address the drifting representation issue when the current DVQ-AE is not a good approximation for new data. The local codebook can be fine-tuned by updating the atoms of the codebook with an exponential moving average instead of the loss term [22], where it was used to reduce the variance of codebook updates. Specifically, $\{z_{i,1},z_{i,2},\cdots,z_{i,n_{i}}\}$ denotes the set of $n_{i}$ outputs from the encoder that are closest to atom $e_{i}$ in the codebook so that the loss used to update the local codebook is:

The optimal value for $e_{i}$ is simply the average of elements in the set:

The exponential moving average procedure is conducted as:

with $\gamma=0.99$. The codebook update is conducted in a relatively lower frequency, such as monthly updates using the weekly samples $e^{(t)}_{i}$, then sent to the server.

In this section, we conduct quantitative evaluations of the communication overheads for Octopus, compared to federated learning and split learning. Let $N_{C}$ denote the amount of clients, $N_{M}$ for the size of model parameters, $N_{D}$ for the size of data, $N_{E}$ for training epoch, $N_{Z}$ as the size of latent codes of Octopus. $N_{S},~{}\eta$ denote the size of the smashed layer and the fraction of parameters for the clients for split learning. Communication efficiency here measures the data volume transmitted across all clients and servers for training and synchronization. The data set is assumed to be equally distributed among clients for all settings for simplification. We also ignore the temporal heterogeneity that the distribution of each local dataset may vary with time.

Overheads of ordinary FL. For ordinary federated learning schemes, the communication for each client is to upload gradients of parameters and then download the averaged gradient with the server for each training epoch. Then the total communication is $2\times N_{C}\times N_{M}\times N_{E}$.

Overheads of gradient compression for communication efficiency. Gradient quantization and sparsification are two commonly adopted approaches to compress the uploading gradients towards the communication efficiency of FL [30, 31, 32, 9, 33]. Generally, the communication is conducted on some selected clients via uploading compressed gradients of parameters and then downloading the averaged gradient with the server for each training epoch. The total communication is $(N_{C}^{selected}\times N_{M}^{up}+N_{C}\times N_{M})\times N^{\prime}_{E}$. The communication compression via quantization, sparsification or clustering strategies could reduce the $N_{C}^{selected}\times N_{M}^{up}\times N^{\prime}_{E}$ part. However, the compressed/quantized communication results in distortion of the gradients and a much lower learning rate, which further significantly increases the more communication rounds necessary to reach convergence, i.e., $N^{\prime}_{E}\gg N_{E}$, even fail to converge. Besides, extra complex procedures are required to handle the hard-to-converge issue, which is also not well generalized to some models such as ResNet101. For complex models (e.g., BERT), quantization-based approaches incorporate a significant decrease of accuracy. Furthermore, downstream update part $N_{C}\times N_{M}\times N^{\prime}_{E}$ will not even be compressed at all.

Overheads of split learning. For split learning, the communication for each client, is to upload activations during forwarding propagation and download gradients during backward propagation. If considering weight sharing among clients to enhance synchronization, even with the cost of more information leakage, the total communication is $(2\times N_{S}\times N_{D}+\eta N_{C}\times N_{M})\times N_{E}$. The communication efficiency is then defined as the ratio of data transfers of federated learning and split learning, i.e., $\rho=\frac{2\times N_{C}\times N_{M}\times N_{E}}{2\times N_{S}\times N_{D}+\eta N_{C}\times N_{M}}$ [34], which could be used to decide the learning scheme according to the ratio.

Overheads of heterogeneity. If taking the extra strategies to address the data heterogeneity issues into consideration, besides the computation overhead and decrease of the accuracy, the direct influence for the communication is the increased $N_{E}$ caused by more iterations for model convergence. Further, if we also consider the temporal heterogeneity that the distribution of each local dataset may vary with time, a large number of iterations is necessary to fine-tune the model using new data, even retraining.

Overheads of multi-task scenarios. In this work, we aim at the multi-task scenarios as well, where various models are required for massive downstream tasks. For federated learning and split learning, the entire training procedures are needed to be rerun many times, resulting in multiplied communication and computation overheads.

Overheads of Octopus. For Octopus, the communication for each client is to upload the latent representation of the collected data and then download the trained model for once-off. Then the total communication is $N_{D}\times N_{Z}+N_{M}+\pi N_{B}+N_{A}$. Generally, the communication overhead of Octopus is far less than the federated learning and split learning. The advantages of the Octopus in the bias of overheads are summarized as follows, compared to existing FL solutions.
(1) Communication size and round. The general communication size for Octopus is $N_{Z}$, which is $\ll N_{M}$ and $N_{S}$ for others. The communication round for Octopus only involves (a) once-off data collection and finalized model download, i.e., the $N_{E}=1$, avoiding massive communication rounds $N_{E}$ used for training iterations in other solutions. The initial autoencoder and the final downstream task models are trained at the server, and only once-off download communication $N_{A}$ is required from the server to the clients. (b) few-shot codebook updates. $N_{B}$ is the size of the codebook, at most, $256\times 64$ (e.g., for 1024*1024*3 images) in our experiments, and $\pi$ is a low-frequency updating rate of the codebook, generally less than 10.
(2) Model independent and lightweight. As the complexity of the learned model grows, so does the communication overhead and the number of iterations for the existing FL scheme. Existing communication compression and optimization strategies may also be ineffective when dealing with complex models. In Octopus, the distributed pre-trained encoders serve as the data collector and feature extractor, allowing model-agnostic and independent communication and distributed learning, via lightweight models.
(3) No/Tiny burden to handle heterogeneity. Due to the global data collection, Octopus does not have to add an extra communication burden in order to handle the heterogeneity issues. Octopus uses a non-training strategy to deal with temporal heterogeneity, i.e., a simple exponential moving average. The communication overhead is only increased by few-shot updating iterations on the codebook.
(4) Multi-task. For Octopus, there is also no uploading communication burden added to the multi-task training, and only once-off communication to download trained models to clients. As the latent representation learning could be treated as generalized feature extraction for all downstream tasks, the computation overhead is also reduced for Octopus compared to training massive models for different tasks from scratch independently.
(5) Trade-offs. Existing strategies to handle heterogeneity, privacy, and communication are typically incorporated into the learning procedure, resulting in negative impacts on accuracy and convergence. Data collection/feature extraction and learning are separated in Octopus, which provides a better balance between accuracy and other objectives. Octopus simplifies the heterogeneity, communication efficiency, and privacy challenges into a single, less expensive task.

We present the results of experiments on the MNIST [35], CelebA [36, 37] (resize to $128\times 128$) and Speech [38] datasets to demonstrate the effectiveness of our approach in terms of performance on performance and privatization. We split all samples into the training set $Tr$ (80%) and hold out the rest as a test set $Te$ (20%). For the Octopus training, 85% of the Tr is divided into different subsets as the distributed data sources, where the data is sorted by class, and each node receives data partition from only a single class. The remaining 15% is used as the additional data (ATD, e.g., as the public-released relevant datasets). For the Centralized scenario: The set of uncompressed Tr data is used as the entire collected data (CTD) to train the downstream tasks in a centralized manner. Then, the test set Te is used to evaluate the performance metrics to set the baseline. For the Federated scenario, the $Tr$ dataset is divided into different subsets as the distributed data sources for federated learning to train the same downstream tasks. The test set is then used to obtain the needed performance metrics. We mimic non-IID data partitions through dividing datasets based on data labels to evaluate performance under heterogeneous data. In an independent and identically distributed (IID) setting, each node is randomly assigned a uniform distribution across all classes, which is the best-case scenario of the non-IID setting. We consider the worst-case of the non-IID setting, where the data is sorted by class, and each node receives data partition from only a single class. Additionally, we also control the skewness through varying the proportion of data that are non-IID to demonstrate the effect of varying the skewness. We apply 20% non-IID as the moderate-case of the non-IID settings, in which 20% of the dataset is separated in terms of labels, while the remaining 80% is divided uniformly at random. To make a fair comparison, we also consider the additional data(ATD, e.g., as the publicly-released relevant datasets) as the globally shared between all the client devices [39].

The same notations for Federated Averaging (FedAvg) algorithm in [40] are adopted. We apply 100 clients, each running one local epoch on top of 100 global communication rounds. We also consider $FedProx$ [41] as the approach to accelerate tuning hyperparameters for algorithms of FedAvg in the case of federated learning with personalization. We also evaluate the performance when the federated learning is incorporated with differential privacy to address the privacy of personally identifiable information, at $(\epsilon,\delta)=(10,10^{-5})$-DP.

OCTOPUS scenario: The ATD data here is the additional data that we assume could be obtained from a public-released relevant dataset. The server uses it to initialize the DVQ-AE, instead of sharing globally. The next step is to fine-tune the local model or codebook using the distributed data source (used in the federated scenario). Each sample from every distributed data source is mapped to the node-specific autoencoder’s encoder. The server gathers the encoded latent codes to be the training set to train the same downstream tasks. After that, the model’s performance evaluations are obtained using the encoded version of the test set.

In our experiments, we explore four group setups for the codebook structure with $N_{v}=1,4$ and $N_{h}=1,41$, respectively. Let the codebook size $B_{x}$ reveal the compression size, namely the size of compressed communication, and each group setup is conducted on $B_{32}$, $B_{64}$, $B_{128}$, $B_{256}$, and $B_{512}$ settings.

We evaluate downstream task models’ testing accuracy when trained and tested in Centralized, Federated, and OCTOPUS scenarios with MNIST, CelebA, and Speech datasets.

Figure 4 compares the downstream task models’ testing accuracy in the OCTOPUS scenario using various compression sizes compared to centralized and federated scenarios. Testing accuracy for the centralized scenario (uncompressed data), the baseline of accuracy, is the highest among all cases, as all features in the raw data can be used to train models. Additionally, the testing accuracy for OCTOPUS is higher than that of Federated in both non-IID and privacy-preserving settings on all three datasets. Also, it has been found that the more complex the image, the more significant the gap. As shown, there is a general degradation in the testing accuracy when the compression size increases from $B_{512}$ to $B_{32}$, as the number of features utilized to train models is decreased. The smaller the compression size, the more features the model has to train a stranger model.

The rate of reduction of the model’s testing accuracy varies across the models on two image datasets, with MNIST being the highest of all the compression sizes. The reason is that the number of features for simple data of small dimensions is even smaller than encoded features. The significant degree of degradation is also visible in Speech, since the complex downstream task requires more features to train the model. We further demonstrate that there is significant degradation of the utility for federated learning in the non-IID scenarios (approximately 10%-30% drop in the downstream task accuracy for the worst-case and moderate-case of non-IID settings) and after incorporating the privacy-preserving procedure (approximately 30%-50% drop in the downstream task accuracy). The OCTOPUS avoids such degradation by incorporating advantages from both centralized learning and federated learning.

As differential privacy (DP) is the commonly adopted extra privacy-preserving approach used in FL, we also demonstrate the utility of the DP-based FL scheme after applying the perturbation to the communication. For the MNIST dataset, as shown in Figure 4, original samples can easily be recognized with more than 99% accuracy on average for the centralized scenario. However, the DP-based privacy-preserving centralized case appears to have lower accuracy than expected in comparison to the centralized scenario. Even under $B_{32}$ compression size, the accuracy of OCTOPUS is generally better than that of centralized learning with DP and federated in non-IID and with DP cases. For the CelebA dataset, the classifier trained on the centralized case can achieve around 90% test accuracy on both utility tasks (gender and smiling) on average, while our trained downstream task using OCTOPUS can achieve a comparable accuracy (87% on average) on the compact latent representations. The accuracy of OCTOPUS is better than that of centralized learning with DP (82% on average) and federated in non-IID (51% for the worst-case and 56% for the moderate-case) and with DP (30% on average) cases. The evaluation on Speech confirms that OCTOPUS does not degenerate the utility of the expected downstream task too much compared to the centralized scenario and is better than the federated scenario.

Generally, applying FedProx for tuning hyperparameters could enhance the performance (accuracy of the tasks) of FedAvg under non-IID settings by around 10-20% on average compared to the worst-case of non-IID scenarios, but still less than the best-case. Besides the extra cost for conducting these improvements, it is also required to apply further privacy-preserving approaches, such as differential privacy, resulting in accuracy degradation. After applying data-sharing strategy [39] to improve FedAvg with non-IID data via globally sharing the ATD data among all the client devices, the test accuracy can be increased by 10-30% on the image datasets compared to the worst-case of non-IID scenarios, though still less than the best-case as well. Even for the best-case of the non-IID case ( uniform distribution over all classes), the accuracy of the federated learning decreases by 30%. For federated learning with hyperparameter tuning and data-sharing strategies, the accuracy enhancement cannot defeat the degradation after deploying privacy-preserving mechanisms (e.g., differential privacy).

In order to protect FL communication privacy, additional privacy-preserving strategies such as secure computation, secure aggregation, and differential privacy schemes are usually necessary, resulting in additional computational overhead or reduced utility. In this section, we demonstrate the privatization ability of the OCTOPUS with no extra privacy-preserving mechanisms incorporated. We evaluate the privacy leakage via the recognition accuracy of the sensitive information and the conditional entropy in bits on the classifier testing set.

The first metric we evaluate privatization is the recognition accuracy of sensitive information. For both FL and centralized settings, the identity classification accuracy remains high, which indicates that a significant amount of sensitive information has been collected from raw data during training. Adopting privacy-preserving technologies (e.g., differential privacy) could reduce the accuracy of identity classification, however, the values are still high (around 70%). For the MNIST datasets, the digit number itself is considered as the private component. The task of downstream recognition is of determining whether the digit contains a circle (i.e., 0,6,8,9) or not. Figure 5 (a) reports the recognition accuracy of the sensitive information as a privacy evaluation metric under the settings given in Section 3.1.1. Classification accuracy for the private component via OCTOPUS decreases rapidly, from 95% to below 10%, while classification accuracy for circle recognition remains similar to that of the centralized scenario. It follows that the OCTOPUS does not degrade the utility of the expected downstream task too much while also significantly limiting the leakage of PII. In the CelebA dataset, we consider the identification as a privacy component, and gender and smiling recognition as downstream tasks utilizing the model in [42, 43]. As shown in Figure 5, the OCTOPUS model can reduce the test accuracy of private information (identification) from 85% down to around 10% on average, demonstrating the ability to protect private information. It demonstrates that the filtered data can still serve the desired classification tasks at the cost of only a small drop in utility accuracy (4%).

Considering differential privacy is associated with a lower computational cost to protect privacy via perturbation, we compare the performance between sensitive information perturbation and sensitive information filtering/replacing in terms of privacy protection. Specifically, we compare the privatization performance between OCTOPUS and centralized/federated learning with differential privacy. As shown in Figure 5, the inference probability of personally identifiable information for OCTOPUS (between 10-20%) is much better than the other (between 30-80%), avoiding introducing additional cost due to perturbation. The results demonstrate that sensitive filtering/replacement can achieve better privatization than perturbation.

To further show the reconstruction with different private component settings and the possibility of generating anonymous copies and style transformations, we demonstrate the reconstructed samples in Figure 6. First, the downstream tasks and privacy-preserving evaluations are conducted only on the public component (the $Z_{\bullet}$), which is derived from the pre-trained encoder and VQ module. Reconstruction is not required for these downstream tasks and privacy-preserving evaluations. During the reconstruction, both the public component $Z_{\bullet}$ and the private component $Z_{\circ}$ are required to decode back to pixel space (images), which could be considered as sampled points from the learned distribution. Since the private component contains sensitive information, this is the reason we filter them from the released features. When the public component $Z_{\bullet}$ is combined with empty or totally random private component $Z_{\circ}$, the decoder would produce a blurry reconstruction, as the decoder is trained on the combination of both $Z_{\bullet}$ and $Z_{\circ}$ from learned distribution. Also, it would be interesting to know what would happen if an adversary has the ability to feed private components into reconstruction, particularly those derived from publicly available training data. Therefore, we demonstrate two possible style transformations with available private components $Z_{\circ}$ in Figure 6: (1) perturbed $Z^{\prime}_{\circ}$ by adding random noise to the extracted private component, and (2) replacing with a different private component $Z^{\prime\prime}_{\circ}$ derived from publicly available training data, such as ATD data. After reconstruction with perturbed $Z_{\circ}$ , the sensitive information is sufficient to fool a well-trained classifier and to be difficult for humans to recognize, as demonstrated in Figure 6 (a). It is possible that a tiny perturbation on the private component could blur the digit identity while retaining the circle recognition information for MNIST. The replacing approach could be considered as a strong case for the perturbation approach. For example, we might use the private component (or further perturbed version) of fake face images to replace the original private component of the sensitive raw data. In Figure 6 (b), it is challenging to obtain sensitive information (e.g., human identification) for both machine and human perception after replacing the private component (extracted from reference samples from ATD) during reconstruction. This shows that the reconstruction with private components replaced maintains the public features such as facial expression but differs in the identification. For the Speech dataset, we consider the speaker’s identification as the privacy component, the phoneme identity as the downstream task used the model in [38]. The evaluation of Speech confirms that OCTOPUS significantly reduces the privacy leakage of the PII.

Conditional entropy as privacy. After this, we evaluate the privacy leakage of both private and public components in terms of the conditional entropy in bits on the classifier testing set, i.e., the results of the classifier trained on transmitted private, public, and both components. As shown in Figure 7, taking MNIST and Face data, for example, the performance of the classifier trained on the private components of latent codes has a lower conditional entropy value and higher test accuracy than the fully released latent codes (private + public) and the public components. It reveals that the private components of the latent codes $Z_{\circ}$ are more informative about the sensitive label. Additionally, the performance of the classifier trained on the public components of latent codes has a higher conditional entropy value and lower test accuracy than the fully released latent codes and the public components. It means the public components of the latent code $Z_{bullet}$ are uninformative in terms of the sensitive class label.

Taking the Speech recognition scenario as an example, we test the privacy component’s recognition accuracy using speaker voice excluded from the training dataset, as the degree of disentanglement of OCTOPUS. The results are summarized in Figure 8 and Table I. The test accuracy of speaker identification is reduced by half on the codebook size for OCTOPUS, compared to the same scheme without our disentanglement strategies. This demonstrates that the latent codes would cause privacy leakage, e.g., speaker information. Namely, the fewer codes in the codebook, the less the privacy component is incorporated. The recognition accuracy increases, as the codebook size is enlarged from $32$ to $512$. However, the re-identification accuracy remains at low probability, even with the $B_{512}$ codebook size. We find a trade-off between the disentanglement and utility of codes for downstream tasks. If we compress the codebook into only a few codes, the disentanglement ability is strong while being hard to contain sufficient features of content information. In our experience, $B_{256}$ is a proper size of codebook to balance the trade-off. The experiments demonstrate that the OCTOPUS scheme can achieve good disentanglement performance without explicitly imposing any objectives or constraints on the encoder or decoder.

The improvements in communication, computing, and storage performance are intuitive. For example, DVQ-AE can compress the high-dimensional raw face image with $128\times 128\times 3$ input size to at most $32\times 32$ latent codes for communication, computing, and storage, achieving comparable accuracy performance as using the raw data. In the speech, the latent space is 64 times smaller than the original waveform input. In addition, we evaluated the normalized amount of time required for training the downstream models for MNIST, CelebA, and Speech datasets. We observed training time decreases across different compression sizes and models. Such reduction may be caused by the reduction of the amount of the input data and smaller number of parameters to be learned when applying the OCTOPUS algorithm with a suitable compression size. Detailed evaluations are provided in the following sections.

As there are clear annotations on CelebA dataset, we select 20 annotations to demonstrate the performance of the Octopus on multiple downstream tasks, e.g., binary attribute classification with respect to Hairline, Eyeglasses, Male, Mouth, Smiling, Hair Style, etc. For Octopus, as the collected latent codes can be considered as the output derived from a generalized feature extractor, we apply a simple linear classifier to conduct multi-task classification on the latent codes with a 512 size. Specifically, the input of the linear classifier is the collected latent codes, followed by three linear classifier layers are (512, 512), (512, 128), and (128, 40) sequentially, and the Sigmoid activation function, with around 0.3M parameters.

The baseline approaches are three competitive deep neural network approaches, i.e., LNet+ANet, LNet+ANet with pre-training, and the MobileNets model. LNets + ANet [44] applies two DNNs to recognize faces and then detect the facial attributes, where both LNeto and LNets have network structures similar to AlexNet (10 convolutional layers), and ANet has four convolutional layers and one fully-connected layer, with around 10M parameters. MobileNet [45] consists of 28 depth-wise convolutional layers and point-wise convolutional layers with around 4M parameters. To demonstrate the performance of Octopus, we allow the baseline models to be trained on the centralized raw data and select the best performance of the binary classification models. The dataset is divided with the ratio of 7:2:1 for training, validation, and test. All these models are trained with a learning rate of 0.001 and for 50 epochs. The results are reported in Figure 9, the performance of Octopus is equivalent to or better than the other three competitive models while using far less computation and parameters. This increased accuracy is due to the fact that our autoencoder functions as an enhanced feature extractor and the Vector-Quantized procedure also reduces noise in the data.

When the fraction of data used for training the autoencoder is increased, the performance of the autoencoder will increase. For example, the performance of the model increases by around 12-15% when increasing the fraction of ATD from 10% to 20% on the image datasets, with respect to Inception Score, accounting for 90% performance of the model trained on the entire training data (ATD is 100%). One promising solution to loosen the assumption on the amount of publicly available data to train the autoencoder is data augmentation, such as Differentiable Augmentation (DiffAugment) [46], where the performance of the model trained using only 100 images for augmentation could match that trained on the entire dataset. One of our next step studies is to leverage data augmentation and transfer learning to address the assumption on publicly available data.

To further illustrate the performance of Octopus, we have evaluated the time of latent code inference, downstream task training, and once-off autoencoder training. Taking the centralized CelebA dataset (128*128 size) as an example, the training time for a VGG-16 based classifier of gender from scratch will take more than 1 hour using one GPU with three epochs, which could be considered as the best performance limit for federated learning as well. Training time grows approximately linearly with the number of downstream tasks. However, with the same computing resources, the training time of the classifier with three linear classifier layers on the centralized latent codes is only within 5 min toward similar accuracy. Although the auto-encoder training takes hours, it is a once-off procedure on the server-side for multiple downstream tasks that could be carried out in parallel across several GPUs or by using the pre-trained model for transfer learning to reduce the training time further. Given the once-off trained autoencoder, the time for extracting and disentangling latent code of each input sample is less than 0.3s for MNIST and CelebA (0.15s and 0.29s, respectively) for the CPU-Only (i5 8200 8GB) end-user. With the distributed pre-trained feature extractor, the training of multiple downstream tasks can be considered transfer learning, resulting in reduced training time and lower generalization error for the neural network model.

Vast amounts of data are necessary to deliver high-quality machine learning models [1]. The conventional framework in machine learning considers a centralized dataset concocted in the integrated scheme. However, real-world data is usually decentralized across multiple parties. Various research works have been conducted in the last few years to apply Federated Learning (FL) to train machine learning models collaboratively while keeping the data decentralized [2, 3, 4].

Generally, a federated learning scheme has fundamentally four principal operations in an iterative manner:

• •

  Sampling Clients. Desired participants are selected by the server from a pool of clients;

• •

  Broadcasting Parameters. The parameters of the initialized commonly-shared model are broadcasted from the server to each selected clients;

• •

  Updating Local Parameters: Each participant retrains the broadcasting model in a parallel manner using the local data;

• •

  Aggregating Global Model: The server will aggregate the gathered uploaded parameters from each client, resulting in the updated global model.

Being a distributed system, federated learning also encounters several challenges in terms of communication, system and data heterogeneity, as well as privatization and security. Several FL architectures are developed to address either the accuracy and computational costs or other challenges in the FL field [47, 48, 49, 50, 51].

Communication performance. Even if the models are commonly less expensive to be transmitted than raw data gathering in centralized learning mechanisms, the communication burden is one of the main bottlenecks for federated learning. The gradient updates, frequently communicated between nodes in distributed and federated learning, are an inevitable burden, resulting in expensive communication, and scalability constraints [9, 10]. Existing solutions proposed to address the communication overheads are either data compression [52] or by restricting only the relevant outputs by the clients [53, 54] to be uploaded to the parameter server.

Heterogeneity performance. The heterogeneity of the data and system in the network, as well as the non-identical nature of the distributed data, impact the performance of the federated learning approaches [40, 47]. Due to lack of global data, there is spatial heterogeneity such that each node may have a data value bias and data size variation with respect to the general population, and temporal heterogeneity such that the distribution of each local dataset may vary with time. FedAvg is proposed as a solution to address the heterogeneity, however, it is still not robust enough to deal with system heterogeneity. Enhancements on the model aggregation approaches have also been explored to solve the heterogeneity [55, 56].

Privacy and security performance. Even though the raw user data does not leave the local clients in FL, communicating model updates can also reveal sensitive information, requiring additional privacy-preserving strategies, such as secure computation, differential privacy schemes, or running in a trusted execution environment. One common privacy-preserving methodology is based on differential privacy to provide a certified privacy guarantee. The differentially private federated learning scheme [57, 58] protects a user’s privacy by adding noise to the clipped model parameters before model aggregation at the cost of model accuracy. Besides, it is time- and resource-consuming to conduct perturbation or encryption for a huge amount of parameters with limited resources. Moreover, they all fail when the server is distrusted. There are also alternative privacy-preserving approaches to communicate gradients without incurring accuracy loss, such as secure aggregation [59]. Secure aggregation protocols only require the averages of the local model weights to be computed from subsets of participants to compute SGD and global model updates. SMC-based techniques only work with honest participants. There is no guarantee that the protocol will be available or correct for Byzantine participants, particularly if they collude with a malicious server to reveal the inputs of a targeted client [60].

Lack of access to global training data can cause security concerns, such as data poisoning or backdoor attacks. As the federated learning scheme is a distributed system, it might be even harder to recognize the misbehaving devices. The attacker can poison the data either by directly inserting poisoned data to the targeted device or injecting poisoned data through other devices [61]. The adversaries could influence either the prediction of a subset of classes or the global model accuracy [62]. To protect the confidentiality of the training data, the aggregator has no visibility into how these updates are generated. A model-poisoning attack is proposed as a more powerful attack than poisoning attacks that target only the training data [63]. As secure aggregation prevents the server from examining individual models, it is hard to detect the poisoning model [60]. In contrast, Octopus’s global data collection methods are capable of handling such erratic and malicious input via prepossessing from a global perspective.

Even though some variants of federated learning, such as federated averaging, could perform more local computation and less communication, three significant challenges need to be considered to promote federated learning being widely adopted by the industry, i.e., heterogeneity, communication efficiency, and privacy concerns. Due to the data heterogeneity, how FedAvg aggregates device models generates a bias in the final model, resulting in obvious accuracy degradation. Even sampling SGD in parallel on a small subset of devices, the total communication scale is still large. Consequently, different extra optimization mechanisms are required to address the communication complexities, data or system heterogeneity, and privacy concerns, resulting in further computation or accuracy degradation. In contrast, Octopus addresses the heterogeneity, communication efficiency, and privacy challenges into one single task with less cost. We highlight the differences between Octopus and existing approaches concerning these three challenges in the following parts.