Federated Learning for Internet of Things: Applications, Challenges, and Opportunities

The rapid development in the Industrial Internet of Things (IIoT) brings several advances in information technology applications for the manufacturing field. The concept of Industry 4.0, also known as the fourth industrial revolution, has been proposed based on the emergence of significance for the inter-connectivity of IIoT and the access to real-time data. With unprecedented connectivity, Industry 4.0 will bring greater insight, control, and data visibility for the supply chain in many industries. Currently, some mature implementations of Industry 4.0 include the Optical Character Recognition (OCR) for the labels, smart and automatic Incoming Quality Control (IQC), and smart Process Quality Control (PQC). However, there are still some real-world problems challenging the deployment of Industry 4.0. First, the amount of data generated from a single factory may not be sufficient enough for training a reliable model comprehensively. Second, the data collected by the industrial IoT devices is highly related to the commercial value, which makes privacy-preserving important. For example, eavesdroppers may infer the capacity for manufacture from its electricity usage for industrial IoT users. The federated learning framework will become an inspired solution to address the above challenges.

As IoT devices become more pervasive in individuals’ daily lives, the privacy of the collected data becomes a significant matter. An example to illustrate privacy concerns is IoT E-health. Nowadays, smart wearable devices are used to monitor the health status of patients, such as heartbeat, blood pressure, and glucose level. Compared to the other types of data, personal healthcare data is most sensitive to users’ privacy and highly restricted by government laws and regulations for any kind of data sharing. Therefore, techniques such as FL are a requirement for investigators and researchers to develop state-of-the-art ML models over a fractured and highly regulated data landscape. The ability to train machine learning models at scale across multiple medical institutions without pooling data is a critical technology to solve the problem of patient privacy and data protection. Successful implementation of federated learning in healthcare could hold significant potential for enabling precision medicine at a large scale, helping match the right treatment to the right patient at the right time.

Smart home systems enabled by consumer IoT devices have achieved great popularity in the last few years as they improve the comfort and quality of life for the residents. The wireless smart IoT home devices, such as smart bulbs, smart doorbells, and smart cameras, are capable of communicating with each other and controlled remotely by smartphones and microcontrollers. The implementations of Wake-Up-Word speech recognition and Automatic Speech Recognition (ASR) on IoT devices bring great convenience to everyday living, and people now tend to rely on smart IoT gateways with intelligent virtual assistants to control their home hands-free. FL has thus become a critical technology that is able to improve the on-device speaker verification while reducing the risk of raw data leakage.

IoT-enabled smart cities are bringing significant advancements by making city operations efficient while improving the quality of life for citizens. Various IoT devices enable city managers to control the physical objects in real-time and provide intelligent information to citizens in terms of the traffic system, transportation, public safety, healthcare, smart parking, smart agriculture, and so on. Due to the concerns of data privacy, smart infrastructures are moving to compute resources close to where data reside, which makes FL framework suitable for deployment. For example, the FL-based smart grid system enables collaborative learning of power consumption patterns without leaking individual power traces and contributes to the establishment of an interconnected and intelligent energy exchange network in the city.

Along with the advancement of the vehicular IoT, autonomous driving technology is making its way into everyday cars. For a reliable self-driving system, it needs frequent real-time communication with a multi-access communication environment. Also, the spatial and temporal changes of the vehicular environment require an intelligent approach that can evolve with the change of environment. For the traditional centralized-over-cloud method, the driving system needs to transmit a large amount of raw data to the server, which would cause potential privacy leakage. The communication overhead triggered by the large-size data transmission and limited network bandwidth may also lead the driving system to not being able to respond to the real-time spatial changes precisely. Adopting federated learning in vehicular edge computing for autonomous driving has thus become a promising direction to mitigate the above challenges. With FL, each vehicle only needs to transmit a limited size of data to the cloud and can adapt to real-time local changes more sensitively.

The metaverse is a hypothesized next generation of the internet, providing fully connected, immersive, and engaging online 3D virtual experiences through conventional personal computing, as well as virtual and augmented reality devices. In metaverse, users own their avatars and can interact with virtual objects and other participants. One of the fundamental building blocks of the metaverse is the digital twins duplicated in virtual environment that reflect the real-time physical world status. The connection between the virtual and physical world is tied by the data collected from IoT devices. Federated learning is a promising solution to enable collaboration between edge and server for better global performance and also boost the security and privacy of the metaverse. For example, the eye tracking or motion tracking data collected by the wearables of millions of users can be trained in local devices and aggregated via an FL server. Hence, users can enjoy services in the metaverse without leaking their privacy.

The deployment of FL on the network edge is severely impeded by the limited resources of the IoT devices. Existing machine learning models, especially deep neural networks, are known to be computation-intensive, which presents strict requirements on hardware and may result in low training efficiency on edge devices. Thus, developing customized and specialized hardware for machine learning applications on the edge is a promising direction to accelerate inference and training tasks while using much less energy compared with general-purpose processors. Edge devices have limited resources not only in terms of computation but also in terms of memory for storage and data access. Recent neural network architectures require accessing a vast amount of memory locations for storing not only model weights and parameters but also the intermediate results produced by the computations. Therefore, a significant challenge for processing neural network models on a resource constraint device is reducing the memory accesses and keeping the data on-chip as to avoid costly reads and writes to the external memory modules. Finally, in contrast to servers with CPUs and GPUs that can use a substantial amount of power, edge devices with embedded processors have a limited energy budget, which further imposes restrictions on the hardware performance. Despite the fact that current edge devices are increasingly powerful, training some deep learning models on-device is still time-consuming and inefficient.

To make models more applicable for the edge environment, researchers mainly focus on two research directions: design lightweight and hardware-friendly models/algorithms, and compress existing models to obtain thinner and smaller models, which are more computation and energy-efficient. As an example, FedMask [6] is proposed as a joint computation and communication-efficient FL framework. By applying FedMask, each device can learn a heterogeneous and structured sparse binary mask; based on the mask, it is able to generate a sparse model with reduced computation cost, memory footprint, and energy consumption. However, this approach is hardware-agnostic; to further reduce the resource demands of federated training, we envision that the approach of hardware and algorithm co-design, which sparsifies the model by taking the IoT hardware architecture into consideration during federated training, is a promising future direction.

Communication bottleneck is considered one of the major challenges in an FL-based IoT environment. Currently, most IoT devices communicate using wireless networks whose bandwidth is much smaller than wired network bandwidth in datacenters. For example, under a smart home scenario, the sum of the overall networking bandwidth is constant for the whole IoT system, no matter how many devices are connected. As more devices join the system, the communication problem is advent when clients possess different resources allocations. The limited network bandwidth not only makes the communication between clients and the server inefficient but also triggers the presence of straggler clients, which fail to share their local updates with the server during the communication round. They both serve as the bottlenecks for the performance of FL deployment in the large-scale IoT scenario.

To reduce the bandwidth demand during federated training, methods such as gradient compression have been heavily explored. However, these methods compromise the training quality to gain training efficiency. Mercury [7] proposed a sampling-based framework that enables efficient on-device distributed training without compromising the training quality as a new inspiration for solving this challenge. In addition, Chen et al.’s work [8] formulates the bandwidth resource allocation and user selection problem during training federated learning models as an optimization problem whose goal is to minimize the training loss while meeting the delay and energy consumption requirements. Liu et al. also proposed a client-edge-cloud hierarchical aggregation framework as a communication resource-efficient method to operate the federated learning in edge computing [9]. Each client is able to offload its data samples and learning tasks from its device to the edge in proximity (e.g., edge gateway at home) for fast computation in the client-edge-cloud paradigm, which allows multiple edge servers to perform partial model aggregation. These works proposed promising and orthogonal techniques to reduce the bandwidth demand in the context of IoT. We envision that those techniques can be combined together in the scenario where IoT devices are confronted with extremely limited network bandwidth.

Apart from the previous challenge of bandwidth limitations, the intermittent connectivity of the IoT devices signifies an unstable network connection that drops the device out of the system in the middle of the training round. Especially in large-scale IoT systems, the dropout problem followed by the intermittent connectivity and availability of various devices will become a serious obstacle for the FL framework to efficiently manage and schedule clients. Currently, most of the FL studies are based on the synchronous update at the server, which implies that the server will not start the model aggregation until it receives the information sent from the slowest client. However, in real-world settings, due to the unbalanced communication abilities and training data distribution, the local training speed varies from device to device, and even some clients will be temporarily disconnected during the training phase, which makes the synchronous update nearly impossible. Also, not all of them will be simultaneously available for FL updating. In the asynchronous FL scenarios, a client could join the active learning group even in the middle of the training progress, which endangers the convergence of the federated training.

To address this challenge, some researchers proposed an asynchronous aggregation scheme with the implementation of coding theory to resist the stragglers in the FL system. In [10], an asynchronous aggregation protocol known as FedBuff has been proposed to mitigate stragglers and enable secure aggregation jointly. Specifically, the individual updates are not incorporated by the server as soon they arrive. Instead, the server will keep receiving local updates in a secure buffer of size K, which is a tunable parameter, and then update the global model when the buffer is full. However, in real-world settings, IoT devices are by nature heterogeneous with diverse computing speeds. IoT devices with higher computing speed would be able to send in their local updates faster than IoT devices with slower computing speed, which inevitably leads to training bias. We envision that an asynchronous approach that can take the heterogeneity of IoT devices into account could be a better and promising solution.

Within cross-device settings, clients under the FL framework have diverse system metrics in terms of both hardware and software. Various devices with different hardware architectures or even different device vendors are used to perform the learning tasks in different operating systems and different software APIs. Clients may choose different deep learning frameworks such as TensorFlow, PyTorch, and Caffe to train the local models, resulting in different model formats for aggregation. All the diversities have not only posed a significant challenge to system design but also exacerbated the asynchronous communication problem as mentioned above. Moreover, in IoT settings, the data collected by different devices can be very different in terms of the feature and dimensions, and various types of devices can also have different temporal and spatial preferences for their data collection, which may create a discrepancy in the local data structure among all the participants under FL framework. For example, a surveillance camera will record videos in real-time (24x7 hours), while the data generated by a doorbell is intermittent. However, the central server could not examine the impact of the data heterogeneity until the training is done.

An FL framework for IoT should enable graceful adaptation of the data and compute load across different devices based on their resource availability. To address this challenge, we envision that the training quality and speed will be improved if we can determine the heterogeneity and make adjustments accordingly before the training starts. Diao et al. [11] proposed a heterogeneous FL framework that can produce a single global inference model from training heterogeneous local models on the clients. It is the first time to challenge the underlying assumption of existing work that local models have to share the same architecture as the global model, which inspires a solution to address the system heterogeneity among IoT devices.

IoT sensing devices will, by their very nature, continuously collect new data, which will be used to update the model for lifelong or continual learning. With the objective of keeping providing services accommodating to newly collected data, the continued model update training poses a new challenge for resource-limited IoT devices. Specifically, as most of the IoT devices are memory-limited, their memory resources are not sufficient enough to handle both model inference and training. Furthermore, the lack of sufficient memory to keep past collected data may exacerbate catastrophic forgetting, which is one of the most critical problems in continual learning.

To address this challenge, we envision that the light-weighted machine learning engine is needed to reduce the memory consumption for on-device training. As an example, FedGKT [12] is a potential method to reduce the training memory footprint for efficient on-device learning. With FedGKT, IoT devices could transfer knowledge from many compact CNN models to a large CNN at a cloud server, which reformulates FL as a group knowledge transfer training model for the large-size model training on resource-constrained edge devices. To avoid catastrophic forgetting, we envision the use of clustering approaches that identify and store a few core data samples from each time interval. Moreover, leveraging IoT “Hubs” that can store non-sensitive/public datasets to inject memory in the training system is another promising solution. Furthermore, approaches that can detect temporal distribution shifts at each IoT node to determine when to update the model would also be needed.

In practical deployment, IoT devices are attractive targets for adversaries seeking to launch attacks such as phishing, identity theft, and distributed denial of service (DDoS). With the expansion of the IoT networks, the potential traffic volume of IoT-based DDoS attacks is reaching unprecedented levels, as witnessed during the Mirai botnet attack leveraging infected webcams and home routers. Attacks through the Internet have raised awareness of the need for IoT risk assessment and security, e.g., in fields such as healthcare. Even though these attacks could be easily defended by installing security patches, many IoT devices lack the requisite computation resources to do so. Moreover, within a cross-device system setting, it is difficult to identify whether the coming participant is malicious or not before it joins the system. Therefore, it is crucial for the IoT system to detect malicious or broken IoT devices that will ruin the model training with limited resources. To address this challenge, one of the promising directions is to implement a lightweight security protocol in the IoT system for the detection of broken and malicious devices. With the distributed nature, FL can offer an alternative approach for IoT cybersecurity by protecting the system from malicious attacks as close as possible to the IoT devices. DIoT [13] is the first system to employ an FL approach to anomaly-detection-based intrusion detection in gateways to IoT devices without centralizing the on-device data, where it demonstrates the efficacy of federated learning in detecting a wider range of attack types occurring at multiple devices.

Although FL shows its efficacy in cybersecurity for the IoT system, the privacy leakage from on-device sensitive data still matters as the participants share the model gradients or weight parameters with the server during the training process, which are derived from the participants’ private training data as a statistical representation of the data it was trained on. The attacker could initiate the model inversion attack on the FL server first to achieve the individual model of each participant, and then recover the personal training data by inverting these personal models. One of the representative works for the attacking above is the inverting gradients attack [14], which proves that personal data reconstruction from gradient information is possible in federated learning setups. Therefore, a critical consideration in FL design is to ensure that the server as a blackbox for aggregation that does not learn the locally trained model of each user during model aggregation. Currently, the state-of-the-art secure aggregation protocols in FL essentially rely on two main principles: the pairwise random-seed agreement between users in order to generate masks that hide users’ models while having an additive structure that allows their cancellation when added at the server; and the secret sharing of the random-seeds so as to enable the reconstruction and cancellation of masks belonging to dropped users. The main drawback of such approaches is that the number of mask reconstructions at the server substantially grows as more users are dropped, causing a major computational bottleneck. Especially for the low-end IoT devices, the additional operator for the secure aggregation becomes an excessive burden to the limited on-device computational resources. To address this challenge, one of the promising directions is to implement lightweight and secure aggregation protocols that could provide the same level of privacy and dropout resiliency guarantees while substantially reducing the aggregation complexity, which meets the constraint in the IoT setting.

There are many concerns that the researchers need to take into account when designing a federated learning system on IoT networks. Issues such as different communication APIs, dataflow models, network configurations, and device properties have to be considered. As an emerging field, FL for IoT has not been standardized and appropriately implemented. Therefore, the research and development for standardization could help expedite the widespread deployment of FL systems on IoT networks and create an open environment for content sharing. Additionally, in light of the complexity involved in federated learning, researchers and industries need to further build upon existing FL developing and benchmarking tools such as TensorFlow Federated, PySyft, and FedML to accommodate the scenarios of IoT applications. From the application-level perspective, user-friendly integrated simulation environments are needed to help design and evaluate the entire FL system on a large scale of IoT networks and its feasibility without implementing the model in real-world settings. From the system design perspective, ideally, we are looking for tools that can help developers accomplish system-level tasks such as load balancing, resource management, task scheduling, or data migrations easily.

One work of note along this direction is FedIoT[15] which provides a mature systems-level framework that the developer can use to deploy their federated learning applications on CPU or GPU-enabled IoT devices, such as Raspberry Pi and NVIDIA Jetson Nano. To make federated learning more ubiquitous on IoT devices, we believe that researchers should pay attention to extending the current training frameworks to edge FL setting with awareness of the challenges mentioned above. It is worth mentioning that current edge computing solutions such as TensorFlow Lite, MNN, and TVM are focusing on improving the performance and efficiency of edge inference instead of training, much less taking FL setting into consideration, which is an under-explored area that would bring significant values to the federated learning and IoT communities.