FedAVO: Improving Communication Efficiency in Federated Learning with African Vultures Optimizer

The proliferation of diverse consumer electronics, including IoT devices, home appliances, smartphones, connected autonomous vehicles (CAVs), drones, Virtual Reality (VR), and Augmented Reality (AR) devices, has witnessed substantial growth in recent years. These devices capture and store various forms of data such as images, audio, and text.Image, audio, and text are just a few of the different forms of data that such devices acquire and store. Every day brings a slight increase in the number of IoT-related applications, their users, and the amount of data they produce. This phenomenon is pioneering the scope of data-driven decision-making. Due to the heterogeneous nature of these collected data, training machine learning models comes with the following challenges: widely distributed: data points are stored in a large number of clients, which can be significantly higher than the average number of training samples retained on a given client; non-IID Data: each client’s data may not accurately represent the entire distribution [1] of a dataset; unbalanced Data: clients may hold different magnitudes of data points; communication constraints: Mobile or IoT devices frequently may experience poor, expensive, or no internet connectivity; communication cost: data collection from edge devices is significantly high. Moreover, transferring the data to the central server for training requires additional network bandwidth.

The study FL has made substantial progress in addressing the aforementioned challenges. FL is a machine learning (ML) paradigm that trains ML models utilizing decentralized data. FL protects data privacy by preventing data transfer from local machines to a centralized server. Each edge devices train the model on its own data locally using its computation resources. Instead of sending local data to the central server, only the local updates are sent. By providing only the local updates, FL cuts down the bandwidth overhead and also the communication cost. For these distinctive qualities, FL has been adopted in a wide range of applications such as speech recognition [2, 3], surveillance system [4, 5], health care system [6, 7], Internet of Things (IoT)[8], human stroke prevention [9, 10], and cybersecurity [11]. An FL paradigm includes local training, client-server communication, and model aggregation [1, 12]. Often communication overhead in FL comes from model broadcast from the server to all clients and vice versa. Furthermore, there is a feasibility risk in every communication round in terms of constrained network bandwidth, packet transmission loss, and privacy invasion. Apart from communication overhead and privacy invasion, training on heterogeneous setups opposes some more challenges. After the centralized server broadcasts its model, the clients train the model while considering some hyperparameters such as learning rate, batch dimension, and rounds per epoch. These diversely trained client models are difficult to aggregate since computational power and data attributes (complexity, ambiguity) differ vastly between each edge device. In an idealistic scenario of hundreds, even thousands of devices, the updated global model may never converge to a global optimum due to the heterogeneous behavior of the clients. Existing aggregation techniques (e.g., FedAvg [1], FedMa [13]) only focus on integrating weights of local models. To increase generalization and convergence rate, some novel approaches suggested model aggregation by using feature fusion [14] of global and local models or using multiple global models [15].

Along with the typical hyperparameters of model training, such as learning rates, optimizers, and mini-batch sizes, FL also has particular hyperparameters, such as local epochs and participant selection. The selection of these hyperparameters can drastically affect the performance of FL. The impact of hyperparameter tuning is much higher in Non-IID data distribution because data samples are radically different from each other across the clients. Although hyperparameter tuning optimization (HTO) has been widely explored in centralized machine learning, several aspects of HTO in FL settings yet need to be studied. In centralized machine learning, the model often trains on the entire dataset, which is often not viable in FL settings. In addition to that, models train on a wide range of hyperparameter configurations, which is extremely expensive in terms of communication and training time in FL settings because each FL cycle consists of multiple phases and communication rounds, and the model must complete each communication round in order to evaluate or change any hyperparameters. Finally, the same hyperparameter configuration functions less well as centralized machine learning due to the heterogeneous behavior of the clients and their data in FL settings. A few strategies for FL-HTO have recently been put forth; however, they concentrate on handling HTO using personalization techniques and neural networks, and they often lack in optimizing the communication cost and convergence rate. In this study, we optimize automated hyperparameter tuning with the help of a meta-heuristic algorithm. We propose FedAVO, which can configure and tune hyperparameters locally concerning the data distribution and their quality. To summarize, our proposed work outlines three novel contributions, which are listed as follows:
$\bullet$ To the best of our knowledge, FedAVO is the first FL hyperparameter tuning algorithm that reveals the potential of AVO as an optimizer for automated hyperparameter tuning during FL model training, specifically tailored for consumer electronics functioning as edge devices.
$\bullet$ FedAVO can be employed to tune the hyperparameters automatically for any state-of-the-art FL algorithm (such as FedAvg [1], FedProx [16], and FedNova [12]). Moreover, FedAVO can automatically configure and tune the hyperparameters locally on each communication rounds.
$\bullet$ We conduct an in-depth evaluation of FedAVO, comparing its performance with the state-of-the-art FL algorithms. Our findings reveal a notable decrease in communication overhead and an increase in convergence rate. Compared to FedAvg, FedAVO reduces the requirement of communication rounds to reach convergence by 80 on average. In addition, FedAVO converges much faster and achieves better accuracy than the state-of-the-art FL algorithms. For instance, FedAVO achieves 99.47% accuracy with MNIST Non-IID distribution and 72.64% accuracy with CIFAR-10 Non-IID distribution. Moreover, for Fashion MNIST and LISA datasets, FedAVO achieves consecutive global accuracies of 85.28% and 92.88%, respectively. The rest of the paper is organized as follows: Section 2 depicts the fundamental concepts of FL and the key elements of our proposed method: AVO algorithm. Related works are reviewed in Section 3, while Section 4 explains the algorithm and operating procedure of FedAVO. Section 5 presents the experimental analysis and highlights the significant findings. Finally, the study is concluded with a discussion of future directions to enhance our work in section 6.

African Vulture Optimization Algorithm (AVOA) is a nature inspired population based metaheuristic algorithm proposed in [17]. The algorithm was derived from the behavior and characteristics of African vultures, leveraging their adaptive and efficient foraging strategies observed in the natural world. AVOA offers a promising method for addressing complex optimization challenges across wide-range of domains. We presented the steps of the AVOA in below:

The rising prevalence of smart and IoT devices in daily life has amplified the need for FL, primarily due to its capability to safeguard data privacy during the model training process. Consequently, there has been a surge in the exploration of efficient optimization techniques tailored to FL. Researchers have directed their attention to various facets of FL, encompassing aspects such as communication efficiency [20, 21], client selection [22, 23], the intricate dynamics of statistical and system heterogeneity [24, 25] and cybersecurity [26]. One particular area of interest that aligns with our research focus revolves around the reduction of communication rounds in federated learning. Our approach to achieving this reduction centers on the optimization of hyperparameters. These hyperparameters, pivotal components of a model, govern its ability to glean insights from a specific dataset.In order to achieve better performance, the hyperparameters need to be fine-tuned with different datasets and different models. The most straightforward automated approach for adjusting hyperparameters involves conducting random searches [27] within the hyperparameter space to identify the best-performing set. However, this method was surpassed by Bayesian techniques [28], which leverage the performance of previously chosen hyperparameters to inform the selection of new ones. Notably, these methods demand substantial computational resources, as the complete training process must be executed to assess hyperparameter fitness. To mitigate the computational burden, there is a shift toward employing partial training [29, 30] with pre-selected hyperparameters. Nevertheless, even with this approach, running full or partial training to identify optimal hyperparameters within resource-constrained FL environments, where communication budgets are limited, remains prohibitively costly and cumbersome. While several researchers have explored the optimization of machine learning (ML) models and their hyperparameters using evolutionary algorithms [31], including techniques like whale optimization [32] and genetic algorithms [33], the same level of attention has not been given to FL. There have been only a few attempts to address FL HTO problems. For instance, the FLoRA framework [34] introduces a novel HTO strategy where global hyperparameters are determined by selecting well-performing hyperparameters from local clients. FedEx [35], on the other hand, optimizes hyperparameter tuning by leveraging Neural Architecture Search (NAS) techniques with a weight-sharing architecture. Nevertheless, several optimization approaches encounter challenges in certain FL scenarios by inadvertently introducing overhead during hyperparameter tuning. Our proposed HTO method strives to bridge the existing gaps, presenting an efficient approach to optimize hyperparameters within the FL framework.

Unlike FedAvg [1], which employs Stochastic Gradient Descent (SGD) as its optimizer, our proposed method introduces a crucial modification aimed at automating the hyperparameter tuning process for SGD. This adjustment has proven to yield substantial improvements in empirical performance, enabling faster convergence with a reduced number of communication rounds. To facilitate a better understanding of the approach, we have detailed the specific parameters and their notations in Table 1. Fig. 2 illustrates the distinct phases of FedAVO, which consist of five key stages.
Local training on client devices: In the first phase, each client performs independent local model training using the initial parameters received from the central server. They update their local model parameters by minimizing the loss function described in Equation 13 using Stochastic Gradient Descent (SGD) on their local data. Notably, this updating process occurs in complete isolation, devoid of any communication between clients or with the server. This strict isolation ensures that clients preserve the privacy of their local data and refrain from sharing it with others.

The hyperparameter tuning phase operates concurrently with the local training phase, as illustrated in Figure 2. To obtain hyperparameter tuning with AVO, an initial population of vultures is initialized to represent candidate solutions. Each vulture explores the problem space through random movement and evaluates its fitness based on an objective function. Here a problem space for any parameter $x$ can be denoted as $x^{p}={\{x_{u},x_{l}}\}$. Where $x_{l}$ and $x_{u}$ represent the lower and upper bounds of the problem space, respectively. Vultures from the population forage through the given problem spaces for an optimal solution for a given problem. The vultures’ movement is influenced by their individual experiences and interactions with other vultures within the population. Furthermore, to enhance foraging efficiency, vultures that repeatedly fail to obtain optimal solutions after a predefined number of attempts share their findings with other vultures. This iterative foraging process continues until a predetermined number of iterations or until a satisfactory solution is achieved, thereby optimizing the hyperparameters for the FL system effectively.

AVO for hyperparameter tuning: In this study, we focus on optimizing the tuning of the following SGD hyperparameters: local epoch, which is denoted as $\mathcal{E}$, learning rate ($\eta$), momentum ($\beta$) and weight decay, which denoted as $\lambda$. The local epoch ($\mathcal{E}$), which is also an FL parameter plays an important role in convergence. Performing more local epochs on clients allows more local computation and potentially reduced communication, resulting in overall global convergence. On the contrary, a larger number of local epochs may lead each device toward the optima of its local objective as opposed to the global objective due to the heterogeneous nature of clients. This may lead to slower convergence or even cause the method to diverge and overfit. Finally, we employ objective function $E_{k}$ for which a problem space gets initialized with hyperparameters. The vultures forage through the problem space and search for optimal hyperparameters to solve the objective function. The optimization can be written as follows:

Here, ‘y’ represents the accurate classification for data point ‘o’, ‘p’ stands for the predicted probability for the same data point, and ‘M’ represents the total number of classes.

Server aggregation: In this phase, the central server aggregates the locally trained model and takes a weighted average of the local updates from the selected clients. The process is repeated until global model reaches the convergence.

Global model broadcast: Finally, the updated global model again gets sent to all selected clients and updates the local model. In the next iteration, the local data will train on the updated local model and selected hyperparameter pool by the AVO. This whole process will repeat itself until a predetermined number of communication rounds or convergence is reached. Our Proposed FedAVO is presented with a complete pseudocode in algorithm 1.

Our study aims to improve the convergence rate and performance of FL algorithms. In order to evaluate the proposed algorithm (FedAVO), we conduct experiments and examine the convergence speed and accuracy. Furthermore, we assess if the model achieves acceptable convergence speed and accuracy after the automated HTO with AVO compared to the baseline algorithm Vanilla FedAvg [1]. In addition, we compare our experimental results with other benchmark FL algorithms such as FedProx [16] and FedEnsemble [36].

This research focuses on reducing communication overhead in federated learning (FL) by fine-tuning the hyperparameters of consumer electronics devices involved in edge learning. FedAVO, our advanced FL algorithm, optimizes hyperparameter adjustments to significantly enhance FL performance. We conducted a thorough assessment on benchmark datasets, comparing FedAVO against state-of-the-art FL algorithms. The results unequivocally establish FedAVO’s superiority over widely adopted FL algorithms like FedAvg, FedPSO, FedEnsemble, and FedGWO, showcasing an average global accuracy improvement of up to 6%. Notably, FedAVO exhibits exceptional capabilities in reducing communication rounds, achieving an average reduction of 30 rounds for the MNIST dataset compared to FedAvg and a substantial reduction of around 150 rounds for the CIFAR-10 dataset. Even when compared to FedGWO and FedPSO, FedAVO significantly reduces communication rounds by up to 50 for CIFAR-10 and an average reduction of 20 for MNIST. Moreover, FedAVO’s adaptability extends to various FL contexts and time-sensitive systems, promising performance enhancements and reduced round requirements in practical real-world scenarios. Future research will explore further improvements in network communication performance by addressing challenges like the risk of local minima and facilitating P2P-AVO communication among clients, potentially through the implementation of dynamic multi-vulture AVO.