Edge AI as a Service with Coordinated Deep Neural Networks

Multi-task learning (MTL) is a technique in which a shared model is used to learn multiple tasks simultaneously [10]. MTL leverages task correlations to efficiently share representations and process high-level features among multiple tasks. Neural architecture search methods explore architectural space to optimize MTL model performance, incorporating branches, skip-connections [11], model refinements, or building a model from scratch [12]. Researchers have developed various techniques for constructing MTL models. In [13], a feature fusion technique is used to combine two distinct models to improve the overall performance. BERT models [14] use adapter modules [15] to adapt pre-trained knowledge to new tasks and enable task-specificity without modifying the base models’ parameters. We use this concept to change data representation when passing data to other inference paths.

Another technique to reduce the computational complexity of DNN models and improve their efficiencies is the early exits (EE) method. With this approach, a number of exit branches are added among the model layers. These EE branches make alternative pathways for the DNN services and can reduce the computation for each task with the cost of reduced precision and increased model parameters. Data is routed through the exit branches when the application is time-sensitive or the system is under heavy load [5]. Alongside EE branches, current methods like SPINN [7] employ synergistic approaches to distribute computation among end devices, edge nodes, and cloud services. It requires the sharing of DNN models and increases resource usage for service replication, posing challenges when computing resources are exhausted.

Instead of offloading entire tasks, we can split models across several devices. Each device works on a part of the model and then sends its results to a central unit that combines everything for the final inference. In [16], data is sent across devices with smaller models, halting computation upon acceptable certainty or forwarding outputs to edge nodes. Similarly, in [6], models are distributed across devices, each performing a portion of computation and forwarding results to aggregators.

The mentioned studies showcased the effectiveness of partitioning and aggregation techniques for distributing DNN models. Nevertheless, we still need to investigate how models can collaborate, exchange knowledge, and utilize each other’s computational resources.

In this paper, we propose a coordinated DNN algorithm at the edge, namely CoDE, to enhance the services offered by SPs. CoDE consolidates individual DNN models on a unified platform by knowledge transferring among them [17] and enhances DNN services by employing resources from other DNN services while ensuring the integrity of DNN models.

Our proposed algorithm establishes new computational and inference paths by linking different DNN models. When SPs are under heavy load, they can use the paths to reduce their computation by offloading the tasks. In this way, SPs increase the system capacity and maintain their quality of services (QoS) [18]. In our algorithm, DNN services can bypass their own obstacles using these pathways, as illustrated in Fig. 2b, while coordinating with other SPs.

Sharing the models may compromise the privacy of SPs. To address this concern, our proposed algorithm does not require the SPs to publish their model’s types or parameters. For example, host SPs may indicate support for images or text without revealing task specifics or disclosing model architecture (e.g., CNN, Transformer) without exposing parameters.

Our main contributions are as follows:

• •

  We introduce coordination among DNN models within or across SPs to allow task offloading to other DNN models and provide more service options. We freeze the models’ parameters and split them by a number of blocks. We add learnable links among the models’ layers to create new paths for the tasks. Our algorithm reduces redundant parameters by preventing replication and avoiding EEs.

• •

  We enable the DNN services to either skip their blocks directly by using skip-connections, or use other services’ blocks by using cross-connections. These connections help DNN services decrease their local computation.

• •

  We then propose CoDE that facilitates the coordination of DNN model sharing and SPs’ resource utilization. Our algorithm obtains the optimal paths when maximizing a reward function which reflects the model accuracy and the inference throughput. The algorithm compares the reward with that of the original model so as to either add the new path to the system or discard it.

• •

  We conduct four experiments to assess CoDE’s performance. Results show CoDE can generate paths with a slight decrease in precision with no extra local parameters. In particular, CoDE increases the local service throughput by up to $40\%$ while the average accuracy is reduced by only $2.3\%$. Compared to the EE method [4], it achieves superior accuracy with less local computation.

This paper is organized as follows. In Section II, we introduce the system model and the proposed algorithm. In Section III, we explain the experiments and provide the results. Section IV concludes the paper.

Each SP offers one or more DNN services, and each DNN service comprises a DNN model. We split pre-trained DNN models into $N$ DNN blocks, as shown in Fig. 2a. We maintain model integrity by freezing the parameters of these blocks. It means that these parameters are not getting updated. Each block can make connections with either local front blocks or host blocks with links, as can be seen in Fig. 2b. Links are small neural network modules with learnable parameters. Two types of connections are defined as follow:

• •

  Skip-connection: It connects local blocks to blocks placed ahead in the path with a link, allowing them to skip over the local blocks in the path.

• •

  Cross-connection: It connects two different DNN models with links. These links are established between the host blocks that are included in the generated path. All of the cross-connection links are located on the host server.

SPs design the links’ architecture for their DNNs based on preceding and succeeding block sizes and their architectures. For example, if the SP aims to make skip-connections [11] for one of its DNNs, it can use any links. Otherwise, if it aims to use the host blocks, it just sends data to the host SP, and it takes care of making the new links between the DNN blocks requested by the local SP. A cross-connection allows the local service to offload its computation partially on the host service. On the other hand, a skip-connection can be established on a single service locally to improve efficiency. The batch size of the local and host services are denoted as $b_{l}$ and $b_{h}$, respectively. The local service offloads $s$ samples of its batch on the path. To accommodate this offloaded data, the host service reserves space for these $s$ samples when a new cross-connection is established, as shown in Fig. 3.

The local and host SPs partition their models into $N_{l}$ and $N_{h}$ blocks, respectively. Let $\mathcal{P}$ denote the set of all possible paths while $p\in\mathcal{P}$ represents a path. Each path $p\in\mathcal{P}$ is represented by $\mathbf{r}_{p}=[lout_{p},hin_{p},hout_{p},lin_{p}]$ where $lout_{p}$ denotes the local block that connects to the $hin_{p}$ block in the host app with the corresponding link, as shown in Fig. 2b. Similarly, the $hout_{p}$ block in the host app connects to the $l{in}_{p}$ block in the local app. We ensure that $0\leq l{out}_{p}<l{in}_{p}<N_{l}$ and $0\leq h{in}_{p}\leq h{out}_{p}<N_{h}$. Alternatively, skip-connections only involve $l{out}_{p}$ and $l{in}_{p}$ since the local server does not communicate with the host. For such cases, we assign $h{in}_{p}=h{out}_{p}=N_{f}$, where $N_{f}\gg N_{l}$.

We define the inference throughput as the number of predictions a model can make in a given time. The local service has two processing streams: A main stream for its own workload, and a host stream for the offloaded tasks. It allows the service to process its own tasks faster by reducing computation per batch, resulting in an enhanced throughput. We define the total throughput as $Th_{p}$, which is the sum of the main and host streams throughput as denoted by $Th_{p}^{l}$ and $Th_{p}^{h}$, respectively. The block processing time allows SPs to determine the throughput. Also, SPs can simply calculate them with a test batch. We introduce the average accuracy $A^{av}_{p}$ and total throughput $Th_{p}$ for each path $p\in\mathcal{P}$ as follows:

where $A_{p}$ and $A_{0}$ are the accuracy of path $p$ and the main path, respectively. We define function $\text{F}(p)$ for each $p\in\mathcal{P}$ to represent its reward as below:

where $\xi(A^{av}_{p})$ is defined as the accuracy reward function and $\zeta(Th_{p})$ is the throughput reward function. In order to reward the desired accuracy range $A^{av}_{p}>A_{min}$, where $A_{min}$ is set by the local SP, we choose $\xi(A^{av}_{p})=Sigmoid(k(A^{av}_{p}-A_{min}))$, where $k$ is a constant. We also define the throughput reward function $\zeta(Th_{p})=Th_{p}-Th_{0}$ that shows the additional throughput path $p$ can provide compared to the original throughput $Th_{0}$. The number of the possible paths is $\mathcal{O}(N^{4})$. Searching through the entire space would take a long time since we need to train all the paths. Hence, we employ a mechanism to predict the accuracy values $A_{p}$. We use a multi-stage optimization algorithm where, at each stage $n$, we aim to estimate the optimal path, denoted as $p^{n}$, using the $n-1$ previously calculated accuracy values of the paths as denoted by set $\mathcal{P}^{n-1}=\{p^{1},...,p^{n-1}\}$.

We define the distance between $p^{i}$ and $p^{j}$ as $d_{p^{i},p^{j}}=\|\mathbf{r}_{p^{i}}-\mathbf{r}_{p^{j}}\|_{2}$. By subtracting $\mathbf{r}_{p^{j}}$ from $\mathbf{r}_{p^{i}}$, we can identify the blocks that exist in only one of the paths $p^{i}$ and $p^{j}$. We predict the accuracy of path $p^{n}$ which is shown by $A_{p^{n}}$ with a weighted average of paths in $\mathcal{P}^{n-1}$. The predicted value is:

The weighted approach leverages the correlation between paths, as closer paths often lead to similar outcomes. Additionally, we calculate $\hat{A}_{p}^{av}$ similar to (1). Thus, we formulate the following problem for stage $n$ which aims to maximize the reward function:

The above problem is a combinatorial optimization problem. As the number of possible paths increases, the complexity of solving this problem escalates tremendously. The complexity of solving such problem would be $O(2^{|\mathcal{P}|})$ in general. Thus, we propose CoDE, as presented in Algorithm 1, to overcome the complexity of this problem. At each stage, the algorithm predicts $F(p)$ for all paths with a total throughput higher than $Th_{0}$. Subsequently, it predicts the path with the highest reward, trains that path, and calculates its actual reward. If the algorithm fails to achieve higher rewards after $c_{stop}$ stages, it terminates the search. The desired path will be generated by linking the blocks of the local and host DNN services. This algorithm handles both cross- and skip-connections. However, by adding a skip-connection, we increase the local service’s parameter count.

By using Algorithm 1, all the links of the cross-connections are positioned on the host DNN services, and the backpropagation process terminates after the entry link. This approach allows services to expand their functionalities without additional parameters on the local device. Generally speaking, it fosters a more flexible network with enhanced capabilities through collaboration among DNN services.

In this experiment, both models are AlexNet [19], with a cross-connection established between them. The objective is to evaluate cross-connection performance for two DNN services with the same architectures but different tasks, depicted in Figs. 5a and 5b. Fig. 4 shows a 6-block architecture, but we add another block at the beginning of the model. The local model is optimized for CIFAR-10, and its accuracy is 86.7%. On the other hand, the host model is optimized separately for ImageNet and Food-101 datasets to assess the performance across different tasks. In addition, to measure the effect of the models, we use random parameters in another task with the same setup. We set the $lout_{p}$ and $hin_{p}$ to 0 and 1, respectively, and conduct the experiments when varying the $l{in}_{p}$ and $hout_{p}$, which means that we skip blocks 1 to $lin_{p}$ in the local app. We also vary the number of host blocks from 1 to $hout_{p}$. All the links are 2D CNN layers. When the actual output size of a block is not equal to the input size of the next block of the path, the transition between the local and the host app or skip-connections, we use a 2D CNN layer.

Fig. 5 shows the accuracy, number of parameters, and the model architectures. The results show a considerable difference between the pre-trained host models and the random model. With $hout_{p}=2$ and $l{in}_{p}=5$, we reduce $1.88$ million local parameters, and we achieve an accuracy of $80.2\%$ with Food-101 and $80\%$ with ImageNet. However, the accuracy for the random host model is $71.3\%$. It means that if considering the random model as the baseline for the accuracy drop, we can compensate $58\%$ for the accuracy drop with the pre-trained models. Furthermore, if we consider the skip-connection from lout:0 to lin:5 with accuracy of $61\%$ as the baseline, then we can compensate $78\%$, which is significant.

If we increase the host blocks, we do not necessarily raise the paths’ precision. For example, when $hout_{p}$ is raised from 4 to 5, the accuracy drops from $83.3\%$ to $82.4\%$, contradicting assumptions in EE models. In [4], the author used early exit on VGG-16, which performs more powerfully than AlexNet on CIFAR-10, but its accuracy is dropped from $93.5\%$ down to $80\%$, and even EEs add a large number of parameters in the local servers. Our method achieves improvements in both local computation and model accuracy compared to [4].

We also assess the effect of the link size on the path performance by reducing CNN filter numbers using two CNN layers. The first layer decreases input layers to either 25 or 5, as shown in Fig. 6, and then the next CNN layer extends it to the next block’s input size. In these cases, if we set $\textbf{r}_{p}=[0,1,4,5]$, links shrink by almost $5$ and $25$ times, respectively. By setting the layer’s filter size to 25, the accuracy decreases from $82.7\%$ to $81.5\%$, and if we set it to 5, then the accuracy would be $73.8\%$. As a result, SPs can adjust their performance, not just by changing the number of blocks but also with the link size.

To investigate the host models’ effect on the accuracy, we then remove the host blocks and just use the relevant links to measure the effect of the links in the performance alone. The accuracy is significantly reduced to $58.8\%$ because most of the links are just bottleneck CNN layers, which cannot effectively extract the input’s features.

In this experiment with an AlexNet model, skip-connections are established between its blocks. Additionally, we investigate how the number of link parameters affects path performance. In this experiment, links are either one CNN layer in normal mode or two CNN layers with one MaxPool layer in reduced mode. In the normal mode, links are generated in order to convert the output size of the blocks, but in the reduced mode, we use two CNN layers to reduce the number of filters.

When we set $lout_{p}$ to 0 for skip-connections, the accuracy notably decreases. It means that the links cannot be learned properly. On the other hand, when we set $lout_{p}$ to 1, the performance rises, and as the gap between $lout_{p}$ and $lin_{p}$ increases, the accuracy drops more.

Fig. 7a shows that if we reduce the link size by 5 times, it decreases the accuracy slightly. For example, when we set $lout_{p}=1$ and $lin_{p}=5$, the accuracy would be $83.3\%$ and $80.3\%$ for the normal and reduced mode, respectively. Both modes have comparatively fewer parameters than the skipped parameters. Fig. 7b shows the number of the link’s parameters and the skipped parameters of the main model. All in all, skip-connection performs less effectively in parameter reduction compared to cross-connection. With $lout_{p}=1$ and $lin_{p}=4$, inference time is $1.4ms$, $30\%$ faster than the main model’s inference time of $2$ms.

In this experiment, we use MobileNet [20] as the host DNN service. Fig. 6 shows that we can compensate for the performance degradation with a host model, even with a different architecture. The host model is optimized for ImageNet ¹¹1PyTorch pre-trained model, as shown in Fig. 7c. In this experiment, $lout_{p}$, $lin_{p}$, and $hin_{p}$ are set to 0, 5, and 1, respectively, and $hout_{p}$ is varied from 2 to 5. Here, the accuracy increases monotonically with the rise of $hout$, which did not happen in Experiment 1. In this scenario, again the number of the added-host parameters, which is the summation of the link’s and the added host’s parameters, is lower than the skipped-local parameters, as shown in Fig. 7d.

By using MobileNet for ImageNet, we can revive the accuracy up to $84.5\%$, which is sufficiently high to serve as a coordinated model. This means, we reduce local computation by $75\%$ just by $2\%$ reduction in the overall accuracy.

We now explore the impact of accuracy and computation time on path selection. We use two P100 GPUs on the same machine, where the stream’s delay is negligible. In order to calculate the throughput of each path, we use the computation time of each block. We also consider that the host architecture is AlexNet, and use the proposed algorithm and the reward function to find a path. For this test, we set $b_{l}=b_{h}=32$, $N_{l}=N_{h}=6$, $s=8$, $A_{min}=86\%$, $c_{stop}=3$, and $\epsilon=0.01$.

In this experiment, $\mathbf{r}_{p}=[0,1,4,5]$ exhibits the best overall performance, as can be seen in Fig. 8, with average accuracy $84.4\%$ and $Th_{p}/Th_{0}=1.4$. In simpler terms, this means that SP1 can generate a path with a $40\%$ increase in throughput and only a $2.3\%$ decrease in the total accuracy. As a result, it is highly efficient for SP1 to generate a path with a host model optimized for ImageNet.