AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications

Due to the dynamic natures of DNN deployment context, we aim to continually tune the DNN compression configurations to directly/indirectly optimize the application-driven system performance (i.e., accuracy, energy efficiency, latency). The hybrid dependency of multiple platform-relative performance metrics and DNN-dependent metrics are shown in Figure 4. To further understand the performance requirements of DNN for the continuously running mobile applications, we ask $60$ mobile users and $10$ Android developers to rate the importance of different DNN performance aspects on mobiles. And we summarize the results as our design goals. Specifically, a DNN for continuously running mobile Apps needs to fulfill the following requirements:

• •

  Accurate: the DNN is accurate enough to guarantee a high-quality task. The model weights at different scales are well-trained to represent the generic information of recognition objects.

• •

  Responsive: the complexity of the DNN should be controllable to satisfy diverse user demands on latency constraints, especially on low-end (e.g. CPU-powered) mobiles.

• •

  Energy-Efficient: the energy consumption of the DNN should be continually optimized, which is the bottleneck metric for continuously sensing applications (Yang et al., 2017).

• •

  Runtime-evolutionary: both the DNN architecture and parameter weights are runtime-evolutionary to meet the dynamic deployment context for continually optimizing the above three requirements (i.e., accurate, responsive, energy-efficient) at runtime.

Unfortunately, none of previous efforts satisfy all these requirements (as discussed in $\S$ 2). To this end, this paper proposes AdaSpring, a context-adaptive and runtime-evolutionary DNN compression framework to automatically optimize the requirements mentioned above, which are closely related to the user experience.

As shown in Figure 4, AdaSpring intends to provide a systematic method to automatically select the compression operator combination for tuning the above conflicting and interdependent performance metrics. Mathematically, AdaSpring explores an efficient solution to the following dynamic optimization problem:

where $\mathrm{\Delta}$ represents the set of all optional convolutional compression operators (as enumerated in $\S$ 5.1). Given an backbone-net architecture $\Omega$, $\Omega(\delta_{i})$ represents the re-configured model architecture compressed by the selected compression operator $\delta_{i}$. $A$, $E$, $T$ and $S$ denote the measured accuracy, energy efficiency, latency, and memory footprint of a given model running on the target mobile platform. The two objectives on $A$ and $E$ are combined by relative importance coefficients $\lambda_{1}$ and $\lambda_{2}$, which dynamically depend on the platform’s remaining battery. We express the dynamic deployment contexts as a set of time-varying constraints, i.e., the threshold of accuracy loss $A_{threshold}(t)$, the latency budget $T_{bgt}(t)$, the storage budget $S_{bgt}(t)$, and relative importance coefficients of objectives ($\lambda_{1}(t)$, $\lambda_{2}(t)$). The latency budget $T_{bgt}(t)$ is application-specified. And the storage budget $S_{bgt}$(t) is platform-imposed. For example, reducing the model size $S$ to satisfy the budget of L2-Cache $S_{bgt}$ helps to fit it into the on-chip memory and avoids the expensive off-chip access. We note that $Norm(.)$ is a normalization operation for objective aggregation, e.g. $log(.)$. We then propose a heuristic optimization solution as adjusting the model architecture for satisfying dynamic performance requirements. In particular, the model architecture $\Omega(\delta_{i})$ can directly determine both $S$ and $A$ (see Figure 4). While the quantification of hardware-dependent metrics $E$ and $T$ are not straightforward. Therefore, AdaSpring’s goal turns to adaptively select the compression operator combination $\delta_{i}$ from a discrete set of all possible combinations $\mathrm{\Delta}$, so that it can directly/indirectly tune model performance metrics.

The above challenging problem motivates the AdaSpring design. As shown in Figure 4, the AdaSpring framework consists of a self-evolutionary network, a runtime adaptive compression block, and a dynamic context awareness block. (i) The self-evolutionary network is an ensemble of a backbone-net and multiple retraining-free compression operator-variants, which enables weight recycle between numerous variants while avoiding catastrophic interference. We initialize the backbone-net’s hyperparameters at design time using an on-demand DNN generation framework, i.e., AdaDeep (Liu et al., 2020), for satisfying mobile application performance demands on a target platform. (ii) The runtime adaptive compression block is capable of selecting a deterministic optimal combination of compression operators for reconfiguring and evolving the backbone-net at runtime. And (iii) the dynamic deployment context awareness block detects the evolution demands and triggers the runtime adaptive compression block. The triggering station can be modeled as the noticeable context changes or by a pre-defined frequency (e.g. time slice) for continuously running Apps in regular days.

This paper focus on the configuration optimization of convolutional architecture, operations, and activations. Because recent successful DNN models tend to shift more parameters on convolutional layers and use fewer fully-connected layers (Wu et al., 2018b; Chen et al., 2016; Cai et al., 2019). Built upon the existing compression experience, we propose the following alternative convolutional compression operators that synthesize multiple scaling dimensions (e.g. width, depth, and connection).

• •

  Compression operator $\delta_{1}$: multi-branch channel merging techniques(e.g. Fire block (Iandola et al., 2016)) increase the model depth with less parameters by replacing one conv layer using two conv layers (i.e., squeeze layer and expand layer) which is elaborately designed to decrease the kernel size and channel size per unit.

• •

  Compression operator $\delta_{2}$: low-rank convolution factorization techniques (e.g. SVD-based (Wu et al., 2018b), sparse coding-based (Bhattacharya and Lane, 2016) factorization, or depth/group-wise convolution (Li et al., 2019)) decompose a conv layer into several conv layers with smaller kernel size, hence leads to a growing model depth with less parameters.

• •

  Compression operator $\delta_{3}$: channel-wise scaling techniques (e.g. channel-level pruning (Cai et al., 2019) and channel-wise architecture noise injection (Yang et al., 2020)) can tune variable operator-variant sampling.

• •

  Compression operator $\delta_{4}$: depth scaling techniques (e.g. depth-elastic pruning (Cai et al., 2019), residual connection (He et al., 2016)) can derive a shallower variant-network from a backbone-network via skipping connections.

We put the retraining process ahead in the ensemble training of the self-evolutionary network at design time to get rid of weight retraining during dynamic inference. Therefore, the self-evolutionary network training is an ensemble of a backbone-net and multiple variant-nets derived by various convolutional compression operators.

To evolve DNN architecture and weight to an optimal configuration at runtime, we propose the runtime search strategy based on the above flexible and elite search space.

We first present the settings for our evaluation.

System Implementation. We implement AdaSpring’s offline block with TensorFlow (Google, 2017) in Python on the server side to train the self-evolutionary network (see $\S$ 4). And we realize the AdaSpring’s online blocks on the mobile and embedded platforms to adjust the DNN configurations on the fly for better inference performance. The self-evolutionary network (i.e., a backbone-net and multiple variant compression operators), generated by AdaSpring’s offline component, is then loaded into the target platform. To further reduce the memory access cost, we load DNN parameters from L2-Cache memory.

Evaluation Applications/Datasets. We use five commonly used mobile applications/datasets to evaluate AdaSpring’s performance as elaborated in Table 1. Specifically, we test AdaSpring for mobile image classification (D1: Cifar100 (Krizhevsky, 2009b), D2: ImageNet (Deng et al., 2009)), mobile acoustic event awareness (D3: UbiSound (Sicong et al., 2017)), mobile human activity sensing (D4: Har (UCI, 2017)), and mobile driver behavior prediction (D5: StateFarm (Kaggle, 2019)).

Mobile Platforms with Dynamic Context Settings. We evaluate AdaSpring on three categories of commonly used mobile and embedded platforms, including one personal smartphones, i.e., Xiaomi RedMi 3S (device1), one embedded development board, i.e., raspberry Pi 4B (device3), and one mobile robot platform i.e., NVIDIA Jetbot (device4) loaded with the mobile development board. They are equipped with diverse processors, storage and battery capacity. The dynamic context is formulated by the time-varying latency budget $T_{bgt}(t)$, storage budget $S_{p}(t)$, and the relative importance coefficient of accuracy and energy efficiency objectives.

Comparison Baselines. We employ three categories of DNN specialization baselines to evaluate ${\sf AdaSpring}$. The detailed settings of ten baselines from three categories are as below. Firstly, the hand-crafted compression baselines relay on manual design to realize efficient DNN compression. They provide the high standard for AdaSpring to tune the specialized DNNs’ performance tradeoff between accuracy, latency, and resource efficiency.

• •

  Fire (Iandola et al., 2016) presented in SqueezeNet reduces filter size and decreases input channels using squeeze layers.

• •

  MobileNetV2 (Sandler et al., 2018) replaces the traditional convolutional operation by an inverted residual with the linear bottleneck to expand module to high dimension and then filter with a depth-wise convolution.

• •

  SVD-based convolutional decomposition technique (Lane et al., 2016) introduces an extra conv layer between $conv_{i}$ and $conv_{(i+1)}$ using the singular value decomposition (SVD) based parameter matrix factorization. The number of neurons $k$ in the inserted layer is set according to the dynamic neuron numbers $m$ in $conv_{i}$, i.e., $k=m/12$.

• •

  Sparse coding-based convolutional decomposition technique (Bhattacharya and Lane, 2016) insert a conv layer between $conv_{i}$ and $conv_{(i+1)}$ using the sparse coding-based parameter matrix factorization. The k-basis dictionary is dynamically determined by the neuron number $m$ in $conv_{i}$, i.e., $k=m/6$.

Secondly, the on-demand DNN compression baseline methods adopt a trainable optimizer to automatically find the most suitable DNN compression strategies for various mobile platforms.These baselines provide a strict benchmark against which we can validate that both searching and retraining costs are bottleneck limitations for the runtime adaptation demands.

• •

  AdaDeep (Liu et al., 2020) automatically selects and combines compression techniques to generate a specialized DNN that balance accuracy and resource constraints.

• •

  ProxylessNAS (Cai et al., 2018) directly learns architectures without any proxy while still allowing a large candidate set and removing the restriction of repeating blocks.

• •

  Once-for-all(OFA) (Cai et al., 2019) obtains a specialized sub-network by selecting from the once-for-all network that supports diverse architectural settings without additional training.

Thirdly, the runtime adaptive DNN compression requires to search for the most suitable combination of retraining-free compression techniques quickly, we select two baseline optimization methods to compare with AdaSpring. Here, the baseline optimizers represent two intuitive searching ideas for the runtime adaptive compression of DNN configurations.

• •

  Exhaustive optimizer tests all combinations of compression operators’ performance on the validation and then selects the one variety with the best tradeoff based on the fixed performance ranking. And then it fixes the compression operators and only scale down the compression operators’ hyperparameters, i.e., compression ratio, to satisfy the dynamic resource budgets.

• •

  Greedy optimizer selects the best compression operator layer-by-layer that obtains the best tradeoff between accuracy and parameter size, in which the relative importance is equally set to a fixed value of $0.5$.

• •

  AdaSpring selects and applies the most suitable combination of compression operators into the self-evolutionary backbone network for accuracy and resource efficiency tradeoff.

We evaluate AdaSpring in terms of the specialized DNNs’ running performance (i.e., accuracy $A$, amount of MACs $C$, parameter arithmetic intensity $C/S_{p}$, activation arithmetic intensity $C/S_{a}$, and energy consumption $EC$) and the specialization methods’ all-around performance (i.e., search cost, retraining cost, and scaling flexibility). As shown in table 2, we compare AdaSpring’s performance with ten baselines. In this thread of experiments, we leverage the same mobile sensing task (i.e., image recognition using CIFAR-100 ($D_{1}$) datasets) and target embedded platform (i.e., Raspberry Pi 4B) for six state-of-the-art baselines and AdaSpring for a fair comparison. We adopt different baseline methods to specialize the DNN architectures and weights for optimizing accuracy and resource efficiency objectives with dynamically specified constraints (see $\S$ Equ. 3.2). Here, the relative importance coefficients (i.e., $\lambda_{1}$ and $\lambda_{2}$) are dynamically determined by the remaining battery percentage $E_{remaining}$ of the target platform, i.e., $\lambda_{2}=max\{0.3,E_{remaining}\}$, and $\lambda_{1}=1-\lambda_{2}$ . Afterwards, we test the specialized DNN’s running performance on a Raspberry Pi 4B platform. To mitigate the effect of noise and increase the robustness of performance measurements, we repeat the steps mentioned above five times and take an average over them.

Performance comparison. Table 2 summarizes the performance comparison between ten baseline methods and AdaSpring. First, AdaSpring achieves the best overall performance in terms of accuracy $A$, MAC amount $C$, parameter arithmetic intensity $C/S_{p}$, activation arithmetic intensity $C/S_{a}$, and energy consumption $En$, while incurring negligible accuracy loss, compared to the DNNs specialized by other baseline methods. The AdaSpring reduces the model inference latency to $15.6$ms, the energy consumption to $1.9mJ$. And it increases the parameter arithmetic intensity $C/S_{p}$ to $158.9$, the activation arithmetic intensity $C/S_{a}$ to $358.7$. Notably, AdaSpring generates DNN to get the largest activation arithmetic intensity $C/S_{a}$ and second-largest parameter arithmetic intensity $C_{a}$. Compared to the parameter size, the influence of activation arithmetic intensity upon energy consumption is equally or even more critical. The DNN specialized by the hand-crafted Fire, MobileNetV2, SVD, and sparse coding techniques consumes energy by $3.1mJ$, $5.2mJ$, $4.8mJ$, and $4.6mJ$, respectively. The accuracy of exhaustive optimizer is much lower than the proposed design, since it shows low accuracy when it fixes the compression operator categories and only over-compresses their hyperparameters. This outcome demonstrates that the reselection of different compression operators are necessary. The specialized DNN’s accuracy achieved by AdaSpring is at least as good as ProxylessNAS, and sometimes even better than the hand-crafted compression techniques. Second, the AdaSpring’s specialization scheme is the most efficient in reducing the searching cost and retraining cost. The adaptive compression baselines involve a high overhead in retraining. For example, AdaDeep requires an average of $19\sim 38$ hours for retraining (e.g. retraining the deep reinforcement learning model-based optimizer) offline on the GPU platform for each adjustment of compression strategies. AdaDeep and ProxylessNAS need $18N$ and $196N$ hours, respectively, to search from the candidate configurations, which increases linearly with the number of dynamic contexts. Although OFA and AdaSpring do not need retaining. OFA needs $41hours$ to search per adaptation, while AdaSpring only needs $3.8ms$ to do that. This is because AdaSpring leverages the elite compression operator space, rather than the basic kernel size or channel number space in OFA, to avoid the redundant search exploration.

Summary. AdaSpring outperforms the other ten baselines in terms of the DNN performance tradeoff between accuracy, latency, arithmetic intensity of parameters and activations, and energy consumption. Meanwhile, it incurs the modest searching cost without retraining, making it ideal for runtime adaptive DNN compression.

To illustrate the AdaSpring’s performance over different tasks, we evaluate it using all the five applications/datasets (see $\S$ 6.1) on a Raspberry Pi 4B platforms (Device 3) which is powered by a mobile $3800mAh$ battery. AdaSpring dynamically detects the platform’s remaining battery and sets the coefficients between accuracy and energy efficiency in Equ. 3.2 according to the percentage of remaining power $E_{remaining}$, i.e., $\lambda_{2}=max\{0.3,1-E_{remaining}\}$. In addition, we specify the storage budget as 2MB that is capacity of the L2-Cache. We set a accuracy loss threshold to be $0.5,0.3,0.6,0.5$ for image classification tasks ($D_{1}$, $D_{2}$), sound sensing (i.e., $D_{3}$), human activity prediction task ($D_{4}$), and driver behavior recognition task ($D_{5}$), respectively. And assume the latency sensitivity as the latency budget of $20ms,10ms$, $30ms,20ms$ for $D_{1}\sim D_{5}$.

Performance. Figure 8 compares the performance of the DNN configurations specialized by AdaSpring on five different tasks in terms of user experience metrics (i.e., inference accuracy $A$, energy efficiency $E$, and inference latency $T$) and direct DNN metrics (i.e., computation $C$, parameter size$S_{p}$, activation size $S_{a}$). And we compute the mean and standard deviation of the running performance of the DNN specialized in five dynamic moments, at which the percentage of remaining battery is $0.85$, $0.75$, $0.62$, $0.52$, and $0.38$, respectively. These affect the tradeoff demands on objectives. The storage budget $S_{bgt}$ for parameters dynamically depends on the available Cache capacity. We simulate the unpredictable resource contention by other software using the randomization noise $\sigma$ injection to Cache’s available capacity, i.e., $(2-\sigma)$MB. For different tasks, datsets, and deployment contexts, AdaSpring selects the various combinations of compression operators to scale up/down the model configurations to optimize and balance multiple performances. It achieves the inference latency $1.2\sim 2.8$, the parameter arithmetic intensity $106\sim 229$, activation arithmetic intensity $143\sim 220$, with a negligible accuracy loss ($\leq 0.5\%$ ) or even accuracy improvement ($\leq 2.2\%$).

Summary. For different tasks with diverse backbone model shapes and various sensitivity to accuracy loss and latency, the DNN specialized by AdaSpring varies. As for the same task, the DNN’s compression configurations founded by AdaSpring also differ according to the dynamic deployment context.

In this experiment, we compare the AdaSpring’s performance for mobile sound sensing application ($D_{3}$), tested in three different platforms. We adopt the same self-evolutionary network comprising of the same backbone-net and some optional compression operator-variants. Different platforms have different resource characteristics, which are further affected by dynamic deployment contexts. In particular, we adopt the RedMi 3S smartphone equipped with Qualcomm B21 processor, $2MB$ L2-Cache, and $4100mAh$ battery; the Raspberry Pi 4B with $2MB$ L2-Cache, and $3800mAh$ battery; and the NVIDIA Jetbot with quad-core ARM Cortex-A57 processor, $2MB$ L2-Cache, and $7200mAh$ battery. We adopt the similar dynamic deployment context settings with $\S$ 6.3.

Performance. Figure 9 summarizes the performance of DNN specialized by AdaSpring along with the dynamic changes of deployment context. We first initialize the different DNN configurations for various platform constraints and then leverage AdaSpring to update the compression operator-variant selections according to the specific platform’s dynamic contexts. We select four points of dynamic contexts. AdaSpring identifies DNN configurations to obtain latency of $1.1\sim 1.8$, parameter arithmetic intensity $81\sim 151$, and activation arithmetic intensity $192\sim 397$ while have slightly degraded or even better accuracy $91.2\%\sim 98.6\%$. We pick a time fragment to show four moments with dynamic deployment contexts, as shown in Table 4. As the gradual reduction of battery power and the dynamic fluctuation of Cache capacity, we show the performance changes of the DNN, which is continually scaled-down/up by selecting and combining different compression operators. Moreover, AdaSpring supports scale up the model architecture again when the dynamic constraints on resource efficiency are relaxed, bringing better flexibility.

Summary. AdaSpring adaptively selects the proper combination of compression operators to optimize DNN performance continually to meet dynamic context demands. Moreover, AdaSpring accomplishes a flexible evolution, i.e., support both scaling up and down the DNN configurations as the context demands change.

In this subsection, we evaluate the impact of different factors on AdaSpring’s design.

We deploy AdaSpring on a commercial mobile robot platform (i.e., NVIDIA Jetbot, device4) and conduct a one-day experiment (09:00 to 17:00) to continually optimize the DNN configurations for a sound assistant application (i.e., UbiEar (Sicong et al., 2017)). This application adopts a DNN to realize a sound recognition and notification tool for hard-of-hearing people to sense emergency (e.g. fire alarms, smoke alarms, kettle boiling whistle) and social events (e.g. doorbell ring, knocking door, people crying). We simulate the dynamic mobile context of the DNN (as described in $\S$ 3) as follows. On the one hand, we artificially play some audio clips for emergency events and generate social events to control the happening frequency of acoustic events, affecting the DNN inference frequency. On the other hand, we simulate the unpredictable storage resource contention by other software using the randomization noise (e.g. Gaussian noise) $\sigma$ injection to the available capacity of L2-Cache, i.e., $(2-\sigma)$MB. Here, the maximum capacity of L2-Cache on NVIDIA Jetbot platform is $2MB$, and we update the randomized resource contention value of $\sigma$ per hour. We do not artificially change the battery power, which is continuously consumed in the real-world as the application runs. Therefore, the remaining battery is dynamically changing, e.g., 86%, 72%, and 63%, as shown in Figure 13, which forms the dynamic energy budgets.

Figure 13 illustrates the dynamic deployment context (i.e., energy, storage, event happening frequency) of the DNN for the continuous sound sensing application. The battery’s remaining energy formulate the importance coefficient $\lambda_{2}$ in the runtime optimization problem (Equ. 3.2). The available capacity of L2-Cache decides the storage budget of parameters $S_{bgt}$. And the sound emergency frequency will indirectly influence the battery’s power. Different deployment contexts have various resource constraints and performance objective sensitivity, which lead to further performance and budget demands on the DNN. AdaSpring triggers the runtime DNN evolution block by a pre-defined frequency (e.g. every two hours) to shrink the DNN configurations in this regular day. Figure 12shows the runing performance of the DNNs specialized by AdaSpring. AdaSpring can continually and adaptively select the best compression strategy to shrink the DNN configurations given diverse user demands. Specifically, it selects the $\delta_{1}$ (Fire) + $\delta_{3}$ (pruning 50% channel) for the regular resource-constrained moment, $\delta_{1}$ (Fire)+ $\delta_{4}$ (pruning 1 layer) for the tight memory constraint moment, and $\delta_{2}$ (SVD-based decomposition) + $\delta_{3}$ (pruning 65% channel) for the tight battery-bounded moment. The evolved models can achieve $\geq 95.6\%$ accuracy and $168.2\sim 202.6$ arithmetic intensity. AdaSpring searches the proper combinations of compression operators that satisfy diverse demands on accuracy and resource efficiency within $2.8ms\sim 3.1ms$.