Accelerate CNNs from Three Dimensions: A Comprehensive
Pruning Framework

Following DBP (Wang et al., 2019b), we put a linear classifier after each layer of model $M_{0}$ and test its accuracy on the evaluation dataset. The accuracy of each linear classifier indicates the discrimination of its corresponding layer’s features. Further, each layer’s discrimination enhancement compared with its preceding layer is seen as the importance of the layer. With this importance metric, we pick out the least important $(1-d^{\star}/d_{0})\times 100\%$ layers and remove them from $M_{0}$, yielding $M_{p_{1}}$.

Filter-level pruning is performed over $M_{p_{1}}$. In particular, we use the scaling factor of BN layers as the importance metric, just like Slimming (Liu et al., 2017). However, different from Slimming that compares the importances of all filters globally, we only compare the importances of filters in the same layer, and the least important $(1-w^{\star}/w_{0})\times 100\%$ filters of each layer will be pruned. Through such a modification, the pruned ratios of all layers are kept the same. Assume the model after filter-pruning to be $M_{p_{2}}$.

After pruning, the pruned model $M_{p_{2}}$ is fine-tuned with images of size $r^{\star}$. The images are resized by bilinear down-sampling, which is the most common down-sampling scheme for images. The model will be fine-tuned with a small learning rate till convergence, leading to the finally pruned model $M_{p}$.

We take three popular datasets as testbeds of our algorithm: CIFAR-10 (Krizhevsky et al., 2009), TinyImageNet (Wu et al., 2017), and ImageNet (Russakovsky et al., 2015). These three datasets differ in their image-resolutions (32$\times$32 to 224$\times$224), number of classes (10 to 1000), and scale of datasets (50K to 1000K images). For all the datasets, images are augmented by symmetric padding, random clipping, and randomly horizontal flip, all of which are common (He et al., 2016; Howard et al., 2017; Wang et al., 2019a) augmentation methods for these datasets.

We test our algorithm on three popular network architectures: ResNet (He et al., 2016), DenseNet (Huang et al., 2017), and EfficientNet (Tan & Le, 2019). Their basic blocks vary from residual blocks to densely connected blocks and NAS-searched blocks, representing three of the most popular designs for deep CNNs.

Following the conventions of previous work (Li et al., 2020; Lin et al., 2020; Ye et al., 2020a), we take the accuracy, parameters-reduction-ratio ($Prr$), and FLOPs-reduction-ratio ($Frr$) as the evaluation protocol of our model acceleration algorithm. $Prr$ and $Frr$ are defined as Eq. (9), where $M_{0}$ and $M_{p}$ represent the base model and the pruned model, respectively.

The compared algorithms fall into three categories: (1) Algorithms solely pruning the model along one dimension (i.e., depth, width, or resolution), including $\mathcal{R}$-only (Howard et al., 2017), $\mathcal{W}$-only (Liu et al., 2017), FPGM (He et al., 2019), DBP (Wang et al., 2019b), PScratch (Wang et al., 2020), DHP (Li et al., 2020), and HRank (Lin et al., 2020); (2) Algorithms that prune along multi-dimensions, such as GAL (Lin et al., 2019); (3) NAS-based algorithms, including EfficientNet (Tan & Le, 2019) and TinyNet (Han et al., 2020), which balance the size of the three dimensions from the NAS perspective.

For base models trained on CIFAR-10, we set batch size to $64$ for DenseNet and $128$ for ResNet, respectively. Weight decay is set to $10^{-4}$. The models are trained for $160$ epochs with the learning rate starting from $0.1$ and divided by $10$ at epochs $80$ and $120$. These are all the most common training settings (He et al., 2016; Howard et al., 2017; Wang et al., 2019a) for models trained on CIFAR-10. For ResNet and DenseNet trained on TinyImageNet and ImageNet, batch size is set to $256$, and weight decay is $10^{-4}$. Models are trained for $100$ epochs. The learning rate is set to $0.1$ at the beginning and is multiplied by $0.1$ at epochs $30$, $60$, and $90$. For EfficientNet, we apply the same training policy as (Han et al., 2020), which is also the most common for EfficientNet implemented with PyTorch (Paszke et al., 2017).

The MAP’s hyper-parameters are set to $\mathcal{R}=1$ and $\mathcal{K}=3$ in our pruning experiments. When collecting training data (i.e., $\{(d_{n},w_{n},r_{n})\}_{n=1}^{N}$) for the polynomial regression, the model is pruned along each dimension for four times (i.e., $rds=4$ in Algorithm 1). ResNet and DenseNet trained on CIFAR-10 are fine-tuned for $40$ epochs at each round, and for $80$ epochs after comprehensive pruning. Therefore, the data collection process consumes as much time as training $3$ models (training one model from scratch costs $160$ epochs). Similarly, models trained on TinyImageNet and ImageNet are fine-tuned for $30$ epochs at each round of the iterative pruning process. Thus, it takes about the same time as training $3.6$ models for the data collection process. The finally pruned models trained on TinyImageNet and ImageNet are fine-tuned for $60$ epochs after comprehensive pruning.

The experimental results on CIFAR-10 and TinyImageNet are shown in Table 1. As we can see, $\mathcal{W}$-only induces greater loss than $\mathcal{D}$-only for ResNet-32 and ResNet-56, while for ResNet-101, the situation is opposite. In other words, the importance of different dimensions lies in the original size of depth, width, and resolution, and we cannot deduce it from a simple prior, which further shows the essentiality of our algorithm. We balance the size of the three dimensions dynamically and always achieve better results than pruning one or two dimensions. The most competitive opponent of our algorithm is DHP, which achieves similar accuracy and $Frr$ to our algorithm for ResNet-56 trained on both datasets. However, we show higher accuracy than DHP for DenseNet-40 on CIFAR-10 (94.54% vs. 93.94%), for ResNet-101 (65.27% vs. 64.82%) on TinyImageNet, and for DenseNet-100 (60.22% vs. 59.40%) on TinyImageNet with similar $Prr$ and $Frr$, which sheds light on the robustness of our algorithm across different architectures and datasets.

Experiments with ImageNet are done on ResNet-50 and DenseNet-121. From Table 2, we can see that our algorithm achieves 0.45% higher accuracy on ResNet-50 than the state-of-the-art algorithms (DHP and PScratch) with the same $Frr$. The improvement on DenseNet-121 is marginal compared with $\mathcal{W}$-only, because our algorithm also prunes DenseNet-121 mainly along width dimension, which indicates that DenseNet-121’s width is large and has much redundancy. By contrast, images do not need to be pruned. With a comprehensive consideration, our algorithm also deems that we should mainly prune filters of DenseNet-121 for acceleration. Therefore, it produces similar pruning results to filter-level pruning. However, the results do not imply that our algorithm is powerless. On the contrary, a pruning policy with a comprehensive consideration is always better than an arbitrary one, though they may produce similar results sometimes.

Algorithms that balance the three dimensions (i.e., depth, width, and resolution) in a NAS manner are also compared, and the results are shown in Table 3. GPU-days is the most common metric to evaluate the search cost of NAS algorithms, which indicates natural days they spend if running with only one GPU. Both EfficientNet (Tan & Le, 2019) and TinyNet (Han et al., 2020) employ so many resources in searching the optimal $(d^{\star},w^{\star},r^{\star})$, while we do not have enough GPUs to reproduce their searching process. Thus, the results of EfficientNet and TinyNet are both drawn from (Han et al., 2020), and their search costs are estimated through the number of models they trained. For example, training an EfficientNet for $300$ epochs takes about $26$ hours with $8\times$V100 GPUs, while TinyNet requires to train $60$ EfficientNet models from scratch. Hence, its search cost is about $520$ GPU days. Instead, our algorithm only spends about $\frac{1}{25}$ as much time as TinyNet on searching but achieves similar accuracy.

In Section 3.2.2, our proposed MAP is

where the rank of $\Theta$ is less than or equal to $\mathcal{R}$. Experimentally, we find that $\mathcal{R}=1$ works well in most cases. To further explore this interesting phenomenon, ResNets with different $(d,w,r)$ are trained on CIFAR-10. The base model (i.e., $(d,w,r)=(1.0,1.0,1.0)$) is ResNet-32 with images of size $32$, and the results are plotted in Figure 3. Observations from the first three sub-figures are shown in their titles. We can deduce from these observations⁵⁵5More results and the proof are put in our supplementary material.:

which coincides with Eq. (10) once $\mathcal{R}=1$. In other words, three variables $(d,w,r)$ in the MAP can be approximately separated from each other. We also test the MAP with different $\mathcal{R}$, as shown in the 4th sub-figure of Figure 3. The MAP with larger $\mathcal{R}$ yields similar $(d^{\star},w^{\star},r^{\star})$ to that of $\mathcal{R}=1$, which also indicates that $\mathcal{R}=1$ is enough for obtaining a well-performed MAP.

Besides restricting the rank of $\Theta$, we also try two extra methods of avoiding overfitting, i.e., decreasing the degree $\mathcal{K}$ of polynomials and applying regression with regularization terms. The results are reported in Table 4. Specifically, a set of $13$ items $(d_{n},w_{n},r_{n},a_{n})$ is used to fit the MAP, and a set of the other $80$ items is used for evaluation. All data is collected with ResNet-56 trained on CIFAR-10. Training error and evaluation error are both reported. As we can see, normal polynomial regression induces severe overfitting and high evaluation loss, and $\ell_{2}$-regularization has a limited effect on dealing with the overfitting issue. Still, lowering the degree of polynomials is not a wise choice because it makes the polynomials fail to converge even on training data. Instead, our specialized MAP shows a lower error rate on both training data ($0.08\%$) and evaluation data ($0.25\%$).

Figure 4 shows the pruning results when adjusting the MAP’s degree $\mathcal{K}$. Especially, the polynomial regression degrades to linear regression when $\mathcal{K}=1$. It turns out that for polynomials with $\mathcal{K}\leq 2$, the predicted optimal $(d^{\star},w^{\star},r^{\star})$ actually leads to a sub-optimal pruning policy, which indicates that the MAP is too simple to use. For polynomials with $\mathcal{K}\geq 3$, all MAPs generate similar predictions about the optimal $(d,w,r)$, i.e., $(0.78,0.82,0.98)$ for ResNet-32 trained on CIFAR-10 and $(0.65,1.0,0.63)$ for ResNet-56 trained on TinyImageNet. These results corroborate that our algorithm is relatively robust with respect to different degrees $\mathcal{K}$, so practitioners do not need to choose the polynomial degree carefully.

In order to further understand why pruning the three dimensions simultaneously yields better results than pruning only one, Figure 5 compares the feature maps for models with different pruning policies. Specifically, all models are pruned from the same baseline model — ResNet-56 pre-trained on CIFAR-10. Input images are randomly chosen from the Internet. For visualization, we extract the last convolutional layer’s feature maps and compute their mean absolute values across channels. Figure 5 shows that the feature maps our pruned model outputs look most similar to those of the original model. This finding reveals that our algorithm preserves most information of the original model by pruning the three dimensions comprehensively.