UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration

The SENet presented by [47] with the SE blocks, which can be directly integrated into various convolutional neural networks, ranked first in the ILSVRC 2017 classification competition. The original SE module is shown in Figure 1. For any given input X, it is transformed into a three-dimensional feature map ${\rm{U}}\in{R^{H\times W\times C}}$ through the convolution operation. First, the feature map is compressed, and the H $\times$ W dimensional data of each channel is aggregated into a tensor of 1 $\times$ 1 $\times$ C through global average pooling and then transforms it into a tensor of 1 $\times$ 1 $\times$ C/r through a fully connected layer followed by ReLU. After that, a fully connected layer was used to produce a tensor of 1 $\times$ 1 $\times$ C, and then the C results are restricted between 0 and 1 based on the sigmoid. Finally, the original feature map is multiplied by this tensor as the input of the next layer.

We use the tensor of 1 $\times$ 1 $\times$ C in the SE module after Sigmoid as the evaluation index of the importance for each channel in the feature map. Considering the feature map have both positive and negative values, if the original global average pooling is used, the positive and negative terms will counteract each other. In order to retain as much information as possible in the feature map, we take the absolute value of each item of the feature map in the original SE module and then followed by the global average pooling. The fully connected layers in the SE module is replaced with convolutional layers, which is more conducive to capture the spatial information in the feature map, and our MSEB module is obtained. In our method, given the feature map ${\rm{U=}}\left[{{{\rm{u}}_{1}},{{\rm{u}}_{2}},\ldots,{{\rm{u}}_{C}}}\right]$, where ${{\rm{u}}_{c}}\in{R^{H\times W}}$, the c-th element in the compressed tensor ${\rm{z}}\in{R^{C}}$ of 1 $\times$ 1 $\times$ C is calculated as follows:

The operation of the excitation layer after compressed is as follows:

where, $\sigma$ refers to sigmoid activation, $\delta$ refers to ReLU and weights ${{\mathop{\rm W}\nolimits}_{1}}\in{R^{\frac{C}{{\rm{r}}}\times C}}$, ${{\mathop{\rm W}\nolimits}_{2}}\in{R^{C\times\frac{C}{{\rm{r}}}}}$. Finally, we use the above activation s and input feature map U to obtain the final output:

where, $\widetilde{\mathop{\rm X}\nolimits}=\left[{{{\widetilde{\mathop{\rm x}\nolimits}}_{1}},{{\widetilde{\mathop{\rm x}\nolimits}}_{2}},\ldots,{{\widetilde{\mathop{\rm x}\nolimits}}_{C}}}\right]$ and ${{\bf{F}}_{scale}}\left({{\mathop{\rm u}\nolimits}{}_{c},{s_{c}}}\right)$ refers to Channel-wise product of ${s_{c}}$ and input feature map ${{\rm{u}}_{c}}\in{R^{H\times W}}$.

We design a network pruning method for traditional convolutional neural networks and ResNets with special residual structures. The output tensor of 1 $\times$ 1 $\times$ C of the MSEB module is used as the importance evaluation index of the feature maps. The unimportant feature maps and all the filters related to them are pruned together to adequately reduce the parameters and FLOPs. The MSEB module obtain C values between 0 and 1 for the C-dimensional feature maps of each convolutional layer. According to the characteristics of the output results of the MSEB module in the experiment, we propose a pruning threshold setting method as follows:

where, $\lambda=1\times 10^{-\beta}$, $i$ is the index of the feature map, and $Thre$ refers to the pruning threshold. The feature maps corresponding to the output values that are smaller than the threshold will be deleted. As shown in Figure 2, The left is the original CNN, and the middle is a pruning schematic diagram of the 8-channel convolutional layer. According to our method, the original 8-channel output feature maps are trimmed according to the threshold. The channels pruned and relevant filters are displayed in the upper right corner. In the experiment, we determine the number of clipped channels by controlling $\beta$ and the sign of $\lambda$. In the rest of this paper, unless specified, the sign of $\lambda$ is negative by default.

After the feature maps are pruned, the filters connected to them are also deleted so that the network is further compressed. Unlike some existing layer-wise network pruning methods, we firstly use the pretrained CNN to generate the MSEB output for each layer and then set a huperparameter $\beta$. Finally, we clip all channels in each layer that are smaller than the threshold to obtain the final compact network. Our method prunes all layers at the same time. According to the specific characteristics of the feature maps, the number of clipping channels of each layer is different, even under the same value of $\beta$. In this way, each layer can be controlled by $\beta$ to prune as many of the less important channels as possible. In the experiment, some output of the MSEB layer are 0.5 due to the sigmoid. When the pruning amplitude is large, we directly cut these layers in half.

The main difference between ResNet and the traditional network is its special residual module, which has two types. One is composed of basic residual blocks that contain two 3 $\times$ 3 convolutional layers. The other is composed of bottleneck residual blocks, which include two 1 $\times$ 1 convolutional layers and a 3 $\times$ 3 convolutional layer between them.

For the first type of ResNet, due to its limitation of special residual structure, only the intermediate output channels of the residual blocks can be pruned. All the existing cutting methods use different pruning strategies to perform the intermediate channels to protect the structure. We attempt to prune the input channels and output channels of the residual block separately so that the flexibility of pruning is greater. We add a 1 $\times$ 1 convolutional layer between every two residual blocks, as shown in Figure 3(d), to ensure that the input and output channels of the residual blocks can be pruned separately without affecting the normal dimensional restriction of the ResNet and without introducing too many extra parameters and FLOPs for the network. We reconstruct ResNet-56 according to the above method and perform experiments on CIFAR-10. Under the same experimental environment and parameter configuration, the classification accuracy of the network constructed in this way decreases by approximately 1.7% compared to the original ResNet. After adding the MSEB module to the reconstructed ResNet-56, the accuracy decreased even more seriously, by approximately 2.5%. It causes the performance deteriorating even more severely after pruning. Therefore, we propose another pruning method for ResNet. Since each stage of the ResNet contains a certain number of residual blocks, the input and output dimensions of these residual blocks are consistent. We take the same trimming range for all residual blocks in this stage, and only prune the input and output channels and the corresponding filters, reserving the intermediate feature maps.

In the implementation process, we add the MSEB module after the last convolutional layer of each residual block, using the same method to train ResNet. Then, we collect and analyze the output values of the sigmoid in the MSEB module. We determine the consistent trimming range at each stage through its overall characteristic and finally experimentally verify that the pruning strategy is effective and better than some existing network pruning methods.

For the second type of ResNet that consisting of bottleneck residual blocks, since the 1 $\times$ 1 convolutional layers have considerably fewer parameters and FLOPs than the 3 $\times$ 3 layers, we perform channel-wise pruning on the middle 3 $\times$ 3 convolutional layers and delete the filters related to them. In the implementation, we embed the MSEB module behind the 3 $\times$ 3 convolutional layer in the middle of the bottleneck residual block. After training the ResNet to convergence, we set a hyperparameter to prune the network according to the traditional CNN pruning method introduced in the previous section.

We apply our method to classical network compression. Given a convolutional neural network, we embed the MSEB module into a specified position. For VGGNet, an MSEB module is added directly after each convolutional layer. In order to maximize the role of the MSEB module, we place it before the ReLU activation layer. For the ResNet, different embedding methods of the MSEB module are performed according to the type of the residual block. For the basic block, each block contains two 3 $\times$ 3 convolutional layers. We want to prune the middle channel, so the MSEB module is placed after the first convolution layer and before the Batch Normalization and the ReLU. But for the bottleneck, each block contains three convolutional layers. In order to better measure the importance of each channel of the entire residual block, we add the MSEB module after the third convolutional layer. The detailed network structure is shown in Figure 4.

Given a dataset, we first train the convolutional neural network with the MSEB module. After converged, the model with the highest classification accuracy is selected as our target model, and we extract the output value of the MSEB module. We determine the hyperparameter according to the output value characteristics of the middle layer in the MSEB module, and then calculate the pruning threshold. In the training process, we use the following cross entropy as the loss function:

where ${y^{\left(i\right)}}$ and ${\widehat{y}^{\left(i\right)}}$ refer to the real probability and estimated probability of the i-th category respectively. N is the total number of image categories. In order to control the magnitude of the weights in the back-propagation and avoid gradient explosion, we introduce $penalty$ in the loss function, which is also conducive to increasing the sparsity of the weights and improving the generalization ability. For the penalty term, we choose L2 norm. The final loss function is as follows:

In the back-propagation, we choose the stochastic gradient descent with momentum. Selecting a small batch of m samples $\left\{{{x^{\left(1\right)}},{x^{\left(2\right)}},\ldots,{x^{\left(m\right)}}}\right\}$ from the training set, the corresponding target real probability is ${y^{\left(i\right)}}$. We calculate the gradient estimate of the parameter after the forward propagation:

where, ${f\left({{x^{\left(i\right)}};W}\right)={{\widehat{y}}^{\left(i\right)}}}$. Then the gradient update method is as follows:

where, $v$ is the initial gradient accumulation, ${v_{0}}=0$, $\alpha$ is the coefficient of momentum, and $\eta$ is the learning rate. We use this method to train the CNNs on the image classification dataset. Our pruning algorithm is show as Algorithm 1.

After the network is pruned, most of the existing methods inherit the weights and biases from the original pretrained network and restore the performance of the compact network as much as possible by fine-tuning [29, 33, 37, 38, 34]. However, when the network pruning amplitude is large, considerable parameter information is lost, and the performance recovery is not obvious after fine-tuning, so the actual performance of the compact network cannot be fully displayed. [48] exposed a surprising character in structured network pruning: the performance of the compact network by fine-tuning with inherited parameters is not better than training from scratch.

The experimental results obtained by using VGG-16 on the CIFAR-10 further verify the correctness of the conclusions in [48]. To adequately reflect the performance of the compact network, we retrain the pruned networks from scratch and keep the FLOPs before and after pruning consistent in the experiments. Specifically, we multiply the training epochs of the original network by the FLOP compression rate as the retraining epochs of the pruned network.

We experiment with VGG-16 and ResNet-56 on CIFAR-10. In VGG-16 pruning, we set $\beta$ to 4 and 8, respectively, and the sign of $\lambda$ is $``$-$"$. For ResNet-56 with basic blocks, we perform consistent pruning on each stage according to the characteristics of the output of the MSEB module in each layer. As shown in Figure 5, the MSEB outputs of ResNet-56 in the first stage are relatively stable. The MSEB output of the first block in the second stage fluctuates greatly, while that of the next 8 blocks changes slightly, with 4 blocks constantly equal to 0.5. Finally, the outputs of 9 blocks in the last stage vary greatly. For stages with large fluctuations, we use a larger pruning range, and for stages with fewer fluctuations, we carefully cut or not. In the experiment, we prune the structure with three stages into 8-32-32 and 8-16-16. The specific experimental results are shown in Table 1.

The experiment shows that for VGG-16, our pruning method can prune the network parameters and FLOPs to a similar extent. When $\beta=8$, the compression ratio of the parameters and FLOPs in the compact VGG-16 exceed 68%, and after retraining from scratch, the accuracy improves by 0.20%. For MSEB-VGG-16, when $\beta=4$, the accuracy of the compact network after retraining can improve by 0.16%, but it decreases with $\beta$ set to 8. This situation also occurs in our later experiments. We assume this may be due to overfitting by adding the MSEB module when the reduction in parameters and FLOPs in the original network influences the generalization ability to a certain extent. For ResNet-56, when the parameters and FLOPs are pruned nearly 40% and 33.3%, respectively, the performance after retraining improves by 0.13%.

On CIFAR-100, we experiment with two networks, VGG-19 and PreResNet-164. For VGG-19, we set $\beta$ to 3 and 8. The number of channels in VGG-19 before and after compression is shown in Table 2, where the last two columns indicate the number of channels after pruning. The “3” and “8” in parentheses correspond to $\beta$. The performance of PreResNet proposed by He Kaiming et al. in [50] is exposed better than ResNet. We use our method to prune the PreResNet-164, which uses the bottleneck with $\beta$ setting to 2 and 8.

For the output characteristics of the MSEB module in each residual block of PreResNet-164, we have done data analysis, and the results are shown in Figure 6. In the first stage, the MSEB output of some residual blocks varies greatly. In the second stage, the MSEB output of all residual blocks covers a very large range between 0-1, and the distribution is relatively on average and has no concentration. The third stage is more stable. Only a few residual blocks have slight fluctuation in MSEB output. This reveals that when performing the image classification task of the CIFAR-100, the parameters in residual blocks of the second stage of PreResNet-164 have greater redundancy than the other two stages.

The results of VGG-19 and PreResNet-164 are shown in Table 3. We prune the original VGG-19, and after reducing the parameters by 68.3% and FLOPs by 37.7%, the accuracy of compact VGG-19 after retraining improves by 0.73. When $\beta=8$, the parameter cropping rate reaches 81.3%, and FLOPs are reduced by 62.8%; however, the accuracy of the compressed network is improved by approximately 0.54 compared with the original network. For PreResNet-164, our method can cut the parameters and FLOPs by approximately 26.6% and 34.2%, respectively, when $\beta$ is set to 2, and the final accuracy increases by 0.12.

Our UCP is to uniformly prune the channels of each layer in the CNN, including its related filters. According to different thresholds, each layer will automatically select unimportant channels to delete based on the specific situation of the feature map generated during the training process, so that the cropping rate of each layer is different. For some layers that are more important for image classification task, the cropping amplitude is relatively small, and for those layers with more redundancy, the cropping ratio will become very large. For example, the cropping ratio of the channels in the penultimate layer of VGG-19 in CIFAR-100 even exceeds 98%. Therefore, our method can prune channels as much as possible in the image classification task, while keeping the experimental performance of the network without significant loss.

To evaluate the effectiveness of our proposed UCP method on large datasets, we experimented with ResNet-18 and ResNet-34 on the ILSVRC-2012 data set. For ResNet-18, we set the values of $\beta$ to 1 and +1, and for ResNet-34 we set the values of $\beta$ to 2 and +1. As shown in the table 4, the results are compared with existing methods whose results are from the original paper.

As can be seen from the table, our method is equally effective on large datasets. For ResNet-18, the baseline Top-1 classification accuracy rate is 69.69, and the Top-5 is 89.42. When the value of $\beta$ is 1, the amount of network parameters is reduced by 29.34%, and the number of FLOPs is reduced by 21.78%. At this time, the top-1 accuracy of the compact network on the ILSVRC-2012 drops by 0.62, and the Top-5 accuracy drops by 0.61. When the value of $\beta$ is +1, the pruning rate of parameters and FLOPs exceeds 50%, and the Top-1 accuracy of the compressed network drops by 2.65, while the Top-5 accuracy drops by 1.62. For ResNet-34, the baseline Top-1 and Top-5 accuracy are 73.30 and 91.45 respectively. When the value of $\beta$ is 2, the amount of parameters is reduced by 33.94%, and the FLOPs are reduced by 30.27%. At this time, the Top-1 and Top-5 accuracy of the compact network drops by 0.22 and 0.13, respectively. When the value of $\beta$ is +1, the pruning rates of the parameters and FLOPs are 62.34% and 57.25%, respectively. The Top-1 and Top-5 accuracy of the compact network drops 2.78 and 1.65 respectively. Furthermore, the compact ResNet-34 has smaller parameters and FLOPs than ResNet-18, but its performance is better.

Then, we conduct ablation analysis on the proposed UCP method. This section consists of two parts: hyperparameter study and comparison of compression ratio and accuracy drop.

In this part, we compare our UCP method with the convolutional neural network pruning algorithms proposed in [28, 33, 37, 38]. Our method is the same as these four methods, which are structured pruning methods and based on the importance of convolution kernels or channels in the convolutional layer. Among them, Weight-Level-Pruning(WLP)[28] prunes unimportant connections with small weights according to a predefined pruning rate. Soft Flter Pruning(SFP)[33] uses L2-norm to evaluate the importance of the filters and then prune the unimportant filters and the corresponding channels. Network Slimming(NS)[37] performs sparse induction regularization on the scaling factor of the batch normalization (BN) layer, and then prunes the channels with smaller scaling factors according to the pruning ratio. L1-norm[38] uses the channel selection method based on LASSO regression to delete unimportant channels.

We compare the above methods in CIFAR-10 with ResNet-56 under the same hardware environment and parameter settings. The classification accuracy drop of each method at different compression rates is compared. The experimental results are shown in Figure 7. It can be seen that as the network compression rate increases, the effect of our proposed method on experimental accuracy is always smaller than that of other methods. This shows that our structured uniform pruning method is more accurate and effective in evaluating the importance of channels.