ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN

The magnitude-based pruning focuses on the importance of parameters and connections. According to the proposed importance evaluation index, the unimportant parameters and connections in the network are directly deleted, and then the obtained compact network is fine-tuned or retrained to restore the experimental accuracy. The core of this type of algorithms is to design a reasonable and effective parameter importance evaluation index. [26] uses the scaling factor of the batch normalization layers to evaluate the importance of the channels and deletes the unimportant channels to realize the channel-level network pruning. [13] uses the magnitude of the weights to measure and deletes all insignificant connections in the pre-trained network whose weight is lower than the threshold. [15] calculates and sorts the L1 norm of the filters and prunes them with smaller values and their corresponding feature maps according to the preset pruning rate. [27] utilizes the L2 norm to measure the importance of the filters, and will continuously update the filters that were pruned last time during the training phase. [28] adopts the fully connected layer as a linear classifier to extract the feature representation of the middle layers and prune them that with less improvement according to a predefined threshold. [29] judges the redundancy of one filter based on the Euclidean Distance of it and the geometric mean of all filters in the layer and deletes the redundant filters in the network according to the preset pruning rate. [30] applies the rank of the feature map to determine how much information it contains. The low-rank feature maps contain less information and can be deleted with confidence. The high-rank feature maps hold much more message, even if fixing some parameters will not damage the final performance. [16] proposes a new filter pruning method, which analyses the diversity and similarity of feature maps to prune redundant filters in the network. [31] prunes the 3D ConvNets according to the magnitude and information score of the filters.

Clustering is the task of dividing the data points into several groups such that data points in the same groups are more similar to each other than those in other groups. In simple words, the aim is to segregate groups with similar traits and assign them into clusters. Broadly speaking, clustering can be divided into three subgroups: prototype-based clustering, such as k-means algorithm [32], etc.; hierarchical clustering, such as BIRCH algorithm [33], etc. and density-based clustering, such as DBSCAN algorithm [34],[35] and OPTICS Algorithm [36], etc. According to specific magnitude characteristics such as similarity of the parameters, the original network can be clustered and pruned.

Sensitivity-based methods require analyzing the influence of pruning on the original network. Concretely, by considering the impact of pruning strategies on network performance in the compressing process, the CNN is pruned layer by layer, and the scheme is adjusted continually during the accelerating phase. The core of this type of algorithm is to design a rational and effective network pruning adjustment strategy. [17] considers the importance score propagation of the neurons and uses feature ranking techniques to measure the importance of each neuron in the ”final response layer” to prune the filters in earlier layers. [37] introduces connection sensitivity to evaluate the importance of structural connections and prunes the network during the parameter initialization stage before training. [38] comprehensively considers the neuron connections before and after the pruned layer and compresses the network by minimizing the formulated Frobenius distortion. [39] introduces a variational technique to estimate channel saliency in the training process and looks for a suitable probability distribution to further deletes redundant channels. [40] proposes a feature boosting and suppression method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. This method retains the complete network structure. [41] develops a differentiable pruning criteria sampler that is learnable and optimized by the validation loss of the pruned network obtained from the sampled criteria. So they can adaptively select the appropriate pruning criteria for different functional layers. [42] directly optimizes channel-wise differentiable discrete gate under resource constraint to controls the deletion of channels in the convolutional layers. [43] proposes a differentiable Markov channel pruning method to efficiently compress the network that is differentiable and can be directly optimized by gradient descent with respect to standard task loss and budget regularization. [44] proposes an effective channel selection layer, which automatically finds less important filters in a joint training manner. The method takes previous activation responses as an input and generates a binary index code for pruning. [45] proposes a network channel pruning scheme based on sparse learning and genetic algorithm.

Swarm intelligence optimization algorithms are a form of nature-based optimization algorithms that mainly simulates the cooperative behavior of insects, birds, and animals, etc. The individuals of the community continuously change the direction to search by sharing knowledge between them until they achieve their goals. Famous swarm intelligence optimization algorithms include ant colony algorithm [46], particle swarm optimization algorithm [47],[48], and artificial bee colony algorithm [49], etc.

Fig.1 shows the overall framework of ACP. Given an image classification dataset, we randomly sample numerous images and infer them via the pre-trained CNN. The feature maps are clustered based on the density and the number of clusters in each layer is regarded as the number of channels after pruning. Then the obtained network substructure is initialized as a structure candidate population. According to the accuracy loss of the subnetwork, we iteratively search and optimize each candidate using PSO. Finally, after a few cycles, we can obtain the optimal compact structure via automatic channel pruning. For different tasks and datasets, the compressed network structure is generally different.

Given a CNN with L convolutional layers and the dataset $D$, the convolutional operation is formulated as $Conv={{\cal F}}\left({D,W;C}\right)$. $C=\left({{c_{1}},{c_{2}},\ldots,{c_{L}}}\right)$ is the original network structure, where ${c_{l}}$ is the number of channels in the $l$-th layer. The size of the filter $W\in{{\mathbb{R}}^{{{c}_{out}}\times{{c}_{in}}\times k\times k}}$ is $k\times k$, where ${c_{out}}$ and ${c_{in}}$ denote the number of output and input channels, respectively. The feature map is a 4D tensor $M\in{{\mathbb{R}}^{s\times c\times W\times H}}$ with the size of $W\times H$, where $s$ is the number of samples from the dataset.

Here, we perform layer-wise clustering pruning based on the similarity of feature maps to achieve the preliminary network compressing. The number of groups formed by clustering is regarded as the number of channels after pruning. Given an input image, the feature map generated in the $l$-th layer is ${{M}_{l}}\in{{\mathbb{R}}^{{{c}_{l}}\times{{W}_{l}}\times{{H}_{l}}}}$, and then the ${{c}_{l}}$ channels of which the size is ${{W}_{l}}\times{{H}_{l}}$ are clustered. We draw on the idea of the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. The density-based clustering can separate regions with adequate density into clusters with arbitrary shapes in the dataset containing noise. Moreover, there is no need to specifically assign the number of classes to be formed before clustering.

Different filters perform convolutional operations on the input to generate various feature maps, and some of them in the same layer may have similarities. For a given dataset containing $m$ training images, we randomly pick $s$ $\left(s\leq m\right)$ samples and process them through the CNN. To fully consider the impact of different types of figures on the output feature maps and obtain more accurate similarity, we superimpose and average the feature maps at the same position produced by $s$ samples, and then calculate the cosine distance to estimate the similarity between any two feature maps ${{M}_{li}}$ and ${{M}_{lj}}$ in the same layer. The specific definition is as follows:

where $M_{li}^{r}$ is the $i$-th feature map produced by the $r$-th image in the $l$-th layer. The neighborhood parameter of the cluster is ($\varepsilon$,$MinPts$), where $\varepsilon$ is the neighborhood radius, and $MinPts$ refers to the minimum number of feature maps in the $\varepsilon$-neighborhood of the core feature map. If the $\varepsilon$-neighborhood of the feature map ${{M}_{li}}$ contains at least $MinPts$ feature maps, that is $\left|{{\mathbb{N}}_{\varepsilon}}\left({{M}_{li}}\right)\right|\geq MinPts$, then ${{M}_{li}}$ is assigned as a core feature map, and all of the core feature maps form a set. We randomly select a core feature map and cluster all feature maps including non-core ones in its $\varepsilon$-neighborhood into a group. If there is a core feature map, then all the feature maps in its $\varepsilon$-neighborhood also belong to this group. Feature maps that are neither core feature maps nor belong to $\varepsilon$-neighborhood of any core feature maps are regarded as noises. According to this scheme, we cluster the channels in each layer and consider the sum of the number of groups and noises obtained by clustering as the number of channels after clustering pruning. In this way, the primary compressed substructure ${C}^{\prime}=\left({{c}_{1}^{\prime}},{{c}_{2}^{\prime}},\ldots,{{c}_{L}^{\prime}}\right)$ is gained. Fig.2 depicts the process of clustering and pruning in one layer. The upper right corner shows the substructure after pruning, and the feature maps in the same color belong to the same cluster. The number of feature maps in the solid line indicates the number of remaining channels.

In the previous section, we have conducted primary clustering pruning for the original network structure $C$. In this section, the compressed substructure ${C}^{\prime}$ is further optimized through structure searching. We use the PSO algorithm to optimize ${C}^{\prime}$ and find the optimal compact network ${{C}^{*}}$ which satisfies:

where ${{D}_{Train}}$ denotes training set and ${{D}_{Test}}$ is test set. $Acc\left(\centerdot\right)$ is the test accuracy of the structure ${C}^{\prime}$. The structure searching optimization can be divided into the following two steps.

Since different CNNs have different network structures, different pruning schemes need to be adopted for different networks to maintain structural stability. We perform experiments on VGGNet, ResNet, and GoogLeNet respectively. Among these CNNs, VGGNet is a common layer-by-layer CNN. During the compressing process, the output channels of each layer can be deleted independently. The number of input channels of each layer remains the same as the output of the previous layer. ResNet and GoogLeNet leverage the Residual and Inception module to make the network deeper and wider respectively. If the channels are discarded arbitrarily, the dimensional matching of the network will be destroyed. For ResNet with the basic block, we only prune the output of the first layer, and the input of the second layer is changed accordingly. The bottleneck consists of three layers, among which the size of the filters in the first and third layers is 1$\times$1, while that in the middle layer is 3$\times$3. We prune the channels in the middle layer of the bottleneck. And the output channels in the first layer and the input channels of the third layer will be modified accordingly. GoogLeNet is a more complex network, composed of multiple Inception V3 modules that contain four branches. To prune the GoogLeNet, we compress the branches containing two and three convolutional layers. The specific structure and pruning scheme of the Inception V3 module is plotted in Fig.3.

In order to demonstrate the effectiveness of our proposed method, we prune VGGNet, ResNet, and GoogLeNet on CIFAR-10, CIFAR-100, and ILSVRC-2012 for image classification. Moreover, we prune SSD on PASCAL VOC for object detection. We implement all experiments using PyTorch. The network pre-training and hyper-parameter settings use the method presented in [21] for sake of obtaining better benchmark accuracy and be able to compare fairly with some existing network pruning methods. The learning rate is initially set to 0.1 and then decreased by a factor of 10 on half and three-quarter epochs. We train 160 epochs on CIFAR-10 and CIFAR-100 and 100 epochs on ILSVRC-2012. During training, all the networks are trained using SGD, and batch size is set to be 64 on CIFAR-10/100 and 16 on ILSVRC-2012. We use a weight decay of $1e$-$4$ and a momentum of 0.9. In the retraining stage, we adjust the learning rate as described above on the CIFAR while adopting the cosine annealing adjustment strategy on the ILSVRC-2012. We only change the neighborhood radius in the clustering pruning to control the compression ratio of the networks. Other hyper-parameter settings are as follows: $MinPts$=5, $T$=5, $N$=6. We perform specific ablation analysis on $\varepsilon$ and MinPts later. We train each compact network for three epochs before calculating its fitness.

We first prune VGG-16 and ResNet-56/110 on CIFAR-10. The results are shown in TABLE I. It can be seen that for the three networks when pruning more than half of the parameters and FLOPs, the performance of all the compact networks even has a slight improvement compared with the vanilla network. For VGG-16, we remove 65.72% parameters and 51.48% FLOPs while still keeping the accuracy at 94.03%, which is even 0.43% higher than the baseline. When the parameters and FLOPs of ResNet-56 are discarded by 58.82% and 54.42%, respectively, the precision is improved by 0.21%. While the pruning rates of parameters and FLOPs for ResNet-110 are 62.43% and 63.95%, the performance of the compact network surprisingly increased by 1.01%. Generally speaking, with the increase of the compression ratio, the accuracy of the pruned model drops gradually. Our method reduces up to 91.66% parameters and 73.44% FLOPs for VGG-16 on CIFAR 10 while maintaining the accuracy as high as 93.45%, which is only declined by 0.15% compared with the original network. We remove 64.71% parameters and 57.84% FLOPs for ResNet-56 with only 0.27% performance drop of accuracy. For ResNet-110, although compressing 80.92% parameters and 78.32% FLOPs, the performance of the network is still improved by 0.33% compared to the original network.

The experimental results show that the deep CNN does have certain redundancy, and the moderate compression will not affect its accuracy. When the pruning ratio is small, removing some redundant parameters in the network can improve its generalization, which leads to an increase in performance. As ResNet-56 is compressed to 0.59M, the classification performance on CIFAR-10 even exceeds the original VGG-16 of 14.73M. It also supports the excellent performance of the residual structure in image feature extraction and classification tasks from the side. When ResNet-110 is pruned to 0.33M, the parameters and FLOPs are both less than the pruned ResNet-56 of 0.35M, but the accuracy is 0.26% ahead. The result sheds light on that increasing the depth of the ResNet can improve its performance when the number of parameters of networks are similar.

To further validate the effectiveness of the proposed ACP, we also perform experiments on GoogLeNet. Due to the integration of the Inception module, GoogLeNet increases the width of the network, and finally, the baseline accuracy reach 94.72%, which is ahead of VGGNet and ResNet. We remove 66.29% parameters and 70.44% FLOPs while still holding the accuracy at 95.09%, which is even 0.37% higher than the baseline model. It further manifests that the proposed method can achieve effective compression and acceleration for CNN. And the ACP also plays the role of regularization while reducing the redundancy of the network.

We continue to experiment with the CNN on CIFAR-100 and the results are reported in TABLE II. CIFAR-100 and CIFAR-10 have the same number of images, but the category has increased from 10 to 100. As the training images of each class decrease, the performance of the network also drops significantly. The initial classification accuracy of VGG-19 on CIFAR-100 is only 72.01%. When pruning 74.02% parameter and 63.24% FLOPs, the accuracy only drops by 0.06%. We remove 57.23% parameters and 53.39% FLOPs of ResNet-110 while even 0.34% higher than the baseline model. Under the circumstance, the parameter and FLOPs of 0.74M ResNet-110 are both smaller than the 0.86M original ResNet-56, but the performance is 1.13% higher than the ResNet-56. The result further verifies the conclusion in the previous section.

To sufficiently demonstrate the effectiveness of our ACP, we prune ResNet-18/34/50 on the large-scale dataset ILSVRC-2012 that contains 1000 categories. Among them, ResNet-18/34 use basic block, while ResNet-50 uses bottleneck. We arrange two pruning ratios for each network, and the results are presented in TABLE III. We can see that the accuracy of all the three networks after compressing dose have a loss compared with the original network, moreover as the pruning ratio increases the accuracy loss also raises. It reveals that the three ResNets have less parameter redundancy on ILSVRC-2012. In spite of this, our ACP can still obtain flexible compression and acceleration with slight accuracy drop. When delete 56.72% parameters and 34.17% FLOPs of ResNet-18, the Top-1 and Top-5 accuracy drop by 2.20% and 0.82%, respectively. For ResNet-34, the initial Top-1 and Top-5 accuracy are 73.40% and 91.44%. When the pruning rate of the parameters and FLOPs exceeds 50%, the Top-1 and Top-5 accuracy only drop 1.32% and 0.87%. Because of the bottleneck, the Top-1 and Top-5 accuracy of ResNet-50 increase by 2.54% and 1.49% respectively, while the parameters only increase by 3.76M compared to ResNet-34. We remove 54.69% parameters and 46.82% FLOPs of ResNet-50, and the performance of the compact network is hardly reduced (0.41% Top-1 loss and 0.21% Top-5 loss). When we discard 64.36% parameters and 63.34% FLOPs, the compact ResNet-50 are smaller than the baseline ResNet-18, but the accuracy is significantly improved (Top-1: 74.64% versus 70.02%, Top-5: 92.19% versus 89.23%).

In this subsection, we compare the method on three datasets including CIFAR-10, CIFAR-100, and ILSVRC-2012. Among the compared methods, Li et al. [15], SFP [51], DCP [52], FPGM [29], EDP [53], CNN-FCF [54], CCP [55], Taylor-FO-BN [56] and HRank [30] are the state-of-the-art channel pruning methods. The results of these competing methods are reported according to the original article.

Then, we conduct ablation analysis on the proposed ACP method. This section is composed by six parts: the influence of $MinPts$, the influence of $\varepsilon$, the influence of different similarity criteria, the influence of PSO, the effectiveness of ACP and generalization ability on detection tasks.