RepBNN: towards a precise Binary Neural Network with Enhanced Feature Map via Repeating

As we all know, the applications of convolutional neural networks (CNNs) have achieved tremendous success in computer vision fields such as image classification, object detection, object tracking, and depth estimation. But this success comes at the cost of a huge amount of computation, which binds the application of convolutional neural networks generally to high-end hardware, such as GPU, TPU, etc. However, the efficient application of CNN models on mobile devices and embedded devices, where the storage space and computing resource are limited, is still quite challenging. In order to solve this problem, various lightweight network technologies have emerged, mainly including network architecture design[21][25], network architecture search[8][12], knowledge distillation[7], pruning[16][5], and quantization[27][23].

Among these lightweight network technologies, binarization contributes to an extreme version of a convolutional neural network , where all feature maps and weights are represented by just 1-bit. Binarization greatly improves the efficiency of CNN models, because it not only reduces the volume of parameters, the requirement of storage capacity, but also its replacement of MAC operations with efficient XNOR and bit-count operations saves a lot computational expense. The work in [19] shows that a binary neural network has achieve 32× parameter compression and 58× speedup than its full-precision network.

However, the speedup of calculation and the compression of parameter by binarization have obvious impact on the accuracy of CNN models. It is the declined representation capability of feature maps that degrades the performance of binarized CNNs. Unlike full-precision networks, the value range of binarized convolutions is so extremely restricted [15] that the accuracy loss becomes more serious. If the number of input channels is $C_{in}$, and the size of the convolution kernel is $k_{h}$ and $k_{w}$, then the value range of the binary convolution is $\left(-C_{in}*k_{h}*k_{w},C_{in}*k_{h}*k_{w}\right)$, a total of $C_{in}*k_{h}*k_{w}+1$ quantization levels. Obviously, increasing the number of input channels $C_{in}$ can improve the feature representation capability of binary neurual networks (BNNs).

Inspired by this observation, we propose a new replaceable convolution module RepConv, which reshapes the weight kernel through expanding its channel dimension while compressing its number of groups. The total number of parameters and the amount of related convolutional calculation remain unchanged after the reshape. To convolve with such a reshaped weight kernel, RepConv has to increase the number of both input and output channels simply through a replication operation. In such a manner, RepConv enriches the network information.

The binary version of RepConv is called RepBconv. If replacing the normal binarized convolution with RepBconv and further modify the remaining modules such as the first convolution layer, batch normalization, bypasses and fully connected layer following the transformation rule of RepTran, we translate a set of highly cited binary neural networks [22][14][13] into the corresponding RepBNNs, which have been demonstrated to achieve universally better performance than the original BNN versions.

Experimental results show that after the RepTran transformation, the Top-1 accuracy of Rep-ReCU-ResNet-20, i.e., a RepBconv enhanced ReCU-ResNet-20[22], reaches 88.97% on CIFAR-10, which is 1.47% higher than that of the original network. And Rep-AdamBNN-ReActNet-A [14][13] achieves 71.342% Top-1 accuracy on ImageNet, a fresh state-of-the-art result of BNNs as shown in Fig.1.

To sum up, this paper makes the following contributions:

• •

  We propose a new and easy-to-use convolution module named RepConv that reshapes the weight kernels thus increases the number of input and output channels so as to enrich the representation capability of binarized feature map without extra cost on convolutional operations and parameters.

• •

  We propose a set of universal network transformation rules of RepTran, which makes the common modules in binary neural network adaptable to the application of RepConv.

• •

  We apply RepTran to a large number of state-of-the-art binary neural networks and validate them on CIFAR-10 and ImageNet extensively. It is proved that our proposed structure can significantly improve the performance of binary networks.

Binarization greatly reduces the amount of operations (OPs) and parameters, but encounters a severe drop in accuracy, which is mainly caused by the loss of information accompanied in the network with just 1-bit weights and activations. Previous work mitigated this precision decline through either an efficient training procedure that accurately back-propagates gradient values or methods to improve the representation capability of binary neural networks.

As explained in Section 1, increasing the number of input channels can improve the feature representation capability of BNNs. Inspired by this observation, we propose a new convolution module named RepConv.

Fig.2 shows a normal convolution process. In Fig.2, the structure of a convolution kernel is expressed as $\left(C_{out},C_{in},k_{h},k_{w}\right)$ , where the number of input channels is $C_{in}$, and the number of output channels is $C_{out}$.

RepConv reshapes the convolution kernel to $\left(C_{out}/\beta,C_{in}*\beta,k_{h},k_{w}\right)$ with $C_{in}*\beta$ input channels and $C_{out}/\beta$ output channels respectively. Then RepConv makes $\beta^{2}$ copies of the output feature map and concates them along the channel direction, so that the final channelwise size of the output becomes $C_{out}*\beta$. This step is called a repeat operation. The specific process of RepConv can be seen in Fig.3.

Finally, at the same cost of convolutional computation, RepConv generates feature map of $C_{out}*\beta$ channels, while its counterpart, a normal convolution process, creates a feature map of just $C_{out}$ channels. Since the output of a previous layer is used as the input to the next layer, the expanded information is consistently passed on in networks using RepConv. A batch normalization right after the repeat operation of RepConv and a residual link that transfers the output of last layer to that of this layer make full use of the enriched feature map by RepConv.

Our experiments have demonstrated that the replacement of normal convolution module with RepConv improves the accuracy for both full-precision and binary networks. More discuss on why a simple repeating operation improves the network accuracy and how the modules of batch normalization and residual link impact the feature contents after RepConv is presented in Section 5.1.3.

$\beta$ is a hyperparameter of RepConv that represents the degree of information dilation in a feature map. However a larger number of $\beta$ raises up the computational overhead of non-convolutional operations like batch normalization. In Section 4.4, we give an example to demonstrate this computational overhead of RepConv is negligible to the overall calculation of the entire network. The discuss on how different $\beta$ affects the accuracy is given in Session 5.1.1.

The binary version of RepConv is called RepBconv in Fig.4b, which consists a regular Bconv module with reshaped weight kernels and a subsequent repeating $\beta^{2}$x operation. A complete BNN generally consists structures such as the first full-precision convolution layer, binary convolution layers, bypasses, batch normalizations, and the last fully connected layer. As shown in Fig.4c, the usage of RepBconv instead of the regular BConv increases the size of information flow by $\beta$ times in a binary convolution structure and so as to the entire BNN network.

The transformation rule for these BNN structures accompanying with the usage of RepBconv is called RepTran. When the number of input and output channels is increased by RepBconv, these structures must be changed accordingly to maintain the function while control the extra computation cost related to the enlarged information size. This section explain how RepTran works from the first layer, through the backbone and until the last layer of the network, as illustrated in Fig. 5.

For a fair comparison, we use the open source inferencing and training codes provided by IR-Net[18], RBNN[11], FracBNN[26], ReCU[22], Bi-Real Net[15], ReActNet[14] and AdamBNN[13]. We also apply exactly the same settings in data preprocessing as these recent outstanding BNN works.

Binary neural network (BNN) is an extreme quantization version of convolutional neural networks (CNNs) with all features and weights mapped to just 1-bit. Although BNN saves a lot of memory and computation demand to make CNN applicable on edge or mobile devices, BNN suffers the drop of network performance due to the reduced representation capability after binarization. In this paper, we propose a new replaceable and easy-to-use convolution module RepConv, which enhances feature maps through replicating input or output along channel dimension by $\beta$ times without extra cost on the number of parameters and convolutional computation. We also define a set of RepTran rules to use RepConv throughout BNN modules like binary convolution, fully connected layer and batch normalization. We apply RepTran to a large number of state-of-the-art binarization works, leading to a series of enhanced binary networks, named RepBNNs. These RepBNNs are validated on CIFAR-10[10] and ImageNet[20]. Among them, Rep-ReCU-ResNet-20 achieves 88.97% accuracy on CIFAR-10. Rep-AdamBNN-ReActNet-A achieves 71.34% accuracy on ImageNet.

It is hoped that our work can bring some inspiration to the fields of lightweight network architecture design and network architecture search for binary neural networks.