Basic Binary Convolution Unit for Binarized Image Restoration Network

As pioneer works, SRCNN (Dong et al., 2015b), DnCNN (Zhang et al., 2017), and ARCNN (Dong et al., 2015a) use the compact networks for image super-resolution, image denoising, and compression artifacts reduction, respectively. Since then, researchers had carried out its study with different perspectives and obtained more elaborate neural network architecture designs and learning strategies, such as residual block (Kim et al., 2016; Zhang et al., 2021a; Cavigelli et al., 2017), dense block (Zhang et al., 2018c; 2020; Wang et al., 2018), attention mechanism (Zhang et al., 2018b; Xia et al., 2022a; Dai et al., 2019), GAN (Gulrajani et al., 2017; Wang et al., 2018), and others (Wei et al., 2021; Peng et al., 2019; Jia et al., 2019; Fu et al., 2019; Kim et al., 2019; Fu et al., 2021; Soh et al., 2020; Xia et al., 2022d; c; b), to improve model representation ability. However, these IR networks require high computational and memory costs, which hinders practical application on edge devices. To address this issue, we explore and design a BBCU for binarized IR networks.

Binary neural network (BNN) is the most extreme form of model quantization as it quantizes convolution weights and activations to only 1 bit, achieving great speed-up compared with its full-precision counterpart. As a pioneer work, BNN (Courbariaux et al., 2016) directly applied binarization to a full-precision model with a pre-defined binarization function. Afterward, XNOR-Net (Rastegari et al., 2016) adopted a gain term to compensate for lost information to improve the performance of BNN (Courbariaux et al., 2016). After that, Bi-Real (Liu et al., 2018) introduced residual connection to preserve full-precision information in forward propagation. IRNet (Qin et al., 2020) developed a scheme to retain the information in forward and backward propagations. Recently, ReActNet (Liu et al., 2020) proposed generalized activation functions to learn more representative features. For super-resolution task, Ma et al. (2019) binarized convolution weights to save model size. However, it can hardly speed up inference enough for retaining full-precision activations. Then, Xin et al. (2020) further binarized activations and used the bit accumulation mechanism to approximate the full-precision convolution for SR networks. Besides, Zhang et al. (2021b) introduced a compact uniform prior for better optimization. Subsequently, Jiang et al. (2021) designed a binary training mechanism by adjusting the feature distribution. In this paper, we reconsider the function of each basic component in BC and develop a strong, simple, and efficient BBCU.

As shown in Fig. 2(a), we first construct the BBCU-V1. Specifically, the full-precision convolution $\mathbf{\mathcal{X}^{f}_{j}}\otimes\mathbf{\mathcal{W}^{f}_{j}}$ ($\mathbf{\mathcal{X}^{f}_{j}}$, $\mathbf{\mathcal{W}^{f}_{j}}$, and $\otimes$ are full-precision activations, weights, and Conv respectively) is approximated by the binary convolution $\mathbf{\mathcal{X}^{b}_{j}}\otimes\mathbf{\mathcal{W}^{b}_{j}}$. For binary convolution, both weights and activations are binarized to -1 and +1. Efficient bitwise XNOR and bit counting operations can replace computationally heavy floating-point matrix multiplication, which can be defined as:

$$\mathbf{\mathcal{X}^{b}_{j}}\otimes\mathbf{\mathcal{W}^{b}_{j}}=\operatorname{bitcount}\left(\mathrm{XNOR}\left(\mathbf{\mathcal{X}^{b}_{j}},\mathbf{\mathcal{W}^{b}_{j}}\right)\right),$$

(1)

$$x^{b}_{i,j}=\operatorname{Sign}\left(x^{f}_{i,j}\right)=\left\{\begin{array}[]{l}+1,\text{ if }x^{f}_{i,j}>\alpha_{i,j}\\ -1,\text{ if }x^{f}_{i,j}\leq\alpha_{i,j}\end{array}\right.,x^{f}_{i,j}\in\mathcal{X}^{f}_{j},x^{b}_{i,j}\in\mathcal{X}^{b}_{j},i\in[0,C),$$

(2)

$$w^{b}_{i,j}=\frac{\left\|\mathbf{\mathcal{W}^{f}_{j}}\right\|_{1}}{n}\operatorname{Sign}\left(w^{f}_{i,j}\right)=\left\{\begin{array}[]{l}+\frac{\left\|\mathbf{\mathcal{W}^{f}_{j}}\right\|_{1}}{n},\text{ if }w^{f}_{i,j}>0\\ -\frac{\left\|\mathbf{\mathcal{W}^{f}_{j}}\right\|_{1}}{n},\text{ if }w^{f}_{i,j}\leq 0\end{array}\right.,w^{f}_{i,j}\in\mathbf{\mathcal{W}^{f}_{j}},w^{b}_{i,j}\in\mathbf{\mathcal{W}^{b}_{j}},i\in[0,C),$$

(3)

where $\mathbf{\mathcal{X}^{f}_{j}}\in\mathbb{R}^{C\times H\times W}$ and $\mathbf{\mathcal{W}^{f}_{j}}\in\mathbb{R}^{C_{out}\times C_{in}\times K_{h}\times K_{w}}$ are full-precision activations and convolution weights in $j$-th layer, respectively. Similarly, $\mathbf{\mathcal{X}^{b}_{j}}\in\mathbb{R}^{C\times H\times W}$ and $\mathbf{\mathcal{W}^{b}_{j}}\in\mathbb{R}^{C_{out}\times C_{in}\times K_{h}\times K_{w}}$ are binarized activations and convolution weights in $j$-th layer separately. $x^{f}_{i,j}$, $w^{f}_{i,j}$, $x^{b}_{i,j}$, and $w^{b}_{i,j}$ are the elements of $i$-th channel of $\mathbf{\mathcal{X}^{f}_{j}}$, $\mathbf{\mathcal{W}^{f}_{j}}$, $\mathbf{\mathcal{X}^{b}_{j}}$, and $\mathbf{\mathcal{W}^{b}_{j}}$ respectively. $\alpha_{i,j}$ is the learnable coefficient controlling the threshold of sign function for $i$-th channel of $\mathbf{\mathcal{X}^{f}_{j}}$. It is notable that the weight binarization method is inherited from XONR-Net (Rastegari et al., 2016), of which $\frac{\left\|\mathcal{W}^{f}_{j}\right\|_{1}}{n}$ is the average of absolute weight values and acts as a scaling factor to minimize the difference between binary and full-precision convolution weights.

Then, we use the RPReLU (Liu et al., 2020) as our activation function, which is defined as follows:

$$f\left(y_{i,j}\right)=\left\{\begin{array}[]{l}y_{i,j}-\gamma_{i,j}+\zeta_{i,j},\text{ if }y_{i,j}>\gamma_{i,j}\\ \beta_{i,j}\left(y_{i,j}-\gamma_{i,j}\right)+\zeta_{i,j},\text{ if }y_{i,j}\leq\gamma_{i,j}\end{array}\right.,y_{i,j}\in\mathcal{Y}_{j},i\in[0,C),$$

(4)

where $\mathbf{\mathcal{Y}_{j}}\in\mathbb{R}^{C\times H\times W}$ is the input feature maps of RPReLU function $f(.)$ in $j$-th layer. $y_{i,j}$ is the element of $i$-th channel of $\mathbf{\mathcal{Y}_{j}}$. $\gamma_{i,j}$ and $\zeta_{i,j}$ are learnable shifts for moving the distribution. $\beta_{i,j}$ is a learnable coefficient controlling the slope of the negative part, which acts on $i$-th channel of $\mathbf{\mathcal{Y}_{j}}$.

Different from the common residual block, which consists of two convolutions used in full-precision IR network (Lim et al., 2017), we find that residual connection is essential for binary convolution to supplement the information loss caused by binarization. Thus, we set a residual connection for each binary convolution. Therefore, the BBCU-V1 can be expressed mathematically as:

$$\mathbf{\mathcal{X}^{f}_{j+1}}=f\left(\operatorname{BatchNorm}\left(\mathbf{\mathcal{X}^{b}_{j}}\otimes\mathbf{\mathcal{W}^{b}_{j}}\right)+\mathbf{\mathcal{X}^{f}_{j}}\right)=f\left(\mathbf{\kappa_{j}}\left(\mathbf{\mathcal{X}^{b}_{j}}\otimes\mathbf{\mathcal{W}^{b}_{j}}\right)+\mathbf{\tau_{j}}+\mathbf{\mathcal{X}^{f}_{j}}\right),$$

(5)

where $\mathbf{\kappa_{j}},\mathbf{\tau_{j}}\in\mathbb{R}^{C}$ are learnable parameters of BatchNorm in $j$-th layer.

In BNNs, the derivative of the sign function in Eq. 2 is an impulse function that cannot be utilized in training. Thus, we adopt the approximated derivative function as the derivative of the sign function. It can be expressed mathematically as:

$$\operatorname{Approx}\left(\frac{\partial\operatorname{Sign}\left(x^{f}_{i}\right)}{\partial x^{f}_{i}}\right)=\left\{\begin{array}[]{lr}2+2\left(x^{f}_{i}-\alpha_{i}\right)&\text{ if }\alpha_{i}-1\leqslant x^{f}_{i}<\alpha_{i}\\ 2-2\left(x^{f}_{i}-\alpha_{i}\right)&\text{ if }\alpha_{i}\leqslant x^{f}_{i}<\alpha_{i}+1\\ 0&\text{ otherwise }\end{array}\right..$$

(6)

However, EDSR (Lim et al., 2017) demonstrated that BN changes the distribution of images, which is harmful to accurate pixel prediction in SR. So, can we also directly remove the BN in BBCU-V1 to obtain BBCU-V2 (Fig. 2 (b))? In BBCU-V1 (Fig. 2 (a)), we can see that the bit counting operation of binary convolution tends to output large value ranges (from -15 to 15). In contrast, residual connection preserves the full-precision information, which flows from the front end of IR network with a small value range from around -1 to 1. By adding a BN, its learnable parameters can narrow the value range of binary convolution and make it close to the value range of residual connection to avoid full-precision information being covered. The process can be expressed as:

$$\operatorname{Mean}\left(\left|\mathbf{\kappa_{j}}\left(\mathbf{\mathcal{X}^{b}_{j}}\otimes\mathbf{\mathcal{W}^{b}_{j}}\right)+\mathbf{\tau_{j}}\right|\right)\rightarrow\operatorname{Mean}\left(\left|\mathbf{\mathcal{X}^{f}_{j}}\right|\right),$$

(7)

where $\mathbf{\kappa_{j}},\mathbf{\tau_{j}}\in\mathbb{R}^{C}$ are learnable parameters of BN in $j$-th layer. Thus, compared with BBCU-V1, BBCU-V2 simply removes BN and suffers a huge performance drop.

After the above exploration, we know that BN is essential for BBCU, because it can balance the value range of binary convolution and residual connection. However, BN changes image distributions limiting restoration. In BBCU-V3, we propose residual alignment (RA) scheme by multiplying the value range of input image by an amplification factor $k$ $(k>1)$ to remove BN Figs. 2(c) and 3(b):

$$\operatorname{Mean}\left(\left|\mathbf{\mathcal{X}^{b}_{j}}\right|\right)\leftarrow\operatorname{Mean}\left(\left|k\mathbf{\mathcal{X}^{f}_{j}}\right|\right).$$

(8)

We can see from the Eq. 8, since the residual connection flows from the amplified input image, the value range of $\mathbf{\mathcal{X}^{f}_{j}}$ also is amplified, which we define as $k\mathbf{\mathcal{X}^{f}_{j}}$. Meanwhile, the values of binary convolution are almost not affected by RA, because $\mathbf{\mathcal{X}^{b}_{j}}$ filters amplitude information of $k\mathbf{\mathcal{X}^{f}_{j}}$. Different from BatchNorm, RA makes the value range of residual connection close to binary convolution (-60 to 60). Besides, we find that using RA to remove the BatchNorm has two main benefits: (1) Similar to full-precision IR networks, the binarized IR networks would have better performance without BatchNorm. (2) The structure of BBCU becomes more simple, efficient, and strong.

Based on the above findings, we are aware that the activation function (Eq. 4) in BBCU-V3 (Fig. 2(c)) narrows the negative value range of residual connection, which means it loses negative full-precision information. Thus, we further develop BBCU-V4 by moving the activation function into the residual connection to avoid losing the negative full-precision information. The experiments (Sec. 4.5) show that our design is accurate. We then take BBCU-V4 as the final design of BBCU.

As shown in Fig. 3(a), the existing image restoration (IR) networks could be divided into four parts: head $\mathcal{H}$, body $\mathcal{B}$, upsampling $\mathcal{U}$, and tail $\mathcal{T}$. If the IR networks do not need increasing resolution (Zhang et al., 2017), it can remove the upsampling $\mathcal{U}$ part. Specifically, given an LQ input $I_{LQ}$, the process of IR network restoring HQ output $\hat{I}_{HQ}$ can be formulated as:

$$\hat{I}_{HQ}=\mathcal{T}(\mathcal{U}(\mathcal{B}(\mathcal{H}(I_{LQ})))).$$

(9)

Previous BNN SR works (Xin et al., 2020; Jiang et al., 2021; Zhang et al., 2021b) concentrate on binarizing the body $\mathcal{B}$ part. However, upsampling $\mathcal{U}$ part accounts for 52.3$\%$ total calculations and is essential to be binarized. Besides, the binarized head $\mathcal{H}$ and tail $\mathcal{T}$ parts are also worth exploring.

As shown in Fig. 3(b), we further design different variants of BBCU for these four parts. (1) For the head $\mathcal{H}$ part, its input is $I_{LQ}\in\mathbb{R}^{3\times H\times W}$ and the binary convolution output with $C$ channels. Thus, we cannot directly add $I_{LQ}$ to the binary convolution output for the difference in the number of channels. To address the issue, we develop BBCU-H by repeating $I_{LQ}$ to have $C$ channels. (2) For body $\mathcal{B}$ part, since the input and output channels are the same, we develop BBCU-B by directly adding the input activation to the binary convolution output. (3) For upsampling $\mathcal{U}$ part, we develop BBCU-U by repeating the channels of input activations to add with the binary convolution. (4) For the tail $\mathcal{T}$ part, we develop BBCU-T by adopting $I_{LQ}$ as the residual connection. To better evaluate the benefits of binarizing each part on computations and parameters, we define two metrics:

$$V_{C}=\left(\operatorname{PSNR}^{f}-\operatorname{PSNR}^{b}\right)/\left(\operatorname{OPs}^{f}-\operatorname{OPs}^{b}\right),$$

(10)

$$V_{P}=\left(\operatorname{PSNR}^{f}-\operatorname{PSNR}^{b}\right)/\left(\operatorname{Parms}^{f}-\operatorname{Parms}^{b}\right),\vspace{-1mm}$$

(11)

where PSNR^b, OPs^b, and Parms^b denote the performance, calculations, and parameters after binarizing one part of networks. Similarly, PSNR^f, OPs^f, and Parms^f measure full-precision networks.

We adopt reconstruction loss $L_{rec}$ to guide the image restoration training, which is defined as:

$$L_{rec}=\left\|I_{HQ}-\hat{I}_{HQ}\right\|_{1},$$

(12)

where $I_{HQ}$ and $\hat{I}_{HQ}$ are the real and restored HQ images, respectively. $\|\cdot\|_{1}$ denotes the $L_{1}$ norm.

Training Strategy. We apply our method to three typical image restoration tasks: image super-resolution, color image denoising, and image compression artifacts reduction. We train all models on DIV2K (Agustsson & Timofte, 2017), which contains 800 high-quality images. Besides, we adopt widely used test sets for evaluation and report PSNR and SSIM. For image super-resolution, we take simple and practical SRResNet (Ledig et al., 2017) as the backbone. The mini-batch contains 16 images with the size of 192$\times$192 randomly cropped from training data. We set the initial learning rate to 1$\times$10^-4, train models with 300 epochs, and perform halving every 200 epochs. For image denoising and compression artifacts reduction, we take DnCNN and DnCNN3 as backbone (Zhang et al., 2017). The mini-batch contains 32 images with the size of 64$\times$64 randomly cropped from training data. We set the initial learning rate to 1$\times$10^-4, train models with 300,000 iterations, and perform halving per 200,000 iterations. The amplification factor $k$ in the residual alignment is set to 130. We implement our models with a Tesla V100 GPU.

OPs and Parameters Calculation of BNN. Following Rastegari et al. (2016); Liu et al. (2018), the operations of BNN (OPs^b) is calculated by OPs^b = OPs^f / 64 (OPs^f indicates FLOPs), and the parameters of BNN (Parms^b) is calculated by Parms^b = Parms^f / 32.

Following BAM (Xin et al., 2020), we take SRResNet (Ledig et al., 2017) as backbone. We binarize body $\mathcal{B}$ part of SRResNet with some SOTA BNNs including BNN (Courbariaux et al., 2016), Bi-Real (Liu et al., 2018), IRNet (Qin et al., 2020), ReActNet (Liu et al., 2020), and BAM (Xin et al., 2020), BTM (Jiang et al., 2021). For comparison, we adopt our BBCU to binarize body $\mathcal{B}$ part to obtain BBCU-L. Furthermore, we binarize body $\mathcal{B}$ and upsampling $\mathcal{U}$ parts simultaneously to develop BBCU-M. Besides, we reduce the number of channels of SRResNet to 12, obtaining SRResNet-Lite. In addition, we use Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010), B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017) for evaluation.

The quantitative results (PSNR and SSIM), the number of parameters, and operations of different methods are shown in Tab. 1. Compared with other binarized methods, our BBCU-L achieves the best results on all benchmarks and all scale factors. Specifically, for 4$\times$ SR, our BBCU-L surpasses ReActNet by 0.25dB, 0.27dB, and 0.44dB on Set5, Urban100, and Manga109, respectively. For 2$\times$ SR, BBCU-L also achieves 0.32dB, 0.27dB, and 0.58dB improvement on these three benchmarks compared with ReActNet. Furthermore, our BBCU-M achieves the second best performance on most benchmarks consuming $5\%$ of operations and $13.7\%$ parameters of ReActNet (Liu et al., 2020) on 4$\times$ SR. Besides, BBCU-L significantly outperforms SRResNet-Lite by 0.16dB and 0.3dB with less computational cost, showing the superiority of BNN. The qualitative results are shown in Fig. 4, and our BBCU-L has the best visual quality containing more realistic details close to respective ground-truth HQ images. More qualitative results are provided in appendix.

For image denoising, we use DnCNN (Zhang et al., 2017) as the backbone and binarize its body $\mathcal{B}$ part with BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU. We also develop DnCNN-Lite by reducing the number of channels of DnCNN from 64 to 12. The standard benchmarks: Urban100 (Huang et al., 2015), BSD68 (Martin et al., 2001), and Set12 (Shan et al., 2019) are applied to evaluate each method. Additive white Gaussian noises (AWGN) with different noise levels $\sigma$ (15, 25, 50) are added to the clean images.

The quantitative results of image denoising are shown in Tab. 2, respectively. As one can see, our BBCU achieves the best performance among compared BNNs. In particular, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.82dB, 0.88dB, and 1dB on CBSD68, Kodak24, and Urban100 datasets respectively as $\sigma=15$. Compared with DnCNN-Lite, our BBCU surpasses it by 0.92dB, 1.03dB, and 1.5dB on these three benchmarks as $\sigma=15$ consuming $58\%$ computations of DnCNN-Lite. Qualitative results are provided in appendix.

For this JPEG compression deblocking, we use practical DnCNN-3 (Zhang et al., 2017) as the backbone and replace the full-precision body $\mathcal{B}$ part of DnCNN3 with some competitive BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU. The compressed images are generated by Matlab standard JPEG encoder with quality factors $q\in\{10,20,30,40\}$. We take the widely used LIVE1 (Sheikh, 2005) and Classic5 (Foi et al., 2007) as test sets to evaluate the performance of each method. The quantitative results are presented in Tab. 3. As we can see, our BBCU achieves the best performance on all test sets and quality factors among all compared BNNs. Specifically, our BBCU surpasses the state-of-the-art BNN model ReActNet by 0.38dB and 0.34dB on the Live1 and Classic5 as $q=40$. In addition, our BBCU surpasses DnCNN-3-Lite by 0.16dB and 0.2dB on benchmarks as $q=10$. The visual comparisons are provided in appendix.

Basic Binary Convolution Unit. To validate BBCU for the binarized IR network, we binarize the body part of SRResNet with four variants of BBCU (Fig. 2) separately. The results are shown in Tab. 4. (1) Compared with BBCU-V1, BBCU-V2 declines by 0.14dB and 0.24dB on Urban100 and Manga109. This is because BBCU-V2 simply removes the BN making the value range of binary Conv far larger than residual connection and cover full-precision information (Fig. 2). (2) BBCU-V3 adds residual alignment scheme on BBCU-V2, which addresses the value range imbalance between binary Conv and residual connection and removes the BN. Since the BN is harmful for IR networks, BBCU-V3 surpasses BBCU-V1 by 0.17dB, 0.16dB, and 0.29dB on Set5, Urban100, and Manga109 respectively. (3) BBCU-V4 moves the activation function into the residual connection, which preserves the full-precision negative values of residual connection (Fig. 2). Thus, BBCU-V4 outperforms BBCU-V3.

Residual Connection. We take SRResNet as the backbone with 32 Convs in body part and explore the relationship between performance and the number of binary convolution in residual connection. Specifically, for both full-precision and binarized SRResNet, we set 1,2,4,8,16, and 32 Convs with a residual connection as basic convolution units, respectively. We evaluate SRResNet equipped with these basic convolution units on 4$\times$ Urban100 (see Fig. 6). (1) For full-precision networks, it is best to have 2 Convs with a residual connection to form a basic convolution unit. Besides, if there are more than 4 Convs in a basic convolution unit, its performance would sharply drop. For binarized networks, it is best to equip each binary convolution with a residual connection. In addition, compared with the full-precision network, the binarized network is more insensitive to the growth of the Convs number in the basic convolution unit. (2) For binarized SRResNet, we delete residual connection of a BBCU in a certain position. In Tab. 4.5, it seems that once the residual connection is removed (the full-precision information is lost) at any position, the binarized SRResNet would suffer a severe performance drop.

Amplification Factor. As shown in Fig. 6, the performance of full-precision remain basically unchanged with the variation of amplification factor $k$. However, the binarized network is sensitive to the $k$. The best $k^{*}$ is related to the number of channels $n$, which empirically fits $k^{*}=130n/64$. Intuitively, the larger number of channels makes binary convolution count more elements and have larger results, which needs larger $k$ to increase the value of residual connection for balance.

The Binarization Benefit for Different Parts of IR Network. We binarize one part in SRResNet with BBCU in Fig. 3 while keeping other parts full-precision. The results are shown in Tab. 4.5. We use $V_{C}$ (Eq. 10) and $V_{P}$ (Eq. 11) as metric to value binarization benefit of different parts. We can see that Upsampling part is most worth to be binarized. However, it is ignored by previous works (Xin et al., 2020). The binarization benefit of first and last convolution is relatively low.