Binarized Weight Error Networks With a Transition Regularization Term

As excepted, quantization errors for binary weights are moderately high compared to ternary weights. However, we believe there is room to take additional steps to reduce this error.

In particular, the idea emphasized in our work is based on the fact that an iterative approach can be more suitable for binarization process to increase the capacity of representations than the quantization of full-precision parameters in a single step. For this purpose, the problem definition in Eq. 1 is rewritten as:

An observation for Eq. 5 is that the use of two binary-weight models for both $\mathbf{W}_{0}^{b}$ and $\mathbf{W}_{1}^{b}$ implicitly reproduces a ternary-weight model. This result is critical because it confirms the mathematical proof of our method. Note that this step also shows that binary-representation can be improved further with simple replacements in the formulation.

Second, we compute the optimum scale value $\alpha_{opt}$ as:

As explained in [10], even if full-precision parameters are sampled from either a normal or uniform distribution, the allowable parameter range can still be large. A high approximation error is therefore expected. Furthermore, parameter updates with binary-precision are problematic as accurate error propagation is nearly impossible and greatly overfits the parameters.

We propose a novel regularization term that forces parameters to concentrate around binary transitions and penalizes those that are far from those points. For clarity, transition points correspond to $0$ for binary-weight models, $\{-\Delta,\Delta\}$ for ternary-weight models and $\alpha_{opt}$s for our models. Hence, the effects of parameter changes for binary-precision can be learned more quickly with this regularization term at train time.

To favor these changes, a simple and effective trick is used. Mathematically, full-precision parameters $\mathbf{W}$ are corrupted with a random noise $\mathbf{\tilde{W}}=\mathbf{W}+0.1\cdot\mathcal{N}(0,\alpha)$. Later, these noisy weights are also quantized $\mathbf{\tilde{W}}^{b}$ similarly with Eq. 1. Note that at the boundaries of transition points, binarized-representations for corrupted parameters differ from the original versions:

In the experiments, we will show that this term is particularly useful for classification tasks than inverse problems. The main reason is that as explained in [22], regression tasks can lead to holes due to continuous solution spaces, so that sparsity cannot be maintained correctly in such problems.

For the implementation detail¹¹1https://github.com/savasozkan/BofT, each model is trained with Adam stochastic optimizer at train time. Mini-batch size is set 128, and learning rate scaling is adopted. In the classification task, VGG-6 (K) with batch normalization is used for acceleration and parameter regularization (i.e., 2×(K-C3) + MP2 +2×(2×K-C3) + MP2 + 2×(4×K-C3) + MP2 + 8×K-FC + Softmax and K is the number of filters). Here, ”K-C3” is convolution layer with K filters of a $3\times 3$ kernel. Also, ”MP2” and ”K-FC” denote a max pooling with stride $2$ and a fully-connected layer with K filters respectively. Similarly, espcn architecture with skip-connection is utilized for super-resolution and denoising tasks.

Experimental results are reported on Fashion, ImageNet2012 (64×64), Cifar10 and SVHN datasets. Fashion, Cifar10 and SVHN datasets have 10 object/character classes, while ImageNet2012 has 1000 object classes. Visual samples for these datasets are illustrated in Fig. 2. Random crop, padding and cutout policies are exploited as data augmentation. Moreover, softmax cross-entropy loss is used as the task-specific loss function $\mathcal{L}_{pb}$. Note that all trainable parameters, excluding fully-linear layers, are binarized in our paper.

First, we compare our method with baseline models, and results are illustrated in Table 1. As observed, our method outperforms all its binary-precision counterparts. This result validates that it improves the quantization performance by considering the approximation error. Especially for ImageNet2012 (INet), the increase is significant, and the proposed model yields close results with the full-precision architecture. Moreover, the performance saturates when the number of parameters increases for all-binarized models as expected.

Finally, the impact of transition regularization term is reported in Table 2. Results confirm that this term improves all threshold-based binarized weight models. In particular, these improvements reach up to $1\%$ for the shallow networks (i.e., K is lower) compared to results in Table 1. The reason is that when the number of parameters increases, parameters are automatically concentrated in a narrow range, which has a positive impact on the quantization.

Experiments are carried out on two well-known inverse problem tasks: visual super-resolution and visual denoise. At train time, images from the COCO val2017 dataset are used for both tasks. For high resolution, these images are rescaled by 2, 3, and 4 to simulate changes in resolution. At test time, the data sets Set5, Set14, Urban, and BSD100 are used. Additionally, different sigma values (i.e., with a mean of zero) are used to attach images for the denoise task. Tests are also performed on Set5 and Set14 records.

First, an input image is converted to YUV color space for both tasks, and Y channel is used for both train and test phases. Also, our NN architecture is fully convolutional. It takes an input image and applies consecutive convolution layers with residual-based learning, as explained in [23]. At the output, it is upsampled with a subpixel layer [24] for super-resolution (i.e., (64-C3) + 2×(64-C3) + 2×(64-C3) + SubPixel + (1-C3)). Note that the first and last layers are full-precision networks, while the rest is binarized.

Experimental results for super-resolution are reported in Table 3. Our method outperforms all other baselines. In particular, for small scale factors, negligible performance drops can be observed compared to full-precision networks. Similarly, the reason is that the allowable parameter range is expected to be small for small scale factors, and the sparsity is decreased.

Denoising results are also shown in Table 4. From the results, the proposed method achieves slightly better performance even for full-precision networks. Residual-based learning eventually promotes robustness with binarized networks since parameter range (i.e., additive noise varies $[0.05,0.2]$) is heavily reduced. Hence, binarized weights can converge better solutions compared to other tasks.

As indicated, the proposed regularization term yields similar or slightly worse performance for inverse problems. The main reason is that the learning process may lead to holes in continuous feature space partitioning.