Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

Response: In the rebuttal, we include comparison results with a leading Integer-only quantization method, and a leading structured pruning approach. Furthermore, we include results of comparing our approach (with fine tuning) vs. applying a low-rank promoting compression method where rank selection is needed during training (with fine-tuning). Here, fine-tuning refers to fine-tuning the parameters of the compressed trained model.

Response: Due to time constraints, we have chosen not to run ImageNet experiment. We started the Tiny-ImageNet experiment and will provide the results if completed in time. If the paper is accepted, we will provide these results. We would like to highlight that we validate our approach using CIFAR-100 dataset in Figure 3.

Response: See our response to Reviewer 88Da comment 2. For other empirical evaluations, [REFER to HYPERLINK AND EXP NUMBER]

Response: In the rebuttal, we provide results for $N=4$ for the setting of the FCN-8 model in Table 1. See Experiment 3 in the supplementary link. We note that the test accuracy is 2% less than the one reported for $N=1$. We remark that while in theory, increasing N indicates an increase in the low-rankness of the appended trained weight, in practice, we are faced with multiple challenges. These include (i) the increased training run-time (although its impact is not very significant as training occurs only once, and the goal of compression is to improve testing time performance), (ii) finding the right weight decay parameter that maintains the same (or very similar) test accuracy, and (iii) the increased non-convexity of the training optimization problem. This trade-off will be included in the revised paper.

Response: See our response to Reviewer 88Da comment 1.

Response: For the GSVT results (Figure 2 (right) - second row), our method maintains a low accuracy drop with only 30% of retained singular values. However, we agree about your observation when using the LSVT method.

Response: CIFAR10.

Response: By overparameterization, we mean that, for every weight matrix, we append linear layers as described in Equation (5). During training, we optimize over the these linear layers before activation. At testing time, the SVD truncation is applied on the product of the trained linear weights.

Response:. For MNIST, data augmentation is not needed as the test accuracies are high. We use data augmentation in Table 2 to obtain CIFAR10/100 models ($N=1$) with improved test accuracies. The use of data augmentation to improve the test accuracy on CIFAR10/100 dataset is a standard practice (see the caption of Table 6 of the original ResNet paper ‘Deep Residual Learning for Image Recognition’).

Response: Proposition 4.1 is very similar to Theorems 4 and 5 in Shang et al. (2016). There are no theoretical contributions made by us. The mentioned distinction does not affect the proof. There are no non-trivial differences in the proof, and therefore, we decided not to include it in the paper as it directly follows Shang et al. (2016). Proposition 4.1 is used to theoretically justify why standard weight decay in our method promotes low-rankness for models employing the linear layer composition method used in our paper. We agree with the reviewer that explicitly stating that the proof will not change by our rank assumption is necessary. As such, in the revised paper, we will emphasize this point more clearly.

Response: We agree that as $N$ increases, the required training time will also increase. However, we would like to highlight that shortening test time for large models holds has a particular importance over shortening training time because training is done either once or infrequently, whereas testing/inference (function evaluation) takes place continuously. Furthermore, user experience is influenced by test time, as deployed compressed models are more commonly used in resource-limited devices.

Response: For evaluation with larger networks such as ImageNet, see our response to Reviewer 88Da - Comment 2.

Response: In this rebuttal, we provide a plot in link (Experiment 2) to show the faster decay of LoRITa-trained model ($N>1$) vs. the baseline ($N=1$) for FCN8 in Table 1.

Response: Please see our response to Reviewer 88Da - Comment 1.

Response: We are not applying element-wise multiplication. We are appending linear layers based on Equation (5). The dimension of the 1st matrix is $M\times N$. The dimension of $\mathbf{W}^{k}$, for $k\in\{2,\dots,N\}$, is $N\times N$.

Response: There is no particular reason of our choice. Given Proposition 4.1, what you suggested should also work. Please note that we neither assume that $M>N$ nor $M<N$.

Response: Thanks for your suggestion. We will try to combine our approach (pure compression method) and the lottery ticket hypothesis and provide results/observations in the camera-ready version (if the paper is accepted). WE HAVE YET TO TRY THAT AND CONSIDER IT FOR FUTURE WORK…

Response: By ”the limited range of the FP16 operations would act as a regularization”, Do you mean quantization may help further reduce the rank? (Please correct us if we understand you wrong). In order to see whether or not quantization help with low-rankness, we designed the following simple experiment. We define a low-rank matrix as $A=LR$ where $L,R^{T}\in\mathbb{R}^{n\times r}$ are generated from standard Gaussian distribution. Then we show how the singular values of $A$, a randomly pruned version of $A$ (by $90\%$), and a coarsely quantized representation of $A$ decay. For a fair comparison, we normalized the spectrums so that the first singular values of all three matrices equal to 1. Experiment 1 in the response supplementary link shows the plots. We see that pruning destroys the low-rankness of $A$ while quantization seems to slightly enhance the low-rankness. We will explore this further.

Response: We agree that increasing $N$ will increase the training time as the number of matrices required for training increases. Furthermore, tuning the hyper-parameters become more challenging as the width of the network increases. However, in theory (as given in Proposition 4.1), the product of the trained linearly composed weights will always exhibit lower rank when compared to standard training ($N=1$). In regard to showing results for $N>3$, please see our response to Reviewer HnzA - Comment 2.

Response: See our response to Reviewer 88Da - Comment 1.

Response: Please see the second part of our response to Reviewer 1gem - Comment 3.

Response: That is correct, we are increasing the width of the DNN model by appending, per weight matrix, a composition of linear layers (Equation (5)). That is the reason we use this terminology (the linear layers composition). As for the suggested decomposition and factorization, we agree that they also could be used.

Response: We thank you for pointing us to this. In the revised paper, we will include a citation. For justification, please see Section 2.2 of ‘Low-Rank Matrix Recovery via Efficient Schatten p-Norm Minimization’. We have included a screen shot to the response supplementary link.

Response: Keeping only SVs whose magnitudes are greater than a certain threshold is exactly what we did. As we vary the cutting-off threshold, the percentage of retained SVs will change monotonically, so we can either use the percentage of retained SVs or the threshold value itself as the x-axis of the figure. The sudden drop of accuracy corresponds to meeting a threshold close to one or more significant singular values.

Response: Yes. We will fix it in the revised version of the paper.

Response: [LOOK FOR ATTENTION LAYER LITERATURE TO SEE HOW EQUATION 3 THAT COMBINES THE INPUT WITH THE OUTPUT OF A ONE ATTENTION LAYER…]

Response: We agree that there could be a very small drop (first three rows) or small increase (row 4 and row 6) in accuracy, but it is very marginal. We will re-phrase this sentence in the revised paper.