Nonlinear Tensor Ring Network

A fully-connected layer maps an input vector $\boldsymbol{x}\in\mathbb{R}^{I}$ into an output vector $\boldsymbol{y}\in\mathbb{R}^{O}$ via a weight matrix $\boldsymbol{W}\in\mathbb{R}^{I\times O}$. Mathematically, it is formulated as

where $(\cdot)^{T}$ denotes the transpose operator. Neural network compression by low-rank matrix/tensor factorization is to decompose $\boldsymbol{W}$ into some small-size factors. To compress $\boldsymbol{W}$ in tensor structure, we first reshape $\boldsymbol{W}$ into a high order tensor $\boldsymbol{\mathcal{W}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{N}\times O_{1}\times\cdots\times O_{M}}$ with

Then, based on TR format, $\boldsymbol{\mathcal{W}}$ is factorized into $(N+M)$ 3rd-order tensors

where $\circ$ implies the outer product operation, $\boldsymbol{\mathcal{G}}_{k}\in\mathbb{R}^{R_{k}\times I_{k}\times R_{k+1}}$ with $k\in[1,N]$, $\boldsymbol{\mathcal{G}}_{k}\in\mathbb{R}^{R_{k}\times O_{k}\times R_{k+1}}$ with $k\in[N+1,M]$, and $\boldsymbol{\mathcal{G}}_{k}(r_{k},:,r_{k+1})$ is the vertical fiber of $\boldsymbol{\mathcal{G}}_{k}$. Moreover, $\boldsymbol{x}$ and $\boldsymbol{y}$ need to be tensorized as $\boldsymbol{\mathcal{X}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{N}}$ and $\boldsymbol{\mathcal{Y}}\in\mathbb{R}^{O_{1}\times\cdots\times O_{M}}$, respectively. Thereby, (3) can be rewritten in the tensor format:

where $\times_{k}^{1}$ and $\times_{k,l}^{1,3}$ are the tensor contraction operations [27]. Consider $\boldsymbol{\mathcal{A}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{k-1}\times I_{k}\times I_{k+1}\times\cdots\times I_{l-1}\times I_{l}\times I_{l+1}\times\cdots\times I_{N}}$ and $\boldsymbol{\mathcal{B}}\in\mathbb{R}^{J_{1}\times J_{2}\times J_{3}}$ with $I_{k}=J_{1}$ and $I_{l}=J_{3}$, $\boldsymbol{\mathcal{A}}\times_{k}^{1}\boldsymbol{\mathcal{B}}$ yields a $(N+1)$th-order tensor $\boldsymbol{\mathcal{C}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{k-1}\times I_{k+1}\times\cdots\times I_{N}\times J_{2}\times J_{3}}$ whose entries are calculated by

Similarly, $\boldsymbol{\mathcal{A}}\times_{k,l}^{1,3}\boldsymbol{\mathcal{B}}$ generates a $(N-1)$th-order tensor $\boldsymbol{\mathcal{D}}\in\mathbb{R}^{I_{1}\times\cdots\times I_{k-1}\times I_{k+1}\times\cdots\times I_{l-1}\times I_{l+1}\times\cdots\times I_{N}\times J_{2}}$ with entries being

It is worth noting that (III-A) is the same as the fully-connected TR network in [17]. To boost the accuracy, we propose to include a nonlinear activation function after each tensor contraction in (III-A), resulting in

where $f(\cdot)$ signifies the nonlinear activation function, e.g., Tanh in Fig. 1. Note that $f(\cdot)$ is not added after $\boldsymbol{\mathcal{G}}_{N+M}$ since the intrinsic activation function outside the current layer replaces $f(\cdot)$. Fig. 1 illustrates the fully-connected layer in NTRN.

At the convolutional layer, an input tensor $\boldsymbol{\mathcal{X}}\in\mathbb{R}^{H\times W\times I}$ is convoluted with a 4th-order kernel tensor $\boldsymbol{\mathcal{W}}\in\mathbb{R}^{D\times D\times I\times O}$ to output a tensor $\boldsymbol{\mathcal{Y}}\in\mathbb{R}^{\widetilde{H}\times\widetilde{W}\times O}$, formulated as

where $*$ denotes the convolution operation in CNNs. To compress $\boldsymbol{\mathcal{W}}$, TR decomposition factorizes it into three core tensors, viz. $\boldsymbol{\mathcal{V}}\in\mathbb{R}^{R_{1}\times D\times D\times R_{2}}$, $\boldsymbol{\mathcal{U}}\in\mathbb{R}^{R_{2}\times I\times R_{3}}$ and $\boldsymbol{\mathcal{\widetilde{U}}}\in\mathbb{R}^{R_{3}\times O\times R_{1}}$. Therefore, $\boldsymbol{\mathcal{W}}$ is computed by

It can be known that $\boldsymbol{\mathcal{V}}$ retains convolution kernel, which is different from NTT format. In NTRN, the convolution operation (10) is decomposed into the following procedure:

where $\boldsymbol{\mathcal{Z}}_{1}\in\mathbb{R}^{H\times W\times R_{2}\times R_{3}}$, $\boldsymbol{\mathcal{Z}}_{2}\in\mathbb{R}^{\widetilde{H}\times\widetilde{W}\times R_{1}\times R_{3}}$ and $(\cdot)^{p_{1}}$ is the 1st tensor permutation. The relationship between $\boldsymbol{\mathcal{V}}^{p_{1}}$ and $\boldsymbol{\mathcal{V}}$ obeys

For the convolutional layer in NTRN, the input tensor $\boldsymbol{\mathcal{X}}$ is first contracted with the intermediate factor $\boldsymbol{\mathcal{U}}$, then convoluted with the first core $\boldsymbol{\mathcal{V}}$, and finally contracted with the last core $\boldsymbol{\mathcal{\widetilde{U}}}$. It is worth mentioning that this procedure is entirely different from the operation in NTT format. On the other hand, when $I$ and $O$ are large, $\boldsymbol{\mathcal{U}}$ and $\boldsymbol{\mathcal{\widetilde{U}}}$ can be further factorized into small-size core tensors for achieving a higher CR. For instance, $\boldsymbol{\mathcal{U}}$ can be reshaped as $\boldsymbol{\mathcal{\widehat{U}}}\in\mathbb{R}^{R_{2}\times I_{1}\times I_{2}\times I_{3}\times R_{3}}$ with $I_{1}\times I_{2}\times I_{3}=I$, and then $\boldsymbol{\mathcal{\widehat{U}}}$ is factorized into three tensors, namely, $\boldsymbol{\mathcal{G}}_{1}\in\mathbb{R}^{R_{2}\times I_{1}\times R_{22}}$, $\boldsymbol{\mathcal{G}}_{2}\in\mathbb{R}^{R_{22}\times I_{2}\times R_{23}}$ and $\boldsymbol{\mathcal{G}}_{3}\in\mathbb{R}^{R_{23}\times I_{3}\times R_{3}}$. To calculate $\boldsymbol{\mathcal{\widehat{U}}}$ from its core tensors, the proposed NTRN adopts a nonlinear contraction operation

The convolutional layer in NTRN is illustrated in Fig. 2.

CR is defined as:

For a fully-connected layer, the number of parameters is $IO$ given a weight matrix $\boldsymbol{W}\in\mathbb{R}^{I\times O}$. The proposed NTRN requires $(\sum_{n=1}^{N}I_{n}+\sum_{m=1}^{M}O_{m})R^{2}$ parameters with the assumption of $R_{1}=\cdots=R_{M+N}=R$. On the other hand, the convolutional layer of conventional neural network needs to store $D^{2}IO$ parameters for $\boldsymbol{\mathcal{W}}\in\mathbb{R}^{D\times D\times I\times O}$. In NTRN, the dimensions of $\boldsymbol{\mathcal{W}}$ are reshaped into ${D\times D\times I_{1}\times\cdots\times I_{N}\times O_{1}\times\cdots\times O_{M}}$ for a high CR. Hence, only $(\sum_{n=1}^{N}I_{n}+\sum_{m=1}^{M}O_{m}+D^{2})R^{2}$ parameters are required. It is known that decreasing $R$ is able to increase CR.

We first test the proposed NTRN using a MLP on MNIST dataset. The MLP consists of input, two hidden and output layers with 784, 1024, 512 and 10 nodes, respectively. All images in the MNIST are reshaped as a vector of length $784$. Therefore, the weight matrices of the MLP are $\boldsymbol{W}_{1}\in\mathbb{R}^{784\times 1024}$, $\boldsymbol{W}_{2}\in\mathbb{R}^{1024\times 512}$ and $\boldsymbol{W}_{3}\in\mathbb{R}^{512\times 10}$. To compress them, $\boldsymbol{W}_{1}$, $\boldsymbol{W}_{2}$ and $\boldsymbol{W}_{3}$ are tensorized, and hence their dimensions are $4\times 7\times 4\times 7\times 4\times 8\times 4\times 8$, $4\times 8\times 4\times 8\times 8\times 8\times 8$ and $8\times 8\times 8\times 10$, respectively. Epoch is set as 50 for training. The experimental results based on 20 independent trials are tabulated in Table I, where Acc and Std denote the average accuracy on the test set and the standard deviation of all results, respectively. In addition, the highest and lowest accuracy among all trials are listed in columns H and L, respectively. Moreover, storage indicates the disk space occupied by these parameters. At 57x CR, the ranks of three layers are $\{16,14,8\}$, while the ranks are set as $\{6,5,5\}$ 359x CR. We can see that the proposed NTRN outperforms TRN [17] at both low and high CRs in terms of the average accuracy, the standard deviation, the highest and lowest accuracy.

We now investigate the compression performance via a CNN with two hidden convolutional and one fully-connected layers on MNIST. The first and second convolutional layers contain 16 and 32 kernels with kernel-size equaling 3, respectively. Moreover, the stride is set to 1, and the padding is 0. Since the dimensions of input data are $28\times 28\times 1$, the weight tensor at the first convolutional layer is $\boldsymbol{\mathcal{W}}_{1}\in\mathbb{R}^{3\times 3\times 1\times 16}$. Besides, the weight arrays at the second convolutional and fully-connected layers are expressed as $\boldsymbol{\mathcal{W}}_{2}\in\mathbb{R}^{3\times 3\times 16\times 32}$ and $\boldsymbol{\mathcal{W}}_{3}\in\mathbb{R}^{320\times 10}$, respectively. To compress weight arrays of the convolutional layers, the dimensions of $\boldsymbol{\mathcal{W}}_{1}$ and $\boldsymbol{\mathcal{W}}_{2}$ are reshaped as $3\times 3\times 1\times 4\times 4$ and $3\times 3\times 4\times 4\times 4\times 8$, respectively. Following the ablation study, the fully-connected layer in CNN is not compressed. Hence, we do not resize its weight matrix.

The results are shown in Table II. The ranks at 4.3x and 8.9x CRs are $\{2,6\}$ and $\{2,4\}$, respectively. It is seen that the proposed NTRN attains higher accuracy than TRN at the same CR. While the standard deviation of NTRN is smaller than that of TRN. Note that the storage information is not provided as the fully-connected layer is not compressed.

With promising results on monotypic layer compression, we conduct experiments using LeNet-5 [23] on the Fashion MNIST dataset to further evaluate the proposed NTRN, where the LeNet-5 is made up of two convolutional and two fully-connected layers. The resized weight tensors from input to output layers are $\boldsymbol{\mathcal{W}}_{1}\in\mathbb{R}^{5\times 5\times 1\times 4\times 5}$, $\boldsymbol{\mathcal{W}}_{2}\in\mathbb{R}^{5\times 5\times 4\times 5\times 5\times 10}$, $\boldsymbol{\mathcal{W}}_{3}\in\mathbb{R}^{5\times 10\times 5\times 5\times 8\times 8\times 8}$ and $\boldsymbol{\mathcal{W}}_{4}\in\mathbb{R}^{8\times 8\times 8\times 10}$. As shown in Table III, the proposed NTRN is superior to the TRN at both 13x and 72x CRs. Herein, the ranks are $\{3,10,30,8\}$ for 13x CR, and $\{3,8,10,5\}$ for 72x CR.

Furthermore, the developed NTRN is examined by VGG-11 on two datasets, namely, Cifar-10 and Fashion MNIST. The VGG-11 consists of eight convolutional and three fully-connected layers. For the Cifar-10 dataset, the weight tensors from input to output layers are $\boldsymbol{\mathcal{W}}_{1}\in\mathbb{R}^{3\times 3\times 3\times 4\times 4\times 4}$, $\boldsymbol{\mathcal{W}}_{2}\in\mathbb{R}^{3\times 3\times 4\times 4\times 4\times 2\times 4\times 4\times 4}$, $\boldsymbol{\mathcal{W}}_{3}\in\mathbb{R}^{3\times 3\times 2\times 4\times 4\times 4\times 4\times 4\times 4\times 4}$, $\boldsymbol{\mathcal{W}}_{4}\in\mathbb{R}^{3\times 3\times 4\times 4\times 4\times 4\times 4\times 4\times 4\times 4}$, $\boldsymbol{\mathcal{W}}_{5}\in\mathbb{R}^{3\times 3\times 4\times 4\times 4\times 4\times 4\times 4\times 4\times 4\times 2}$, $\boldsymbol{\mathcal{W}}_{6}=\boldsymbol{\mathcal{W}}_{7}=\boldsymbol{\mathcal{W}}_{8}\in\mathbb{R}^{3\times 3\times 4\times 4\times 4\times 4\times 2\times 4\times 4\times 4\times 4\times 2}$, $\boldsymbol{\mathcal{W}}_{9}\in\mathbb{R}^{4\times 4\times 4\times 4\times 2\times 4\times 4\times 4\times 4\times 2}$, $\boldsymbol{\mathcal{W}}_{10}\in\mathbb{R}^{4\times 4\times 4\times 4\times 2\times 4\times 4\times 4\times 4}$ and $\boldsymbol{\mathcal{W}}_{11}\!\in\!\mathbb{R}^{4\times 4\times 4\times 4\times 10}$, respectively. Excluding the first weight tensor $\boldsymbol{\mathcal{W}}_{1}\in\mathbb{R}^{3\times 3\times 1\times 4\times 4\times 4}$ for the Fashion MNIST dataset, the dimensions of the others are kept the same as those in Cifar-10. The ranks are set to $\{8,\!25,\!40,\!50,\!70,\!70,\!70,\!70,\!15,\!15,\!5\}$ for 9x CR. By contrast, the ranks are $\{8,\!25,\!30,\!40,\!50,\!50,\!50,\!50,\!10,\!10,\!5\}$ at 17x CR.

Table IV lists the results on the Cifar-10 dataset. It is known that NTRN demonstrates overwhelming superiority compared with TRN. Specifically, the average accuracy by the proposed NTRN increases by 1.4% and 1.8% at 9x and 17x CRs, respectively. The corresponding results for the Fashion MNIST are tabulated in Table V. Compared with TRN, the proposed NTRN effectively mitigates the loss of accuracy caused by compression.