Cascaded Compressed Sensing Networks: A Reversible Architecture for Layerwise Learning

The error backpropagation (BP) method has achieved great success in the supervised learning of deep neural networks [1]. Due to taking the entire network as a whole to optimize, the method can hardly disclose the contribution of each layer, causing difficulty to network analysis. As an alternative to BP, the emerging layerwise learning method [2, 3] seems more suitable for network analysis. The method aims to learn the network layer by layer and thus reduces the network analysis to the layer level.

For layerwise learning, the key is to derive an optimization target for each layer, simply called local target hereafter. Currently, the most simple method is to directly employ the global target [4], namely the optimization target of the entire network, as the target of each layer. However, the global target cannot guarantee to be optimal for each layer to minimize the global loss. To achieve BP level performance, the methods in [5, 6] propose to append a classifier to the layer of interest, and then optimize them together with the global target. This forms a shallow network with at least one hidden layer, in which the layer of interest is still hidden and thus hard to analyze [7, 8, 9]. To directly optimize and analyze each layer, we resort to another emerging layerwise learning method known as target propagation (TP) [2, 3, 10], which derives the local target of each layer by inversely propagating the global target, and optimizes the layer by matching its output to the target. In the method, local targets are required to be of two properties. First, their feedforward outputs should be identical or close to the global target. This ensures that the optimization of each layer could reduce the global loss to the maximum extent. Second, the local target of each layer should be close to the forward output of the layer. This will help to reduce the variation of each layer over optimization, while the low variations incline to improve the network’s generalization [11, 12, 13]. However, the two properties are hard to achieve because the inverse propagation of global target involves the inversions of nonlinear activations from low-dimensional to high-dimensional spaces. This is an ill-posed problem. For the problem, the existing solution is to learn an auxiliary network to specially propagate the target. Obviously, the method will increase the complexity of network learning, and moreover, the auxiliary network has no guarantee for stable and accurate target propagation. In the letter, we show that auxiliary networks are not necessary for layerwise learning, and target propagation could be achieved with guaranteed accuracy by modeling the network’s each layer with compressed sensing.

In compressed sensing [14, 15], it has been proved that a high-dimensional signal $x\in\mathbb{R}^{n}$ with sparse transformations $s=Dx$ over an orthonormal dictionary $D\in\mathbb{R}^{n\times n}$, could be recovered from its low-dimensional projection $y=WDx=Ws$, where $W\in\mathbb{R}^{m\times n}$, $m\ll n$. The recovery is implemented by first recovering the sparse vector $s$ from $y$ via solving an $\ell_{0}$ or $\ell_{1}$-regularized least square problem [16], and then simply deriving $x=D^{\top}s$. Inspired by this reverse process, we propose to model each network layer to be a compressed sensing process. Specifically, we formulate each layer as a cascade of $W$ and $D$, such that the input $s$ of $W$ could be recovered from its output $y$ via sparse recovery, and the input $x$ of $D$ (namely the output $y$ of $W$) could be simply recovered from its output $s$ by multiplying with $D^{\top}$. Repeating this process layer by layer, target propagation will be finally accomplished. To obtain better performance, as usual, we suggest to further deploy a nonlinear function $f(\cdot)$ following each dictionary $D$, in order to remove small activations in magnitude. This operation brings two benefits. First, it further improves the sparseness of the output activations of $D$, which is beneficial to the sparse recovery on the following $\mathbf{W}$. Second, it tends to reduce within-class scattering and thus improve the classification accuracy [17]. Overall, with the perfect combination of $W$, $D$ and $f(\cdot)$, we establish a compressed sensing framework for each layer, and then obtain a cascaded compressed sensing architecture with all layers. To the best of our knowledge, this is the first time that compressed sensing is introduced for layerwise learning to handle the target propagation problem.

Compared to auxiliary networks-based TP methods, the proposed compressed sensing-based TP method has two major advantages. First, it avoids the usage of auxiliary networks and thus reduces the complexity of network learning. Second, it could achieve target propagation with guaranteed accuracy. In layerwise learning, the amount of local targets we need to compute and store is equal to the number of activations in the layer. Considering convolution networks contain huge amounts of activations and pose great computational challenges to layerwise learning, as in [3, 10], we only test the proposed layerwise learning method on fully-connected networks for tractable computation. Our performance advantage is verified by the classification experiments on MNIST and CIFAR10.

Let us consider a deep fully connected network with $L$ layers as shown in Fig. 1. Each layer consists of two sublayers $\{W_{l},D_{l}\}$, where $W_{l}\in\mathbb{R}^{m_{l}\times n_{l}}$ with $m_{l}\ll n_{l}$ denotes a projection matrix and $D_{l}\in\mathbb{R}^{m_{l}\times m_{l}}$ means an orthonormal dictionary, $D^{\top}D=I$. Following $D_{l}$ is a pointwise thresholding function $f_{l}(\cdot)$ which zeroes out all but the $k_{l}$ ($\ll m_{l}$) largest positive elements, acting similarly to the ReLU function. Note that the output layer is a classifier which comprises $W_{L}$ but no $D_{L}$. Suppose $h_{W}^{l}\in\mathbb{R}^{m_{l}}$ and $h_{D}^{l}\in\mathbb{R}^{m_{l}}$ are the output activations of $W_{l}$ and $D_{l}$, respectively. The feedforward network structure is formulated by

where $g(\cdot)$ is a softmax function, and the notations $h_{D}^{0}$ and $h_{W}^{L}$ denote the network’s input and output, respectively.

As sketched in Algorithm 1, the proposed layerwise learning is implemented within three phases. First, we initialize the network via feeding forward the network input $h_{D}^{0}$. During this process, the projection matrices $W_{l}$ are randomly generated, and the orthonormal dictionaries $D_{l}$ are learned with their input activations ${h}_{W}^{l-1}$. Second, as shown in the lower part of Fig.1, by compressed sensing the global targets $\tilde{h}_{W}^{L}$ are inversely propagated layer by layer, producing local targets $\tilde{h}_{W}^{l}$ and $\tilde{h}_{D}^{l}$ respectively for $W_{l}$ and $D_{l}$, $1\leq l<L$. Finally, we feed forward the input $h_{D}^{0}$ again, in order to successively minimize the local losses $\mathcal{L}_{l}({h}_{W}^{l}$, $\tilde{h}_{W}^{l})$ and $\mathcal{L}_{l}({h}_{D}^{l},\tilde{h}_{D}^{l})$ of each layer by updating $W_{l}$ and $D_{l}$, $1\leq l<L$. This finally minimizes the global loss $\mathcal{L}({h}_{W}^{L}$, $\tilde{h}_{W}^{L})$. In the letter, we simply define the local losses with Euclidean distances and the global loss with cross entropy. In the following, we detail the three phases.

In this section, we aim to prove that the proposed compressed sensing-based target propagation (CSTP) method could achieve better performance than the auxiliary network-based target propagation (ANTP) method [3], on the layerwise learning of deep networks. Moreover, we analyze the proposed CSTP method by ablation study. For comparison, as in [3], we test the proposed CSTP method on two typical datasets, MNIST [29] and CIFAR10 [30]. The MINIST dataset consists of $28\times 28$ gray-scale images of handwritten digits. It has a training set of 60,000 samples, and a test set of 10,000 samples. The CIFAR10 dataset consists of $32\times 32$ natural color images in 10 classes. There are 50,000 samples in the training set and 10,000 samples in the testing set.

In the letter, we have shown that the cascade of compressed sensing can be established as a reversible network, which allows for layerwise learning and analysis. The optimization target of each layer is derived by inversely propagating the target of the network via compressed sensing. The method avoids learning an auxiliary network to specially propagate the target, reducing the complexity of network learning. Also, the method could achieve favorable performance owing to the stableness of compressed sensing. The proposed target propagation involves two major computations: dictionary learning and sparse recovery, which both have polynomial complexity. To reduce the complexity, it is interesting to sparsify the structures of the dictionaries and projection matrices, such that some combinatorial algorithms with linear complexity could be adopted [32, 33]. This will be left for our future work. Note that in the letter we focus our interest on the layerwise learning method that optimizes exactly one layer at a time and thus reduces the network analysis to the layer level. Currently, the method seems difficult to achieve BP level performance, especially on large-scale data [31]. The reason is that the finite number of parameters in one layer can hardly handle the matching between poorly-separable features and strong supervision constraints [31]. The problem could be addressed if more than one layer is learned at a time [5, 6].