Attn-HybridNet: Improving Discriminability of Hybrid Features with Attention Fusion

We briefly summarize PCANet’s $2$-layer architecture and provide background on tensor preliminaries in this section.

Tensors are simply multi-mode arrays or higher-order⁴⁴4Also known as modes (dimensions) of a tensor and are analogous to rows and columns of a matrix. matrices of dimension $>2$. In this paper, the vectors are denoted as $\boldsymbol{x}$ are called first-order tensors, whereas the matrices are denoted as $\boldsymbol{X}$ are called second-order tensors. Analogously, matrices of order-$3$ or higher are called tensors and are denoted as $\boldsymbol{\mathscr{X}}$. A few important multilinear algebraic operations utilized in this paper are described below.

Similar to the first layer in PCANet, we begin by collecting all overlapping patches of size $\textit{k}_{1}\times\textit{k}_{2}$ around each pixel from the image $\boldsymbol{I}_{i}$. However, contrary to PCANet the spatial structure of these patches are preserved and instead of matrix and we obtain a $3$-mode tensor $\boldsymbol{\mathscr{X}}_{i}\in\mathbb{R}^{\textit{k}_{1}\times\textit{k}_{2}\times\tilde{m}\tilde{n}}$. The mode-$1$ and mode-$2$ of this tensor represent the row-space, and the column-space spanned by the pixels in the image. Whereas the mode-$3$ of this tensor represents the total number of image patches obtained from the input image. Iterating this process for all the training images, we obtain $\boldsymbol{\mathscr{X}}\in\mathbb{R}^{k_{1}\times k_{2}\times N\tilde{m}\tilde{n}}$ as our final patch-tensor. The matrix factors utilized to generate our convolution-tensorial filters for to the first two modes of $\boldsymbol{\mathscr{X}}$ are obtained by utilizing our custom-designed LoMOI (presented in Alg. 1) in Eq. 9.

where $\hat{\boldsymbol{\mathscr{X}}}\in\mathbb{R}^{r_{1}\times r_{2}\times N\tilde{m}\tilde{n}}$, $\boldsymbol{U}^{(1)}\in\mathbb{R}^{\textit{k}_{1}\times r_{1}}$, and $\boldsymbol{U}^{(2)}\in\mathbb{R}^{\textit{k}_{2}\times r_{2}}$. We discard obtaining the matrix factors from mode-$3$ of tensor $\boldsymbol{\mathscr{X}}$ (which is $\boldsymbol{X}_{3}$) as this is equivalent to the transpose of the patches matrix $\boldsymbol{X}$ in layer 1 of the PCANet which is not factorized in the PCANet while obtaining weights for its convolution filters. Moreover, the matrix factors for this mode span the sample space of the data which is trivial. A total of $L_{1}=r_{1}\times r_{2}$ convolution-tensor filters are obtained from the factor matrices $\boldsymbol{U}^{(1)}$ and $\boldsymbol{U}^{(2)}$ as in Eq. 10.

where ‘$\otimes$’ is the outer-product between two vectors, $i=[1,r_{1}]$, $j=[1,r_{2}]$, $l=[1,L_{1}]$, and $\boldsymbol{U}^{(\textit{m})}_{(:,i)}$ represents ‘$i^{th}$’ column of the ‘$m^{th}$’ factor matrix. Importantly, our convolution-tensorial filters do not require any explicit reshaping as the outer-product between two vectors naturally results in a matrix. Therefore, we can straightforwardly convolve the input images with our obtained convolution-tensorial filters as described in Eq. 11 where $i=[1,N]$ and $l=[1,L_{1}]$ are the image and filter indices respectively.

However, whenever the data is an $RGB$-image, each extracted patch from the image is a $3$-order tensor $\boldsymbol{\mathscr{X}}\in\mathbb{R}^{\textit{k}_{1}\times\textit{k}_{2}\times 3}$ (i.e., RowPixels$\times$ColPixels$\times$Color). After collecting patches from all the training images, we obtain a $4$-mode tensor as $\boldsymbol{\mathscr{X}}\in\mathbb{R}^{{k}_{1}\times\textit{k}_{2}\times 3\times N\tilde{m}\tilde{n}}$ which is decomposed by utilizing LoMOI ($[\hat{\boldsymbol{\mathscr{X}}},\boldsymbol{U}^{(1)},\boldsymbol{U}^{(2)},\boldsymbol{U}^{(3)}]\leftarrow LoMOI(\boldsymbol{\mathscr{X}},r_{1},r_{2},r_{3})$) for obtaining the convolution-tensorial filters in Eq. 12.

where $i\in[1,r_{1}]$, $j\in[1,r_{2}]$, and $k\in[1,r_{3}]$.

Similar to the first layer, we extract overlapping patches from the input images and zero-center them to build a $3$-mode patch-tensor denoted as $\boldsymbol{\mathscr{Y}}\in\mathbb{R}^{k_{1}\times k_{2}\times NL_{1}\tilde{m}\tilde{n}}$ which is decomposed as $[\hat{\boldsymbol{\mathscr{Y}}},\boldsymbol{V}^{(1)},\boldsymbol{V}^{(2)}]\leftarrow LoMOI(\boldsymbol{\mathscr{Y}},r_{1},r_{2})$ to obtain the convolution-tensor filters for layer 2 in Eq. 13.

where, $\hat{\boldsymbol{\mathscr{Y}}}\in\mathbb{R}^{r_{1}\times r_{2}\times NL_{1}\tilde{m}\tilde{n}}$, $\boldsymbol{V}^{(1)}\in\mathbb{R}^{\textit{k}_{1}\times r_{1}}$, and $\boldsymbol{V}^{(2)}\in\mathbb{R}^{\textit{k}_{2}\times r_{2}}$, $i=[1,r_{1}]$, $j=[1,r_{2}]$, and $l=[1,L_{2}]$. We, now convolve each of the $L_{1}$ input images from the first layer with the convolution-tensorial filters obtained as below in Eq. 14.

The number of output images obtained here is equal to $L_{1}\times L_{2}$ which is identical to the number of images obtained at layer 2 of PCANet. Finally, we utilize the output layer of PCANet (Sec. II-A3) to obtain the feature vectors from the minutiae view of the image in Eq. 15.

Despite having close resemblance between the feature extraction mechanism of the PCANet and the TFNet, these two networks capture visibly distinguishable features from the two view of the images as shown in Fig. 1. These plots are obtained by convolving image of a $cat$ with the convolution filters obtained in the first layer of the networks.

Undoubtedly, each of the $L_{1}$ convolution responses within the PCANet is visibly distinct. Whereas the convolution responses within the TFNet shows visual similarity, i.e., the images in a triplet sequence show similarity consecutively. These plots demonstrate that the TFNet emphasize mining the common information from the minutiae view of the data. Whereas the PCANet emphasizes mining the unique information from the amalgamated view of the data. Both these kinds of information are proven beneficial for classification in [17, 15] and motivate the development of HybridNet.

Besides, we also present convolution responses of MPCANet [25] in the comparisons. MPCANet employs tensorized-convolution but differs from our TFNet in two ways: 1) construction of convolution kernels and 2) convolution operation. Technically, the convolution kernel and convolution operation in MPCANet in [25] (and its predecessor in [26]) are combined together as conventional tensor operations, obtaining a) factor matrices for each mode from the patch-tensor and b) n-mode products of factor matrices with the patch-tensor. In other words, the convolution in MPCANet is a n-mode product of the patch-tensor and factor matrices, whereas in TFNet the convolution kernels are obtained by performing outer-procut of factor matrices in Eq. 10 followed by convolution in Eq. 11. From Figure 1, it is visible that the convolution responses in MPCANet is less diversified than those of TFNet although visible resemblance is still observable. We believe that this is because of how the convolution filters and convolution operation are performed in MPCANet as analyzed above.

Similar to the previous networks, we begin the feature extraction process by collecting all overlapping patches of size $\textit{k}_{1}\times\textit{k}_{2}$ around each pixel from the image $I_{i}$. Importantly, the first layer of HybridNet consists of image-patches expressed both as tensors $\boldsymbol{\mathscr{X}}\in\mathbb{R}^{{k}_{1}\times\textit{k}_{2}\times 3\times N\tilde{m}\tilde{n}}$ and matrices $\boldsymbol{X}\in\mathbb{R}^{k_{1}k_{2}\times N\tilde{m}\tilde{n}}$ which are utilized for obtaining weights of convolution filters in layer 1 of HybridNet.

This enables this layer (and the subsequent layers) of HybridNet to learn superior filters as they perceive more information from both views of the data. The weights for the pca-filters are obtained as the principal-eigenvectors as $\boldsymbol{W}_{l_{PCA}}^{1}=mat_{k_{1},k_{2}}(ql(\boldsymbol{X}\boldsymbol{X}^{T}))$, and the weights for convolution-tensor filters are obtained by utilizing LoMOI as $\boldsymbol{W}_{l_{TF}}^{1}=\boldsymbol{U}^{(1)}_{(:,i)}\otimes\boldsymbol{U}^{(2)}_{(:,j)}\otimes\boldsymbol{U}^{(3)}_{(:,k)}$. Furthermore, the output from this layer is obtained by convolving input images with a) the PCA-filters and b) the convolution-tensorial filters in Eq. 16. This injects more diversity to the output in succeeding layer of HybridNet.

Since we obtain of $L_{1}$ pca filters and $L_{1}$ convolution-tensor filters, a total of $2\times L_{1}$ outputs are obtained in this layer.

Similar to the first layer, we begin with collecting all overlapping patches of size $\textit{k}_{1}\times\textit{k}_{2}$ around each pixel from the images. However, contrary to the above layer, the weights of the pca-filters $\boldsymbol{W}_{l_{PCA}}^{2}$ and convolution-tensor filters $\boldsymbol{W}_{l_{TF}}^{2}$ are learned from the data obtained by convolving input images with the pca filters and the convolution-tensor filters i.e. both $\boldsymbol{I}_{i_{PCA}}$ and $\boldsymbol{I}_{i_{TF}}$. Hence both the patch-matrix $\boldsymbol{Y}\in\mathbb{R}^{k_{1}k_{2}\times 2L_{1}N\tilde{m}\tilde{n}}$ and the patch-tensor $\boldsymbol{\mathscr{Y}}\in\mathbb{R}^{k_{1}\times k_{2}\times 2L_{1}N\tilde{m}\tilde{n}}$ contain image patches obtained from $[\boldsymbol{I}_{i_{PCA}},\boldsymbol{I}_{i_{TF}}]$. This enables the hybrid filters to assimilate more variability present in the data while obtaining weights of their convolution filters as evident in Fig. 3.

The plot in Fig. 3(a) compares the eigenvalues obtained in layer 2 (we exclude eigenvalues from layer 1 as they completely overlap as their expected behavior). The leading eigenvalues obtained in layer 2 of the HybridNet by $principal$ $components$ has much higher magnitude than the corresponding eigenvalues obtained by $principal$ $components$ in PCANet. This demonstrates that the $pca$ filters in the HybridNet capture more variability than those in the PCANet.

Similarly, Fig 3(b) compares the core-tensor strength in different layers of the HybridNet and the TFNet. We plot the norm of the core-tensor for both the networks as the values in the core-tensor is analogous to eigenvalues for higher-order matrices, and its norm signifies the compression strength of the factorization [27]. Again, the norm of the core-tensor in layer 2 of HybridNet is much lower than that of the TFNet, suggesting relatively higher factorization strength in HybridNet. Besides, as expected, the norm of the core-tensor in layer 1 for both the networks coincides and signifies equal factorization strength at this layer. Consequently, this leads to attainment of better-disentangled feature representations with the HybridNet and hence enhances its generalization performance over the PCANet and the TFNet by integrating information from the two views of the data.

In the second layer, the weights of pca filters are obtained by principal components as $\boldsymbol{W}_{l_{PCA}}^{2}=mat_{k_{1},k_{2}}(ql(\boldsymbol{Y}\boldsymbol{Y}^{T}))$ and the weights for convolution-tensor filters are obtained as $\boldsymbol{W}_{l_{TF}}^{2}=\boldsymbol{V}^{(1)}_{(:,i)}\otimes\boldsymbol{V}^{(2)}_{(:,j)}$, where the matrix factors are obtained using LoMOI $[\hat{\boldsymbol{\mathscr{Y}}},\boldsymbol{V}^{(1)},\boldsymbol{V}^{(2)}]\leftarrow LoMOI(\boldsymbol{\mathscr{Y}},r_{1},r_{2})$. Analogous to the previous networks, the output images from this layer of HybridNet are obtained by a) convolving the $L_{1}$ images corresponding to the output from the PCA-filters in the first layer with the $L_{2}$ $pca$ filters obtained in the second layer (Eq. 17), and b) convolving the $L_{1}$ images corresponding to the output from the convolution-tensorial filters in the first layer with the $L_{2}$ convolution-tensorial filters obtained in the second layer (Eq. 18). This generates a total of $2\times L_{1}\times L_{2}$ output images in this layer.

The output images obtained from the pca-filters ($\boldsymbol{O}_{i_{PCA}}^{l}$) in layer 2 are then processed with the output layer of the PCANet (Sec. II-A3) to obtain $f_{i_{PCA}}$ as the information from amalgamated view of the image. Similarly, the output images obtained from the convolution-tensor filters ($\boldsymbol{O}_{i_{TF}}^{l}$) are processed to obtain $f_{i_{TF}}$ as the information from minutiae view of the image. Finally, these two kinds information are concatenated to obtain the hybrid features as in Eq. 19.

The whole procedure of obtaining hybrid features is detailed in Alg. 2. Although these hybrid features bring the best of both the common and the unique information obtained respectively from the two views of the data, they still suffer from feature redundancy problems induced by the spatial pooling operation in the output layer. To alleviate this drawback, we propose the Attn-HybridNet, which further enhances the discriminability of the hybrid features.

To calculate the computational complexity of Attn-HybridNet, we assume the HybridNet is composed of two-layers with a patch size of $k_{1}=k_{2}=k$ in each layer followed by our attention-based fusion scheme.

In each layer of the HybridNet, we have to compute the time complexities arising from learning convolution weights from the two views of the data. The formation of the zero-centered patch-matrix $\boldsymbol{X}$ and zero-centered patch-tensor $\boldsymbol{\mathscr{X}}$ have identical complexities as $k^{2}(1+\tilde{m}\tilde{n})$. The complexity of eigen-decomposition for patch-matrix and tensor factorization with LoMOI for patch-tensor are also identical and equal to $\mathcal{O}((k^{2})^{3})$, where $k$ is a whole number $<$ 7 in our experiments. Further, the complexity for convolving images with the convolution filters at stage $i$ requires $L_{i}k^{2}mn$ flops. The conversion of $L_{2}$ binary bits to a decimal number in the output layer costs $2L_{2}\tilde{m}\tilde{n}$, where $\tilde{m}=m-\lceil{\frac{k}{2}}\rceil,\tilde{n}=n-\lceil{\frac{k}{2}}\rceil$ and the naive histogram operation for this conversion results in complexity equal to $\mathcal{O}(mnBL_{2}log2)$.

The complexity of performing matrix multiplication in Attn-HybridNet is $\mathcal{O}\Big{(}2L_{1}B\big{(}d(1+2^{L_{2}})+2^{L_{2}}\big{)}\Big{)}$ which can be efficiently handled with modern deep learning packages like Tensorflow [29] for stochastic updates. To optimize the parameters in the attention-based fusion scheme ($\boldsymbol{W}$, $f_{c}$, and $w$), we back-propagate the loss through the attention network until convergence of the error on the training features.

In our experiments, we utilized a two-layer architecture for each of the networks in comparison, while the number of convolution filters in the first and the second layer are optimized via cross-validation on the training datasets. The dimensionality of the feature vectors extracted from PCANet and TFNet is then $BL_{1}2^{L_{2}}$, where $L_{1}$ and $L_{2}$ are the number of convolution filters in layer 1 and layer 2 respectively. The dimensionality of feature vector with HybridNet is then $2BL_{1}2^{L_{2}}$. We utilized Linear-SVM [30] as the classifier incorporating features obtained with the PCANet, TFNet, and the HybridNet.

The attention-based fusion scheme is performed by following the procedure as described in Alg. 3, where we obtained the optimal attention dimension on training data for the context level feature vector $w\in\mathbb{R}^{d}$ in $[10,50,100,150,200,400]$. The obtained attention features i.e. $F_{attn}\in R^{BL_{1}2^{L_{2}}}$ are utilized with $softmax$-layer for classification. The parameters of attention-based fusion scheme ($\boldsymbol{W}$, $f_{c}$, and $w$) are optimized via back-propagation on the classification loss implemented in TensorFlow [29]. We observed that the attention-network’s optimization took less than 15 epochs for convergence on all the datasets utilized in this paper.

The details of datasets and hyper-parameters are as below:

• 1

  MNIST variations [31], which consist of gray scale handwritten digits of size $28\times 28$ with controlled factors of variations such as background noise, rotations, etc. Each variation contains $10K$ training and $50K$ testing images. We cite the results for baselines techniques like 2-stage ScatNet [32] (ScatNet-2) and 2-stage Contractive auto-encoders [33] (CAE-2) as reported in [10]. The parameters of HybridNet (and other networks) are set as $L_{1}$ = 9, $L_{2}$ = 8, k₁ = k₂ = 7, with a block size of $B=7\times 7$ keeping size of overlapping regions equal to half of the block size for feature pooling.

• 2

  CUReT dataset [34], consists of $61$ texture categories, where each category has images of the same material with different poses, illumination, specularity, shadowing, and surface normals. Following the standard procedure in [34, 10] a subset of $92$ cropped images were taken from each category and randomly partitioned into train and test sets with a split ratio of 50%. The classification results are averaged over $10$ different trails with hyper-parameters as $L_{1}$ = 9, $L_{2}$ = 8, k₁ = k₂ = 5, block size $B$ = $50\times 50$ with overlapping regions equal to half of the block size. Again, we cite the results of the baselines techniques as published in [10].

• 3

  ORL⁵⁵5The ORL database is publicly available and can be obtained from the website: http://www.uk.research.att.com/facedatabase.html and Extended Yale-B [35] datasets are utilized to investigate the performance for face recognition. The ORL dataset consists of 10 different images of 40 distinct subjects taken at different times, varying the lightning, facial expression, and facial details. The Extended Yale-B dataset consists of face images from 38 individuals under 9 poses and 64 illumination conditions. The images in both the datasets are cropped to size $64\times 64$ pixels followed by unit length normalization. The classification results are averaged over $5$ different trails by progressively increasing the number of training examples⁶⁶6The training and test split are obtained from http://www.cad.zju.edu.cn/home/dengcai/Data/FaceData.html with hyper-parameters as $L_{1}$ = 9, $L_{2}$ = 8, k₁ = k₂ = 5 with a non-overlapping block of size $B$ = $7\times 7$.

• 4

  CIFAR-10 [11] dataset consists of $RGB$ images of dimensions $32\times 32$ for object recognition consisting of $50K$ and $10K$ images for training and testing respectively. These images are distributed among $10$ classes and vary significantly in object positions, object scales, colors, and textures within each class. We varied the number of filters in layer 1 i.e., $L_{1}$ as 9 and 27 and kept the number of filters in layer 2 i.e. $L_{2}$ = 8. The patch-size k₁ and k₂ are kept equal and varied as 5, 7, and 9 with block size $B$ = $8\times 8$. Following [10] we also applied spatial pyramid pooling (SPP) [36] to the output layer of HybridNet (and similarly to the out layer of other networks). We additionally applied PCA to reduce the dimension of each pooled feature to 100⁷⁷7Results does not vary significantly on increasing the projection dimensions.. These features are utilized with Linear-$SVM$ for classification and Attn-HybridNet for obtaining attention features $F_{attn}$.

• 5

  CIFAR-100 [11] dataset closely follows CIFAR-10 dataset and consists 50k training and 10k testing images roughly distributed among 100 categories. We use the same experimental setup as in CIFAR-10 on this dataset.

On face recognition datasets, we compare the performance of Attn-HybridNet and HybridNet against three recent baselines: Deep-NMF [37], PCANet+ [38], and PCANet-II [39]⁸⁸8The paper did not provide its source code and the results are based on our independent implementation of their second order pooling technique.. On MNIST variations and CuReT dataset we select the baselines as in [10]. Besides, the hyper-parameters of these baselines are set equal to those in HybridNet.

On CIFAR-10 dataset, we compare our schemes against multiple comparable baselines such as Tiled CNN [40], CUDA-Convnet [41], VGG style CNN (VGG-CIFAR-10 reported by [42]), K-means (tri) [43], Spatial Pyramid Pooling for CNN (SPP) [44], Convolution Bag-of-Features (CBoF) [45], and Spatial-CBoF [4]. Note that, we do not compare our schemes against schemes which aim to either compress deep neural networks or transfer pre-learned CNN filters such as in [46, 47, 48, 49, 50, 51] as these schemes do not train a deep network from scratch whereas our proposed schemes and comparable baselines do so. Besides, we also report the performances of ResNet [19] and DenseNet [52] on CIFAR datasets, as mentioned in their respective publication.

Lastly, we perform a qualitative case study on the CIFAR-10 dataset by studying the performance of baselines and our scheme by varying the size of the training data. Besides, we also studied the effect on the discriminability of hybrid features with attention-based fusion scheme.