Attention Mechanism Meets with Hybrid Dense Network for Hyperspectral Image Classification

Hyperspectral Imaging (HSI) systems based on collections of electromagnetic spectrum, ranging from visible to near-infrared region, reflected by the objects of interest [1]. The images thus obtained are usually generated by a pre-configured HSI camera installed on either mobile (e.g. satellites, drones, aircrafts) or static (e.g., indoor, rooms, labs) setups depending upon the problem at hand [2, 3, 4, 5, 6, 7, 8]. Thereby, HSI sensors gather a huge amount of data from hundreds of contiguous spectral bands [9, 10].

Such big and rich HSI dataset, including the different spectral bands data related by the (spatial) geo-located position, may contain hidden information and patterns. HSI Classification (HSIC) [11] aims at discover, detect, identify and recognize such patterns. However, the spectral dataset size usually increases combinatorially with the problem size (e.g. the area, the resolution), leading to the curse of dimensionality and thus making traditional HSIC methods inefficient [12]. To mitigate such issues, several dimensionality reduction and feature selection techniques have been proposed [13, 14]. Conventional feature extraction/selection methods rely on hand-crafted features, however, due to the spatial variability of spectral information [15], the extraction of discriminative and most informative features is still a big challenge [16].

Hand-crafted features may be insubstantial in the case of HSI data, therefore, it is challenging to achieve a trade-off between discriminability and robustness on features which also considerably differ on the different HSI datasets. Furthermore, the process of feature design and selection is strongly affected by the designer and architect knowledge and skills [17, 16, 18, 19]. To such a purpose, an automatic approach to hierarchically identify the features was developed by Hinton back in 2006 [17, 20, 21], based on a deep learning model developed in growing semantic layers until a desirable representation is achieved. Similarly, other models have been proposed on feature learning and classification as well [22, 17, 23, 24], automatically learning and improving the underlying system representation from the available data without any prior knowledge. They can extract both linear and non-linear features, thus capable of handling HSI data in both spatial and spectral domains [25].

In nature, HSI datasets are usually non-linear due to the undesired light-scattering phenomena given by land cover objects and particles in the atmosphere [26]. Thus rendering the use of linear transformation or feature learning methods [27] for HSIC. To overcome the non-linearity issues, Convolutional Neural Network (CNN) was proposed to extract both high as well as low-level features which ultimately lead to the extraction of abstract and invariant features [28]. As a result, 2D CNN achieved remarkable performance but unfortunately not so good for HSIC due to the missing channel-related information, i.e. 2D CNN are not able to learn spectrally discriminative features. Unlike 2D CNN, 3D ones can jointly extract the spatial-spectral information for HSIC providing higher accuracy than 2D CNN [29]. However, 3D CNN models are computationally-/time-intensive due to the high number of parameters involved by 3D convolutional filters on each layer.

For instance, in [30], a Spatial-Spectral Residual Network (SSRN) implemented a 3D CNN residual network based on ResNet [31]. Despite SSRN achieves remarkable classification results, the summation method used to aggregate features at each layer requires output feature maps to have consistent scale as the residual feature maps. Hence each layer has self weights which overall lead to an explosion of network parameters [26]. Thereby, high accuracy comes at expenses of the computational power, i.e. SSRN is more complex than traditional 3D CNN. Similarly, [17] proposed a fast and compact 3D CNN (FC-3D-CNN) model to overcome the limitations of computational cost and reduce the number of training parameters. FC-3D-CNN achieves better results in a computationally efficient manner than SSRN due to the reduced spectral information used in the experimental process.

In general, CNN models tend to poorly perform especially in the case of pixels of different classes but similar texture over contiguous spectral bands [32]. To deal with that, [33, 34, 35] proposed hybrid models that combine the power of 2D and 3D convolutional layers to extract high and low-level features i.e., extraction of abstract and invariant features. The hybrid models achieve better accuracy than state-of-the-art 2D and 3D CNN solutions. Despite the high accuracy, hybrid models still require a large number of parameters as compared to the 3D CNN network [17] while, on the other hand, 3D CNN/SRNN has a longer processing time than hybrid models. Therefore, inception models have proved that the network topology significantly affects the complexity and the accuracy [36]. Recently, graph convolutional network (GCN) shows the superiority in HSCI, Hong et al. [37] proposed a novel GCN in a mini-batch fashion, called miniGCN, which solves the problem of large-scale graph computation and learning. Apart from the complexity and accuracy trade-off, all the inception models have one common property i.e., a split-transform-merge strategy which proved to be a good strategy for HSIC. Traditional 2D, 3D, hybrid, and inception models exploit the fixed convolution kernel size, however, HSI class distribution is complicated thus conventional CNN with fixed kernel size is not flexible enough. Convolutions with different spatial sizes may capture more discriminative and important information for pixel-based HSIC.

Nowadays, an attention mechanism has been extensively used to suppress redundant information while extracting features for classification. SENet [38] was the first network proposed to suppress the redundant features by weighting channel direction features. The work [39] proposed CBAM (convolutional block attention module) combines spatial attention with channel attention through pooling whereas the work [40] proposed a NLNet which combines the convolution operations with matrix multiplication operations to capture the long-range relationship in the global space. A combination of SENEt and NLNet was proposed in [41], which consists of a simplified lightweight module GCNet for effectively extracting global context. Very recently, a novel transformer framework was rapidly developed and its spectral version, called SpectralFormer, was for the first time proposed with the application to HSIC, yielding state-of-the-art classification performance [42].

Though, attention networks have achieved remarkable results for HSIC based on the internal architecture of the attention module. These works, to some extent, put attention weights in either one or two dimensions and ignore the rest of the HSI dimensions. For instance, single and double attention networks were proposed in [43, 44], in which the work [43] only consider the spectral information and ignore the spatial information, whereas, the work [44] was proposed to reduce the interference between the spatial and channel information. The work [45] was proposed to jointly explore the spectral and spatial information, where spectral-spatial dimensions were weighted by the spectral-spatial attention module. The combination of attention in more or less in one or two dimensions may improve the performance, however, it is highly recommended to integrate all channel information for better classification.

Wang, et.al. [46] significantly improved the squeeze and excitation structure attention mechanism proposed in [38], reducing the model complexity by a local cross-channel interaction strategy without any preprocessing, i.e., dimensional reduction. Zheng, et.al. [47] worked to overcome the limitations of inconsistent class ratio and over-parameterization using a stratified sample-based training strategy. While the spectral attention module was proposed to render the soft band selection process to further refine the redundant spectrum information. However, all these spatial or spectral attention models are to some extent isolated in which the spatial attention ignores to make a connection between spectral dimension whereas, the spectral attention ignores the spatial connection and uses only the correlation between different spectral bands.

Moreover, it has been proven fact that accuracy improves while increasing the depth of the model, thus require more parameters and higher computational burden which makes optimization an NP hard problem. Therefore, to overcome the aforesaid issues, this work explicitly investigates the possibilities of combining the core idea of 2D as well as 3D inception models into an Hybrid attention architecture to boost the pixel-based HSIC performance. We tested the model on several publicly available HSI datasets which shows competitive results compared to the state-of-the-art methods. Our proposed pipeline achieved an overall accuracy of 97% for the Indian Pines dataset, 100% for Botswana, 99% for both Pavia University, Pavia Center, and Salinas datasets, respectively. In a nutshell, the contributions made in this work as summarized as follows.

1. 1.

   A densely connected hybrid inception net is proposed to enrich the spatial-spectral feature learning process. Different from the conventional inception model which consists of a single branch in each block, the proposed densely connected hybrid blocks are composed of parallel multiple sized filters which significantly improves the propagation and reuse of features with less number of tune-able parameters. Moreover, the proposed hybrid inception net comprehensively utilizes features of different scales from HSI dataset.

2. 2.

   A triple-branch attention fusion block is introduced which boosts the robustness of the discriminative network. As compared with recently published attention blocks for HSIC, the proposed pipeline simultaneously model interactions across different spectral bands and spatial regions by re-weighting the significance of features. The triple-branch attention block adaptively emphasizes the important information and significantly suppresses the redundant and ineffective information.

3. 3.

   A hybrid spectral convolutional block is used to reduce the required number of parameters for the HSIC model. Moreover, the activation inducted convolutional layer can further improve the non-linear representation capacity of the whole network.

The rest of the paper is structured as follows. Section II presents the pipeline proposed in this paper. Section III describes the experimental settings along the metrics used to compute the accuracies. Sections III-A-III-C exhibits the experimental datasets used to conduct the experiments and to validate the proposed methodology. Moreover, sections III-A-III-C demonstrates the experimental results as compared with the state-of-the-art methods proposed in the literature. Finally, section IV concludes the paper with possible future research directions.

Lets assume $R=[r_{1},r_{2},r_{3},...,r_{L}]^{T}\in R^{L\times n}$ be the HSI cube, where $n=N\times M$ samples associated with $C$ classes and $L$ band images. Each $r_{i}=(r_{i},c_{j})$ where $c_{j}$ be the class associated with $r_{i}$ sample. In nature, $r_{i}$ exhibit high intra-class variability and inter-class similarity, overlapping and nested regions. Therefore, HSI cube has been first divided into small spatial patches to overcome the aforesaid issues. For each patch, the ground truths are formed based on the central pixel of the patch. Principle Component Analysis (PCA) has been used before creating the patches which eliminate the redundancy among the band images i.e. $L\rightarrow B$, where $B\ll L$.

The patching process creates neighboring patches $P\in R^{S\times S\times B}$ centered at the spatial location $(a,b)$ covering $S\times S$ spatial windows [17, 33]. The total of $Z$ patches given by $(U-S+1)\times(V-S+1)$. Thus, in total, these patches cover the width from $\frac{a+(S-1)}{2}$ to $\frac{a-(S-1)}{2}$ and height from $\frac{b+(S-1)}{2}$ to $\frac{b-(S-1)}{2}$ [33]. The $Z$ patches are first convolved with a kernel function which computes the sum of the dot product between the input patch and kernel function to introduces the nonlinearity [17, 33, 48, 21]. The activation maps for spatial-spectral position $(x,y,z)$ at $i-th$ feature map and $j-th$ layer can be represented as $v_{i,j}^{(x,y,z)}$.

where $d_{i-1}$ be the total number of feature maps at $(i-1)-th$ layer, $w_{i,j}$ and $b_{i,j}$ be the depth of the kernel and bias, respectively. Moreover, $2\gamma+1$, $2\delta+1$, and $2\nu+1$ be the height, width, and depth of the kernel [17]. Similarly, 2D modules do the same process with 2D input as well as the 2D kernel function. In both 3D and 2D layers, the kernel is striding over the input to cover the whole spatial dimension. More specifically, as the proposed model combines the power of 3D and 2D kernel functions, thus, 2D convolution $V^{x,y}_{i,j}$ represents the activation value of $i-th$ feature map at $(x,y)$ spatial position on $j-th$ layer and can be formulated as $v^{x,y}_{i,j}$ and finally can be formed as follows.

where $2\gamma+1$ and $2\delta+1$ be the height and width of the kernel, respectively. In short, the proposed hybrid AfNet convolutional filters are as follows with the input of $9\times 9\times 15$. The size of 3D filters are $3D_{1}=(7\times 7\times 9)$, $K_{1}^{1}=7,K_{1}^{2}=7$, $K_{1}^{3}=9$, $3D_{2}=(5\times 5\times 7)$, $K_{2}^{1}=5,K_{2}^{2}=5$, $K_{2}^{3}=7$, and $3D_{3}=(3\times 3\times 5)$, $K_{3}^{1}=3,K_{3}^{2}=3$ and $K_{3}^{3}=5$ for each layer on each block with different number of filters, i.e., $(30,20,10)$ for first block, $(40,20,10)$ for second block and $(60,30,10)$ for third block. Similarly the size of 2D filters are $2D_{1}=(3\times 3)$, $K_{1}^{1}=3,K_{1}^{2}=3$, $2D_{2}=(3\times 3)$, $K_{2}^{1}=3,K_{2}^{2}=3$, and $2D_{3}=(1\times 1)$, $K_{3}^{1}=1,K_{3}^{2}=1$ for each layer on each block with different number of filters, i.e., $(16,32,64)$ for first block, $(16,32,64)$ for second block and $(16,32,64)$ for third block. A 2D fusion module has been used to incorporate the information learned hierarchically at different blocks. Finally, a 2D convolutional layer is used with $(1\times 1)$ kernel size with total of $128$ filters to better represent the low to high level information.

To decrease the number of spectral-spatial feature maps, nineteen densely connected 3D and 2D convolutional layers are used prior to the flatten layer to make sure the convolutional process discriminate the spatial information while considering different spectral bands with no loss and less number of parameters to boost the performance [17]. The weights are randomized initially and optimized using Adam optimizer based on back-propagation with a soft-max loss function. Later the randomized weights are updated using a mini-batch size of $256$ for $50$ epochs. The overall structure of the proposed hybrid AfNet using the Indian Pines dataset as an example is presented in Figure 1.

The experimental results explained in this work have been obtained through Google Colab, an online platform to execute any python environment with Graphical Process Unit (GPU), up to 358$+$ GB of cloud storage, and up to 25 GB of Random Access Memory (RAM). In all the stated experiments (not only for the proposed method but for all comparative methods as well), the training, validation, and test sets are randomly divided into three parts; 15%/15%/70% (i.e., Training/Validation/Test sets).

For fair comparative analysis and to make the claims more reliable, the learning rate for all experimental methods are set to 0.001, Relu as the activation function for all hidden layers except the final (output) later on which Softmax activation function is used. Patch size is set to $9\times 9$ for all experimental results, and $15$ most informative spectral dimensions were selected using PCA to reduce the computational burden in terms of time and space. All the models are evaluated on 100 epochs without batch normalization.

To compute the experimental results, several metrics such as Average Accuracy (AA), Overall Accuracy (OA), and Kappa ($\kappa$) have been used. Where $\kappa$ metric is known as a statistical metric that considered the mutual information regarding a strong agreement among the classification and ground-truth maps. AA represents the average class-wise classification performance, whereas the OA is computed as the number of correctly classified examples out of the total test examples.

where

where $C$ be the total number of classes presents in HSI dataset. Moreover, $FP$, $FN$, $TP$, and $TN$ are false positive, false negative, true positive, and false negative, respectively.

Convolution Neural Networks (CNNs) are known to overcome the nonlinearity issues with fixed kernel sizes which are not flexible enough because these kernels are specific and not conducive to feature learning thus impairs classification accuracy. However, CNN with different kernel sizes may capture more discriminative and important features. Thus, taking into account the aforesaid advantages, this work proposed a hybrid (3D-2D) Inception net with an attention mechanism to boost the classification performance. The proposed attention fused hybrid network (AfNet) used attention-based six parallel hybrid sub-nets with different kernels in each sub-block to enhance the final ground-truth maps. The proposed AfNet selectively filters out the discriminative feature i.e., the critical features for classification. AfNet has been tested on several Hyperspectral datasets and shows competitive results as compared to the state-of-the-art models except for a few expensive choices.