3D-RCNet: Learning from Transformer to Build a 3D Relational ConvNet for Hyperspectral Image Classification

With the widespread application of deep learning, HSI classification tasks have entered a phase of rapid development, wherein convolutional neural networks play a crucial role. This primarily includes methods using 1D ConvNet, 2D ConvNet, and 3D ConvNet. HSI classification methods based on 1D ConvNet and 2D ConvNet typically focus on either spectral or spatial information [19][20][21][22][23], failing to effectively extract the rich spectral and complex spatial information contained in HSI data. Therefore, current research predominantly focuses on 3D ConvNet, which demonstrate better extraction performance for three-dimensional structured data. The convolutional kernels of 3D ConvNet slide along both spatial and spectral dimensions of HSI data, thus processing the spatial location and spectral band of each pixel during feature extraction. Liu et al.’s team validated the effectiveness of 3D ConvNet in feature extraction from HSI data through comparisons with 1D ConvNet and 2D ConvNet combined with principal component analysis [24]. Ladi et al. demonstrated that the structure of 3D ConvNet exhibits superior performance in processing the three-dimensional data of HSI compared to 1D ConvNet and 2D ConvNet [25]. Chen et al. proposed a model combining hyperspectral target detection and 3D ConvNet, achieving effective HSI classification [26]. Jiang et al. proposed a 3-D separable ResNet, which demonstrated excellent performance in HSI classification with small training samples[27].Due to its three-dimensional structure, 3D ConvNet can avoid the information loss caused by traditional convolutional downsampling methods and effectively extract both spectral and spatial features, making it widely used in HSI classification. Inspired by these, our proposed 3D-RCNet continues the module design similar to the 3D ConvNet three-dimensional structure at the shallow level, which is used to extract spectral and spatial features efficiently. However, 3D ConvNet use static convolutional kernels when processing different datasets, which limits their adaptability and flexibility in capturing features of diverse data distributions.

Dosovitskiy et al. first applied the Transformer model to computer vision tasks, proposing the Vision Transformer (ViT), which divides images into fixed-size patches, then flattens these patches into a one-dimensional sequence to be processed by the Transformer model[13]. The ViT architecture is highly flexible, capable of accommodating images of different sizes and varying numbers of categories, and it achieves better performance than traditional ConvNet when trained on large-scale datasets.

Recently, with the deepening research of ViT in the field of machine vision, especially in image classification tasks, using ViT to extract HSI data features for classification has become a major research direction. Hong et al. proposed using grouped spectral embeddings instead of all spectral bands for feature extraction to model local detailed spectral differences[14]. This work added cross-layer skip connections in multiple encoders of the Transformer to reduce information loss during layer-by-layer propagation. Sun et al. proposed a model combining spectral-spatial feature tokenization with the Transformer, improving the accuracy of HSI classification by segmenting and encoding spatial features of the images[15]. Jiang et al. proposed a Graph Generative Structure-Aware Transformer architecture that integrates spatial-spectral features and graph structure information in HSI classification to enhance classification performance[28]. In HSI classification tasks, ViT demonstrates excellent feature extraction performance for HSI data with different scale three-dimensional structures by leveraging self-attention mechanisms and global feature capturing capabilities. However, when training on HSI data, the ViT has a high demand for training samples, and the self-attention mechanism entails significant computational complexity. Additionally, due to the excessive flexibility of the self-attention mechanism, the training process may become unstable.

Both 3D ConvNet and ViT exhibit excellent performance in HSI classification tasks. However, due to the three-dimensional structural characteristics of HSI data and its complex spectral and spatial information, single 3D ConvNet and ViT models have certain application limitations. 3D ConvNet are better at handling three-dimensional structural data but cannot effectively manage long-range dependencies, making them more suitable for capturing low-level features of HSI data at shallow layers. ViT, with its attention mechanism, can handle long-range dependencies well, but its computational complexity is exponentially influenced by the scale of image data, making it more suitable for capturing spectral and spatial global information of HSI data at deeper layers. In their LeViT model, Graham et al. combined Transformer structures with ConvNet, optimizing the balance between inference speed and accuracy, significantly outperforming previous convolutional and ViT models[29]. Fu et al. proposed IncepFormer, using deep convolutional and pooling for spatial reduction followed by attention operations, and employed a pyramid structure to capture rich features of objects of different sizes[30]. In MobileViT, the deeper layers of MobileNetv2 were replaced with their proposed MobileViT blocks[31]. In ParCNet, Zhang et al. introduced Position-Aware Circular Convolutions (ParC) to capture global features and designed a hybrid structure by combining the proposed ParC operations with traditional convolutions[32].

Recently, researchers have introduced hybrid structures into HSI classification. Ghaderizadeh et al. proposed a Multi-Scale Dual-Branch Residual Spectral-Spatial Network (MDBRSSN), which uses 3D convolutions with different kernels in parallel within two branches to extract spatial-spectral features, subsequently inputting these features into an attention mechanism for feature extraction[33]. This work fused the features extracted by the two branches, followed by 2D ConvNet and GAP for feature compression, and achieved good classification results after passing through a fully connected layer. The Spectral-Spatial Feature Tokenization Transformer (SSFTT) employs 3D ConvNet to extract shallow spectral features and 2D ConvNet to extract shallow spatial features, which are then Gaussian-weighted and tokenized before being input into the Transformer module for feature learning[15], yielding effective results in HSI classification tasks. The T-SST-based classification framework (T-SST-L) uses ConvNet to extract spatial features and densely connected Transformer to capture sequential spectral relationships[34]. These studies combine ConvNet and Transformer modules from a model structure perspective, effectively mixing ConvNet and Transformer modules to improve model accuracy.

While previous hybrid models combining 3D ConvNet and ViT have shown promise in HSI classification, they typically rely on straightforward concatenation of these technologies, which can be limiting. Building on this foundational hybrid approach, our research takes a significant leap forward by more deeply integrating the convolutional capabilities of 3D ConvNet with the self-attention mechanisms of Transformers. This integration is realized through our novel 3D-RCNet model, which utilizes the specially designed 3D-RCBlock to seamlessly blend the strengths of both architectures, resulting in enhanced classification performance and operational efficiency.

Network structure: Fig. 1 shows the overall framework of our proposed 3D-RCNet, which is designed for HSI classification and seamlessly integrates the 3D-RCBlock, equipped with a 3D relational convolutional operation. The framework is divided into three stages: the first stage is the sample extraction for HSI data; the second stage involves multi-layer feature extraction based on the 3D-RCNet framework; the third stage performs HSI classification using Global Average Pooling.

Sample extraction: In this stage, a data cube block with spatial dimensions S and spectral bands L is extracted as a sample, where the block size is defined as S × S × L. The block, centered around a pixel within its domain window in the HSI data, assigns the label of this central pixel to the entire data cube block. S is set as an odd number to ensure the central pixel is properly positioned for classification. Subsequently, this extracted data cube is input into the Stem layer, where the number of spectral bands is compressed and initial spectral information is extracted.

HSI feature multi-stage feature extraction architecture: This multi-stage feature extraction architecture seamlessly incorporating 3D-RCBlock, utilizing a hybrid design of 3D ConvNet and 3D-RCBlock. In Stage 1, convolutional downsampling is applied to the input data to increase channel quantity while simultaneously reducing the resolution of the HSI to minimize feature information loss. Basic semantic-free features are extracted using a 3D-ConvBlock. Stage 2 continues with a similar setup, employing two 3D-ConvBlock to extract more complex and abstract features. In Stage 3 and Stage 4, 3D-RCBlock take over, initially reducing resolution and increasing channel count through 3D relational convolutional downsampling, and subsequently extracting more abstract and semantic features. This downsampling strategy across the four stages effectively reduces both spectral and spatial resolution at rates of [2,4,8,16], facilitating a progressive enhancement in feature representation. Experiments in Section IV-C have shown that the framework, which utilizes 3D-ConvBlock in shallow layers and 3D-RCBlock in deeper layers, exhibits good performance. Moreover, ablation studies in Section V demonstrate that 3D-RCBlock can more effectively extract high-level semantic features in deeper layers.

HSI classification stage: By employing global average pooling, the entire feature map is compressed into a fixed-length vector. This global feature vector captures the comprehensive information of the image, which not only enables more efficient classification by the model but also enhances its ability to generalize to positional changes and unseen data within the input image.

This section will elaborate on the design philosophy and core mechanisms of our proposed 3D-RCBlock, which combines the advantages of traditional convolution and self-attention mechanisms. We will use the same input data to compare the effectiveness of traditional convolution, self-attention mechanisms, and 3D-RCBlock in feature extraction, and will present Multiply-Accumulate Counts (MACs) for these three approaches. Fig. 2 displays the feature extraction processes utilized by these three modules. The input is denoted as I and the output as O, with H and W indicating the height and width of the input data space, and S representing the number of spectral bands. The design specifics and benefits of our 3D-RCBlock are detailed through its data feature extraction process.
Convolutional: Convolutional aggregates information by considering the local domain of each element of the input features. For convolution, sequential feature extraction using overlapping windows for input I enables effective extraction of local features and the parameters of the convolutional layer are shared over the entire image, reducing the overall number of parameters. Most of all, the translation invariance of convolutional can enable the model to process images at different locations and better generalise to new data, which can be expressed as follows:

Where O is the output feature map, I is the input image or feature map, and K is the convolutional kernel. However, this reuse of the same convolutional kernel parameters at different locations prevents the convolutional from performing adaptive extraction of features.

As shown in Fig. 2(a), when 3D ConvNet is used to process HSI data, the input I is H×W×S, the depthwise convolutional kernel is 3×3×3. And the output of the convolutional is also H×W×S, the MACs for depthwise 3D convolutional operation is $H\times W\times S\times 3^{3}\times C$.
Self-attention: In self-attention, three learnable weight matrices, $W^{Q}$, $W^{K}$, and $W^{V}$, are predefined to facilitate the learning of multiple meanings associated with Query, Key, and Value. Each input token is linearly mapped through these matrices to produce the query matrix Q, key matrix K, and value matrix V. The attention score is calculated using the dot product between the query matrix Q and key matrix K, while the score weights are derived through the softmax function. This function models the relationships within the image, enabling effective capture of dependencies of different pixels and enhancing image understanding. The self-attention mechanism then aggregates embeddings by computing a weighted sum, with weights based on the pairwise relationships among the embeddings, which can be expressed as follows:

Where $q_{ijl}$ is a row of the query matrix representing a particular query vector. $q_{ijl}$ is used in the transpose $K^{T}$ dot product operation on the key matrix to compute the similarity of $q_{ijl}$ to all keys. $K$ and $V$ are flatten features, each of which consists of $N=H\times W\times S$ feature vectors, shown as follows:

where $d$ is the dimension of attention heads. Generally $C=n_{head}d$, and $n_{head}$ is the number of attention heads. The corresponding theoretical MACs is $2\times(H\times W\times S)^{2}\times C$. It can be seen that self-attention does not require a specific setting for the resolution, is highly flexible. However, for high-resolution inputs, self-attention is computational cost because its computational complexity is proportional to the square of the input resolution.
3D-RCBlock: By analysing convolutional and self-attention, we design an 3D-RCBlock that combines the advantages of both to enable better extraction of feature information on 3D data and reduce its computational effort. Specifically, for the input feature I, we use the same sliding overlapping window as convolutional for data extraction, with the window size set to 3. We perform attention operations within the extracted window, but unlike self-attention, we use the centre pixel within the window as query matrix Q, which can be expressed as follows:

where m, n, and z represent the computation of the attention weights on three independent dimensions with the corresponding values $V_{(}i-m)(j-n)(l-z)$. $e^{-(q_{ijl}+K_{(}i-m)(j-n)(l-z))}$ is an exponential weight where $q_{ijl}$ is the query vector and $K_{(}i-m)(j-n)(l-z)$ is the key vector, which is represented by their sum to indicate their similarity at a specific offset. And $Norm$ is a normalisation factor that ensures that the weights of all the values of V used for weighting sum to one. As shown in Fig. 2(c) , the proposed 3D-RCBlock generates dynamic kernels for the input cube according to the relationship among pixels in this cube. The MACs of the 3D-RCBlock is $2\times(H\times W\times S)\times 3^{3}\times C$. Table I lists the MACs for three operations when process 3D HSI data. Although the computational load of the 3D-RCBlock is proportional to the input resolution, the reduction in computational load is quite significant compared to the MACs of self-attention, and the computational load is greatly reduced in high-resolution scenarios.

In our 3D-RCBlock, we do the attention operation within a 3 × 3 × 3 window by a convolution-like operation, which greatly reduces the amount of operations. Our 3D-RCBlock performs the attention operation within the window, which makes our 3D-RCBlock also has the translation invariance of convolutional network and has the same flexibility as self-attention. The 3D-RCBlock combines the advantages of convolutional and self-attention, which reduces the computation amount and at the same time, it can effectively extract the deeper features of the HSI data. Unlike traditional static convolutional kernels, the 3D-RCBlock utilizes dynamic convolutional kernels inspired by the self-attention mechanism, which adapt based on the distribution characteristics of the data. This has been demonstrated experimentally in Section V-C.

The Indian Pine dataset, the Pavia University dataset, and the Houston 2013 dataset were chosen for their distinctive features and widespread use in remote sensing. The Indian Pine dataset presents varied agricultural landscapes, the Pavia University dataset includes detailed urban settings, and the Houston 2013 dataset offers diverse urban and semi-urban scenes with high-resolution data, ideal for testing the granularity of classification methods. These datasets ensure a comprehensive evaluation across a wide range of real-world conditions, effectively testing the robustness and accuracy of the methods in diverse and challenging environments. Fig. 3 displays the false color composites and the ground truth of these three datasets. A detailed description of these datasets follows.
Indian Pines Dataset: Hyperspectral remote sensing images captured by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) over northwestern Indiana, USA, in 1992. This dataset features a 145x145 pixel grid with 220 spectral bands, covering a wavelength range from 0.4µm to 2.5µm. Designed primarily for agricultural and forestry applications, each pixel in the dataset captures a rich series of reflectance values across these bands. This detailed spectral information makes the dataset highly valuable for analyzing and distinguishing between various natural and anthropogenic features, such as farmlands, forests, and built-up areas. Although the dataset originally included all 220 bands, analyses often exclude certain bands that fall within the water absorption regions, typically using around 200 bands for conducting detailed land cover classifications into 16 distinct categories. The Indian Pines Dataset is widely utilized in remote sensing research for developing and testing algorithms for hyperspectral image processing and classification.
Pavia University Dataset: Hyperspectral remote sensing image data acquired by the Reflectance Optical System Imaging Spectroradiometer (ROSIS) sensor from the urban area near the University of Pavia, Italy, in 2001. The dataset contains multispectral images with a total of 610x340 pixels, each pixel having information in 115 spectral bands with wavelengths ranging from 0.43 µm to 0.86 µm.These bands represent different electromagnetic wavelengths used to capture a variety of features on the surface. The image contains many different features of the urban area such as buildings, roads, grass, trees, etc. After removing 12 noise bands, the remaining 103 bands are used for 9 land cover classifications.
Houston 2013 Dataset: Hyperspectral airborne remote sensing imagery acquired over the University of Houston campus and surrounding urban areas by the ITRESCASI 1500 hyperspectral images, Houston, Texas, USA, in 2012, was initially used in the 2013 IEEE GRSS data fusion competition. The dataset contains 349x1905 pixel hyperspectral images with 144 simultaneous spectral bands in the wavelength range of 0.36µm to 1.05µm for 15 land cover classifications. Since the images cover a variety of different features in urban areas such as buildings, roads, fields, water bodies, etc., this dataset is challenging to develop effective classification algorithms to differentiate between these feature types.

In our experiments, we divided the above three datasets into training and test sets. For the Pavia University and Houston 2013 dataset, we took 150 samples from each category as the training set and used the others as the test set. For the IndianPines dataset, since 6 of its 16 classes have less than 300 total samples, especially class1 (Alfalfa), class7 (Grass-pasture-mowed), and class9 (Oats) have only 46, 28, and 20 samples, respectively, for these classes with relatively small number of samples, we randomly select 10 samples as the training set and used the others as the test set. Table II lists the types of land cover classes, the number of samples per class, and the number of samples used for training in each of the three datasets.

The experiments were conducted using a high-specification server equipped with a 2.10GHz Intel(R) Xeon(R) Gold 6230 CPU, 512 GB RAM, and Nvidia Tesla V100 32 GB graphics cards. The proposed 3D-RCNet model was implemented using the Pytorch 1.12.0 open source framework. The model training was performed exclusively on the Nvidia Tesla V100 32 GB graphics card.

For the training setup, the model processed three different datasets, maintaining the same network architecture across all. Each dataset was uniformly cropped into 3D cubes with a spatial resolution of 27x27 for input, while the spectral dimensions were left unaltered. The batch size was set at 64 for all datasets with the training running for 300 epochs. The optimization was done using AdamW, with a learning rate of 5e-4 and a weight decay of 1e-5. The learning rate scheduling was managed by a cosine annealing strategy, beginning with a warm-up phase covering the first 30 epochs where the learning rate increased from 10% of its initial value. For the remaining 270 epochs, the learning rate decayed from 1e-5 to 5e-6.

In the Indian Pine dataset, 3D-RCNet’s classification performance significantly surpasses five other methods across 16 land cover categories, with the relevant data listed in Table III and the corresponding visual comparison results shown in Fig. 4. Table III indicates that 3D-RCNet achieved an OA of 98.50%, AA of 98.03%, and a Kappa of 98.26%. In these three key performance indicators, 3D-RCNet outperforms the other five comparison methods. Notably, although SSFTT demonstrates excellent performance due to its superior deep semantic feature extraction capabilities, surpassing several other advanced methods, 3D-RCNet’s OA is still 1.32% higher than SSFTT.

Furthermore, the visual comparison results of 3D-RCNet on the Indian Pine dataset are displayed in Fig. 4(g), where the bottom right corner provides an enlarged view for easier visual comparison. Other methods exhibit varying degrees of classification errors, such as the salt-and-pepper noise-like visual effects shown in Fig. 4(f), but the proposed 3D-RCNet’s classification results are noticeably superior to the comparison group methods, much closer to the ground truth.

Table IV lists the classification performance on the Pavia University dataset, where our proposed 3D-RCNet achieved an OA of 98.96%, significantly higher than other advanced methods. Among the rest, the best performing ConvNet-based method was LWNet with an OA of 96.86%, and the best performing ViT-based method was GraphGST, with an OA of 98.06%. In comparison, 3D-RCNet exceeds these two methods by 2.1% and 0.9%, respectively.

Fig. 5(g) shows the visual classification results of 3D-RCNet, which has fewer misclassified points, particularly in the classification of grassland edge areas, demonstrating results close to the ground truth. Compared to the other five control group experiments, 3D-RCNet exhibits the most outstanding performance in the classification task on the Pavia University dataset.

Similarly, our proposed 3D-RCNet shows excellent performance on the Houston 2013 dataset. As shown in Table V, 3D-RCNet has an OA of 99.23%, an AA of 99.41%, and a Kappa of 99.17%, which once again outperforms all the control group methods in this set of performance metrics. As shown in Fig. 6, 3D-RCNet’s visualisation results on the Houston 2013 dataset show effective separability, with fewer misclassified points in the zoomed-in detail section at the bottom right.

Overall, the comparative experiment results conducted on these three datasets for six different methods demonstrate that our proposed 3D-RCNet offers the most stable performance and optimal results. Furthermore, the results from other methods further validate the necessity of developing 3D-RCNet.

1. 1.

   On the three datasets, the performance of the two methods based on ConvNet has shown to be stable, whereas methods based on ViT exhibit significant fluctuations due to their sensitivity to data. Nonetheless, when optimized for data, the ViT-based methods can outperform traditional ConvNet methods. This also underscores the necessity of adopting a hybrid structure in 3D-RCNet, which aims to combine the high performance of ViT-based methods with the stability of ConvNet methods.

2. 2.

   We note that classical ViT-based methods such as SpectralFormer perform well when there is a sufficient amount of data, but their performance falters in experiments with limited data due to fewer training samples. In the field of hyperspectral image (HSI) classification, where the cost of labeling data is high, reducing dependence on large data volumes poses a challenge. 3D-RCNet maintains excellent performance even with fewer samples, a fact that is corroborated by our comparative experiments.

3. 3.

   The experimental results also display the sensitivity of different methods to data volume. ViT-based methods are sensitive to data, which results in varying performance across different datasets. For instance, SSFTT significantly outperforms GraphGST on the Indian Pine dataset, whereas the opposite is true on the Pavia University dataset. Such variability in adaptability to data highlights the importance of maintaining consistent performance across different data qualities or volumes. 3D-RCNet is precisely such a method that can adapt to varying data requirements while maintaining stability.

In convolutional neural networks, generally speaking, the larger the size of the convolutional kernel, the larger the window size on the feature map and the better the feature extraction ability. In order to verify that different convolutional kernel sizes comply with this property in our proposed 3D-RCBlock, we set different kernel sizes to determine the most suitable convolutional kernel size for 3D-RCNet.

As shown in Fig. 7, we set the convolutional kernel size of 3D-RCBlock for Stage 4 to 3×3, 5×5,7×7 and tested the classification performance on each of the three datasets, and there is but only a small improvement in the classification accuracy as the kernel size increases. Similarly on different datasets, the degree of improvement is not the same, and even the accuracy decreases with increasing kernel size instead. This is because in Stage 4, 3D-RCBlock extracts the deep semantic information in the feature map output from the previous stage, and the kernel size of the module has little effect on the classification accuracy in the face of the deep semantic information in different data respectively. Therefore, we set the kernel size of 3D-RCBlock as 3×3 to reduce the computational overhead during operation.

We propose the 3D-RCBlock as a plug-and-play module. To analyze the effects of inserting this module at different stages of the model, we designed the experiments in this section, where the last module of the traditional 3D-ConvBlock at each stage is replaced with the 3D-RCBlock. This setup helps us verify the performance of the 3D-RCBlock at various stages.

Results in Table VI show that the performance advantages of using the 3D-RCBlock in the shallow stages (Stage 1, Stage 2) of the model are not significant, especially on the Indian Pines dataset, where it even performs slightly worse than the original model. We believe that this may be due to the fact that the primary task in the shallow stages of the model is to map data space to feature space. Using the highly data-sensitive 3D-RCBlock at this stage may hinder the model from extracting basic semantic features from the raw data.

Moreover, when the 3D-RCBlock is used in the deeper stages (Stage 3, Stage 4) of the model, there is a significant improvement in performance. This improvement may be due to the richness of the feature space in the deep stages, which contains more in-depth semantic features, requiring a more flexible module for further refined extraction. This strategy of using more flexible modules in the deeper layers of the model aligns with current advanced research, which typically uses traditional ConvNet in the shallow layers and structures like ViT in the deeper stages[29][35][36]. We further validated that inserting the 3D-RCBlock in all four stages of the model can achieve superior performance. However, when designing the 3D-RCNet, we also considered the overall simplicity of the system and the convenience of deployment. Therefore, in the proposed 3D-RCNet, we still chose the same pattern as current advanced research, using the 3D-RCBlock in the deeper layers (Stage 3, Stage 4), to strike a balance between the performance and practicality of the model.

The performance of using 3D-RCBlock in the deeper layers was verified through experiments, but the number of 3D-RCBlocks to be used in model design still requires analysis. As shown in Table VI, using 3D-RCBlock in Stage 4 shows better performance than in Stage 3. Initially, we replaced all blocks in Stage 4 with 3D-RCBlock and achieved good performance, as shown in Fig. 8. Then, we replaced all blocks in both Stage 4 and Stage 3 with 3D-RCBlock, and found that replacing all blocks in both stages performed better than replacing only in Stage 4. Therefore, in the proposed 3D-RCNet, we decided to construct all blocks in both Stage 3 and Stage 4 using 3D-RCBlock, to fully utilize its flexibility and performance in capturing deep semantic information.

To verify that the convolutional kernels of the 3D-RCBlock differ from traditional static convolutional kernels, we conducted feature extraction using both 3D-ConvBlock and 3D-RCBlock on the Indian Pine dataset. In the experiments, we randomly selected two sets of convolution steps to examine the state of the convolutional kernels used by the 3D-ConvBlock and 3D-RCBlock and the output feature maps at these steps. As shown in Fig. 9, the state of both types of kernels during different operations is depicted using heat maps. The results clearly show that the 3D-ConvBlock’s kernels remain static throughout the data processing, whereas the 3D-RCBlock’s kernels dynamically adapt and vary based on the data distribution extracted by the convolution window at different steps. This dynamic nature of the kernels not only effectively extracts data features but also demonstrates flexibility in adapting to various data distributions.