Spectral-Spatial Global Graph Reasoning for Hyperspectral Image Classification

Early HSIC community extracts deep features by fully connected networks [23, 24, 25]. However, these networks require sufficient computations since each neuron needs to connect with all units in the next layer. To tackle these problems, many researchers use CNNs for pixel-level classification. In addition, compared with the DNNs receiving 1-D spectral vectors, CNNs process the clipped spatial patches around target pixels and have larger vision fields. In the past years, regarding abundant channels, mainstream deep learning-based methods are classification networks that receive spatial patches or spectral vectors of target pixels [4, 5, 6, 7, 8, 9]. In addition to being directly used as an extractor for single-scale features, CNNs can be modified to generate enhanced features for further accuracy improvement [26, 21, 22, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37]. However, as networks deepen, their abilities are soon saturated, because of the limited sizes of input patches despite theoretical receptive field may extraordinarily large. To this end, many methods adopt the fully convolutional network (FCN) — a special family of CNNs, where the whole image can be directly input into the network and all pixels are simultaneously classified [38, 39, 40, 41]. Although CNN has made promising achievements in HSIC, traditional convolutions still cannot meet the requirements of obtaining the relationships between objects with irregular distributions. Therefore, GCN-based methods are being developed. We will introduce the related technologies later.

Most existing deep learning-based segmentation methods adopt FCN [42]. For example, UNet [43] and RefineNet [44] use encoder-decoder architecture to carefully recover details for upsampling. However, simple stacking ordinary $3\times 3$ convolution causes limited perceptions. To expand the vision field, GCN (global convolution network) [45] uses larger kernels, BiSeNet [46] adopts global pooling, while an effective encoding layer is introduced in EncNet [47]. Multiscale features are employed to improve segmentation performance, such as Deeplab [48] and PSPNet [49], which separately benefit from ASPP or PPM modules that adopt multiple dilated convolutions or spatial poolings in different scales. With flexible long-distance perception, self-attention mechanisms have also been introduced into the segmentation community. For instance, DANet [50] utilizes position and channel attention to separately obtain spatial and channel contexts. Pyramid-OCNet [51] restricts the spatial context capturing in grids generated by the PPM module. While in CCNet [52], only pixels lying in a criss-cross area can communicate with each other. Recently, descriptor-based methods are gradually being valued by researchers. To obtain descriptors, ACFNet [53] aggregates features of the same category in the label map, OCRNet [54] adopts the generated soft regions with the help of an auxiliary loss branch, while EMANet [55] directly defines and learns descriptors by Expectation-Maximization algorithm. In addition, the descriptors can interact with each other in various ways, and a typical implementation is with graph convolution [56], whose applications in the HSIC field will be detailed later.

In the HSIC domain, the development history of using segmentation networks is similar to the natural image. Nevertheless, there are still some differences because of the distinct characteristics of HSI. The earliest SSFCN [38] applies dilated convolutions into spatial and spectral feature extractions for image-level classification. Although FPGA [39] also adopts an encoder-decoder structure, it extra designs a stochastic stratified sampler to promote network convergence. ENL-FCN [57] and FullyContNet [58] employ self-attention mechanisms, too, and the latter captures sufficient contextual information by simultaneously introducing spatial, channel, and scale attentions. Unlike them, the proposed methods follow the pattern of employing descriptors to save computational overhead. Concretely, our SAGRN obtains the descriptors by gathering features in homogeneous areas that are generated by superpixel segmentation inside networks, while the counterparts in SEGRN are obtained by aggregating closely related adjacent channels.

According to our literature survey, GCN is being widely used in the HSIC community, and many approaches [11, 12, 13, 14, 15, 16, 17, 18, 19] are constructed based on superpixel segmentation, since it can provide natural graph structure. For example, MDGCN [11] generates descriptors by directly superpixel segmenting the original image, and then performs the superpixel-level labeling. [12] initializes graph nodes with superpixel centers, and then the nodes are adjusted by modeling the relationships between the center and the pixels within a superpixel, the final classification map is obtained through a region-pixel transformation. Besides using a predefined graph. DGCN [59] separately generates point and distribution graphs with deep features to optimize the relationships between training samples, while EMS-GCN [16] conducts the superpixel segmentation procedure inside the network to generate an adaptive graph, too. Similar to multiple conventional convolutions, DIGCN [15] and XGPN [60] adopt parallel graph convolutions in different scales. While CEGCN [14] and LA-DG-GCN [61] further combine the features from GCN and CNN, respectively. Besides the spatial aspect, spectral information is also drawing researchers’ attention, and it can be exemplified by DASGCN [13]. In addition, some approaches aim to simplify the calculation of the original GCN for efficient HSIC. For instance, [62] operates on smaller graphs using mini-batch samples and the extracted sub-adjacent matrices. Some other schemes, such as spatial pooling [63], novel distance metric [64] and position coordinates [65] are being involved in the graph convolution procedure to achieve HSIC.

Compared with the above attempts, we must argue that the proposed methods are different from them. Firstly, in SEGRN, a spectral graph is adaptively generated since the nodes are obtained by aggregating adjacent channels of changing intermediate features in the network, while the corresponding graph in DASGCN is predefined. What’s more, although the descriptors in SAGRN are also built by superpixel segmentation on the intermediate feature, our spatial graph is dense, since our adjacent matrix is obtained by modeling the relationships between any descriptor with all other partners. Thus, our descriptors can capture global contexts, which are completely different from EMS-GCN, whose adjacent matrix is still produced through traditional similarity computation between neighboring nodes. At last, the proposed SSGRN can adaptively obtain spectral and spatial global graph contextual information.

The original GCN [10] is defined as follows

Here, $X$ is the input feature, $Z=\widehat{D}^{-\frac{1}{2}}\widehat{A}\widehat{D}^{-\frac{1}{2}}$, where $\widehat{D}_{ii}=\sum_{j=1}^{K}\widehat{A}_{ij},\widehat{A}=\widetilde{A}+I_{K}$. $\widetilde{A}$ is the adjacency matrix of the current graph that has $K$ nodes, and $I_{K}$ is the identity matrix, $\mathcal{G}^{(l)}$ and $W^{(l)}$ are separately the output and trainable parameter matrices of the $l$th layer, $\sigma$ is the ReLU activation function.

From the above formula, we can see that the critical parts for graph reasoning are to effectively obtain $Z$ and $X$, while the proposed networks realize similar ideas.

Hundreds of bands and high spatial resolution in HSI provide a close spectral-spatial relationship between pixels, and this contextual information is easily utilized by superpixel segmentation, which can be implemented by adopting the SLIC algorithm [66] to generate a series of compact superpixels. However, the original SLIC algorithm is difficult to be directly placed into the network for end-to-end training because of undifferentiable min-max operations. [67] addresses this problem by transforming these operations to differentiable weighted addition. With this technique, we successfully obtain superpixel segments inside the network and generate effective descriptors. In addition, it should be noticed that the generated superpixels bring graph structure and the number of clustered districts is significantly less than the original pixels, benefitting conducting graph reasoning with high efficiency.

In SAGRN, the obtained superpixels and descriptors are actually homogeneous regions and a group of vectors $D=\left\{d_{1},d_{2},\cdots,d_{K}\right\}\in\mathbb{R}^{K\times C}$. Concretely, each descriptor is computed by taking the average of the features in the corresponding area. Then, the input $X=\left\{x_{1},x_{2},\cdots,x_{K}\right\}$ of the GCN is obtained from $D$ using linear mapping. This process is shown as follows

where $F\in\mathbb{R}^{C\times H\times W}$ is the input feature, $\mathcal{S}$ and $S$ represent the superpixel generating procedure and the corresponding segmentation result map. There are a total of $K$ descriptors and $i$ is the index, $C$, $H$ and $W$ are separately the number of channels, height, and width of $F$. $\mathbb{I}(S_{j}=i)$ is a binary indicator that judges whether the value of the $j$th pixel in $S$ is equal to $i$. $\xi$ represents a 1 $\times$ 1 convolutional layer, which is used to conduct information integration for different channels.

To obtain $Z$, different from the conventional adjacent matrix that only considers neighbor nodes, we treat it as a dense graph where each node possesses a relationship with all the other nodes to better capture graph contexts. Specifically, the relationship in each pair of nodes is measured by computing their similarity in a mapped latent space

where $\phi$ and $\psi$ are mapping functions, both of which can be implemented with a 1 $\times$ 1 convolution. Then, we normalize the similarity matrix using the softmax function.

After obtaining $Z\in\mathbb{R}^{K\times K}$ and $X\in\mathbb{R}^{K\times C_{1}}$, spatial graph reasoning is achieved by directly adopting the GCN formula

where $W\in\mathbb{R}^{C_{1}\times C_{2}}$ is trainable parameters and ReLU is used as the activation function $\sigma$. It can be seen that each district-level feature $x$ is enhanced since it has a global view that captures the contextual information lying in all the other nodes.

At last, these enhanced nodes need to be reprojected for recovering the shape of pixel-level features. The construction of pixel-level features depends on a reasonable combination of descriptors since they possess specific connotations that are various from each other. In this paper, the node vectors after reasoning are considered as a group of bases that can form an effective feature space, where the information at any point can be inferred based on a linear aggregation of these vectors for more complex semantic understanding.

To this end, assume $G=\mathcal{G}(Z,X)=\left\{g_{1},g_{2},\cdots,g_{K}\right\}\in\mathbb{R}^{K\times C_{2}},F=\left\{f_{1},f_{2},\cdots,f_{HW}\right\}$. The affinities $A\in\mathbb{R}^{K\times HW}$ between feature $F$ and node set $G$ are firstly measured in a newly transformed space

Then the target pixel-level feature $F_{sa\_main}\in\mathbb{R}^{HW\times C_{3}}$ is subsequently obtained by linearly combining these graph nodes, where the affinity matrix $A$ is served as corresponding weights. Thus, $F_{sa\_main}=A^{T}\zeta(G)$, and we subsequently reshape $F_{sa\_main}$ to $\mathbb{R}^{C_{3}\times H\times W}$. Here, $\rho$, $\eta$, and $\zeta$ are all implemented with a 1 $\times$ 1 convolution, and the subscript main means the main branch to distinguish the later introduced auxiliary path in the network. In the above procedure, $C=C_{1}=C_{2}=C_{3}$ for convenience. After obtaining the $F_{sa\_main}$, through a series of layers, denoted as $\delta_{sa\_main}$, including a 3 $\times$ 3 convolutional layer followed by a group normalization (GN) layer, a ReLU function, a 1 $\times$ 1 convolutional layer, and a bilinear upsampling function, the probability matrix $P_{sa\_main}\in\mathbb{R}^{C_{n}\times H\times W}$ is acquired for computing loss, where $C_{n}$ is the number of categories. This procedure is symbolized as follows.

However, in the early stage of training, disorganized high-level features may be unfavorable to the homogeneous region generation, and classification quality is unavoidably suffered with the affected descriptors. To produce more stable superpixels, we add an auxiliary branch (AB) to achieve fast convergence and obtain probability matrix $P_{sa\_aux}$ like the main branch.

In the proposed methods, the loss function is defined to $L_{(\cdot)}=l(P_{(\cdot)},Y)$, where $Y$ is the ground truth and $l$ is implemented with the cross-entropy loss, thus the total loss of SAGRN is

The diagram of SAGRN is presented in Figure 1 (A)-(a).

In SEGRN, we adopt a similar idea as SAGRN. Since the homogeneous areas in SAGRN are regarded as clusters of pixels. Thus, it is natural to consider reasonably obtaining channel clusters. For this purpose, inspired by [21, 22], we directly take the mean value of adjacent channels as spectral descriptors. This can be realized by grouping the feature maps in the channel direction.

Concretely, for a input feature $F^{\prime}=\left\{b_{1},b_{2},\cdots,b_{C}\right\}\in\mathbb{R}^{C\times H^{\prime}\times W^{\prime}}$ with $C$ bands, assume they are separately assigned to $M$ groups $F^{\prime}=\left\{r_{1},r_{2},\cdots,r_{M}\right\}$, then the $i$th group is

Thus the $i$th spectral descriptor can be obtained through

The remaining steps are similar to SAGRN. It should be noticed that $F^{\prime}$ is downsampled from $F$ by average pooling before conducting graph reasoning to save computational resources. Therefore, $H^{\prime}<H$, $W^{\prime}<W$, and we group the downsampled $F^{\prime}$ to generate spectral descriptors $d$. The pixel-level feature $F_{se}\in\mathbb{R}^{C\times H\times W}$ is reconstructed after a reasonable descriptor linear combination. Then, a bilinear interpolation is employed to ensure the feature sizes are consistent in subsequent aggregations. Through SEGRN, we successfully perform graph reasoning in spectral direction since the contextual information lying in different channels is perceived, and we obtain an enhanced feature, where each channel is improved by capturing the contexts of other bands.

At last, the loss of SEGRN is computed by

The diagram of SEGRN is depicted in Figure 1 (A)-(b).

The feature $F$ is obtained through a backbone network containing three blocks. Each block includes a convolutional layer followed by a GN layer and a ReLU function. There is a 2x downsampling after the first block to reduce memory consumption. The whole network is trained from scratch and does not need any pre-trained parameters of existing popular models.

After passing through SAGRN and SEGRN, we obtain the corresponding enhanced features $F_{sa\_main}$ and $F_{se}$. To preserve the original information of the input feature $F$, we adopt a residual skip connection, thus the spectral-spatial fused feature $F_{fused}\in\mathbb{R}^{C\times H\times W}$ is defined as

And the corresponding loss is also obtained by

At last, the total loss of the proposed SSGRN is computed through

The whole diagram of SSGRN is shown in Figure 1 (A).

• 1)

  Indian Pines: This scene was gathered in North-western Indiana by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor in 1992, consisting 200 bands with the size of 145 $\times$ 145 pixels that are in 20m spatial resolution after water absorption bands were removed and in the wavelength range of 0.4-2.5$\mu$m. 16 vegetation classes are involved in this scene. The settings of category color, training, validation, and testing samples have been presented in Table I.

• 2)

  Pavia University: This scene was obtained over Pavia University in Northern Italy by Reflective Optics System Imaging Spectrometer (ROSIS) in 2001, consisting 103 bands with the size of 610 $\times$ 340 pixels that are in 1.3m spatial resolution and the wavelength range of 0.43-0.86$\mu$m. 9 categories are included in this data. The experimental settings are shown in Table II.

• 3)

  Salinas Valley: This scene was collected over Salinas Valley, California by the AVIRIS sensor, consisting 204 bands with the size of 512 $\times$ 217 pixels in 3.7m spatial resolution. Same as the Indian Pines scene, 20 water absorption bands were discarded. 16 categories including vegetables, bare soils, and vineyard fields are involved, see Table III for experimental settings.

• 4)

  University of Houston: This scene was acquired over the University of Houston campus and its neighboring regions by ITRES-CASI 1500 sensor in 2012, containing 144 bands with the size of 349 $\times$ 1905 pixels that in 2.5m spatial resolution and in the wavelength range of 0.4-1.0$\mu$m. 15 categories are included in this data. Table IV lists the experimental configurations. This scene was also used in the 2013 IEEE GRSS Data Fusion Contest.

We employ Pytorch to implement the proposed methods. The base learning rate is set to 1e-3, which is adaptively adjusted with the poly scheduling strategy and multiplied by $\left(1-\frac{iter}{max\_iter}\right)^{0.9}$. The SGD with momentum optimization algorithm is used for training, where the momentum is set to 0.9 and the weight decay is configured as 0.0001. During training, iteration and batch size are always 1 in each epoch since the whole image is directly input into the network, and the total training iteration number is set to 1K. The structure details of the proposed networks are shown in Table V and the values of “Parameter” separately represent “channel number”, “kernel size”, “stride”, “padding size” and “dilation rate” of the corresponding layer. In addition, this table also indicates whether GN and ReLU are used after the convolutional layer. Besides standardization, we do not adopt any data augmentations on the proposed methods.

In our implementations, training samples are obtained by randomly choosing. The remainder consists of a validation set and a testing set. Accuracies are evaluated on the testing set, while the validation set is used for monitoring model status during training. All experiments are repeated 10 times. Three commonly used evaluation criteria in the HSIC community are applied in the experiments, including overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa). OA is the most popular evaluation criterion, which is calculated by dividing the number of correctly classified pixels by the number of pixels that need to be judged. However, OA is usually affected by the phenomenon of unbalance categories. To tackle this problem, AA and Kappa are separately computed based on the confusion matrix. The recall values of all categories are averaged to get AA, while the Kappa is used for measuring classification consistency to penalize the model possessing category preference. All experiments are conducted using an NVIDIA Tesla V100 GPU.

In the proposed methods, only the descriptor numbers in SAGRN and SEGRN need to be configured manually. They are exactly the number of nodes in spatial and spectral graphs. They also implicitly determine the average size of homogeneous regions or channels, which affect the scale of context aggregations when generating descriptors. Because of this, we analyze the influences of different descriptor numbers on the network performance, and the corresponding accuracies including OA, AA, and Kappa have been displayed in Figure 2. It can be observed that the OA keeps increasing as the descriptors grow. Concretely, when increasing descriptors, accuracies are improved quickly in the early stage, but growth rates gradually slow down, and the turning points happen on 16 and 32 in SAGRN and SEGRN, respectively. However, when the descriptor number of SAGRN is larger than 256, accuracies have a slight decline. In our consideration, too manly homogeneous areas fragment the feature, and these small areas narrow the scope of region aggregation, which may weaken the representative ability of the generated descriptors. When there are few descriptors, accuracies may decrease instead even if the descriptor numbers are improved, especially in SEGRN. Moreover, in SEGRN, we also notice that accuracies keep at a low level when the descriptor number is less than 16 because spectral information is lost when a large number of bands are aggregated at once. In our experiments, to obtain relatively high accuracies while reducing the computational complexity as much as possible, we set the descriptor number in SAGRN and SEGRN both to 256.

We evaluate different parts of the proposed methods on the Indian Pines dataset, including the branches of spatial or spectral graph reasoning (SAGR or SEGR) and the AB in SAGRN. In addition, the influence of multiple loss functions (we mainly consider the additional $L_{sa}$ and $L_{se}$, while $L_{fused}$ is not included) are also considered, and the results are shown in Table VI. It can be seen that the proposed methods improve FCN a lot with the help of graph reasoning. However, without the AB, the accuracies of SAGRN degrade seriously since a rough feature cannot produce high-quality homogeneous areas and effective descriptors. The extra-added AB directly supervises the corresponding feature rather than the reprojected feature like the SAGR path, shortening the distance from feature $F$ to the classifier end, overcoming the difficult learning phenomenon that is caused by weak gradient propagation, and further promoting the network convergence. Thus, the abilities of SAGR are helped to be more fully exploited with the high-quality feature generated under the aid of AB, obtaining better results. Compared with FCN, SEGRN performs better since the contexts lying in different channels are well extracted by implementing SEGR, showing the importance of spectral information in HSI interpretation. It is also a critical characteristic to distinguish HSI from natural images. By jointly taking the advantage of SAGR and SEGR, SSGRN achieves the best performance, especially when equipped with $L_{sa}$ and $L_{se}$, since the gradients gain better propagations in the corresponding SAGRN and SEGRN.

To more comprehensively analyze the proposed methods, we assess the parameter number (Params) and computational complexity of SAGRN and SEGRN, while the latter is shown in the form of floating-point operations per second (FLOPs), and the results are shown in Table VII. Based on the backbone network of FCN, we simultaneously compare performances of the proposed SAGR and SEGR with other existing commonly employed non-local context capturing modules that utilize self-attention mechanisms, including PAM[50], RCCA[52] and CAM[50], where the recurrent number $R$ of RCCA is set to 2 to keep consistent with the original literature. It can be seen that our methods perform better than other modules since they achieve competitive accuracies with fewer computations. Concretely, on the square Indian Pines dataset, the inner product operations in PAM and RCCA are separately implemented $N^{2}$ and $4N\sqrt{N}$ times, while the proposed SAGR only needs $K^{2}+NK$ times, where $N$ is the pixel number of input features, $K$ is the number of spatial descriptors, in practice $K\ll N$. In SEGRN, the complexity of SEGR is less than CAM since the feature $F$ is downsampled before implementing graph reasoning. Although the accuracies of FCN+PAM on the Indian Pines dataset are close to that of FCN+SAGR+AB, due to memory issues, it may not be able to handle some large scenes at once, such as the University of Houston dataset, which unavoidably affects practical applications. Actually, in our experiments, the combination of FCN, SAGR, and AB equals SAGRN, while FCN+SEGR equals SEGRN.

We conduct a comparison between the proposed methods with other classical or state-of-the-art approaches, including typical CNN-based networks: 1-DCNN [6] for spectral classification with spectral vector, 2-DCNN [6] for spatial classification using spatial patches, and an FCN-based algorithm SSFCN [38] for spectral-spatial joint classification. As for approaches on the foundation of graph convolution, we choose classical GCN [10] and Mini-GCN [62], while MDGCN[11], CADGCN [12], DASGCN [13] and EMS-GCN [16] are selected as advanced GCN relevant techniques. The implementation of GCN is in the pattern of transductive learning, where training and testing sets are simultaneously fed into the model since each pixel sample is regarded as a graph node, and it unavoidably consumes too many computational resources. Mini-GCN simplifies this problem by using a shrunk adjacency matrix that is obtained through only computing on the nodes in the current mini-batch, while the graph nodes in MDGCN are obtained from the segmented superpixels and the classification is performed at the superpixel-level, too. CADGCN also uses superpixels to generate graph nodes and performs a pixel-region-pixel transformation. By using predefined superpixels, DASGCN simultaneously obtains spatial and spectral information and samples different pixels as nodes for performing graph convolutions. While EMS-GCN captures spatial graph contexts by dynamic superpixel maps that are generated by networks, they additionally employ channel attention to enhance spectral features.

Table VIII-XI lists the classification accuracies of each algorithm, where the mean value and standard deviation are reported at the same time. It can be seen that the FCN-based SSFCN usually performs better than the CNN-based 1-DCNN and 2-DCNN, showing the importance of spectral-spatial combination in HSIC. However, the accuracies of SSFCN are still limited because SSFCN only perceives local information since its convolutions are local operators, and regular context capturing can not align with irregular object distributions. For graph convolution-based methods, redundant calculations of adjacent pixels in GCN affect the classification while Mini-GCN attempts to use a simplified adjacency matrix to accelerate this procedure with a mini-batch training strategy. However, in GCN and Mini-GCN, pixels without representation of various contexts are directly set as descriptors, degrading the classification. Some effective descriptors are obtained in MDGCN, CADGCN, and DASGCN with the help of superpixel segmentation on the original image. Thus, they get higher accuracies than GCN and Mini-GCN, especially on Pavia University and Salinas Valley datasets. What needs to be noticed is that the adjacency matrices in MDGCN, CADGCN, and DASGCN are both calculated by only considering the relationships between neighbor nodes, which may be limited in the long strip University of Houston scene. Among these methods, DASGCN performs the worst since it conducts the pixel-level graph convolution that has shorter context ranges. We also notice EMS-GCN does not perform well in almost all datasets. It is probably because the obtained dynamic superpixel map during network training is not stable. At this time, the produced graph features by local context aggregations may be not effective. Compared with the above methods, although homogeneous areas are produced by implementing superpixel segmentation on intermediate features in the network, our SAGRN performs better since we adopt a global view where the relationships between each node and all the other nodes are measured, and more effective descriptors can be flexibly and adaptively generated. By implementing spectral reasoning, SEGRN successfully captures the relationships lying in different channels. At last, combining the spatial and spectral graph contexts that are separately obtained by SAGRN and SEGRN, the proposed SSGRN achieves the best overall accuracies of 97.87%, 98.70%, 99.34%, and 95.59%, respectively, exceeding the second place by 3.50%, 3.00%, 1.41%, and 2.54% on Indian Pines, Pavia University, Salinas Valley and University of Houston datasets.

The classification maps of the above methods have been depicted in Figure 3-6. On the Indian Pines dataset, it can be seen that the spectral vector-based methods 1-DCNN, GCN, and Mini-GCN have serious point noises. The introduced spatial information alleviates this issue, making the maps of 2-DCNN and SSFCN cleaner. However, their conventional convolutional filters can not capture irregular contexts, especially in areas where objects are densely distributed, such as red and pink boxes. In GCN-based methods, MDGCN and CADGCN partly solve this problem by adopting convolutions on graph structures. Compared with them, the proposed methods have longer graph context perception distances, and we also consider spectral information. Therefore, SSGRN can simultaneously obtain discriminative results where the objects possess continuous surfaces, well-maintained edges, and are close to the ground truth in the box displayed areas. On the Pavia University dataset, our method produces more smooth surfaces inside objects, see red and pink boxes. In the region of the red box at the Salinas Valley scene, the proposed method generates a preferable visual result with fewer misclassifications compared to other approaches. On the challenging University of Houston dataset, whose classification maps are difficult to be distinguished, since a large number of small objects are distributed dispersedly. Thus, we select and enlarge two typical regions for comparison, which have been presented in red and blue boxes. We can find that MDGCN and the proposed methods perform well on relatively large objects, so we further strengthen restrictions with pink and purple circles, respectively. It can be seen that benefitting from the global graph context capturing, only our SAGRN and SSGRN correctly classify all small objects in these areas. In addition, in our intuition, SEGRN is probably not as good as SAGRN since it only exploits spectral contexts. Surprisingly, SEGRN performs better than SAGRN on the Indian Pines dataset. It may be because the Indian Pines image possesses more channels. This result implies the importance of channel information in HSIC.

We also evaluate the stability of different methods. Specifically, we adopt different percentages for training samples in Table I-IV. To highlight comparison, we only evaluate the methods that have relatively high accuracies in previous performance comparison experiments, including SSFCN, MDGCN, CADGCN, SAGRN, SEGRN, and SSGRN. The corresponding overall accuracies are shown in Figure 7. In an overall view, SSFCN performs the worst and has no advantage in almost all cases since its conventional convolutions cannot perceive irregularly distributed objects. By addressing this problem using graph convolution with the help of superpixel segmentation, MDGCN and CADGCN perform better than SSFCN. In addition, since their superpixels are always generated on the original image, the number of superpixel regions is far less than the pixel number. Thus, the performances of MDGCN and CADGCN are still stable when using fewer samples, especially for MDGCN that adopts superpixel-level prediction. Owing to the proposed graph reasonings are based on high-level features, whose qualities heavily depend on the learnable knowledge of the training data. Thus, our methods may not be as effective as MDGCN and CADGCN when training samples are extremely scarce (e.g. 20%). Nonetheless, the proposed SAGRN and SEGRN are rapidly improved once the number of training samples is increased. Notice that SEGRN is more preferring the Indian Pines scene, while SAGRN does the opposite. By combining spatial and spectral graph information, SSGRN achieves promising results in most sample scenarios and performs to be comparable even if in cases of few samples.

To more intuitively understand our methods, we separately visualize the affinity matrices $A$ in SAGRN and SEGRN in Figure 8-9, which separately indicates the similarity of the selected descriptors in different spatial positions or channels of the feature $F$, and the responsibility intensity is represented by different colors. In Figure 8, the red color means high affinities. It can be seen that different descriptors highlight different areas, demonstrating that they separately possess closer relationships with corresponding regions. In other words, the connotations of these descriptors are certainly the meaning of emphasized areas. These concepts serve as basic components that can be organized by linearly aggregating the transformed graph nodes in $G$ to generate more complex semantic information for understanding other positions. In Figure 9, each row represents the weights on different channels for a specific descriptor, while the columns represent different channels. If one descriptor pays more attention to some channels, then the colors of these columns will be closer to white. To our surprise, most descriptors present similar characteristics. For example, almost all descriptors show high responsibilities on channels in the red box. Nevertheless, there are also some descriptors behaving differently and bringing diverse spectral information, which can be exemplified by the descriptor in the orange box. Although only a few descriptors perform in special, SEGRN still achieves comparable accuracies. The visualization of spectral descriptors also indicates spectral features still have great potentials to be mined for HSIC.

We also assess the distinguishability of extracted features in the proposed SSGRN at a three-dimensional space by utilizing t-SNE dimension reduction [68]. The distributions for reference samples at the corresponding input image, the feature $F$ generated by the backbone network, the probability matrix $P_{sa\_aux}$ in AB, the $F_{sa\_main}$ of SAGRN, the $F_{se}$ of SEGRN, and the fused feature $F_{fused}$ are separately shown in Figure 10. It can be observed that the samples on the original image space are mixed-up and difficult to be identified, while after backbone network encoding, the obtained $F$ starts to be separable, and AB further strengthens the distinction. Benefitting from graph reasoning modules, the pixel representations of $F_{sa\_main}$, $F_{se}$ and $F_{fused}$ possess high separability with larger inter-class distances, and the points of each category constitute a unique manifold, indicating that the latent patterns of corresponding categories have been perceived, demonstrating the effectiveness of the proposed SSGRN.