Integrating global spatial features in CNN based Hyperspectral/SAR imagery classification.

The structure of CNN can be roughly divided into two parts. The first part is the feature extraction part composed of convolution and pooling operations, and the second part is the classification part aims to map the extracted high-level abstract features to the classification labels. The convolution operation of CNN can be seen as a feature selection of input data with different filters. Through appropriate training, the network structure can learn filter parameters effectively to replace the manual design of features in the traditional feature extraction method and get better characteristics at the same time. The architecture of 1D-CNN in this paper is mainly designed for pixel extraction.

In the 1D-CNN, the relationship between the input and output of the convolution layer can be defined as:

where $V_{j}$ is the output of node $j$, $W^{1}_{ij}$ is the weight matrix, $\odot$ is the convolution operation, $X$ is the pixel feature vector input of the remote sensing image and $b_{j}$ is the bias. $f(\cdot)$ is the activation function, which uses $ReLU$. In the convolution layer, the number of nodes is set to 20, the stride is 1, and the size of the convolution kernel is $1\times 10$.

Then, the $V_{j}$ will be processed by the max-pooling layer and the fully connected layer, which is expressed by

where $W^{2}$ is a weight matrix composed of fully connected layers, $V$ is the result of the pooling layer, and $O$ is the extraction feature of 1D-CNN. In fully connected layer, the dropout rate is 0.75, and the number of node is 100.

Generally, the probability graph result of the classifier can be fused with the neighborhood-information of pixels by segmentation methods to improve the classification accuracy [16, 17]. Their methods transform the classified probability graph into an energy function, which is established by the pixel probability information and its neighborhood information. However, the local segmentation methods lead to excessive smoothness on boundaries. The dense conditional random field (DenseCRF) method is implemented to capture non-local relationship of a pixel connected to all other pixels [18, 19]. In the DenseCRF, the energy is defined as

where $C$ is the number of pixels. The unary potential $\psi_{u}(x_{i})$ is obtained by the probability graph in classification method, and the pairwise-potential function $\psi_{p}(x_{i},x_{j})$ is defined as weighted sum of Gaussian kernels. The entire pairwise potential is given as:

where $I_{i}$ and $I_{j}$ are the color vectors; $\theta_{\alpha}$, $\theta_{\beta}$, and $\theta_{\gamma}$ are the control parameters; $w^{(1)}$ and $w^{(2)}$ are the weight parameters; $p_{i}$ and $p_{j}$ are the positions. Although the $appearance$ kernel has been widely used in deep learning to solve the classification problem, the $smoothness$ kernel, which based on the spatial coordinates $p_{i}$ and $p_{j}$, always be ignored. In fact, global coordinates information is much useful to improve the classification results. Besides, it’s true that the pixel in similar positions usually have the same classification in traditional image classification, and vegetation cover of the same latitude and longitude is usually highly correlated in ecological theory [20, 21, 22]. Inspired by their theories, a dual-branch CNN structure that fuses the global feature of remote sensing images with traditional features of the pixel was designed, as shown in Fig. 1.

Since the coordinate information of remote sensing pixels has only two features, a fully connected neural network is used to extract its primary information. In the two layers, the first layer network has 256 nodes to expand the coordinate vector information, and the second layer network has 100 nodes to reduce the feature dimension of the output result of the previous layer network. Finally, the coordinate feature vector is output from the second network.

Then, a dual-branch neural network is designed to fuse the pixel feature vector of 1D-CNN and the coordinate feature vector of FCN. Assuming that $O_{1}$ is the pixel feature vector of 1D-CNN and $O_{2}$ is the coordinate feature vector of FCN, and their feature vectors are fused by vector addition

Then, the fusion vector $O_{f}$ is operated in the fully connected network

where $softmax$ is used for the activate function, $W$ is the weight matrix of FCN, and the final output of our method is $P$ corresponding to the probability of different labels.

In our experiments, $Adam$ is used in the optimizer, $cross-entropy$ is used in the loss function, and the number of maximum epoch is set to 500 .

The ground truths of two datasets are shown in the Figure 2. In the first dataset, the size of the Indian Pine is $145\times 145$, the wavelength range of the spectrum is 0.4-2.5 microns, and the spatial resolution is 20m. After removing a few poorly performing spectra, 220 spectral channels are retained. The Indian Pines scene contains two-thirds of the agriculture and one-third of the forest or other natural perennial vegetation.

In the second dataset, the size of Flevoland is $750\times 1024$, which acquired by NASA/JPL AIRSAR system in Flevoland, Netherlands, August 1989 [16]. In the experiment, 107 features are adopted from different polarimetric descriptors, elements of coherency $\&$ covariance matrix, and the target decomposition theorems.

To prove the effectiveness of our method, a 1D-CNN [23, 24] and SVM [16] are also performed using for the comparison experiments. The structure of single-channel CNN is same as the first branch of our proposed method in fig. 1, and the parameters of their networks are consistent. The overall accuracy (OA), average accuracy (AA), and Kappa coefficient are used as the evaluation criteria for classification results.

In the first experiment, available ground truth is designated in 16 categories in HSI. Approximately $5\%$ training samples are randomly selected from the labeled samples and the classification result is shown in Table I.

Compared with 1D-CNN and SVM, the classification accuracy has been greatly improved in our method. Although the AA, OA and Kappa of SVM are better than those of 1D-CNN, the SVM method has poor classification of limited samples, e.g., Alfalfa, Grass-pasture-mowed and Oats. The classification accuracy of Alfalfa in 1D-CNN is only 8.7%, while its classification accuracy can reach 97.83% when the coordinate information is introduced in our method. In addition, the classification accuracy of Corn-notill, Corn-mintill, Corn, Oats, Soybean-clean, and Buildings-Grass can be improved by more than 50%. Compared with the 1D-CNN, our method can dramatically improve OA, AA and Kappa to 94.59%, 88.55% and 0.94, respectively.

In the second experiment, 11 different land cover types are marked in the ground truth in PolSAR, and $1\%$ of the labeled samples are selected as the training samples. The classification result is shown in Table II.

Although the improvement of classification accuracy in the second experiment is not as obvious as in the first experiment, the advantage of our method is still significant. The classification performance of 1D-CNN is more superior than that of SVM in OA, AA and Kappa, but the classification performance of our method is much better than that of SVM and 1D-CNN. Compared with 1D-CNN, the classification accuracy of Forest has the largest improvement, increased from 78.73% to 99.94%. The classification accuracy of all categories has been improved to varying degrees, and the classification accuracy of Potato, Wheat, Water and Pea increased by more than 10%. As a result, OA is improved from 87.46% to 98.97%, AA is improved from 89.02% to 98.62%, and the Kappa coefficient is improved from 0.86 to 0.99.