Self Supervised Learning for Few Shot Hyperspectral Image Classification

Let $X\in\mathbb{R}^{N\times M\times C}$ be a hyperspectral image. We denote by $x\in\mathbb{R}^{p\times p\times d}$ a hypercube of size $p\times p$ sampled from the scene (or a pixel when $p=1$). The goal is to pre-train an encoder $f$ on unlabeled pixels using SSL, and use $f$ to train a classification model $g$ on the set of labeled pixels. The overall approach is depicted in Fig. 1.

We follow a two-stages procedure consisting of a pre-training and a classification phase. It is a classical approach in unsupervised learning which has also been used for HSI classification [1]. However, modern SSL methods are still under-studied on hyperspectral data, even though some recent studies for clustering and classification are emerging [5, 6].

For pre-training, we treat all pixels as unlabeled and train an encoder $f$ using Barlow-Twins [7], which we describe below.

Barlow-Twins is an SSL algorithm based on redundancy minimization. Specifically, from an input $x$, two views $x^{A}=T_{A}(x)$ and $x^{B}=T_{B}(x)$ are stochastically generated using two sets of augmentations $T_{A},T_{B}$. In this work, we also leverage spatial information to create the views by sampling partially overlapping patch pairs (see Section 2.4). The views are fed into a shared encoder $f$ to obtain their representations $h^{A}=f(x^{A})$ and $h^{B}=f(x^{B})$. Then, an additional Multi-Layer Percepteron (MLP) head is applied to get $z^{A}=MLP(h^{A})$ and $z^{B}=MLP(h^{B})$. Since $x^{A}$ and $x^{B}$ are augmented views, their representations should be similar. Moreover, following the redundancy minimization principle, the dimensions of $z^{A}$ and $z^{B}$ should be decorrelated. This can be enforced on a batch-level by optimizing the following loss:

The pre-training step produces an encoder $f$ which has learnt useful representations for the pixels in the scene. The goal of the second step is to leverage this pre-trained model to construct a classifier $g$. For this purpose, we use two common approaches: linear classification and finetuning.

1. 1.

   Linear Classification: the linear protocol constructs a linear classifier $l$ on top of the representations produced by $f$. The classifier is $g=l\circ f$, where $f$ is kept frozen.

2. 2.

   Finetuning: the finetuning protocol uses the encoder’s weights as an initialization to train the classifier $g=l\circ f$. All the parameters of $f$ are trainable. Therefore, special care must be taken to avoid overfitting or moving away too quickly from the initial pre-trained model.

SSL algorithms heavily depend on the view-generation strategy, which is traditionally done by stochastically augmenting the same input. In this work, we leverage spatial information combined with data augmentation to generate the views.

We evaluate the approach on Pavia and Houston University datasets. Pavia University is a $610\times 340$ pixels scene with 9 classes and 103 bands ranging from 0.43 to 0.86 µm. Houston University is a 349 × 1905 with 144 bands ranging from 0.36 to 1.04 µm and 15 classes. All bands are used in our experiments without any pre-processing (besides normalization).

For both datasets, we use multiple training maps. Specifically, we consider a few-shot setting where we randomly select $K\in[5,10]$ labeled pixels per class. In addition, we consider an abundant labels scenario using the training maps provided in DASE¹¹1http://dase.grss-ieee.org/, which are much more challenging than random sampling.

We consider simple models for HSI classification. Specifically, in the patch-level setting, we rely on a custom 2D Convolutional Neural Network (CNN) with 2 residual blocks. The patch-size we use is $9\times 9$. For the 1D case, we use a similar 1D CNN architecture with 2 residual blocks.

In all experiments, Barlow-Twins pre-training is ran for 100 epochs with LARS optimizer. Classification models are trained for 100 epochs using stochastic gradient descent with a cross-entropy loss. We report the Overall Accuracy (OA) and the Kappa coefficient ($\kappa$) in the experiments and compare against a supervised baseline using the same networks.

We analyze the impact of data augmentation on the overall accuracy of Barlow-Twins on PaviaU with a linear classifier using the training map from DASE. To do so, we consider up to two augmentations at a time. We run the pre-training and classification phase for every pair of transformations listed in Section 2.4. The transforms are applied in both branches in a symmetric manner with a fixed order and probability $p=0.75$. Results are presented as a symmetric matrix in Fig. 2.

We observe that the transforms which work best on average are random flipping, Gaussian noise, random band dropping and random band swapping. Moreover, a part from a few exceptions, the diagonal values of the matrix (i.e., when a single transform is used) are usually lower than the off-diagonals (i.e., transformation pairs). Additionally, when discarding all transformations, we get $79.4\%$ of accuracy, which confirms the need for data augmentation. Yet, accumulating many transforms can be problematic with hyperspectral images since, contrarily to RGB data, the relevant information lies in the spectrum of every pixel.

We run the proposed SSL-based approach on PaviaU and Houston with a varying number of samples per class using a 2D CNN. For these experiments, we use flipping, Gaussian noise, random band dropping and random band swapping for data augmentation. Results are given in Table 1 and Table 2, where BT denotes Barlow-Twins. In every setting, SSL outperforms vanilla supervised learning by significant margins and proves to be more label efficient. Furthermore, the benefit of using SSL is also visible even when labels are abundant, as shown in Table 3. Moreover, it is interesting to observe that the linear protocol (BT + Lin) and the finetuning protocol (BT + Tune) lead to comparable results, even-though most of the network parameters in the linear case are trained in an unsupervised way. This shows that the model has learnt useful features during the pre-training phase. In addition, the linear protocol is also less prone to overfitting.

We train a BT + Lin classifier on PaviaU using 1D CNN. We generate the pairs using neighboring pixels (in a $5\times 5$ neighborhood). We do not apply any data augmentation because preliminary experiments suggest that it does not always help with 1D CNNs. The results are shown in Table 4. Once again, we observe that SSL outperforms supervised learning when the labels are limited. It is also the case on the full split, where supervised learning reaches $80.9\%$ of accuracy compared to $86.2\%$ for BT + Lin.