A Global Model Approach to Robust Few-Shot SAR Automatic Target Recognition

In Stage 2 (ref. Fig. 1) we use the global model $f_{\phi}$ as a fixed feature extractor and learn a classifier $c_{\theta}$ for the given (M-way, N-shot) FSL task, described by $\mathcal{D}_{support}=\{(x_{i},y_{i})\}_{i=1\ldots(M\times N)}$. To be clear, “ways” refers to the number of categories in the label-space and “shots” is number of labeled training chips per category. Unlike related works in the natural imagery domain that use simple linear regression or nearest neighbor-style classifiers [5], we find it beneficial to utilize a 2-layer neural network which has the flexibility to learn non-linear decision boundaries.

We specifically consider two ways to train $c_{\theta}$. The first, called the basic method, is to train under a vanilla supervised learning objective which minimizes empirical risk on $\mathcal{D}_{support}$ samples only. Mathematically, the basic training method is described as

where $L$ is the cross-entropy loss between the classifier’s prediction and the ground truth label $y$. The intuition for this training method is to learn a classifier that produces accurate and confident predictions on the ID data.

While the basic method can achieve high ID accuracy, it gives no guidance to the model for how to behave when an OOD input is encountered. Without violating or adding any assumptions, we can do something clever to greatly improve the classifier’s ability to handle OOD inputs. Specifically, we can re-purpose the $\mathcal{D}_{pretrain}$ dataset from Stage 1 as an outlier exposure (OE) set in Stage 2. The intuition for OE [7] is to teach the model a confidence calibration on ID and OOD inputs – it should make accurate and confident predictions on ID data and minimally confident predictions on OOD data. Functionally, the minimum confidence state is when the predicted probability is $\nicefrac{{1}}{{\#classes}}$, so to achieve the desired effect we set the training targets of the OE data to be a Uniform distribution over the classes $\mathcal{U}_{\mathcal{C}}$, while the targets of the ID data remain 1-hots. The complete OE objective is

where $\lambda$ is a weighting factor between the ID- and OOD-focused terms that we set to 0.5 here [7].

The final detail of our methodology is how we detect OOD samples during deployment. To conceptualize the process, think about each test input $x$ being assigned a real-valued score $\mathcal{S}_{ID}(x)$ as a measure of its ID-ness. During operation, if $\mathcal{S}_{ID}(x)\geq\beta_{thresh}$ (an application dependent threshold), then the input would be considered ID and the classifier would release its prediction over the set of known classes. Else, if $\mathcal{S}_{ID}(x)<\beta_{thresh}$, the input is considered OOD and the system abstains from releasing the prediction. In this work we use a temperature-scaled Maximum Softmax Probability detector [20] to produce $\mathcal{S}_{ID}(x)$ scores, described as

Here, $\tau$ is a temperature hyperparameter which we set to 100 via validation. The intuition for this detection scheme is straightforward: ID samples should be predicted by the classifier with higher confidence than OOD samples. This fits naturally with the OE training goal, making it a logical choice in our system. We note that OOD detection is currently a popular research topic and future works may investigate alternative scoring mechanisms.

To measure SOC performance on MSTAR we train the models on 17^∘ elevation imagery and test the accuracy of classifying 15^∘ data [22]. As mentioned, because we do not assume to have labeled pretraining data, meta-learning-based FSL methods are inappropriate for comparison (also see Supplemental B). For baselines we instead train supervised models from scratch on the $\mathcal{D}_{support}$ samples only. We examine two architectures of varying complexities: A-ConvNet [22] and RN18 (displayed as Scratch-{AConv,RN18}). The MSTAR SOC results are shown in Table I for #-ways={2,5,10} and #-shots={1,5,10,20,25}, covering a gamut of potential scenarios.

To start, we find several intuitive takeaways. First, accuracy increases with the number of shots; and second, as the number of ways increases the models need more shots to achieve the same performance. In all cases, our SimCLR-based models significantly outperform the models trained from scratch, and in most cases the OE classifier has a slight edge over the basic classifier. When only given 10-shots, our average margin of improvement over the best baseline is +9.8%. On full 10-class MSTAR, we reach 90% average accuracy at $\sim$25-shots, which is a 6% improvement over the best baseline. Lastly, we want to emphasize the implications of these results. The self-supervised SimCLR backbone, which has been trained on mostly ships, is able to separate the MSTAR data by class even though it has never been trained on such data!

For the MSTAR EOC test we train the classification models on 17^∘ elevation imagery and test on 30^∘ data [22]. The large difference in collection geometry causes a sizable change in the target signatures and offers a more challenging test of generalization. Our results are reported in Table II. We see several similar trends to the SOC results w.r.t. the #-shots and #-ways. In all cases, the SimCLR-based models outperform the train-from-scratch baselines, while the basic and OE classifiers perform very similarly to each other. Interestingly, the largest margins over the baselines are at the lowest number of shots. For example, at 5-shots our average improvement over the best baseline is +9%.

Our goal in the third experiment is to examine OOD detection performance across a spectrum of difficulties/granularities. To do this we leverage OOD data from various sources. For a particularly hard OOD test we use a holdout scheme. Starting with a 7-way classifier, where in each test iteration 7 of the 10 MSTAR classes are randomly selected for use in the ID label space, we use data from the remaining 3 classes as Holdout OOD samples. For medium difficulty OOD, we use data from SARSIM-Roads, SARSIM-Medium, and SARSIM-Grass [6]. This is synthetic SAR data which has been generated to specifically match the collection conditions of MSTAR with different backgrounds (note, we remove the Bulldozer and Tank classes to avoid potential ID/OOD ambiguities). For coarse-grained OOD we generate FakeData by creating random Uniform noise chips and use the MNIST test set which contains hand-written digits. Both are clearly OOD w.r.t. any SAR classifier. Fig. 3 shows examples from each of these datasets. Importantly, keep in mind that none of the test OOD data is in $\mathcal{D}_{pretrain}$ (the OE set), meaning the model has never seen it before.

Table III shows the average OOD detection performance (as measured by % AUROC [20]) of a 7-way MSTAR SOC classifier at different #-shots. Note, the AUROC value can be interpreted as the probability that an ID test input would have a greater $\mathcal{S}_{ID}$ score than an OOD input [23], where 100% is a perfect detector. At each setting we examine the performance of the basic versus OE classifier, which we know both achieve high ID accuracy from Sec. III-A. The most important takeaway from Table III is that the OE classifier is better than the basic classifier in all cases by a large margin. Interestingly, for the basic model the coarse-grained OOD sets are the hardest to detect, while the OE classifier detects these almost perfectly. This surprising behavior is something we plan to study in a future work. Another interesting finding is that detection of Holdout samples improves significantly as #-shots increases, while detection of the other OOD types stays relatively constant w.r.t. #-shots. Finally, a somewhat expected result is that for OE models, the Holdout samples are the most difficult to detect, while the medium- and coarse-grained OOD inputs are less challenging. Given that these results still show room for improvement, we highly encourage future system designers (whether in few-shot scenarios or not) to use OOD-aware classifier training schemes such as OE to build robustness to the many forms of OOD inputs.

Our final experiment is an analysis that deliberates the question: can we have it all? Specifically, can we have high accuracy on ID data in SOC and high accuracy on ID classes in EOC and reliable OOD detection? To answer this question we create Fig. 4, which shows density plots of $\mathcal{S}_{ID}(x)$ scores measured on a SimCLR+OE model for ID test data in SOC (SOC-ID), ID test data in EOC (EOC-ID), and Holdout OOD data in SOC (Holdout-OOD).

Firstly, we observe that the SOC-ID and Holdout-OOD are relatively separable, which matches the findings in Table III. However, the EOC-ID and Holdout-OOD are not very separable, as both are predicted with relatively low scores. Thus, with our method if you require good Holdout-OOD detection then most of the EOC-ID would also get flagged as OOD. Conversely, if you require high recall on EOC-ID then many Holdout-OOD would pass through the detector. These results indicate that we cannot have it all with this design, and highlight a serious trade-off to be considered in future work. Finally, while this trade-off may not be completely unique to the FSL setting, our previous observation that less shots makes Holdout-OOD detection harder (Sec. III-C) means this trade-off may be more dire in the few-shot setting.

In this section we will discuss the augmentation pipeline used to train the SimCLR model in more detail. Fig. 6 shows the PyTorch transform block that defines the augmentations. Notice, several of the operations are from torchvision.transforms and we leave description of those to the official docs (https://pytorch.org/vision/0.8/transforms.html). The remaining transforms are defined as follows:

• •

  ClipAndScale(a,b) - linearly re-scale the range of pixel values in the image given a randomly chosen maximum value between a and b. Pseudo: $\text{max\_val}=random.uniform(a,b);x_{new}=clip(x_{orig},0,\text{max\_val})/\text{max\_val}$.

• •

  PowScale(a,b) - pixel-wise exponentiation using a randomly selected value between a and b. Pseudo: $\text{val}=random.choice([random.uniform(a,1),random.uniform(1,b)]);x_{new}=(x_{orig})^{\text{val}}$.

• •

  SpeckleNoise(a,b) - randomly replace a subset of pixel values with values chosen from a Uniform(0,1) distribution. The size of the subset in % total pixels is decided with $random.uniform(a,b)$.

• •

  GaussianNoise(a,b) - apply pixel-wise additive Gaussian noise using $stdev=random.uniform(a,b)$.

• •

  GaussianBlur - convolve the chip with a Gaussian filter.

Finally, Fig. 6 shows several examples of augmented pairs that are generated with this transform block. Recall, the SimCLR training objective encourages the two different views in the pair to lie near each-other in projection space, while views from different pairs should lie far away.

In the main document we do not compare our method to meta-learning-based FSL algorithms. As mentioned several times, this is because meta-learning approaches assume the existence of a labeled dataset for pretraining the feature extractor, which is a violation of our basic assumption set. However, for the sake of completeness we now present a detailed comparison with a state-of-the-art meta-learning approach to few-shot MSTAR called Domain Knowledge Powered Two Stream Deep Network (DKTS-N) [3]. Below are some notable points in no particular order:

• •

  DKTS-N uses the labeled SARSIM dataset [6] as the source of pretraining data for meta-learning. We argue that this is essentially an ideal scenario because SARSIM is specifically/conveniently synthesized to match the conditions of MSTAR (i.e., the downstream FSL task). We believe that this implies a more significant assumption than any we make in our work. Also, it is unclear if the method would be nearly as effective if the pretraining dataset was not so closely related to the downstream task. Finally, although our method is not reliant on such a related dataset, in practice we could leverage more data by adding it to $\mathcal{D}_{pretrain}$.

• •

  DKTS-N reports results using a preprocessing function that performs perfect azimuth angle normalization on both the training and test data. We believe this is done using meta-data provided in the MSTAR dataset. They also show that as the normalization function incurs errors, performance can degrade significantly. It is unclear how one would achieve perfect azimuth angle normalization in the “wild,” and thus we do not use it here. However, if such a function were developed, we believe it could complement our model in a similar way to theirs.

• •

  DKTS-N does not consider OOD detection in the few-shot classifier and thus is susceptible to producing erroneous predictions on OOD data at test time.

• •

  DKTS-N involves a time-intensive inference procedure which scales poorly as #-shots increases (refer Table VII of [3]). This is because for each test input it performs an iterative optimization procedure w.r.t each support sample. Our inference procedure is not nearly as complex, and is a single forward pass through a DNN (e.g., a RN18).

• •

  While DKTS-N shows very high performance in MSTAR SOC, our method actually outperforms it in several of the EOC settings. Specifically, in the (4-way, 10-shot) and (4-way, 25-shot) cases our method is over 3% more accurate.

In the main manuscript we use the SimCLR algorithm to pretrain the feature extractor in Stage 1 of the pipeline. SimCLR is often used as a standard baseline for comparison in Self-Supervised Learning (SSL) literature and our intention is to use it as a representative of the SSL-class of algorithms. To be clear, our framework is not tied to the SimCLR algorithm and we believe that any modern SSL algorithm may be used for pretraining. To show the potential impact of a different SSL algorithm we run experiments with the Bootstrap Your Own Latent^† (BYOL) method. BYOL uses a non-contrastive distillation scheme to perform the representation learning. This is thought to offer some advantages over SimCLR because the learning signal is not reliant on many negative samples to contrast the positive pair with. Because the field of SSL is advancing so quickly it is unclear which SSL paradigm will ultimately emerge as the “best,” but for now these two methodologies make for an interesting comparison. As mentioned in the paper, an important future work is still to consider using different SSL algorithms for pretraining within our framework.

Table IV shows the extended few-shot classification results in MSTAR Standard Operation Conditions, where the Scratch and SimCLR numbers are copy-pasted from Table I in the main paper. The BYOL training parameters are nearly identical to the SimCLR parameters, including the extractor architecture, optimization procedure, augmentation scheme, and training schedule. The only method specific parameter is the EMA momentum which we set to $\tau=0.9995$. In the table we see that BYOL actually outperforms the Scratch and SimCLR models in all scenarios. Interestingly, the benefits of BYOL over SimCLR are most clear in the very low-shot scenarios (i.e., 1- and 5-shots) where the average margin of improvement is $\sim$5%. Focusing on the 10-way scenario, BYOL outperforms the best scratch model by at least 12% in the 1-10 shot range. Finally, we note that BYOL-based models trained with 10-shots perform on par with Scratch models trained with 20-shots! The ease at which BYOL is swapped into our framework is a key advantage and speaks to the flexibility and extendability we were striving for.