Discovering and Explaining the Non-Causality of Deep Learning in SAR ATR

Shapley value[12] is a method for calculating the player contribution solution in a cooperative game. This unique solution satisfies the four properties and is considered a fair attribution method[13, 14]. Consider $N$ is a set of all players with $2^{\left|N\right|}$ possible subsets. A game is a function $v:2^{\left|N\right|}\rightarrow\mathbb{R}$ that maps a subset to a real number. The Shapley value of $i$-th player is given by:

where $S$ is a subset of $N\backslash\{i\}$, $\left|\cdot\right|$ is the number of elements in the set. The Shapley value $\phi(i|N)$ indicates whether $i$-th player has a positive or negative impact on the outcome.

There are many variants for computing Shapley values in deep learning[15, 14], and we use the Baseline Shapley (BShap)[14]. The BShap is also the unique solution method that satisfies certain properties[14]:

where $x_{S};\tilde{x}_{N\backslash S}$ means that $S$ part of input $x$ retains its original value, while other parts are replaced with baseline $\tilde{x}$. We set $f(\cdot)$ as the model classification score corresponding to the true class before Softmax³³3Since deep learning has saturated performance under SOC, we are more interested in the contribution of different regions to correct recognition during training. In addition, analyzing the causes of incorrect recognition in tests is also essential..

The Bivariate Shapley Interaction (BSI)[13] is a way to measure the additional benefit of a coalition $S_{ij}=\{i,j\}$ of two players $i$ and $j$. The contribution of a coalition is often different from the sum of the individual players due to the influence of player interactions. For example, the shadow region responds to target shape information under specific sensor conditions and can form a coalition with the target region[11, 16]. The BSI can be defined below:

where $N^{\prime}=N\backslash\{i,j\}\cup S_{ij}$, $N_{i}=N\backslash\{j\}$, $N_{j}=N\backslash\{i\}$, $\Delta f(S,i,j)=v(S\cup\{i,j\})-v(S\cup\{i\})-v(S\cup\{j\})+v(S)$. The BSI describes whether an alliance between two players benefits the outcome.

The comparable SCR⁴⁴4Please refer to Fig. 3, which provides the comparable SCR curves across classes between the training and test sets is a brief representation of the background correlation. We propose a random SCR re-weighting method inspired by the re-weighting methods in causal inference[17]. We intend to create a similar SCR between classes to remove this spurious correlation. Our experiments show that this intervention reduces the overfitting for clutter, which shows that SCR is indeed one of the factors contributing to the background correlation. Alternatively, SCR re-weighting preserves the clutter texture, so we can further analyze the texture bias in various models.

First, the SCR of SAR images is defined as the ratio of the mean pixel value of target region $\overline{m}^{\rm{tar}}$ to the mean pixel value of clutter region $\overline{m}^{\rm{clu}}$, i.e., $\rm{SCR}=20\ lg(\frac{\overline{m}^{\rm{tar}}}{\overline{m}^{\rm{clu}}})$. For the input magnitude image matrix $\mathbf{X}$, $\rm{SCR}^{\prime}$ is sampled from a specific distribution⁵⁵5A uniform distribution is used to simulate random sampling, which keeps the SCR of different samples in the same 3 dB range. We aim to verify whether the comparable SCR leads to overfitting for clutter., such as a uniform distribution $U(11,14)$. We calculate the re-weighting factor $\alpha=10^{\frac{(\rm{SCR}-\rm{SCR}^{\prime})}{20}}$ and the re-weighting image $\mathbf{X}^{\prime}$ is

where $\mathbf{K}^{\rm{clu}}$ is the mask matrix of the clutter region, $\mathbb{I}$ is a matrix whose elements are all one, $\circ$ is the Hadamard product.

MSTAR, the most cited SAR ATR dataset in recent years, was collected by Sandia National Laboratory in the 1990s[1, 2]. The SAR images in JPEG format are generated by the official conversion tool⁶⁶6The official tool generates JPEG images by linear mapping and contrast enhancement. Since linear mapping normalizes the amplitude to [0, 255], we use SCR (a ratio) instead of amplitude to represent clutter properties. The contrast enhancement enhances the amplitude of targets and some clutter points. This method reduces A-ConvNet overfitting for clutter. and use SARbake⁷⁷7We use SARbake as a benchmark since it is an open-source dataset for segmentation. As SARbake uses 3D CAD models and projection relations to determine regions, some segmentation results do not completely match the actual SAR image. segmentation[18] as the ground truth for target, shadow, and clutter regions. SOC is the recognition of ten classes of targets[2]. The training and test sets are images of different azimuth angles in a similar scene⁸⁸8Although the scenes are all flat grasslands, factors such as the height, sparseness, and water content of vegetation can affect the scattering properties of background clutter., while the depression angle differs by 2°. The benchmark models are three deep learning models (A-ConvNet[19], AM-CNN[20], MVGGNet⁹⁹9MVGGNet used optical pre-training weights in the original paper. Considering our discussion of parameters, the optical pre-training weights are used for models with parameters larger than MVGGNet. Previous work[9] has shown that convolution kernels with pre-training have better texture and shape properties than direct training in the MSTAR dataset. We find this to be critical for ConvNeXtTiny. Optical pre-training weights reduced its overfitting for clutter and improved the test set accuracy from 55% to 98%.[21]) in SAR ATR and other models (EfficientNet[22], ResNet[23] and ConvNeXt[24]) in computer vision.

The total score of BShap is $f(x)-f(\emptyset)=\sum_{i}\phi(i|N)$. Therefore the baseline setting of Shapley values (i.e., absence of input variables) is essential. Belloni et al.[11] set the absence value to zero to calculate the recognition rate change. Considering the dark shadow region[11, 9], the absolute values of a Gaussian distribution $N(0,0.1)$ are used in Fig. 1. This distribution differs from the target, clutter, and shadow areas to ensure that the total score reflects the differences. In the experiments, zero or noise baselines do not guarantee that $f(\emptyset)$ is zero across classes. Moreover, various models have different classification scores $f(x)$. These cause the total score $f(x)-f(\emptyset)$ to vary across models, classes, and baselines. Therefore, we use the Shapley value ratio¹⁰¹⁰10Due to the negative Shapley values, the Shapley value ratio is calculated by $\frac{\phi(i|N)}{\sum_{i}abs(\phi(i|N))}$, where $abs(\cdot)$ is the absolute value. to reflect the relative values. The Shapley values for the following experiments are the average of five replicate experiments, and Fig. 2 is the result of one.

Exp. 1, deep learning uses all correlations to reduce training errors, which does not guarantee causality. The recognition rate and Shapley values increase simultaneously during training in Fig. 2a. However, the Shapley value ratio shows different trends. The clutter curve undergoes an early rising process in Fig. 2b. This curve is because deep learning does not distinguish between correlations that come from causality and data bias. A-ConvNet initially focuses more on background differences. According to Table I, shadow areas have the least influence on recognition, with clutter and targets having the most influence. This finding is similar to the previous study[11]. In summary, the overfitting for clutter reflects the non-causality of deep learning.

Exp. 2, deep learning further exploits interactions between regions to reduce training errors. Fig. 2c shows that interactions between regions increase during training, indicating that the model can exploit correlations on a large scale. Both the target and shadow regions contain class information, but the model also uses clutter differences to form a coalition. Table I shows that the strength of coalition interaction varies across models. Hence, the interactions of regions in deep learning are also influenced by biases.

Exp. 3, data bias leads to class-related clutter properties, one of which is the comparable SCR curves between the training and test sets. According to the SCR curves in Fig. 3, the background clutter is comparable for different target classes in MSTAR. Due to background correlation in training and test sets, overfitting for clutter does not significantly reduce performance under SOC in Table I. Interestingly, the target region contributes negatively to the results when A-ConvNet recognizes the four lowest SCR classes in Fig 3. This phenomenon indicates that the model learns incorrect feature representations to recognize some classes. Similarly, removing targets does not affect the recognition rate for some classes in [11]. Another similar case in computer vision is using snowy backgrounds to distinguish between wolves and huskies[25]. Despite the impressive performance in Table I, our work illustrates that background correlation results in deep learning-based feature representations that are incorrect but valid for biased datasets.

Exp. 4, deep learning models can use background clutter texture for recognition. Fig. 4 shows that clutter contribution is reduced by eliminating the comparable SCR. This result also confirms that comparable SCR is one of the factors contributing to background correlation. As our method only changes the amplitude of clutter, the texture remains intact¹¹¹¹11The relative intensity of clutter scattering points still maintains the original spatial distribution, which keeps a local repeating clutter texture in SAR images.. Our experimental results suggest that the model can more memorize clutter texture in datasets when its capacity increases significantly with the parameter¹²¹²12We use parameters to measure different models. In fact, many factors influence texture bias. We find that optimization methods and model structures affect clutter contribution. Therefore, we use the same optimizer. Furthermore, special model structures such as batch normalization and attentional mechanisms need to be discussed to investigate whether the texture bias is discrepant for the target or clutter..