X-Fake: Juggling Utility Evaluation and Explanation of Simulated SAR Images

Image quality assessment (IQA) methods mainly aim to assign a score to an image that can evaluate its quality from a visual perspective. Subjective and objective assessment are the two main types of IQA methods. Subjective assessment measures quality with human input, while objective assessment measures quality without it [20, 21]. The objective IQA, which is the most popular method, can be divided into full-reference (FR), no-reference (NR), and reduced-reference (RR) approaches [22]. A lot of research papers have used FR metrics like mean square error (MSE), peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and visual information fidelity (VIF) to describe the quality of a simulated SAR image. These methods assess the image quality based on human visual perception. For generated images with deep learning models, the Fréchet Inception Distance (FID) is also widely used in feature distribution [24]. In essence, it utilizes the pre-trained ImageNet Inception network as the feature extractor and compares the distances between features. However, due to the specific imaging mechanism of SAR, there may be a lack of trustworthiness for evaluation [5, 25]. The NR models rely primarily on image quality benchmark datasets that record human beings’ evaluation results for visual perception [26, 27]. However, the quality assessment of human visual systems is not applicable to SAR, as we will discuss in our later experiments.

In real-world applications, we mostly train the deep neural network with simulated SAR images and real data to enhance the algorithm’s generalization ability. As a result, the utility of the simulated SAR images is worth studying. Some related work focuses on utility evaluation in terms of model performance. The SAR target recognition task, for example, is the most explored. For evaluation, Guo et al. proposed the Substitution Rate Curve (RSC) criteria based on the changing recognition rate [7]. Similarly, the hybrid recognition rate curve (HRR) and the class-wise image quality assessment were proposed in the literature [8], respectively. Besides the utility evaluation, some other works aim to improve the data utility of simulated SAR images with transfer learning, domain adaptation, or knowledge distillation [9, 10, 43]. These methods reduce the distribution discrepancy between real and simulated SAR images, enhancing model performance through the use of simulated data. In summary, evaluating and improving the utility of the simulated SAR images are equally important, yet current studies approach them separately. Mostly, the utility improvement takes place implicitly in the feature space, making it difficult to interpret. In contrast, our method aims to address the two issues simultaneously with a unified, trustworthy, and explainable framework.

Methods for estimating uncertainty in a DNN prediction are a popular and vital field of research [29]. They can be categorized into single deterministic [30], Bayesian [61], ensemble [31], and test-time augmentation methods [32]. For Bayesian neural networks, specifically, the posterior distribution over the network parameters is estimated, and the predictive uncertainty of test data can be quantified. Approaches such as variational inference [33], Monto Carlo sampling [34], and Laplace approximation [35] were proposed to infer the posterior distribution of BayesCNN. Apart from uncertainty estimation, it is also important to explain the source of uncertainty in order to achieve more trustworthy artificial intelligence (AI). Researchers in explainable artificial intelligence (XAI) have developed numerous methods for helping users understand the predictions of complex supervised learning models [36]. By contrast, explaining the uncertainty of model outputs has received relatively little attention [37, 38, 39]. The mainstream explanation methods can tell why a model works well, but they will be ineffective in explaining the bad result with very high uncertainty [40]. Some researchers conducted causal counterfactuals to explain the high uncertainty in the input [41]. Recently, the literature [19, 42] proposed methods that can help identify which input features are explicitly responsible for models’ uncertainty.

Inspired by the introduced work, we re-formulate the utility evaluation problem by quantifying the domain-shift uncertainty of simulated SAR images in this paper. Furthermore, we illustrate the inauthentic details of the fake SAR image through an uncertainty explanation.

The proposed X-Fake framework, as shown in Fig. 2, consists of three steps. The probabilistic evaluator (P-Eva) is firstly constructed with a BayesCNN model. The simulated SAR image $x_{0}$ is evaluated by P-Eva to obtain the criteria vector $\mathbf{m}$ containing target class and angle predictions and their uncertainties, denoted as:

where $\mathcal{F}_{\mathrm{Bayes}}(\cdot)$ represents the BayesCNN model. The construction and optimization details will be introduced in Section III-B. Then, an IntroVAE model is pre-trained to obtain the optimized encoder $\mu_{\phi}$ and generator $\mu_{\theta}$, thus preparing for high-quality counterfactual generation. It can be denoted as:

The details will be elaborated in Section III-C. Given the simulated SAR image $x_{0}$, its counterfactual explanation is defined as $x_{c}$ that satisfies the following conditions. On the one hand, $x_{c}$ should be close to $x_{0}$, where the distance metric $d(\cdot,\cdot)$ is applied for constraint. On the other hand, $x_{c}$ should be satisfied with the desired output $\mathbf{m}_{c}$ of P-Eva. To this end, the optimization can be written as:

Note that optimizing $x_{c}$ in the high-dimension image space is difficult. To this end, we transform the optimization problem in the latent space of the pre-trained IntroVAE model. The algorithms will be illustrated in details in Section III-D.

The current SAR target image generation is based on prior parameters, and among them the target class and azimuth angle are most critical. The inauthentic details of a simulated SAR target image exist in the following three aspects: ambiguous scattering centers, inaccurate azimuth angles, and out-of-distribution clutter background. They can be summarized with inconsistent feature distribution between real and simulated SAR images, which leads to inferior utility of simulated data. To this end, we propose to construct a probabilistic evaluator based on Bayesian deep neural network to measure the feature distribution discrepancy in terms of class label and azimuth angle.

Specifically, we construct a BayesCNN model as shown in Fig. 3. It features four trainable layers in the front and is subsequently bifurcated into two branches, one for target category prediction and the other for azimuth vector regression prediction. The azimuth $\theta$ is encoded as a vector $[\cos\theta,\sin\theta]$. In this manner, our BayesCNN network is capable of simultaneously predicting the category label and azimuth angle of the SAR target image.

Compared to its equivalent frequencist CNN, which utilizes a single point estimate for weight parameters, BayesCNN incorporates a probability distribution for its weights. This approach involves conducting posterior inference on the distribution parameters of the neural network using Bayes’ theorem. Therefore, BayesCNN offers results by weighting the posterior probability of all parameter settings, rather than relying solely on a deterministic value of parameters. This approach allows for not only predicting results, but also capturing the uncertainty of those predictions. Our method involves assessing the utility of simulated data by quantifying the uncertainty of the recognition model outputs.

The BayesCNN construction and optimization can be realized in different ways. In this paper, we applied optimization with Bayes by Backprop and Monte Carlo Dropout approximation to construct the P-Eva, denoted as Eva-BBB and Eva-MCD, respectively.

Optimizing the counterfactual images in the high-dimensional space is challenging and cannot ensure the in-distribution constraint [19]. To this end, the optimization can be conducted in the latent space of a probabilistic generative model to ensure the similar data manifold. In order to generate precise counterfactual explanations, it is crucial to first develop a high-quality image generation model. Consequently, we propose to utilize IntroVAE [58] as the foundational structure of the explainer. Similar to VAE, IntroVAE consists of an encoder and decoder that can project the high-dimensional data into low-dimensional latent space, enabling us to optimize in the latent space in order to preserve the data manifold. Additionally, IntroVAE incorporates adversarial training based on VAE to enhance the quality of the generated counterfactual explanations.

As depicted in Fig. 2, the VAE encoder is additionally served as the discriminator, and the VAE decoder is served as the generator in GAN. Adversarial learning like GAN is performed during model training so that the model can self-estimate the difference between the generated sample and the training data, and then update itself to produce a more realistic sample.

In order to match the distribution of the generated samples to the true distribution of the given training data, we use the regularization term $L_{\mathrm{KL}}$ as the adversarial training cost function. The Encoder requires minimizing $L_{\mathrm{KL}}(z)$ so that the posterior $q_{\phi}(z|x)$ of the real data $x$ matches the prior $p(z)$. Simultaneously, it requires maximizing $L_{\mathrm{KL}}(z_{s})$ so that the posterior $q_{\phi}(z_{s}|x_{s})$ of the generated sample deviates from the prior $p(z)$. $z_{s}$ includes $z_{r}$ and $z_{pp}$ which are the latent codes of the reconstructed images $x_{r}$ and $x_{p}$, respectively.

The loss function of the Encoder is:

Conversely, the Generator needs to minimize $L_{\mathrm{KL}}(z_{s})$ so that the posterior $q_{\phi}(z_{s}|x_{s})$ approximately matches the prior $p(z)$:

where $m$ is a positive margin, $[.]^{+}=max(0,.)$. $L_{\mathrm{RE}}$ is the mean square error (MSE) loss function of the reconstruction error. $L_{\mathrm{KL}}$ represents KL-divergence: $L_{\mathrm{KL}}(z;\mu,\sigma)=\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{M_{z}}(1+log(\sigma_{ij}^{2})-\mu_{ij}^{2}-\sigma_{ij}^{2})$. $\alpha_{R}$, $\alpha_{E}$, $\alpha_{G}$ and $\beta$ are weighting parameters used to balance the importance of each item.

Counterfactual analysis aims to examine the question of ”What changes are required to the input in order to get the desired output?” to explain how the model makes decisions. In our study, we aim to minimize the modifications required for SAR images deemed of low quality by the BayesCNN evaluation network in order to optimize their utility for deep learning models. This involves generating an ”explanation” closely resembling the original simulated images, and enabling the evaluator to achieve the desired outcome.

For the input simulated SAR image $x_{0}$, the prediction of P-Eva is denoted as $\mathbf{m}_{0}=[\overline{y}_{0},u_{c},\overline{v}_{0},u_{a}]$ where the overall uncertainty $u_{c}+u_{a}$ could be relatively high, or the label and angle prediction $\overline{y}_{0},\overline{v}_{0}$ are even wrong. The counterfactual explanation $x_{\mathrm{opt}}$ should meet the requirements of correct prediction $\overline{y}_{\mathrm{opt}},\overline{v}_{\mathrm{opt}}$, lower uncertainty, and minimal change compared to $x_{0}$. Therefore, the optimization can be divided into the following three parts. First is the distance constraint, denoted as:

where $d_{1}(\cdot,\cdot)$ is the L1 norm. The following constraints the counterfactual image consistent with the prior category and angle information:

where $C$ denotes the class number, and $d_{2}$ denotes the L2 norm. $\lambda_{y},\lambda_{v}$ are two hyper-parameters. The final term defines the predicted uncertainty of $x$:

The above-mentioned $\overline{y},\overline{v},u_{c},u_{a}$ are predicted by the pre-trained P-Eva model. As a result, the overall optimization can be written as:

In the optimization process, we do not directly optimize $x$ in the high-dimensional space, but map $x$ to the low-dimensional latent space through the pre-trained encoder of IntroVAE to optimize $z$. Therefore, Equation (17) can be re-written as:

Then, the latent counterfactual $z_{\mathrm{opt}}$ can be obtained from:

The counterfactual explanation in the image domain can be generated from the pre-trained decoder of IntroVAE, i.e., $x_{\mathrm{opt}}=\mu_{\theta}(x|z_{\mathrm{opt}})$. During the counterfactual generation, only the latent code $z$ is optimized while the P-Eva, the encoder and generator of IntroVAE are frozen.

The inauthentic details of a simulated SAR image can be reported as the pixel changes of the counterfactual explanations and the original image. For a better visualization, the difference $\Delta x=x_{\mathrm{opt}}-x_{0}$ is multiplied with its absolute value $|\Delta x|$ while keeping the sign. The positive and negative values are colored with red and blue in the experiments, respectively.

In the experiments, a real SAR image dataset and four simulated SAR image datasets are applied to conduct experiments. The simulated datasets include an EM model based simulation dataset and three generated SAR image datasets by advanced generative AI approaches. Some examples are given in Fig. 4.

In this paper, all experiments are conducted on NVIDIA GeForce RTX 3090 Ti with a compute capability of 8.6. The details of training and validation data settings are given in Table II.

The proposed evaluation method aims to assess the distribution inconsistency between real and simulated data. Consequently, different from other full-reference assessment methods, our method does not require paired data. In order to prove the generalization ability of the proposed evaluation method, we applied real SAR images with different depression angle to train the BayesCNN model. To evaluate SAR-ACGAN, SAR-CAE and SAR-NeRF data with depression angle of 17^∘, the BayesCNN model is trained with MSTAR real data at 15^∘, as given in Table II.

The input size of evaluation model is 88$\times$88. The training data are pre-processed with center-crop, logarithm and random grayscale stretching for data augmentation, aiming to improve the generalization of the evaluation model. We used Adam as the optimizer with a learning rate of 0.001. The epoch number and the batch size are set to 300, 25, respectively. The trade-off parameter $\lambda_{a}$ in Equation (7) is set to 20.

At IntroVAE Pre-training stage, the training and validation settings as well as the data pre-processing, follow the settings of evaluation stage. The hyper-parameters are given in Table III. We use Adam as optimizer to optimize Generator and Encoder with a learning rate of 0.0005. The epoch is 500 and the batch size is 25.

The trade-off parameters $\lambda_{d}$, $\lambda_{y}$, and $\lambda_{v}$ during the counterfactual optimization will be discussed in the experiments. The optimal values in our experiments are set to 1, 1, and 30, respectively. The Adam optimizer is applied, and the learning rate is 0.0005. The epoch numbers are set to 50, 150, 50, and 200 for SAR-NeRF, SAR-CAE, SAR-ACGAN, and SAMPLE, respectively.

In order to prove the effectiveness of the proposed evaluation method, we sorted the simulated data with different assessment metrics and split them into ”TOP” and ”LAST” evenly. Then, the classification model is trained with ”TOP” and ”LAST” data separately and tested with real images. The performance gap between ”TOP” and ”LAST” demonstrates the effectiveness of the assessment metrics in terms of utility.

In this part, we applied SAR-NeRF and SAR-CAE datasets for experiments. The ”TOP” and ”LAST” numbers of SAR-NeRF and SAR-CAE are set to 180 and 100 per category, respectively. Four full-reference IQA metrics, i.e., PSNR [48], SSIM [49], VIF [50], FSIM [51], and four non-reference IQA models, i.e., NIQE [53], BRISQUE [55], NIMA [54], DBCNN [52], are compared.

As shown in Fig. 5, the classification accuracy of models trained with ”TOP” data filtered by different assessment metrics are marked in blue, while the ones trained with ”LAST” are presented in orange. The performance gap between ”TOP” and ”LAST” is also plotted. The positive values denote the assessment metrics are effective in terms of utility, otherwise demonstrating ineffective.

The results show that our method achieves the highest performance gap, which demonstrates that our method outperform other full-reference and non-reference IQA methods in terms of utility. Notably, some IQA models fail to evaluate the utility of simulated SAR image with negative performance gaps. The results illustrate that those methods based on human visual perception may not applicable to SAR images. We also explore the effectiveness of different BayesCNN models, including Eva-BBB and Eva-MCD. The results show that Eva-BBB can achieve better performance in evaluation.

Fig. 6 exhibits four paired examples of real and simulated images. It is difficult to distinguish the quality of these four simulated images from visual perspective. The VIF indicators demonstrate they have similar image quality, and example (c) and (d) are with lower quality scores than (a) and (b). However, we observed the uncertainties of simulated sample (a) and (b) are high, denoting their distinct feature distribution discrepancy with real data. The incorrect predicted labels by a recognition model trained with real data can verify this. As a result, our method can successfully assess the simulated SAR images in terms of data utility, even they have similar visual quality.

Some researches also proposed the evaluation metrics based on data utility, such as the recognition versus substitution rate curves (RSC) [7]. Fig. 7 illustrates the RSC results of three simulated SAR image datasets. Note that RSC is a dataset utility evaluation index which can only report the global evaluation result, rather than assigning criteria to individuals. To this end, it cannot filter out samples with low-utility. On the contrary, our method can evaluate data utility for each sample with a quantitative metric so as to filter the high quality data.

To demonstrate that the proposed explanation method can explicitly reveal the inauthentic details of the simulated data, we visualize some examples from different simulated datasets, as shown in Fig. 8. In this figure, we present the original simulated images in (a) and (e); generate counterfactual samples in (b) and (f); explain the inauthentic details in (c) and (g), where red and blue represent the missing and redundant parts of the simulated images, respectively; and present real samples with the closest azimuth angles in (d) and (h). The predicted uncertainty $u$ and class label $\overline{y}$ of the images are also provided.

It can be found that the uncertainty of the generated counterfactual image is reduced to a certain extent (marked in green). Additionally, the wrong prediction can be corrected (marked in red). Compared with the original simulated image, the counterfactual explanation is closer to the real one from a visual perspective. The inauthentic details explicitly explain the reason for the low utility of simulated data.

In addition, we verify the effectiveness of the proposed counterfactual explanation method by visualizing the feature distribution in the latent space. As provided in Fig. 9, the feature distribution of real and original simulated SAR images, and the one of real and counterfactual explanations, are demonstrated in (a) and (b), respectively. It can be seen that the original simulated data and the real ones have a distinct feature distribution inconsistency in the latent space. In contrast, the generated counterfactual samples achieve a more consistent distribution with the real ones, thus improving the data utility.

As illustrated above, the generated counterfactual explanation can improve the data quality and utility both explicitly and implicitly. Accordingly, we conducted several quantitative experiments to verify the potential of our proposed explanation method for improving the simulated data. In Table IV, we applied the A-ConvNet model trained on different simulated data to test the real MSTAR target images (15^∘) and recorded the results. For each simulated dataset, four experiments are conducted. The term Upperbound denotes training with full real data, which demonstrates the recognition model’s upper limit. Before denotes training with full original simulated SAR images, while after (BBB/MCD) denotes training with counterfactual samples generated by Eva-BBB and Eva-MCD, respectively. We maintain the same number of training samples across all four experiments to ensure fairness in comparison. We can see that training with full original simulated data will cause a dramatic decrease in classification accuracy tested on real SAR images. The generated counterfactual explanation, however, can successfully improve the data utility that makes the trained model more applicable.

Furthermore, we also tested the performance of training different deep learning architectures with counterfactual samples, as given in Table V. The experimented simulated dataset is SAR-NeRF. We explore popular convolutional neural network and vision transformer models, such as ResNet-18, ResNet-50, DenseNet-121, and Swin-Transformer, for illustration. The results also demonstrate the effectiveness of the proposed explanation method, which has the potential to improve the utility of the data.

Fig. 10 illustrates the utility evaluation results and the counterfactual explanation results of the three AI-generated SAR image datasets. The x-axis and the y-axis denote the number of training samples of the Gen-AI model, and the test results on real SAR images of a simulated data trained model, respectively. It can be observed that SAR-CAE data are generated by the fewest training samples and can perform comparable utility with SAR-ACGAN where the training data are doubled. With similar training numbers, SAR-NeRF data has better utility than SAR-ACGAN according to our evaluation results. The counterfactual explanations of SAR-NeRF and SAR-CAE data are more effective than those of SAR-ACGAN data, that improve the image utility better.

In order to verify the effectiveness of each module in the counterfactual generation process, we conducted a series of ablation experiments on the SAMPLE dataset, as shown in Table VI.

First, the recognition model is trained directly on original simulated data and tested on real data to obtain the baseline result. Then, we applied the CLUE method to generate the counterfactual explanation, i.e., using VAE as the generator and optimizing with distance and uncertainty. The result shows a slight improvement of 5.15% over the baseline, indicating the effectiveness of the counterfactual explanation primarily. It is worth noting that the utility of the generated counterfactual data has increased significantly after adding the class guidance. Additionally, IntroVAE can improve the visual quality of counterfactual samples compared with VAE, enhancing its utility as well as obtaining high-quality counterfactual interpretation samples.

By introducing the angle guidance, we need to enable the assessment model to predict the azimuth angle, denoted as Eva-BBB. It can be obviously observed from the fourth row and the last row that introducing angle guidance can further improve the utility of counterfactual samples. The performance of predicting class and angle has improved to a certain extent. Moreover, we also verify the superiority of BayesCNN based CNN in predicting uncertainty compared with point-estimated frequency neural networks, as given in the 5th and 6th rows in Table VI.

There are three hyperparameters involved in the process of counterfactual sample generation. $\lambda_{y}$ and $\lambda_{v}$ control the trade-off between the classification loss and angle loss, and $\lambda_{d}$ controls the distance term between the counterfactual and original data. By fixing $\lambda_{d}=1$ and changing the ratio of $\lambda_{y}:\lambda_{v}$, it can be found that with the increase in the proportion of angle loss, the azimuth angle of the generated counterfactual image is more accurate, and the recognition model trained with these counterfactual samples performs better on azimuth angle regression prediction. When $\lambda_{y}:\lambda_{v}=1:30$, the result achieves the highest classification accuracy on the test set. We also experience the influence of $\lambda_{d}$. The result shows that a larger $\lambda_{d}$ would result in less change from the original image, leading to inferior counterfactual results.