Anomaly-Aware Semantic Segmentation by Leveraging Synthetic-Unknown Data

Anomaly detection, also known as out-of-distribution (OOD) detection, is formulated as a problem to determine whether an input is in-distribution ($i.e.$, the training distribution) or out-of-distribution ($i.e.$, the other distributions). The issue has been studied in several aspects, such as selective classification [11] and open-set recognition [3]. Hendrycks et al. [5] demonstrated that deep classifiers have a tendency to assign lower maximum softmax probability (MSP) to anomalous inputs than in-distribution samples. Such property enables models to perform OOD classification by simple thresholding. Liang et al. [6] further proposed temperature scaling and input preprocessing to make the MSP score become a more effective measurement for anomaly detection. Monte Carlo dropout (MCDropout) has also been studied in previous works to measure network uncertainty [12, 13], but the post-processing of such methods is rather expensive. Some approaches based on ensemble learning have shown remarkable performance [14], but training multiple models requires considerable resources.

The threshold-based methods [5] are notable for their computational efficiency, yet the pre-trained classifiers they used might fail to separate in- and out-of-distribution samples due to the overconfident prediction issue. To address the overconfident prediction issue of the threshold-based methods [5, 6], [7, 8] trained anomaly detectors against an auxiliary dataset of outliers [15] so the models could be capable of generalizing to unseen anomalies. In order to obtain more representative samples, [7] leveraged GAN [16] to synthesize adversarial examples around the in-distribution boundaries; [10] proposed a re-sampling strategy to construct a compact yet effective subset for training. Additionally, [8] proposed an extra loss term to penalize non-uniform predictions when out-of-distribution data are given, regularizing the learning so that OOD samples will have much lower confidence scores.

Recent studies have made significant progress on semantic segmentation, which is one of the essential problems in computer vision and has shown great potential in a myriad of applications. FCN [17] is regarded as a fundamental work in semantic segmentation, which included only convolutional layers and can process images of arbitrary sizes. Feature pyramid [18] is also widely used in segmentation models to better capture multi-scale information. DeepLabv3 [19] achieves admirable performance with computational efficiency by exploiting dilated convolutions. Though these approaches have shown robustness and effectiveness, they have a tendency to make overconfident decisions as OOD inputs are given. Such an insufficiency would seriously restrict these methods from deployment to real-world scenarios.

While prior attempts have studied OOD detection in multiple views, the task of anomaly segmentation remains rarely explored. Wilddash dataset [20] was proposed to offer evaluations that reflect algorithmic robustness against visual hazards ($e.g.$, illumination changes, distortions, image noise), yet the OOD detection in [20] still remains at the image-wise level. Therefore, the CAOS benchmark [21] comprised of real-world driving scenes and synthetic anomalies is proposed, attempting to bridge the gap between anomaly detection and semantic segmentation. Xia et al. [22] proposed a framework consisting of both generative and comparison modules, identifying OOD regions by comparing the synthesized image to the original input. Franchi et al. [23] achieved state-of-the-art results by tracking the trajectory of weights and using an ensemble of networks. Both [22, 23] demonstrated remarkable performance, however, these approaches require additional models to refine the output. In this paper, we show our framework achieves superior performance by directly thresholding the softmax scores without exploiting generative modules, and such computational efficiency is highly desired in real-world applications.

In this paper, we consider the problem of anomaly segmentation. Unlike conventional OOD classification [5, 6, 7] treating a whole image as an anomalous sample, in anomaly segmentation, we must segment anomalous regions with pixel-level accuracy. Let $D_{k}$ and $D_{u}$ denote two distinct distributions, known and unknown, respectively. Those pixels of instances within training data are drawn from $D_{k}$, while the pixels of anomalous instances are sampled from $D_{u}$. We further define a synthetic-unknown distribution $D_{s}$ , which indicates the space of those synthetic pixels. We assume a well-trained semantic segmentation model $f$ and the ground truth pixel labels $Z\in\mathbb{R}^{H\times W}$ associated with the input image $X\in\mathbb{R}^{C\times H\times W}$ are available. For the convenience of interpretation, we define the well-known softmax probability as

where $f_{i}(X_{h,w})$ is the model output score of category $i$ at pixel $(h,w)$; $N$ is the number of classes in the dataset. In the inference stage, we follow previous threshold-based methods [5] to classify image pixels into anomalous objects $(D_{u})$ if

where $\delta$ denotes the threshold. An ideal anomaly segmentation model is expected to predict a class distribution $\{f_{i}(X_{h,w})|i=1,2...,N\}$ which is extremely close to the uniform distribution $\mathcal{U}(.)$ as an anomalous pixel $X_{h,w}\in D_{u}$ is given. In the following sections, we will focus on explaining the training process of our anomaly segmentation framework.

The auxiliary data plays a crucial role in our approach and is widely adopted in existing anomaly classification methods [7, 8, 10]. Nonetheless, the selection bias of known-unknown samples raises serious concern and remains rarely explored in anomaly segmentation (see Tab. I). Furthermore, within the more complicated problem setting of semantic segmentation, we cannot simply leverage a whole image drawn from another dataset as an auxiliary sample.

To solve the above problems, we propose Masked Gradient Update (MGU) addressing the aforementioned issues by producing auxiliary images comprised of both in- and out-of-distribution regions in an adaptive manner. As shown in algorithm 1, we use $I_{\tilde{y}}$ to indicate the indices of those pixels belonging $\tilde{y}$, and $P_{\tilde{y}}$ denotes the indices of pixels predicted as $\tilde{y}$ correctly. Given an initialized auxiliary image $X^{\prime}$, we arbitrarily choose one of the classes existing in that image as the adversarial class $\tilde{y}\in\{1,2,...,N\}$. Then, our proposed MGU will iteratively update the pixels included in $P_{\tilde{y}}$ conditioned on the cost function $L_{mgu}$. The cost function is defined as

During the process, the parameters of $f$ are frozen and only the pixels within the target regions will be updated along the descending gradient direction of $L_{mgu}$. By minimizing the softmax probability of $\tilde{y}$, those $X^{\prime}_{h,w}\in I_{\tilde{y}}$ would be transformed into the synthetic-unknown pixels, which can also be regarded as the pseudo-unknown samples used for training an anomaly segmentation model. The auxiliary dataset can therefore be constructed by obtaining such synthetic-unknown images for each of the $N$ categories.

The concept of MGU is illustrated in Fig 2(a). Unlike previous methods [8, 10] treating a known class as unknown objects, we aim to enforce the embeddings of synthetic-unknown instances to locate outside the decision boundary of known categories. Punishing the pixels for having large $S_{\tilde{y}}(X^{\prime}_{h,w})$ can be regarded as a process to enforce the embeddings of these pixels to move from their cluster center towards the decision boundary. Note that once a pixel has been updated and predicted ”incorrectly”, it will immediately stop being updated. Besides, the $\mathcal{L}_{mgu}$ encourages the pixels to be updated against their ground truth labels, so it can be expected the $S_{\tilde{y}}$ of each pixel to be small at the last few updates. Therefore, we can expect the gradient of the last update of most pixels is too small to drag the samples far away from the boundary. Such a property is highly desired because the synthetic-unknown instances would be the most discriminative pseudo-unknown samples that clearly identify the boundary of known categories.

After we generate the synthetic-unknown data through MGU, the next step is to fine-tune $f$, which is pre-trained on a normal semantic segmentation task, with those auxiliary data. As illustrated in Fig. 2(b), the idea is that we could learn to distinguish anomalies between in-distribution samples and the synthetic-unknown samples around the original model decision boundaries. Following the threshold-based method in Equation 2, our goal is to lower the maximum probability value for those regions in $D_{s}$ while maintaining the original semantic segmentation output for regions in $D_{k}$.

For pixels in $D_{s}$, we apply the $\mathcal{L}_{KL}$ loss [7, 8, 10] to minimize KL divergence between the output probability distribution and uniform distribution. $\mathcal{L}_{KL}$ is defined as:

where $X^{\prime}_{h,w}$ is a pixel sampled from $D_{s}$, and $z=1/N$. This forces the output probability distribution towards uniform distribution for the anomaly areas.

Combined with the standard semantic segmentation cross-entropy loss $\mathcal{L}_{k}$, the objective function for fine-tuning is defined as:

where $\alpha$ is weighting the hyper-parameter between the loss of standard semantic segmentation and anomaly segmentation. $\mathcal{L}_{k}$ calculates on all the pixels in $\mathcal{D}_{k}$, and $\mathcal{L}_{KL}$ calculates on all the pixels in the $\mathcal{D}_{s}$.

In this paper, we also explore another loss function option, $\mathcal{L}_{ER}$, for anomaly segmentation by measuring the entropy of softmax output. The entropy ratio loss $\mathcal{L}_{ER}$ is:

where $r$ is a regularization term. The $\mathcal{L}_{ER}$ aims to maximize the output entropy of the pixels belonging to $D_{s}$.

Through the anomaly-aware fine-tuning, using the data generated from MGU, the model is able to perform the original semantic segmentation task and detect anomaly regions through thresholding.

In order to verify the effectiveness of our proposed method, we conduct experiments on two anomaly segmentation datasets (StreetHazards and BDD-Anomaly datasets). StreetHazards dataset [21] is a large-scale anomaly segmentation dataset containing 5125 training images, 1031 validation images, and 1500 testing images with 12 object classes for self-driving applications. Noted that for the testing images, a total of 250 unique anomalous objects appear in the image. BDD-Anomaly dataset derives from the BDD100K semantic segmentation dataset [24], a semantic segmentation dataset with diverse driving conditions. We follow the experimental setup in [21], regarding the classes motorcycle and train as anomalous objects. The dataset contains 6688 training images, 951 validation images, and 361 testing images. For network architectures, we use two state-of-the-art backbones, Deeplabv3 [19] and PSPNet [18], to validate the performance of our method on the StreetHazards dataset, and we train a Deeplabv3 for anomaly-aware semantic segmentation on the BDD-Anomaly dataset.

For both backbones, we first train the model with Adam and the learning rate $0.0001$ for $1000$ epochs with batch size $256$ on standard semantic segmentation task. Then, for auxiliary data generating, we choose $300$ images for each in-distribution class to apply MGU. The Deeplabv3 model is fine-tuned with Adam and the learning rate $0.0001$ for $10$ epochs with batch size $256$. The PSPNet model is finetuned with Adam and the learning rate $0.0001$ for $30$ epochs with batch size $256$. The hyper-parameter $\alpha$ in Equation. 5 is $0.05$, and the regularization term $r$ in Equation. 6 is $(-\sum_{i=1}^{12}z\log z)\times 0.01$, where $z$ is $1/12$ in this case for the both backbones.

We use the following three standard metrics to evaluate the effectiveness of the anomaly-aware semantic segmentation model. In this paper, we treat the unknown pixels as positive samples for all three metrics.

AUPR is the Area Under the Precision-Recall curve. In safety-critical applications, it is necessary to detect all anomalous objects. AUPR score focuses on evaluating the model performance on positive samples (i.e.  anomalous samples), and it is more sensitive to positive and negative class imbalances. When positive samples are relatively small, AUPR is a more comprehensive indicator that reflects the data imbalance in the score. It is a threshold-independent metric, but the random guess score is the ratio between the number of positive and negative samples.

AUROC is the Area Under the Receiver Operating Characteristic curve. It is the probability for a positive sample to get a higher score than a negative sample. AUROC is also a threshold-independent metric, and the score of random guesses is always $50\%$ regardless of the ratio of the number of positive samples to the number of negative samples.

FPR95 means the false positive rate at $95\%$ true positive rate. FPR at $x\%$ TPR tells how many false positives are inevitable for a given method to obtain the desired recall $x\%$.

To investigate the selection bias of known-unknown-based methods [8, 10], we conduct a pilot study with the same settings except for merely different instance categories serving as known-unknown samples. Specifically, the $12$ classes in StreetHazards dataset [21] are split into $4$ subsets. Each subset consists of $3$ classes without overlaps, and in each sub-experiment, one of the subsets will be chosen as the anomaly subset. Those instances in an image belonging to the categories of the anomaly subset should be regarded as anomalies for training. Such an experimental setting is consistent with those methods using auxiliary data [8, 10], except that we explore anomaly segmentation rather than classification.

As shown in Tab. I, the significantly fluctuating performance indicates that known-unknown class selection heavily influences the results. The results reveal our hypothesis of known-unknown class selection correlates to model performance. Therefore, known-unknown classes could barely represent the out-of-distribution regions. On the other hand, our proposed MDG can adaptively generate synthetic-unknown instances based on observing distribution instead of choosing particular categories as known-unknown instances.

Tab. II demonstrates the experimental result on the StreetHazard dataset. Using the backbone of both Deeplabv3 and PSPNet, our MGU+AaFt with entropy ratio loss reaches state-of-the-art performance on AUPR and AUROC metrics while having a competitive FPR95 score. As there are only $1.5\%$ unknown pixels in the StreetHazards dataset, metrics that do not consider data imbalance might underestimate the significance of detecting unknown pixels. The impressive AUPR scores ($5.3\%$ and $1.6\%$ enhancement with Deeplabv3 and PSPNet), which is the only one regarding data imbalance, suggest the capability of anomaly awareness of our proposed method. Besides, our method also performs well on the BDD-Anomaly dataset, as shown in Tab. III. The results reveal that our method outperforms the best previous score by $6.5\%$ in AUPR. The results indicate the effectiveness and the robustness of our proposed components.

Also, more than achieving remarkable performance, our method is more resource-efficient. Prior works either require multiple models (Deep Ensemble [14], TRADI [23]) or reconstruct original testing images (SynthCP [22]). Additionally, our approach is model-agnostic and has huge potential to further boost the performance of future models or applications.

Fig. 3 demonstrates several cases on the StreetHazard dataset. MSP method alone is unable to tackle anomalies on the roads and leads to critical safety issues. With the same backbones, we finetune the model with our generated auxiliary data and enable the model to recognize unknown objects.

In the ablation study, we measure the effectiveness of each designed module. As shown in Tab. II, by training with our MGU generated, we boost the MSP (first block, second row) from 6.6 AUPR to 13.2 (second block, second row). Considering MSP is a relatively simple method, the performance gain is impressive. Additionally, it reveals possible further performance improvement of training modern models with MGU generated data.

Our entropy ratio loss $\mathcal{L}_{ER}$ aims to investigate the influence of loss usage, further improving the performance from 13.2 AUPR to 14.6 (second block, third row). As a step toward, the result suggests a potential research direction for future work.

The AUPR, AUROC, and FPR95 metrics measure the relative output confidence scores between in-distribution pixels and anomalous pixels regardless of the correctness of semantic segmentation. In this section, we introduce how thresholding influences the performance of semantic and anomaly segmentation. Fig. 4 shows the semantic and anomaly segmentation accuracy of our method and MSP. As increasing $\delta$, more pixels are classified as anomalies. Note that the MSP model has not been fine-tuned with anomaly data, so ideally adjusting the threshold $\delta$ affects little in the semantic segmentation performance; however, it will encounter the overconfidence issue when fed with anomalous inputs. Overall, as increasing $\delta$, the anomaly-aware semantic segmentation model trained with our method can significantly improve the anomaly segmentation accuracy, while the model still maintains desirable semantic segmentation performance.