t-RAIN: Robust generalization under weather-aliasing label shift attacks

Tackling image corruptions for improved perception has been a long-standing challenge in the field of computer vision [2, 1, 26]. As newer datasets have introduced weather-based corruptions [3, 17, 41] for improving robustness, the awareness of this subject is on the rise. Recently, multi-weather robustness has been the focus of several works which proposed ideas like stacking [27], ensembles [39] and image restoration [25], the performance of classic benchmark models still fail on extreme weather conditions. In the history of label shift methods, several works study correcting label shift and generalization in general, especially in unsupervised settings [11, 12, 10]. The progress in this field is rapid due to the parallel development of autonomous vehicles and need for explainability for trust-worthy AI systems for the future.

The DAWN dataset [17] and WEDGE dataset [23] present interesting adversarial weather conditions including fog, rain, snow , dust as visible in Figure 1. The most recent benchmark on these datasets [25, 23] presents state-of-the-art results and demonstrates effectiveness of using synthetic data augmentation in the task of overall object detection.

The recent development in large vision-language models [29, 28] and refined generative techniques, have led to creation of more realistic synthetic images. The natural advancement would be adoption of synthetic images to augment limited real-world datasets. However, this adoption is limited by the cost, realism, availability and usability of synthetic images. By incorporating synthetic images [23] in this work, we demonstrate one positive use case of such images.

Finding people in imagery has been a long-standing challenge in computer vision, with several decades of prior work [6, 37, 44] setting up the foundation for examination of finer problems in modern computer vision. The creation of high-quality datasets [6, 42, 9, 13] was a contributing factor to rapid development of powerful algorithms capable of detection in challenging conditions. Classical algorithms combined with novel architectures for robust detection were the focus of many works in vision encompassing multi-scale detection, occlusion invariance and cascaded rejection classifiers [38, 40, 24, 4, 21, 32, 43].More recently, tracking and detection of pedestrians in the real-world has been solved using a variety of deep networks, algorithmic strategies and forecasting approaches [18, 33, 7, 8].

We employ the DAWN Dataset [17] and the WEDGE dataset [23] to test the efficacy of our strategy. The DAWN dataset is a 1000 image object detection dataset that includes traffic imagery in bad weather like rain,fog, dust and snow. WEDGE is a synthetic dataset that employs the DALL-E 2 model [29, 28] with prompts encompassing 16 weather and season conditions with a focus on autonomous vehicle scenarios. It features images captured during severe weather, such as rain, snow, fog, and sandstorms (Refer Figure 1).

In limited data settings, generalization capabilities are naturally limited by the number of examples seen by the classifier. However, with the development of large-scale vision-language models (stable diffusion, DALL-E), access to unlimited synthetic datasets has become much easier. We propose an algorithm that can leverage classical classifiers with synthetically generated data to provide generalization capabilities even in limited data settings ($<160$ images).

The algorithm works by mapping similarity between the source distribution classes (C) and target synthetic classes (C) to sample relevant images and up-sample for each class. The technique currently performs uniform weighting but can be extended to weighted sampling in the future works. Once the source class and target class are compared (this can be between class and prompt keywords as well), we filter the images with greatest similarity (currently cosine similarity 1).

We return this set through the oracle until all classes are mapped and sufficient samples (determined by hyper-parameter $\beta$) from the target dataset (of size $\eta$) are satisfactorily generated. The filtered target samples from class with closest class similarity to source sample ($X_{i}$) are the augmentation set.

As visible in Table 2 and Figure 4, the model performance drops significantly when restricted to limited data environments as expected. The models begin overfitting on training set and are unable to generalize to multi-weather conditions. EfficientNetV2S and ConvNeXtSmall (the weakest learning models) have shown improvement on limited data settings, indicating the pseudo-generalization capabilities of weak learners, albeit extremely poor performance due to underfitting.

We simulate label shift by boosting one class at a time (Figure 2) and a simple random affine transformation on the remaining set sizes. The goal is to observe shift when specific weather conditions dominate the target distribution.

We can see from Table 3 and Figures 3,6,7, some model-specific trends:

1. 1.

   Xception model which had fit very well on snow class, performed better when the snow class was positively biased in the target distribution. Similarly it performs worse when exposed to other shifts which it had learned poorly. Upon applying t-RAIN it shows 3 % increase in accuracy for no-shift and fog shift conditions. It shows 5% increase in accuracy for dust-shift attacks.

2. 2.

   VGG-16 is one of the stronger learning models with higher performance values. It performs similar on all classes and attains high performance under all shifts, especially snow. When synthetic data is sampled via t-RAIN, it consistently outperforms baseline VGG-16 with 1-4% increases. This may point towards better generalization of stronger models due to efficient learning of complex representations from the extended input space.

3. 3.

   VGG-19 shows similar trends to VGG-16 but with 3% under-performing margin for rain and dust shifts.

4. 4.

   ResNet50 is one of the weakest learners with second-last performance in most cases. It generalizes better in limited data due to under-fitting and shows mediocre results which improve marginally for snow shifts. t-RAIN dramatically improves the ResNet performance on all shifts by 2-11 % accuracy. This is an important result, as we observe that variability in data can boost both strong and weak learners, with greater effects on weak learners.

5. 5.

   EfficientNetV2S is a significantly weak learner with worst performance on dust shift. t-RAIN is only able to boost performance by 3-4 % on dust and fog shifts.

6. 6.

   ConvNextSmall is also one of the more poorly performing models with almost zero robustness to weather conditions like dust shift. Although t-RAIN improves robustness under all conditions by 3 - 8 % except rain and no-shift the model suffers under limited data constraints and plummets to bottom rank.

7. 7.

   MobileNet features good performance and fast training. It is not robust to fog corruptions but otherwise provides reasonable results with 1-4% boosts in majority classes and worse under rain and no-shifts.

8. 8.

   DenseNet features similar trends with baseline performance mainly for snow shifts. It features the most common boost of 1-3 % over all shifts uniformly.

9. 9.

   MobileNetV2 features good performance and fast training. The model improved with t-RAIN under rain and fog shifts by 5 % and 3 % respectively.

10. 10.

    Inception V3 is an average learner but suffers adversely from fog shift. Adding t-RAIN to such models significantly improves the performance which is a remarkable result. There is 8 and 16% increase in test accuracy after using t-RAIN to improve no-shift and fog robustness in Inception V3.Adding t-RAIN improves the performance the most dramatically out of all the other results with 16% increase in test accuracy here under fog shift which is a remarkable result.

Some general observations include snow being one of the easiest weather classes to recognize due to significant distinguishing characteristics. Models suffer from easier evaluation when snow is considered as one of the evaluation classes. For true robustness evaluation such classes should be held out and only measured as sanity checks and not robustness measures. As visible in Figure 6, the t-RAIN algorithm is able to improve generalization , for all learners from strong learners like VGG-16 to weak learners like EfficientNetV2S and outperforms the performance on limited data with synthetic augmentation. The averaged improvement across all 5 shifts are 2.1, -0.8,4.4,1.9, 2.7 % respectively. One interesting result presents a question of why t-RAIN boosts specific class -shifts performance inspite of underlying uniform distribution and uniform augmentation. This could potentially be attributed to special variations and robustness introduced by the synthetic data which helped gain generalization capabilities beyond the source distribution.

We use the earlier analysis as a prior to analyze performance of models in detection pedestrians under anomalous weather conditions. As visible in Table 4 and Figure 8, the best detection performance is under Real Snow Conditions with 82.69 AP and Synthetic Fog Conditions with 62.31 AP when evaluated on DAWN and WEDGE datasets. Intuitively another direction we explore the adversarial difficulty of each weather condition for detecting pedestrians. In real data, dust weather appears to obstruct vision for pedestrian detectors the most whereas in synthetic data, snow appears to obstruct vision the most. This is a surprising result as snow is the easiest weather in real data, which is the opposite in synthetic data. Due to this adversary, we can understand now why t-RAIN improves generalization capability of 7/10 models under snow shift with a maximum increase of 10% accuracy. However, conversely, ease in detection like fog conditions does not imply ineffectiveness of t-RAIN. t-RAIN improves generalization capability of 10/10 models under fog shift with a maximum increase of 16% accuracy. Further analysis can be done in the direction of discovering distribution shift induced by synthetic generation. This analysis was not performed in the scope of this study, but there can be multiple possible underlying factors for the results reported in Table 4. Firstly, we can see that generated humans in synthetic data appear out-of-distribution due to the trust and security layers implemented for privacy concerns and obscuration. Thus evaluation of pedestrian detection across Sim2Real data is not actually an informative indicator of generative accuracy but may hint towards undiscovered generative anomalies. Models that work well on real-data and poorly on synthetic data could be suffering due to (a) Real2Sim gap (b) Beneficial adversarial robustness or (c) Harmful generative anomalies that do not help real-world models. Identifying the exact cause for performance shift is an interesting challenge that we propose for future works.