StyleAugment: Learning Texture De-biased Representations
by Style Augmentation without Pre-defined Textures

While previous de-biasing method, such as fairness, assumes the existence of bias labels (e.g., assuming that there exists the protected attribute labels during the training), it is often unrealistic to assume the all bias labels are accessible by the model; labeling requires a huge human annotator costs, and the bias labels are often ill-defined, e.g., it is difficult to categorize natural textures in pre-defined texture sets. To avoid the direct use of bias labels, de-biasing without explicit bias annotation has been actively studied in recent years. Especially, utilizing two networks, one for capturing bias and the other for de-biasing from the biased network, have been one of the major branches in this field. For example, in VQA tasks, the networks are known to be easily biased towards much easier text representations, while ignoring visual representations. To mitigate the issue, RUBi [3] and LearnedMixin [6] adjusted the final logit before the cross-entropy loss of the multi-modal model by using predictions from the uni-modal model where RUBi uses multiplication and LearnedMixinH uses summation. In the uni-modal debiasing tasks, HEX [36] employs a texture extractor model and lets features extracted by the target model be orthogonal to the texture extractor outputs. ReBias [1] makes two models have different capacity so a model is biased towards specific cues, e.g., color or texture. Learning from Failure (LfF) [29] uses the re-weighting cross-entropy where the biased network is trained by the knowledge that networks in the early stage of training rely more on the spurious correlation than networks in the late stage of training.

While previous unsupervised de-biasing methods utilize additional biased model, our method does not need any additional biased model, but only requires a pre-trained style transfer algorithm, such as AdaIN [21]. In the experiments, our method outperforms previous de-biasing methods in ImageNet clean accuracy as well as the ImageNet robustness benchmarks, such as texture de-biased accuarcy [1], ImageNet-C [14] and ImageNet-A [16].

Since deep models are data hungry [27], synthesizing abundant training images by random crop, random flip, and color jittering have become a rule-of-thumb for enhancing the generalizability of deep models [34, 20]. Recent data augmentation methods have attended on either synthesizing mixed samples [42, 40], or applying a series of very strong image filters, such as equalize, solarize, Cutout [9], to make difficult samples [7, 15, 8]. Our method can be viewed as a variant of Mixup approaches, such as Mixup [42], CutMix [40], while our method does not mix labels of samples to prevent a model attending on spurious correlations, (i.e., textures) rather than the objectness (i.e., shape). In this study, we did not compare our method with augmentation methods with visual filters, such as AugMix [15] or RandAugment [8], because these algorithms need to pre-define augmentation types by a strong human prior knowledge, while our goal is to make universally applicable data augmentation algorithm without a strong inductive bias.

Since Geirhos et al. [11] have shown that deep convolutional neural networks are biased towards textures, and a simple stylization-based augmentation (StyleImageNet) can mitigate the texture biases, stylization-based augmentation methods have been studied in a robustness view point. While the previously proposed data augmentation methods showed that the performance improvements in existing robustness benchmarks [42, 40, 15], the data augmentation methods for in-distribution generalization is known to be not helpful when the semantic gap between the training images and the test images is significantly large, such as domain generalization benchmarks [4]. From this motivation, Zhou et al. [43] proposed a Mixup-like stylization augmentation method named MixStyle for domain generalization benchmark. MixStyle mixes two images from different domains to let the model be generalized to diverse domains. Our method focuses on a conventional image recognition task (i.e., the ImageNet classification task) in the de-biasing aspects (i.e., additional ImageNet robustness benchmarks). The stylization-based augmentation strategy also studied by Hong et al. [19], a contemporary work of our study. Hong et al. proposed to utilize both the content loss and the style loss by mixing labels as Mixup [42] or CutMix [40]. Our method does not mix the labels to avoid confounding factors, i.e., textures.

Stylized ImageNet [11] is proposed to mitigate the texture bias of deep models, not only learning shape-biased features but also showing performance improvements in downstream tasks, such as ImageNet classification, object detection, and the ImageNet-C corruption robustness benchmark. Stylized ImageNet, however, has fundamental limitations on its image fidelity (as the significant gap between the artistic paintings and the target natural images) and lack of the style diversity for each image (as the data generation process generating all training images before training). Our method mitigates these two issues by utilizing the natural images from the mini-batch as the style reference, showing better fidelity and diversity as shown in Figure 1.

Arbitrary style transfer tasks [10, 21, 26] aim to generate an image with the given two images, i.e., content and style images. The underlying assumption of style transfer methods since Gatys et al. [10] is that the feature statistics, including mean and standard deviation, represent the texture of images. Primal studies [10] directly optimize the input image to match the feature statistics of the content (mean) and the style (covariance, or Gram matrix) in an iterative manner. Real-time style transfer methods, such as whitening-coloring-transform (WCT) [26] or adaptive instance normalization (AdaIN) [21], approximate the optimization process by replacing feature statistics of content and style images. We use AdaIN style transfer method for real-time data augmentation.

Let $Enc$ and $Dec$ be an encoder and a decoder. AdaIN style transfer first extracts a feature $f$ from the input image $x$ by $f=Enc(x)$. For the content and style images $c,s$ and their corresponding features $f_{c},f_{s}$, the AdaIN operation is defined as follows:

where $\mu(\cdot),\sigma(\cdot)$ denotes the feature statistics from the instance normalization. The transferred feature is decoded by the decoder as $\tilde{x}=Dec(AdaIN(f_{c},f_{s}))$. We use the ImageNet-trained VGG-16 network [33] as the encoder $Enc$, and the official AdaIN decoder as the decoder $Dec$ following the official implmentation²²2https://github.com/xunhuang1995/AdaIN-style.

For the given mini-batch $\mathcal{B}=(x_{1},x_{2},\ldots,x_{n})$ with batch size $n$, StyleAugment generates a new augmented mini-batch $\mathcal{B}^{\prime}$ by augmenting stylized images where the style references are from the other samples in the mini-batch. In each training iteration, StyleAugment randomly combines styles from the mini-batch to make diverse image samples. In practice, we train the model with the original mini-batch and the augmented mini-batch, i.e., $\mathcal{B}+\mathcal{B}^{\prime}$. We illustrate the PyTorch pseudo code for StyleAugment in Figure 2. StyleAugment does not require any prior knowledge on the pre-defined textures (e.g., Stylized ImageNet [11]) or strong image visual filters (e.g., AutoAugment [7], AugMix [15], RandAugment [8]). StyleAugment training strategy enables the mini-batch-level knowledge interaction as Mixup variants [42, 40], but StyleAugmentation does not mix labels which can make the model rely on spurious correlations, such as textures or backgrounds. We will discuss the details in §4.3.

Compared to the artistic stylized images, natural stylized images show benefits on the image fidelity. Figure 1 shows the example stylized images by artistic style transfer (Figure 1(b)) and by the proposed StyleAugment (Figure 1(c)). Interestingly, the images generated by StyleAugment show diverse distribution shifts, rarely observed in real-world. For example, in the second row of the figure, the background of the globefish seems as a grassfield. Similarly, in the fourth row, the dog shows the “eye” texture from the butterfly image, and the background changes from snow to the grass-like texture. In other words, StyleAugment generates an image with uncommon correlations, such as “a globefish on the grassfield” or “a dog with butterfly patterns”. By augmenting rare combinations of the true objectness and the other confounding factors, StyleAugment makes a model learn de-biased representations to the spurious correlations, such as background, textures, or color.

However, StyleAugment still has a limitation on its image quality. For example, as shown in the first row of Figure 1, StyleAugment tends to preserve the original content information compared to artistic style transfer, but there exists the damage of the content information. This may hurt the final performance of the model as shown in [11]. To understand the trade-off between shape-preserving and stylization, we evaluate our method with a photorealistic style transfer method [39], focusing on the edge preserving, but only transferring color information. Our experimental results show that StyleAugment with AdaIN shows worse results in in-distribution generalization performances, but better results in distribution shift generalization. We will discuss the details in §4.3.

We conduct ablation studies of design choices of StyleAugment in ImageNet-9. We tested two variants of StyleAugment in the image quality and the target label. First, as we discussed in the previous sections, the image quality by AdaIN is not perfect (Figure 1). Especially, AdaIN loses edge and shape information of the original image. We use a photorealistic style transfer model WCT², focusing on preserving edge information by the Haar wavelet transform. Note that WCT² transfers color and light information, while AdaIN transfers texture information. In the second row of Table 4, we report the StyleAugment performances when AdaIN encoder and decoder modules are changed to WCT² encoder and decoder. For real-time computation, we modify the original WCT² algorithm from whitening-coloring-transform [26] to adaptive instance normalization (Equation (1)). We observe that StyleAugment with WCT² shows better in-distribution generalization performance (94.1%) than StyleAugment with AdaIN (93.8%), as well as ImageNet-A (29.6% $\rightarrow$ 31.7%) and occlusion performances (73.0% $\rightarrow$ 77.6%). However, StyleAugment with WCT² shows drastic performance drops in ImageNet-C (65.3% $\rightarrow$ 57.0%) by losing texture information. To sum up, the better image quality by content preservation improves in-distribution generalization performances, but cannot generalize the corruption robustness by losing advantages of texture transferring.

We also test a variant on the target labels of StyleAugment. First, we mix content and style labels for the target label as mixup variants. We observe similar phenomenon to the results of WCT²; it improves clean accuracy, but drops ImageNet-C performance. We assume that it is because StyleAugment allows the model to observe a sample with various spurious cues, such as texture and background (as shown in Figure 1). However, if we mix the content and style labels, the model can attend unexpected prediction cues, rather than object information itself.