Supervised Contrastive Learning on Blended Images for Long-tailed Recognition

Re-sampling methods create a subset of training data that consists of class-balanced samples. Oversampling [4, 1] or undersampling [21, 32] methods have been proposed to achieve the class balance of the subset. Oversampling approaches match the number of samples per class by repeatably collecting tail class instances. On the contrary, undersampling methods omit head class samples to equalize the label distribution. Training with the balanced subset reduces defects caused by the class imbalance. However, the lack of diversity of the tail class data still needs to be solved, which endangers the network to overfitting. Re-weighting methods calculate weights of training samples for the training loss function. The weights could be derived from the label distribution [2, 7, 25], or instance-level scoring functions [16, 14]. The weighted loss decreases the dominance of head information, which leads to the trade-off between the head and tail class recognition performances. Using multiple experts [29, 13] also shows good performance. The experts compensate each other to boost the recognition performance. More recently, Mixup-based data augmentation has attracted attention in the field of long-tailed recognition. Remix [6] introduces a class distribution-aware labeling for the image mixtures. Park et al. [22] suggest a MixUp-based data augmentation method, named CMO, to increase the information diversity of head-biased data. The variance-enhanced training data provokes a more generalized feature space, leading to better recognition performances. Our method is in line with Remix and CMO in terms of using data augmentation to adjust the biased distribution of training data. The difference is that SMC utilizes the generalization capability of SCL on top of the mixtures.

Supervised contrastive learning [11] utilizes class labels to define positive and negative pairs of contrastive learning. The positive pairs get closer, while the negative pairs get farther away. SCL shows good performance in various applications [11, 17, 33]. However, in long-tailed recognition, using SCL directly causes a head-biased feature space and a biased classifier that could hinder the network [9]. To reduce the bias, KCL [9] screens the positive pairs during training. Li et al. [15] introduce TSC, a two-stage learning method with regularization on the uniformly pre-defined class center points. BCL [34] utilizes the classifier weights as the additional class information for SCL. Our concept is inspired by the feature regularization approaches mentioned above. The difference is that our method balances the feature space using augmented data without strong regularization.

MixUp [31] is proposed for more robust training of neural networks. It shows that using mixed instances is in line with the risk minimization of neural networks. Yun et al. [30] claim that the linear combination of images in MixUp creates unnatural and ambiguous images that could deteriorate the performance of networks. They propose CutMix, which replaces the linear combination of images in MixUp with random cropping and pasting to create locally meaningful images. Qin et al. [24] point out the possible information loss of the random cropping in CutMix. They introduce ResizeMix, which uses image resizing rather than the random cropping. The high generalization capability of MixUp-based augmentations boosts various computer vision applications [19, 22]. Recently, SDMP [26] has utilized MixUp in self-supervised learning. The main concept of SDMP is using ResizeMix to create positives for self-supervised contrastive learning, similar to our idea. The difference between SDMP and SMC is that our method aims at supervised contrastive learning with class labels on long-tailed recognition, while SDMP focuses on self-supervised learning.

The imbalance of the latent feature space is the main challenge to use SCL in long-tailed recognition. Applying SCL without considering class imbalance results in a head-biased feature space, which discourages tail class recognition. Previous studies add regularization to the network to balance the feature space. However, we suspect that the strong regularization could lead to suboptimal results since tail classes still have small variances. Therefore, we use the MixUp-based [31] data augmentation method to expand the variety of training data. As illustrated in Fig. 1, the interpolated data form a broader data space than the original data, which increases the generalization capability of the network. Furthermore, using the blended data could relax the information imbalance of the long-tailed data.

Among the variations of MixUp, CutMix [30] shows a good performance in the previous long-tailed recognition study [22]. However, the random cropping nature of CutMix may lead to loss of the critical information of foreground classes as illustrated in Fig. 2. Although the information-removed samples could prevent overfitting, the information loss is undesirable since CutMix assumes the presence of multiple class features in the blended image. Thus, we replace the random cropping with a resizing operation to secure the foreground class information, which leads to better generalization capability of the trained model. For foreground and background images $x_{f}$ and $x_{b}$ and their labels $y_{f}$ and $y_{b}$, the mixed image $\tilde{x}$ and its label $\tilde{y}$ are defined as follows:

		$\displaystyle\tilde{x}=\bm{M}\odot R(x_{f})+(\bm{1}-\bm{M})\odot x_{b},$		(1)
		$\displaystyle\tilde{y}=\lambda y_{f}+(1-\lambda)y_{b},$

where $\bm{M}\in\{0,1\}^{W\times H}$ is a binary mask for the images, $R(x_{f})$ is a resized and padded foreground, $\odot$ implies the Hadamard product and $\lambda\sim Beta(\alpha,\alpha)$ indicates the combination ratio that sampled from the beta distribution. We normalized the combination ratio in the range of $[0.2,0.8]$ to increase the diversity of training data by preserving both class features in the mixed output.

The next issue is the way to sample the foregrounds and backgrounds. On long-tailed data, the uniform sampling could lead to oversampling of head samples, which is undesirable in terms of data diversity. Inspired by the previous study [22], we randomly sample background images with a uniform probability, while foreground samples have a weighted sampler that samples the tail class instances more frequently. The weighted sampling for foregrounds balances between head and tail class information in the mixture distribution. The sampling probability for the $k$-th class is defined as follows:

$$q(k)=\frac{n_{k}^{-\gamma}}{\sum_{l=1}^{C}n_{l}^{-\gamma}},$$

(2)

where $n_{k}$ is the number of samples in the $k$-th class, $C$ implies the number of training classes and $\gamma$ is a hyperparameter that determines the sampling strategy. We set $\gamma$ to one based on the previous study [22].

SCL requires to define positives and negatives for training samples. Here, positives share the same class label, and negatives have samples with different labels from the target sample. However, this definition is inappropriate for the mixed data because a blended sample contains multiple class information. Therefore, we suggest three positive pair types as illustrated in Fig. 3: foreground-shared pairs, background-shared pairs, and cross-shared pairs. A foreground-shared and background-shared pairs share the foreground and background classes in a mixed image, respectively. A cross-shared pair is an image that shares at least one same class, but is neither a foreground-shared pair nor a background-shared pair. The negative set consists of the remaining samples except for the positives.

Previous contrastive learning methods augment the target image to produce positives [5, 11]. Among the various augmentation operations, random cropping is a crucial factor in enhancing the generalization capability [5]. However, the random cropping on the mixture is in danger of the information loss of the original images as illustrated in Fig. 4. Therefore, we suggest adjusting the original images before blending to secure the diversity for contrastive learning:

		$\displaystyle\tilde{x}=\bm{M}\odot R(\tilde{x}_{f})+(\bm{1}-\bm{M})\odot\tilde{x}_{b},$		(3)
		$\displaystyle\tilde{y}=\lambda y_{f}+(1-\lambda)y_{b},$

where $\tilde{x}_{f}$ and $\tilde{x}_{b}$ are the augmented foreground and background images, respectively. This approach preserves the information of foreground classes while benefits from the rich generalization capability of image mixing.

Finally, the SMC loss $L_{SMC}$ on a set of features of augmented images $\tilde{x}_{i}\in\tilde{\mathcal{X}}$ is defined as follows:

		$\displaystyle L_{f}=-\frac{1}{|\tilde{\mathcal{X}}|}\sum_{\tilde{x}_{i}\in\tilde{\mathcal{X}}}{\frac{1}{|\tilde{\mathcal{X}}_{i}^{f}|}\sum_{\tilde{x}_{f}\in\tilde{\mathcal{X}}_{i}^{f}}{\log{\frac{\exp{(\tilde{x}_{i}\tilde{x}_{f}/\tau)}}{\sum_{\tilde{x}_{j}\in\tilde{\mathcal{X}}_{i}}{\exp{(\tilde{x}_{i}\tilde{x}_{j}/\tau)}}}}}},$		(4)
		$\displaystyle L_{b}=-\frac{1}{|\tilde{\mathcal{X}}|}\sum_{\tilde{x}_{i}\in\tilde{\mathcal{X}}}{\frac{1}{|\tilde{\mathcal{X}}_{i}^{b}|}\sum_{\tilde{x}_{b}\in\tilde{\mathcal{X}}_{i}^{b}}{\log{\frac{\exp{(\tilde{x}_{i}\tilde{x}_{b}/\tau)}}{\sum_{\tilde{x}_{j}\in\tilde{\mathcal{X}}_{i}}{\exp{(\tilde{x}_{i}\tilde{x}_{j}/\tau)}}}}}},$
		$\displaystyle L_{c}=-\frac{1}{|\tilde{\mathcal{X}}|}\sum_{\tilde{x}_{i}\in\tilde{\mathcal{X}}}{\frac{1}{|\tilde{\mathcal{X}}_{i}^{c}|}\sum_{\tilde{x}_{c}\in\tilde{\mathcal{X}}_{i}^{c}}{\log{\frac{\exp{(\tilde{x}_{i}\tilde{x}_{c}/\tau)}}{\sum_{\tilde{x}_{j}\in\tilde{\mathcal{X}}_{i}}{\exp{(\tilde{x}_{i}\tilde{x}_{j}/\tau)}}}}}},$
		$\displaystyle L_{SMC}=w_{f}L_{f}+w_{b}L_{b}+w_{c}L_{c},$

where $\tilde{\mathcal{X}}_{i}=\tilde{\mathcal{X}}\setminus{\tilde{x}_{i}}$ is a set of the mini-batch features without $\tilde{x}_{i}$, $|\mathcal{X}|$ indicates the mini-batch size. $\tilde{\mathcal{X}}_{i}^{f}$, $\tilde{\mathcal{X}}_{i}^{b}$, and $\tilde{\mathcal{X}}_{i}^{c}$ are a set of foreground-shared pairs, background-shared pairs, and cross-shared pairs, respectively. $|\tilde{\mathcal{X}}_{i}^{f}|$, $|\tilde{\mathcal{X}}_{i}^{b}|$, and $|\tilde{\mathcal{X}}_{i}^{c}|$ indicate set sizes of the corresponding positives, and $\tau$ is a temperature parameter set to $0.1$. The loss weights $w_{f}=\lambda_{w}/1.5$, $w_{b}=(1-\lambda_{w})/1.5$, $w_{c}=0.5/1.5$ follow $\lambda_{w}$, the combination ratio of images in the pair. The weights are normalized to make the total summation of the weights to one. They represent the semantic relationship between images in a pair. Positive pairs with stronger relationships have a greater impact on the network than pairs with smaller relationships.

The cross-entropy function is a typical choice to train classifiers. In the long-tailed environment, however, it often results in a head-biased classifier due to imbalanced training data [25]. The logit compensation methods [3, 20] aim to reduce the bias by adjusting the classifier output with the knowledge on the class distribution. We use the compensation method to balance the classifier with the mixed data. Since the mixing augmentation increases the data variance, the mixtures aid the classifier to explore more broader data space. Furthermore, training the classifier and feature extractor simultaneously achieves the one-stage training of the model. With the blended data, the classifier is trained with the following loss function:

$$L_{BCE}=CE(f(\tilde{x})+\bm{m},\tilde{y}),$$

(5)

where $CE(\cdot,\cdot)$ is the cross-entropy function, and $\bm{m}=\log{(\mathbb{P}_{Y})}$ is a prior vector of the class distribution $\mathbb{P}_{Y}$ with the logarithm function for the compensation. The prior vector gives higher values for head classes, which emphasizes tail class instances during training.

By combining the losses defined above, the total training loss is defined as follows:

$$L_{train}=L_{BCE}+\eta L_{SMC},$$

(6)

where $\eta$ is a weight hyperparameter. Figure 5 illustrates the summary of SMC.

We used CIFAR-100-LT [3], ImageNet-LT [18], and iNaturalist 2018 [28] to evaluate SMC. The datasets have different imbalance ratios. The imbalance ratio $\rho$ is defined as the ratio of the number of instances in the largest head class to that of the smallest tail class. CIFAR-100-LT is a subset of the CIFAR-100 dataset [12]. It artificially excludes training data to make 100 long-tail distributed classes. We used three imbalance ratios for comparisons: 100, 50, and 10. ImageNet-LT is a subset of the ImageNet 2012 dataset [27]. Similar to CIFAR-100-LT, it selects training data to create a set of 1000 imbalanced classes. There are $115.8\text{K}$ images and its imbalance ratio is 256. On the contrary to the previous datasets, the iNaturalist 2018 dataset is imbalanced from the data collecting stage. There are $437,513$ training images of $8,142$ classes, where the imbalance ratio is 500.

We followed the training strategy of previous long-tailed recognition studies [9, 15, 22] for a fair comparison. we used the ResNet-32 [8] backbone on CIFAR-100-LT and the ResNet-50 [8] backbone on ImageNet-LT and iNaturalist 2018. On CIFAR-100-LT, we followed the training strategy in [2]. For ImageNet-LT and iNaturalist 2018, the initial learning rate is set to $0.1$. In the ImageNet-LT case, the number of training epochs is set to 100 and the learning rate is decreased by $0.1$ at epoch 60 and 80. The number of samples in a mini-batch is 128. In the iNaturalist 2018 case, we trained the network 200 epochs and we reduced the learning rate at epoch 75 and 160 by $0.1$. The training mini-batch size is set to 256 for the iNaturalist 2018 dataset. For all datasets, we set the loss weight $\eta$ to $0.1$ and the output dimension of the 2-layer MLP for contrastive learning to 128. The $\alpha$ value for the Beta distribution is set to $1.0$, and the hidden layer size of the head follows the input feature size. Please see the supplementary material for further details.

Since SMC utilizes the power of SCL and MixUp-based augmentation, we select SCL-based methods [9, 15, 34] and MixUp-based methods [6, 15] with previous long-tailed recognition methods [10, 7, 16, 2] as the comparison baselines. The comparison result on CIFAR-100-LT is in Table 1. SMC shows better performances than baselines in all comparisons. This result implies that SMC is effective for various long-tailed conditions. The mixed samples increase the semantic information space of training data, which improves the discriminative power of the network. Furthermore, the blended data gives a soft balancing regularization to the network. The soft regularization reduces the bias in the feature space that leads to better recognition capability.

The performance comparisons on the ImageNet-LT and iNaturalist 2018 datasets are in Table 2 and Table 3, respectively. The overall recognition performance of SMC is still better than the baselines. This comparison result demonstrates that SMC is still effective when the data distribution is complex. Compared with TSC, an SCL-based two-stage learning method, SMC has better performance on medium and few shot classes, while the head class performance is weaker than TSC. It shows that the broader training data space and soft regularization are helpful for low-shot classes, but harms the classification on major classes. Nevertheless, the performance gain on low-shot classes significantly improves the overall recognition capability. We leave the performance balancing between many-shot and low-shot classes as an open problem.

In this subsection, we evaluate the contribution of SMC by analyzing the feature space created with SMC. We use two metrics to assess the trained feature space: Inter-class and Semantic similarity scores.

Inter-class score $IS_{k}$ for the $k$-th class is defined as follows:

$$IS_{k}=\exp\left(-\frac{1}{C}\sum_{j=1}^{C}{\left(\bm{c}_{k}-\bm{c}_{j}\right)/\tau^{\prime}}\right),$$

(7)

where $C$ is the number of classes, $\bm{c}_{k}$ is a class feature vector calculated from an average of the feature vectors of the $k$-th class, and $\tau^{\prime}$ is a temperature value to scale score values. We set $\tau^{\prime}$ to 10 for comparisons. The lower inter-class score means that the class centers are far apart from each other, which implies that the network can distinguish different classes more easily.

The well-trained feature space contains the semantic information of the training classes. If two classes are semantically similar with each other, they will be placed close to each other within the space. Semantic similarity score measures the similarity between class semantic information and the latent space. It measures the difference between cosine similarities of class semantic vectors and that of the class feature vectors. Semantic similarity score $SS$ is defined as follows:

		$\displaystyle SS=\frac{1}{C^{2}}\sum_{i=1}^{C}{\sum_{j=1}^{C}{||\mathbf{S}_{i,j}^{s}-\mathbf{S}_{i,j}^{c}||_{1}}},$		(8)
		$\displaystyle\mathbf{S}^{s}_{i,j}=\frac{\bm{s}_{i}\bm{s}_{j}}{||\bm{s}{i}||_{2}\cdot||\bm{s}_{j}||_{2}}\quad\mathbf{S}^{c}_{i,j}=\frac{\bm{c}_{i}\bm{c}_{j}}{||\bm{c}_{i}||_{2}\cdot||\bm{c}_{j}||_{2}},$

where $\bm{s}_{k}$ is the semantic vector of the $k$-th class, $C$ is the number of classes, $\mathbf{S}^{s}$ and $\mathbf{S}^{c}$ are cosine similarity matrices of the class semantic vectors and center features, respectively. We use GloVe [23] to generate semantic vectors of classes from their names. The lower score implies that the trained feature space embeds semantic knowledge of the classes.

The evaluation results are in Table 4. For the comparison, we train a network that follows our training strategy but using KCL for the contrastive loss. Our method forms better latent space than the other. Furthermore, the feature space of SMC is more similar to the semantic information space. SMC helps the network understand semantic information.

In this subsection, we present an in-depth empirical analysis of our method to validate our proposals. Table 5 displays the comparison results, and the detailed description of the analysis follows.

Normalized combination ratio in the data blending. We normalize the combination ratio to the range of $[0.2,0.8]$ to preserve the foreground information and increase the data variance. We assess the effect of the restriction and show the result in the first block of Table 5. The constrained range help contain the information of original images, which enlarges the data variance. This observation emphasizes the impact of data diversity to the network.

Information loss in cropping. The second block of Table 5 is the comparison result between different data mixing methods. Crop-SMC replaces the resizing operation in SMC into random cropping operation. Using resizing operation shows better performance than using random cropping, which implies that resizing is advantageous than random cropping for long-tailed recognition.

Data augmentation analysis. We measure the importance of random cropping to create positive instances with alternative augmentations and present the comparison result in the third block of Table 5. In line with the previous contrastive learning studies [5], the random cropping dramatically enhances the recognition performance. However, cropping on mixed images inhere the possibility of the information loss. Our augmentation method reduces the information loss, which results in the better performance.

Adaptive weights on positives. In Eq. 4, we weight positives based on their combination ratios. We assess the influence of the adaptive weights by replacing the weighted summation in Eq. 4 with different loss weighting methods. The fourth block of Table 5 illustrates the result. Averaging indicates the simple averaging of the three losses, and the assigning uses the class of the larger component as the class label of the blended sample, then it trains the network with the naïve SCL. If the combination ratio is high, the mixtures are more likely to have the foreground information than the background information. The weights adjust the impact of a positive pair to the network to boost the performance of the model. Furthermore, the lower performance of the assigning method emphasizes the necessity to define the positive and negative pair definitions for blended images.

Loss hyperparameter analysis. We set the loss weight hyperparameter $\eta$ in Eq. 6 to $0.1$ in experiments. The last block of Table 5 shows the performance dependence on the loss weight. Higher weight makes the contrastive objective dominate during training, while lower weight lessens the effect of the contrastive term. To fully utilize the power of SMC, the balance between the two losses is required.

The aforementioned comparisons have been compared on a single network. In this subsection, we show that SMC can benefit from ensemble-based methods. We combine SMC with RIDE [29], a state-of-the-art ensemble-based long-tailed recognition method, and measure the performance differences followed by the change of the number of experts in the ensemble of networks. We added an additional MLP head for each expert and trained the feature extractor with our training loss during the first stage training of RIDE. Table 6 shows the experimental result. As the number of experts increases, the performance gets better. This result demonstrates the expandability of SMC. Our method can be easily merged with additional long-tailed recognition methodologies for better recognition capability.

Our data augmentation method aims to preserve the semantic characteristics of training data. Despite of the efforts to reduce the information losses in the training samples, the loss of critical information could occur. Figure 6 illustrates the several failure cases. The merging operation with a mask could remove the background object, which is undesirable in terms of the data diversity. Moreover, the resizing operation makes the object of interest in a foreground image too small that could drop the geometric information of the foreground object. We believe that the performance of SMC could be improved with a proper knowledge-preserving augmentation method.