SAU: A Dual-Branch Network to Enhance Long-Tailed Recognition via Generative Models

The main purpose of this unit is to create a description of a certain class as a guide prompt for the T2I generation process. Here we have two ways to generate prompts, basic prompts and LLM-enhanced prompts.

After obtaining the texts, we employ them to generate synthetic images by leveraging advanced T2I models. In detail, we use GLIDE [22] to generate synthetic images, which can be formulated as:

where $n\in N_{y}$ represents the number of synthetic images of a certain class, $I^{n}_{y}$ and $P^{n}_{y}$ indicate the corresponding generated images and prompts for the class $y$, respectively.

However, some synthetic images might not be high-quality enough to represent the corresponding class. To address this problem, we use CLIP [25] to evaluate whether synthetic images are well related to their corresponding textual contexts.

First, we apply the MixUp [33] to mix real samples and synthetic samples. Let $x\in\mathbb{R}^{W\times H\times C}$ and $y$ denote a training image and its corresponding label, respectively. For a pair of two images and their labels $(x_{i},y_{i})$ and $(x_{j},y_{j})$, the mixed augmentation image and its label $(\tilde{x}_{ij},\tilde{y}_{ij})$ can be calculated as follows:

where the combination ratio $\lambda$ is sampled from a Beta distribution. $\lambda$ is a hyper-parameter, which is set as 1 here for a more random mixture.

Another mixture technology CutMix [32] also aims to mix real samples and synthetic samples, which combine two images by simply replacing the image region with a path from another image. Similarly, the combining operation can be defined as follows:

where the matrix $\textbf{M}\in\{0,1\}^{W\times H}$ is the binary mask that indicates the randomly selected region from the image $x_{i}$, which is then filled into $x_{j}$. 1 is a binary mask filled with ones. $\odot$ is element-wise multiplication. Specifically, we sample the bounding box coordinates $\textbf{B}=(r_{x},r_{y},r_{w},r_{h})$ indicating the cropping regions on $x_{i}$ and $x_{j}$. The binary mask M is determined by setting to 0 within the bounding box B, and to 1 elsewhere. The box coordinates are uniformly sampled according to:

where the $\lambda$ is also sampled from a Beta distribution, ensuring that the cropped area ratio is equal to $1-\lambda$.

After constructing two kinds of real-synthetic mixed augmented data, we use the cross-entropy loss to calculate two losses in the same way:

here, the $f(\tilde{x})$ denotes the predictions of $\tilde{x}$.

Contrastive learning methods typically use the noise contrastive estimation (NCE) objective to distinguish between similar and dissimilar data points. Specifically, it pulls different views of the same sample together, while pushing others away in a latent space. Similarly, in a supervised classification task, where label information is given, it pulls different views of samples from the same class together while pushing others away. For instance, we randomly sample a mini-batch of images $\{\mathbf{x}\}^{N}_{i=1}$, and then we apply a random augmentation function $T(\cdot)$ to obtain a pair of different augmented views each image. Within this multiviewed batch, we can define that $i\in I\equiv\{1,2,...,2N\}$ as the index of multiviewed batch samples. Following the previous works [7, 17], the loss function can be defined as:

where $\mathbf{z}_{i}=\phi(\mathcal{F}(T(\mathbf{x}_{i})))$ indicates the vector in a latent space of input images, and $I_{y}\equiv\{i\in I:\mathbf{y}_{i}=y\}$ indicates a set of views of same class $y$, and the symbol $|\cdot|$ denotes to the number of the elements, and $\tau>0$ denotes a scalar temperature hyper-parameter.

To further address the problem of low-quality synthetic images, motivated by [36], we propose a K-Nearest-Neighbour(KNN) based method to dynamically detect low-quality synthetic image-label pairs. Specifically, given two sets of representations $\{\mathbf{z}_{real}\}^{N_{1}}_{i=1}$, $\{\mathbf{z}_{syn}\}^{N_{2}}_{i=1}$ and their corresponding labels $\{\mathbf{y}_{real}\}^{N_{1}}_{i=1}$, $\{\mathbf{y}_{syn}\}^{N_{2}}_{i=1}$ of both real images and synthetic images, we generate correction labels for the synthetic images based on the labels of their k nearest real neighbors. Combining them with original labels $\mathbf{y}_{org}$, the final labels for synthetic images can be defined as:

where $\mathbf{y}_{new}$ indicates the new labels for the synthetic data. $\mathbf{y}_{cor}^{1}$ and $\mathbf{y}_{cor}^{2}$ are the two sets of corrective labels of two different augmented views. $\mathbf{y}_{noise}$ represents the noise labels for those low-quality synthetic images.

To ensure all classes of real samples appear in every mini-batch, following previous work [21], we calculate prototypes of each class by averaging the real training data at the end of the previous epoch for the current epoch. Formally, the normalized prototype for a certain class $y$ can be formulated as:

where $\mathcal{P}\in\mathbb{R}^{C\times d}$ indicates that prototypes include $C$ data points with $d$ dimension each. $N_{1}$ is the number of real data, and the symbol ${\lVert\cdot\rVert}_{2}$ denotes the $l^{2}$-norm. Note that since the prototypes $\mathcal{P}$ are the average from real data, we treat them as real data in the following declaration.

The key idea to tackle low-quality synthetic data is treating them as noise within a mini-batch. Here, we propose three forms of loss for this scenario.

First, the most intuitive way is simply to ignore all the noise data points:

here, $A=\{I,I^{{}^{\prime}}\}$ denotes the entire dataset, where $I$ consists of real data, prototypes, and high-quality synthetic data, and $I^{{}^{\prime}}$ is the noise data. The symbol $|\cdot|$ represents the number of elements. Note the original $|I_{y}|-1$ term in Eq. 8 should plus one because prototypes contain one data point for each class.

Second, instead of simply removing all the noise data points, we keep them and treat all the noise data points as individual positive samples. Concretely, we assign unique noise labels $\mathbf{y}_{noise}=\{-1,-2,...,-N_{n}\}$ for noise data points, representing new classes distinct from all the original labels:

where $y_{i}\in\{\mathbf{y}_{org},\ \mathbf{y}_{noise}\}$ indicates there is also a new noise class in addition to the original class. The denominator of the average term should count all samples among $I_{y}$ and $I^{{}^{\prime}}_{y}$.

Third, similar to $\mathcal{L}_{2}$, we keep noise data points, but only as negative samples:

In summary, the overall training loss function for our supervised contrastive learning framework can be expressed as:

where $\lambda$, $\beta$, $\gamma$ are the hyper-parameters. More details in Algorithm 1.

Both CIFAR-10 and CIFAR-100 [18] consist of 60000 32$\times$32 color images with 50000 images for training and 10000 images for testing. There are 10 classes and 100 classes resulting in 6000 images and 600 images for each class respectively. CIFAR-10-LT and CIFAR-100-LT are the subsets of the long-tailed versions of CIFAR-10 and CIFAR-100. In this work, the training datasets consist of real data and generated data. For real data, following the previous works [2, 12, 37], we use the exponential decay function $N_{i}=N_{o}\mu^{\frac{i}{n-1}}$ to obtain long-tailed datasets, where $i$ is the class index (0-indexed), $N_{o}$ is the original number of training images and $\mu$ is the imbalanced factor. The imbalanced factor $\mu=N_{max}/N_{min}$ reflects the degree of imbalance and we use different $\mu$ [10, 50, 100, 200] for both CIFAR-10-LT and CIFAR-100-LT. For synthetic data, they are simply set as the complement of the real data. Following the official GLIDE T2I process, we use different prompts to generate 64$\times$64 color synthetic images.

ImageNet-LT, proposed by [20], is a long-tailed version of the original ImageNet [9] dataset, which is designed to mimic real-world scenarios. Specifically, the original ImageNet consists of 1,280,000 images of 1000 classes in total with 1280 images per class. And ImageNet-LT is a subset of ImageNet with an imbalance factor of 256. To reduce computational costs, we only use basic prompts to generate 256$\times$256 color synthetic images.

We use Pytorch [23] deep learning framework to train models for all datasets.

For CIFAR-10-LT and CIFAR-100-LT, we follow [12, 7, 37] to use ResNet-32 [14] for a fair comparison across different studies. For the optimization, we use a standard SGD optimizer with a momentum of 0.9. We start with an initial learning rate of 0.01 with a cosine annealing scheduler. Additionally, the batch size is set at 128 and the weight decay rate at 5e-3. The model will be trained for 200 epochs, after which the model is evaluated on the test set to assess its performance. In terms of data augmentation strategies, we use random horizontal flipping and cropping for the classification branch, AutoAugment [6], Cutout [10] and SimAugment [5] for contrastive learning branch. And MLP consists of two hidden layers of size 2048 and output vector of size 128, and there are batch normalization and ReLU activation function between two hidden layers.

For ImageNet-LT, we use ResNet-50 [14] and ResNeXt-50-32x4d [31] as backbone. We use a standard SGD optimizer with a momentum of 0.9. We set the initial learning rate as 0.1, which is adjusted by a cosine annealing scheduler. The batch size is set at 256 and the weight decay rate at 1e-4. The model is trained over 180 epochs.

For all datasets, we focus on the top-1 accuracy. We train models on a balanced dataset composed of a mix of real long-tailed training sets and synthetic set and evaluate them on the balanced validation/test dataset. Furthermore, for ImageNet-LT, we report many-shot classes with real training samples > 100, medium-shot classes with real training samples 20 $\sim$ 100, and few-shot classes with real training samples $\leq$ 20, respectively.

The result has been reported in Table 1. For comparison baselines , we mainly consider previous state-of-the-art works including CB-Focal [8], PaCo [7], Hybrid-SC [30], BCL [37], GLMC [12], and SURE [19], whose models are trained from scratch. Furthermore, for a fairer comparison, we also apply recent state-of-the-art work GLMC on real-synthetic mixed balanced datasets. As we can see, our method achieves the best results on all imbalance factors.

Similar to CIFAR, we report the Top-1 accuracy of our models compared to other works in Table 2. For comparison baselines, we mainly consider both current and previous state-of-the-art works including Cross-Entropy, CB-Focal [8], BCL [37], PaCo [7], ProCo [11] and GLMC [12]. From the results, our method achieves the best performance in few-shot.

First, we compare the performances of three different contrastive losses ($\mathcal{L}_{1}$, $\mathcal{L}_{2}$, and $\mathcal{L}_{3}$) to deal with the poor-quality synthetic images. From the results presented in Table 3, all losses are effective and $\mathcal{L}_{2}$ performs best on both CIFAR-10-LT and CIFAR-100-LT.

We also deploy an extensive ablation experiment to examine the effectiveness of primary components. The results are shown in Table 4, which validates the effectiveness of our proposed method. Note the first row is the baseline, which uses cross-entropy as loss function and is trained on the real-synthetic mixed dataset. As we can see, the final row of the table, where both $\mathcal{L}_{MixUp}$ and $\mathcal{L}_{CutMix}$ are utilized alongside $\mathcal{L}_{SC}$, presents the best result with a Top-1 accuracy of 64.47%.