Label-Occurrence-Balanced Mixup for long-tailed recognition

Instead of Empirical Risk Minimization (ERM), mixup method creates new examples in the vicinity of original data and trains model based on the principle of Vicinal Risk Minimization (VRM) [17]. To be specific, given training data $x$ and its label $y$, mixup creates new example $(\tilde{x},\tilde{y})$ by combining two random training data $(x_{i},y_{i})$ and $(x_{j},y_{j})$ linearly

	$$\displaystyle\tilde{x}=\lambda x_{i}+(1-\lambda)x_{j}$$		(1)
	$$\displaystyle\tilde{y}=\lambda y_{i}+(1-\lambda)y_{j},$$		(2)

the combination ratio $\lambda\in(0,1)$ between the two data points is sampled from the beta distribution ${\textup{Beta}}(\alpha,\alpha)$.

Due to the randomness in selecting data pairs and combination ratio, mixup relaxes the constraint of finite training datasets and encourages model to learn on diverse in-between examples. However, the randomness in sampling data will lead to an imbalanced class distribution for long-tailed data.

Here, we first introduce label occurrence ratio to quantitatively demonstrate the phenomenon of label suppression. Formally, for a dataset consisting of $N$ data points from $C$ classes, mixup shuffles the dataset twice and mixes the $2N$ data points to generate $N$ new examples. For convenience, we double the index and let $A\equiv\{1,\dots,2N\}$ be the set of indexes of data points from the two shuffles and $I(k)\equiv\{i\mid y_{i}=k,i\in A\}$ be the set of indexes of data points from class $k$. In addition, the number of data points in class $k$ is denoted by $n_{k}=\left|I(k)\right|$. Assuming the probability of data point $(x_{i},y_{i})$ being selected is $P_{i}$, we define the label occurrence ratio for class $k$ as

$$\gamma_{k}=\frac{\sum_{i\in I(k)}P_{i}\lambda_{i}}{\sum_{j\in A}P_{j}\lambda_{j}},$$

(3)

where $\lambda_{i}$ is the corresponding mixing ratio of $(x_{i},y_{i})$. In mixup, each data point has the same probability of being selected, i.e., $P_{i}=1/(2N)$ for $i\in A$. Because the number of data points $n_{k}$ for each class is imbalanced, the distribution of $\gamma_{k}$ is also imbalanced, which leads to label suppression.

To address the problem of label suppression, the key idea is to balance the distribution of $\gamma_{k}$, either by adjusting $\lambda_{i}$ or $P_{i}$. We find the former method of adjusting $\lambda_{i}$ is sub-optimal, as the mixing ratio for all the head examples will be too small to provide informative feature. Therefore, we re-balance the label occurrence by adjusting $P_{i}$. As shown in Fig. 2, we employ two independent class-balanced samplers $S_{1}$ and $S_{2}$ to generate data pairs. In this case, each class has an equal probability of being selected by the two samplers. The probability of sampling an example with label $k$ is

$$p_{k}=\frac{1}{C}.$$

(4)

Accordingly, the probability of an example $(x_{i},y_{i})$ of being selected is class-conditional:

$$P_{i\mid i\in I(k)}=\frac{1}{n_{k}}p_{k}=\frac{1}{n_{k}C}.$$

(5)

The above process can be seen as a two-stage sampling operation from one class list and $C$ per-class sample lists, which first selects a class $k$ from the class list and then samples an example from the per-class sample list of class $k$ uniformly.

During training, the samplers generate two data points, $(x_{i}^{S_{1}},y_{i}^{S_{1}})$ from $S_{1}$ and $(x_{j}^{S_{2}},y_{j}^{S_{2}})$ from $S_{2}$, respectively. For exhaustively taking advantage of the data diversity, both of the data points are sampled from the whole dataset independently, which are not limited in the same mini-batch. Then we perform mixup on the data pair to create new examples:

	$$\displaystyle\tilde{x}_{\textup{LOB}}=\lambda x_{i}^{S_{1}}+(1-\lambda)x_{j}^{S_{2}}$$		(6)
	$$\displaystyle\tilde{y}_{\textup{LOB}}=\lambda y_{i}^{S_{1}}+(1-\lambda)y_{j}^{S_{2}}.$$		(7)

Due to the dual class-balanced samplers, the new batch composed of $(\tilde{x}_{\textup{LOB}},\tilde{y}_{\textup{LOB}})$ have a balanced distribution of $\gamma_{k}$.

To further improve the training performance, we employ a deferred re-balancing training strategy [9], which first trains with vanilla mixup before annealing the learning rate, and then uses the proposed label-occurrence-balanced mixup.

For sound recognition on ESC-50-LT, we use EnvNet [19] and EnvNet-v2 [2] as backnone networks and follow the training settings of [2]. For visual recognition on CIFAR-10/CIFAR-100-LT, we train a ResNet-32 backnone network for 200 epochs, with a learning rate initialized as 0.1 and decayed at the 160th and 180th epoch. For ImageNet-LT, we choose ResNet-10 as the backbone network and train the model for 90 epochs, following [11]. The base learning rate is set to 0.2, with cosine learning rate decay. The mixing ratio $\lambda\sim{\textup{Beta}}(\alpha,\alpha)$, where we set $\alpha=0.2$ for ImageNet-LT, and $\alpha=1$ for other datasets.

Results for visual recognition are reported in Table 1. Compared with mixup, our method obtains $4.2\%$ relative improvement on ImageNet-LT. Our method also outperforms other mixup-based methods like manifold mixup and Remix. It is worth noting that class-balanced sampling method leads to even worse performance on some datasets, e.g., CIFAR-100-LT, while our method shows consistent performance gains on all the reported benchmarks. Furthermore, by integrating with deferred re-balancing training strategy, our method achieves lower error rates than most of the state-of-the-art methods, such as logit adjustment [20] and BBN [10].

Results for sound recognition show similar trends. As shown in Table 2, class-balanced sampling gets worse performance, probably due to overfitting on repeated samples. BC learning outperforms ERM, while our method further improves the performance for both EnvNet and EnvNet-v2 backbones.