On Robust Learning from Noisy Labels: A Permutation Layer Approach

To combat the effect of label noise, we propose adding an instance-dependent permutation layer, sometimes referred to simply as the permutation layer. In this section, we introduce the notion of an instance-dependent permutation layer and demonstrate how it is constructed for a given training sample.

The permutation layer for the $k^{th}$ training sample $\hbox to0.0pt{\hskip 1.40451pt\leavevmode\hbox{\set@color$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{\set@color$P$}}_{\!\!\boldsymbol{\alpha}^{k}}$ is constructed as a convex combination of the permutation matrices in the set $\{P(\tilde{y}^{k},1),P(\tilde{y}^{k},2),...,P(\tilde{y}^{k},c)\}$. This set consists of all single-swap permutation matrices involving the element $\tilde{y}_{k}$. The convex combination is obtained using the learnable parameter $\boldsymbol{\alpha}^{k}$. Ultimately, we want to learn $\boldsymbol{\alpha}^{k}$ such that $\hbox to0.0pt{\hskip 1.40451pt\leavevmode\hbox{\set@color$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{\set@color$P$}}_{\!\!\boldsymbol{\alpha}^{k}}$ reflects the noise affecting the true label $y^{k}$, i.e., $\hbox to0.0pt{\hskip 1.40451pt\leavevmode\hbox{\set@color$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{\set@color$P$}}_{\!\!\boldsymbol{\alpha}^{k}}=P(\tilde{y}^{k},y^{k})$.

The proposed instance-dependent permutation layer alleviates the impact of label noise by correcting the loss. As demonstrated in Figure 1, we provide two approaches of incorporating the permutation layer into the learning procedure. Both approaches aim to capture the noise influencing $y^{k}$, i.e., $P(\tilde{y}^{k},y^{k})$. The first approach applies the permutation layer to the noisy label $\boldsymbol{e}_{\tilde{y}^{k}}$, as in equation (1). In this case, the permutation layer cleans the noisy label, $P(\tilde{y}^{k},y^{k})\,\boldsymbol{e}_{\tilde{y}^{k}}=\boldsymbol{e}_{y^{k}}$. In the second approach, we instead apply the permutation layer to the model’s prediction $f_{\theta}(\mathbf{x})$, as in equation (2). Thus, the prediction $f_{\theta}(\mathbf{x}^{k})$ is permuted to have a noise matching the noise on $y^{k}$, $P(\tilde{y}^{k},y^{k})$.

Note that these two approaches are analogues to the forward and backward corrections in (Patrini et al. 2017a). Looking closely at equation (1), we can see that it is in fact equivalent to learning the label directly. In other words minimizing $\pazocal{L}_{1}$, with respect to $\theta$ and $\{\alpha^{i}\}_{i=1}^{N}$, is equivalent to minimizing $\sum_{i}\ell(f_{\theta}(\mathbf{x}^{i}),\,S(\boldsymbol{\alpha}^{i}))$. The approach of directly learning the label was first proposed in (Tanaka et al. 2018), and then adopted by (Yi and Wu 2019). The following proposition demonstrates this equivalence by showing that applying a permutation to the label $\hbox to0.0pt{\hskip 1.40451pt\leavevmode\hbox{\set@color$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{\set@color$P$}}_{\!\!\boldsymbol{\alpha}}\,\boldsymbol{e}_{\tilde{y}^{i}}$ yields $S(\boldsymbol{\alpha}^{i})$.

In the next section, we further analyze these two approaches, and analytically show that they possess different theoretical properties.

Although the two approaches of employing the permutation layer seem equivalent, they are in fact substantially different. In this section, we compare these approaches and analytically show that applying the permutation to the prediction has two key advantages: ($i$) The optimization does not get stuck in a local optimum after solving for the permutation layer parameters $\boldsymbol{\alpha}$. ($ii$) The norm of the gradient $||\nabla_{\!\boldsymbol{\alpha}}\pazocal{L}_{2}||$ is affected by the model’s confidence, i.e., $\pazocal{L}_{2}$ is “confidence-aware”.

We simultaneously learn the model’s parameters $\theta$ and the parameters of the permutation layer $\{\boldsymbol{\alpha}^{i}\}_{i=1}^{N}$ using gradient descent. In fact, the optimization problem has a closed-form solution with respect to $\{\boldsymbol{\alpha}^{i}\}_{i=1}^{N}$. However, similar to (Yi and Wu 2019), we use gradient descent to allow for gradual changes of the parameters and to take advantage of the confidence-aware property described in proposition 4. The permutation layer parameters, $\{\boldsymbol{\alpha}^{i}\}_{i=1}^{N}$, are updated as follows:

for $1\leq i\leq N$, where $\eta_{\alpha}{}$ is the learning rate of the permutation layer. The initialization of the permutation layer weights is determined by the hyperparameter $I_{\alpha}{}$. Precisely, $I_{\alpha}{}$ affects the initialization as follows:

where $\frac{1}{c}<I_{\alpha}{}<1$. We avoid setting $I_{\alpha}{}$ very close to 1 to allow for a more desirable gradient flow, i.e., avoiding vanishing gradients. The effects of the choices of $\eta_{\alpha}{}$ and $I_{\alpha}{}$ are analyzed in section 4.4. Note that the permutation layer is discarded at inference time.

In the previous sections, we have established the similarity between permuting the labels and Joint Optimization (Tanaka et al. 2018), and that permuting the predictions has more appealing theoretical properties. Therefore, in the next section, we empirically analyze PermLL when the permutation is applied to the prediction.

To evaluate the effectiveness of PermLL, we use three standard image classification benchmarks: CIFAR-10, CIFAR-100 (Krizhevsky, Hinton et al. 2009) and Clothing-1M (Xiao et al. 2015b). Since the CIFAR10/100 datasets are known to be clean, we simulate a noisy learning setting by injecting synthetic noise into the training labels. Two types of synthetic label noise are used: symmetric and asymmetric noise. For the symmetric noise, we take a fraction of the labels and flip them to one of the c classes with a uniform random probability. Asymmetric noise attempts to mimic the mistakes that naturally occur by annotators in the real world. We follow (Patrini et al. 2017b) for injecting the asymmetric noise in CIFAR-10: truck$\,\to\,$autombile, deer$\,\to\,$horse, bird$\,\to\,$airplane, cat $\leftrightarrow$ dog. In the case of asymmetric noise for CIFAR-100, we flip labels of subclasses within randomly selected superclasses cyclically with a total of 20 superclasses. Following (Liu et al. 2020; Tanaka et al. 2018), we hold out 10% of the training data as a validation set.

Clothing1M, on the other hand, is a large dataset of images from online clothing shops containing realistic noise. The noise level is estimated to be around 38.5% (Xiao et al. 2015b). This dataset contains four sets: a noisy training set (1M samples), a clean training set (50K samples), a clean validation set (14K samples), and a clean testing set (10K samples). In our experiments, we completely exclude the 50k samples in the clean training set as PermLL does not assume the availability of a clean training set.

For CIFAR-10 we use a ResNet34 (He et al. 2016) and an SGD optimizer with learning rate of 0.02, momentum of 0.9, a weight decay of 0.0005 and a batch size 128. The model is trained for 120 epochs and the learning rate is decayed by a factor of 10 at epochs 80 and 100. Additionally, we apply standard augmentation and data preprocessing: padding followed by random cropping, random horizontal flip, and data normalization, similar to (Liu et al. 2020; Tanaka et al. 2018; Yi and Wu 2019). The same setup is used for CIFAR-100 except that the weight decay used is 0.001, and the learning rate decays once by a factor of 10 at epoch 100. For all noise settings in CIFAR10 we set $I_{\alpha}{}=0.35$, and $\eta_{\alpha}{}=1.5$. For CIFAR100 we use $I_{\alpha}{}=0.225$, with $\eta_{\alpha}{}=3$ for symmetric noise and $\eta_{\alpha}{}=6$ for asymmetric noise.

For Clothing1M we use a pretrained Resnet50 from TorchVision (Paszke et al. 2019) and an SGD optimizer with learning rate of 0.001, momentum of 0.9, a weight decay of 0.001, and a batch size of 64. The model is trained for 15 epochs and the learning rate is decayed by a factor of 10 at epochs 5, 10 and 13, with $I_{\alpha}{}=0.275$ and $\eta_{\alpha}{}=80$. Standard augmentation and data preprocessing are applied; the images are normalized, resized to 256x256, randomly cropped to 224x224, and randomly horizontally flipped. Similar to previous methods (Liu et al. 2020; Li, Socher, and Hoi 2020), during training, we sample balanced batches (based on the noisy labels) as Clothing1M contains significant class imbalance. All experiments were implemented using PyTorch (Paszke et al. 2019) and were run on NVIDIA V100 and NVIDIA RTX A6000 GPUs.

Table 1 shows the results of PermLL on CIFAR10/100 with symmetric and asymmetric synthetic noise. We compare our results with state-of-the-art methods that use a similar setup (e.g., same architecture and augmentation). To ensure fairness, we use the same architecture, batch size, and augmentation as other baselines when possible (as described in 4.2). PermLL consistently outperforms all other methods in symmetric noise. This is especially clear in the high noise CIFAR100 setting, in which PermLL achieves an improvement of $\sim$6% and $\sim$12% in terms of test accuracy for 60% and 80% symmetric noise, respectively, while using the same hyperparameters for all noise percentages. In the case of simulated asymmetric noise, our method attains comparable performance at 20% noise. In the 40% simulated asymmetric noise, our method achieves poor performance compared to the baselines. However, we observe that PermLL does not face the same issue when faced with real asymmetric instance-dependent noise.

To validate the efficacy of PermLL in real noise environments, we use the Clothing-1M dataset as a benchmark. We report the test accuracy in Table 2 of PermLL and compare it with state-of-the-art methods. We achieve high performance and outperform all baselines, slightly outperforming UniCon. It is worth noting that the comparison might not be entirely fair as methods denoted with (*), unlike PermLL, use semi-supervised training, heavy augmentation, and multiple networks to achieve their results. Nevertheless, PermLL remarkably achieves very competitive results despite its relative simplicity, demonstrating its practicality and strength.

In this section, we analyze the effect of PermLL hyperparameters on learning correct permutation layers, similar to (Tanaka et al. 2018; Yi and Wu 2019). Recall that, we ultimately want to learn $\boldsymbol{\alpha}^{i}$ such that $\hbox to0.0pt{\hskip 1.40451pt\leavevmode\hbox{\set@color$\overline{\hbox{}}$}\hss}{\leavevmode\hbox{\set@color$P$}}_{\!\!\boldsymbol{\alpha}^{i}}=P(\tilde{y}^{i},y^{i})$ for all $i$, where $\tilde{y}^{i}$ and $y^{i}$ are the clean and noisy labels, respectively. Therefore, we consider a permutation layer to be correct if $\arg\max_{j}\boldsymbol{\alpha}^{i}_{j}=y^{i}$ for all $i$.

Figure 3 (a) shows the impact of changing the permutation layer’s learning rate $\eta_{\alpha}{}$ on the percentage of correct permutations throughout training. A small $\eta_{\alpha}{}$ delays or halts learning the permutation layers, causing the model to memorize the noisy labels. In contrast, a large $\eta_{\alpha}{}$ results in early and erroneous permutations. Furthermore, an extremely large $\eta_{\alpha}{}$ reduces the percentage of correct permutations below its initial value, as early wrong predictions cause harmful large updates on the permutation layer parameters. Similarly in Figure 3 (b), we study the effect of changing $I_{\alpha}{}$. The effect appears to be inversely similar to the effect of changing $\eta_{\alpha}{}$. Analogous to the case of a high $\eta_{\alpha}{}$, extremely small $I_{\alpha}{}$ significantly reduces the permutation accuracy. Figure 3 (c) illustrates the relationship between $I_{\alpha}{}$ and $\eta_{\alpha}{}$, showing their influence on the percentage of correct permutations over a larger scale. The percentage of correct permutations suffers the most when $I_{\alpha}{}$ is low and $\eta_{\alpha}{}$ is high. We can also notice that as we increase $\eta_{\alpha}{}$ we also need to increase $I_{\alpha}{}$ to maintain a good permutation accuracy.