Ensemble Learning with Manifold-Based Data Splitting for Noisy Label Correction

To solve noisy label correction problem, several methods were developed by re-designing loss terms so that the learning system itself can re-weight the importance of training instances. For example, Wang et al. [20] employed an importance re-weighting strategy that enables the training on noisy data to reflect the results of noise free environment. Arazo et al. [21] evaluated their entropy-based per-sample loss for label correction by controlling the confidence of training sample via two weights. Still, some other losses, e.g., mean absolute error (MAE), generalized cross entropy (GCE), and reverse cross entropy (RCE), were proposed to avoid a biased learning system, the gradient saturation problem, or the overfitting/under-learning problem in the presence of noisy labels [22, 12, 23]. Nevertheless, although these methods prevent a learning system from fitting noisy samples, they may still be impracticable to complicated datasets due to the ignorance of hard samples.

Most recent label correction methods work based on i) a noise transition matrix, or ii) class prototypes. As for the former sort of methods, they are based on an instance-independent assumption to derive the transition probability, which depicts how possible a clean label flipping into a noisy one, independent to any type of data structure. However, this assumption cannot prevent the generation of incorrect labels, nor can it prevent the transition matrix from learning noisy labels. Therefore, Xia et al. [8] proposed a transition revision method to address this issue, but their method is still limited by the fairness of the estimated initial transition matrix. Moreover, “class prototype” methods learn to represent each class via some class prototypes based on which data samples can receive corrected label information from their neighborhood [9, 10]. Nevertheless, these methods learn prototypes from a noisy dataset, so the quality and the accuracy of prototypes are still affected by the original noise distribution. Both these two types of methods learn from noisy data. Because there is no effective mechanism to verify the corrected labels suggested by the transition matrix or the class prototypes, the resulting pseudo-label may be incorrect and generate a worse correction suggestion if the class prototypes or the transition matrix overfits noisy training labels.

Some other methods exploit i) ensemble learning that feeds stochastically-augmented inputs to a number of parallel sub-models to derive the corresponding prediction jointly with ii) an averaging method to derive label correction suggestions [16, 19]. These methods are based on the inconsistency among information learned by different sub-networks. For example, MLNT [15] randomly changes the labels of each sub-network’s training data to enhance the difference learned by each subnet, and DivideMix [16] uses an additional regularization term to increase the prediction diversity among different sub-models. However, making sub-models learn to be different from each other cannot ensure the correctness of any sub-model itself, and they may be worse than they are supposed to be due to an unmatched noise distribution. For example, Yan et al.’s method is a method works only on uniformly distributed label noises [24]. In real-world applications, label noise usually distributes in some specific way rather than uniformly. Consequently, it is still intractable for simple ensemble learning approaches to filter out noisy labels.

Label propagation is a graph-based semi-supervised method for generating pseudo-labels of unlabeled data based on given anchors [25, 26, 27]. Compared with the nearest-neighbor strategy that does not take the data manifold into account, label propagation methods can propagate labels from labeled anchors to unlabeled samples through a graph characterizing the manifold of the noisy dataset. For example, the method in [28] uses an affinity matrix to derive an approximate $k$-NN graph, through which it performs label propagation for a large-scale diffusion task. Iscen et al. [26] proposed a transductive label propagation method to make label predictions by using i) a nearest-neighbor graph, ii) conjugate gradient based label propagation, and iii) a loss weighted by label certainty. Moreover, Bontonou et al. [29] proposed a graph smoothness loss, maximizing the distance between samples of different classes, to derive better feature and manifold representations for classification purpose. This loss increases the robustness of a learning model against noisy training inputs, though its performance is still similar to that of cross-entropy because only distances between instances in different classes are considered.

Fig. 2 illustrates the framework of the proposed method for label correction. Let $d_{i}=(x_{i},y_{i})\in\mathcal{D}$ denote a sample-label pair from a data collection $(\mathcal{X},\mathcal{Y})$, where $\mathcal{D}=\{d_{i}\}$ is the source noisy dataset. Our ensemble learning scheme aims to split $\mathcal{D}$ into $M$ disjoint subsets, namely $\mathcal{S}_{m}$ for $m=1,\dots,M$, for training $M$ ensemble branches. During the training, our method first trains $M$ classifiers $\Phi_{m}$ with parameter $\theta_{m}$, through the $M$ ensemble branches, independently on the $M$ disjoint subsets $\mathcal{S}_{m}$ derived by our data splitting strategy. Second, by feeding all training samples $x_{i}$ into each classifier, we extract totally $M$ different feature vector sets, each of which can be used to derive one graph representation of the data manifold of training dataset $\mathcal{X}=\{x_{i}\}$. Third, for given the $m$-th partial-label set $(x^{m}_{j},y^{m}_{j})\in\mathcal{S}_{m}$ where only labels $y^{m}_{j}$ belonging to $\mathcal{S}_{m}$ are available, we construct its graph representation and use the graph to predict the label correction suggestions through label propagation for all data samples in the remaining $M-1$ partial-label sets. As a result, we use the $M$ partial-label sets to predict totally $M\times M$ label correction suggestions based on the $M$ graphs derived in the second step. Fourth, based on the corrected labels $\hat{y}^{m}_{l}\in\hat{\mathcal{Y}}_{m}$ and the $M$ sample-correction sets $\{(x^{m}_{l},\hat{y}^{m}_{l})\,|\,\forall x^{m}_{l}\in\mathcal{S}_{m}\}$ obtained in the $l$-th training epoch, we generate another $M\times M$ label propagation suggestions. Finally, we derive the most likely correction result $\bar{\mathcal{Y}}=\{\bar{y}_{i}\}$ from the total $2\times M^{2}$ label correction suggestions via a majority decision.

Specifically, based on the assumption that real-world noisy labels (e.g., obtained via crowd-sourcing) tend to be locally concentrated near decision borders, we devise a data splitting strategy to partition the noisy source dataset $\mathcal{D}$ into $M$ disjoint subsets $\mathcal{S}_{m}$, each being the training set for one ensemble branch. In this way, $S_{m}$ is expected to retain the shapes of $k$ randomly-selected local neighborhoods, no matter noisy or not, as well as ensemble a random subset of $x_{i}\in\mathcal{X}$. Therefore, by taking a majority decision on the approximate manifolds learned by the $M$ sub-networks, the local influences brought by noisy samples can be mitigated. Finally, the noisy label branch in our design is trained on the whole original dataset $\mathcal{D}$ to derive the feature vectors of all $x_{i}$. Meanwhile, the corrected label branch is trained based on corrected labels, i.e., $(\mathcal{X},\bar{\mathcal{Y}})$, obtained after the l-th training epoch, and it is the only branch used for deployment.

Our key idea is to restrict the influence of any group of locally-concentrated noisy labels to only a minority of sub-networks via our local-patch-based data slitting strategy. As a result, most ensemble branches can learn their own relatively correct approximations of the data manifold around the noisy local patch, so that a suitable label correction result can be yielded by majority decision, accordingly. Also, no matter label noise is locally-concentrated or uniformly distributed in the data space, it can be effectively mitigated by the proposed ensemble method. Overall, each training epoch of our method can be divided into three phases, as will be elaborated in Sec. III-B–III-D.

The first phase of each training epoch is to randomly scatter per local neighborhood of data points on the source data manifold, including noisy labels, into disjoint subsets. Specifically, as illustrated in Fig. 3, for each selected $x_{i}$, we pick only its $k$ nearest-neighbors of the same class to form a local patch around $x_{i}$ with $k=N_{c}/(M\times B)$, where $B$ is a predefined hyper-parameter, and $N_{c}$ denotes the cardinality of the class that $x_{i}$ belongs to. Therefore, because each disjoint subset $\mathcal{S}_{m}$ holds different local feature relationships of each class, we can confine the negative influence of each locally-concentrated noisy neighborhood to primarily one $\mathcal{S}_{m}$ and one ensemble branch by packaging every local $k$-nearest neighbors. This is the main advantage of the proposed method. As a result, on one hand, when a noisy local neighborhood only contaminates a minority of ensemble branches, our method can vote down the negative influence of noisy labels by a majority decision. This case leads to a performance leap with our method: the performance upper-bound. On the other hand, in the extreme case that all noisy labels are uniformly distributed globally, our data splitting strategy will not alter the noise distribution so that our ensemble model leads to a as good performance as typical random-selection schemes: the baseline performance. In sum, our method can be expected to provide a performance in between the upper-bound and the baseline. Since real-word label noise distribution tends to be concentrated on decision boundaries, our method can usually lead to performance improvements.

The proposed data splitting method is composed of the following steps:
Initialization: Determine the hyper-parameters, including the number of ensemble branches $M$ and the number of data packages per class per branch $B$.
$k$-NN Packaging: As illustrated in Fig. 3, for each class, we separate its data into $M\cdot B$ packages, each containing a randomly selected seed sample and its $k$-NN in the feature domain with $k=\frac{N_{c}}{M\cdot B}$, where $N_{c}$ denotes the number of samples belonging to class $c$. Note that we adopt “random sampling without replacement” to guarantee that each sample and its $k$-NN can be assigned into only one package.
Splitting: For each class, we randomly assign the $M\cdot B$ packages into $M$ disjoint subsets. Each subset $\mathcal{S}_{m}$ therefore stands for a coarse global approximation of the source data manifold but holds fine local manifold structures of different places independently.

Moreover, we initialize each training epoch with the data splitting process, and we name this strategy re-splitting. Re-splitting enables each resemble branch to learn a different coarse approximation of the data manifold to prevent the resemble branches from overfitting and being biased to the same training data. Because $\mathcal{S}_{m}$ is changed for each epoch, we need to initialize the model parameter $\theta^{l}_{m}$ for each ensemble branch to guarantee the fast convergence of the $l$-th training epoch by

where $\alpha_{m}^{l}$ is a random value within $[0,1]$; and, $\theta_{\mbox{pseudo}}^{l-1}$ and $\theta_{\mbox{noisy}}^{l-1}$ are respectively the model parameters, obtained after the $(l-1)$-th training epoch, of the corrected-label branch and that of the noisy-label branch.

We then construct the $m$-th graph representing the data manifold based on the features $f^{m}_{i}=f(x_{i},\theta_{m})$ extracted by the $m$-th model. Then, for each graph, we can produce $M+M$ label correction suggestions by taking i) the original sample-label pair $(x_{j}^{m},y_{j}^{m})\in\mathcal{S}_{m}$ and ii) the sample-correction pair $(x_{j}^{m},\hat{y}_{j}^{m,l})$ obtained in the $l$-th training epoch as different starting partial-labels, where $\bar{y}_{j}^{m,l}$ denotes the $j$-th sample’s label correction suggestion given by the $m$-th ensemble branch in the $l$-th training epoch. As a result, our method can hence offer totally $M\cdot(M+M)=2M^{2}$ sets of label correction suggestions.

Our multi-graph label propagation strategy works based on the features extracted by classification models $\Phi_{m}(\cdot;\theta_{m})$ of the $M$ ensemble branches. The first step is to build $M$ normalized weighted undirectional adjacency matrices $\mathcal{W}_{m}$, each representing the $m$-th graph constructed based on $f_{i}^{m}$, for the label propagation process described in (4), and $\mathcal{W}_{m}$ is required to be symmetric [25]. The matrix $\mathcal{W}_{m}$ is defined as

where $D_{m}$ is a diagonal matrix used to normalize $(A_{m}+A^{T}_{m})$, and $A_{m}$ is a directional adjacency matrix whose ($s$,$t$)-th entry is defined as follows:

where $f^{m}_{s}$ is the feature of the $s$-th sample extracted by the model of the $m$-th branch, $\mathcal{N}(f^{m}_{t})$ denotes the collection of the $k$ nearest neighbors of $f^{m}_{t}$, $sim(\cdot,\cdot)$ denotes the cosine similarity metric ranging in $[0,1]$, and $\gamma$ is a predefined hyper-parameter. Note that $A_{m}$ is asymmetric.

Next, we propagate the label information of i) the sample-label pairs $(x^{m}_{j},y^{m}_{j})$ in the $m$-th disjoint subset $\mathcal{S}_{m}$ and ii) sample-correct pairs $(x^{m}_{j},\bar{y}^{m,l}_{j})$ in turn through each of the $M$ graphs to obtain totally $2M^{2}$ label correction suggestions, i.e., partial-labels, for each data sample $x_{i}$. We adopt such a strategy because of the following two concerns. First, while using only original noisy labels $y^{m}_{j}$ to derive the label propagation result, the obtained labels are probably contaminated, so this setting surely limits the accuracy of obtained labels. Second, while using only corrected labels $\hat{y}^{m,l}_{j}$, the propagation result may overfit the corrected labels, thereby misleading the label correction process. Therefore, we exploit both the original noisy labels $y^{m}_{j}$ and corrected labels $\hat{y}^{m,l}_{j}$ in our label propagation process.

Let $Z_{m,j}$ denote the label correction suggestion, the label propagation process [25] is conducted by

where $Y_{j}$ is an $N\times C\times 2$ matrix that records the reference labels associating the data samples belonging to the $j$-th disjoint subset $\mathcal{S}_{j}$, $\mathcal{W}_{m}$ is the normalized weighted undirectional adjacency matrix defined in (2), and $N$ and $C$ denote respectively the cardinality of the original dataset $\mathcal{X}$ and the cardinality of $\mathcal{Y}$, i.e., the number of total classes. Note that the third dimension of $Y_{j}$ contains two partial-labels: the original noisy one $y^{m}_{j}$ and the corrected one $\hat{y}^{m,l}_{j}$.

Finally, the partial-label suggested by the $m$-th ensemble branch based on the label information of $\mathcal{S}_{j}$ becomes

where every $Z_{m,j}$ is an $N\times C\times 2$ matrix, and $Z_{m,j}(n,c,\cdot)$ indicates how possible the $n$-th sample belongs to the $c$-th class. After obtaining $2M^{2}$ different label suggestions $\hat{Y}_{m,j}$, it turns to Phase-3 to derive the final correction result.

We select the most frequent category among all label suggestions $\hat{Y}_{m,j}(n,:)$ via a majority decision as the final label correction result $\bar{y}(n)$.

To guide the label correction, we propose a normalized average confidence level to devise our loss function. The normalized average confidence level $\bar{\omega}(n)$ of the final label correction result $\bar{y}(n)$ is defined as

where

where $\Omega_{n}=\{(m,j,q)\mid\bar{y}(n)=\hat{Y}_{m,j}(n,q)\}$ means that only those samples, whose label correction suggestions $\hat{Y}_{m,j}(n,q)$ are identical to the final corrected label $\bar{y}(n)$, are used to calculate the normalized average confidence level. Therefore, $\bar{\omega}$ can be maximized when i) the predictions have a high confidence level, which implies a relatively low entropy, or ii) the $n$-th corrected labels suggested by all graphs are of the same category.

Note that the confidence level $\omega_{m,j}(n)$ of the label propagation result $Z_{m,j}$ can be assessed based on the certainty weight described in [26]:

where

where $\bar{Z}_{m,j}$ records the occurrence frequency of each label in the label propagation result matrix $Z_{m,j}$, and $H(\cdot)$ denotes the entropy. Therefore, a larger $\omega_{m,j}(n,q)$ implies a label $\hat{Y}_{m,j}(n,q)$ with higher confidence (i.e., lower uncertainty).

After normalizing the final weights described in (6), we use i) the corrected labels and ii) the weighted cross entropy loss to train the next-epoch ensemble branches and the next-epoch corrected label branch. The loss functions used to train our ensemble model are detailed in the next subsection.

We adopt the following weighted cross-entropy to measure the loss of corrected (pseudo) labels branch:

where $\bar{y}(n)$ is the pseudo label of $x(n)$ derived from our label correction process in the current training epoch, and $p_{\mbox{pseudo}}(n,\bar{y}(n))$ denotes the probability, predicted by the softmax layer of the corrected (pseudo) label branch, of $x(n)$ belonging to class $\bar{y}(n)$.

Then, the noisy label branch is trained via the weighted cross-entropy loss, i.e.,

where

where $p_{\mbox{noisy}}$ denotes the probability predicted by the softmax layer of the noisy label branch. This loss means that if the n-th sample is corrected, we give it a complementary confidence level $1-w(n)$ to highlight its low confidence.

Moreover, because the $M$ ensemble branches are mainly designed to construct the $M$ graphs rather than predicting the classification probability of each sample, we exploit a graph smoothness loss modified from [29] to derive a set of graph embedding via which the inter-class distances are maximized, similar to the concept described in [30]. This loss uses radial basis functions to measure the weighted distance between the features of any two samples of different classes. Because the two samples may come from different $\mathcal{S}_{m}$, this loss can lead to a large enough between-class margin on all graphs. That is,

where $(s,t)\in\Omega=\{(s,t)\mid\bar{y}(s)\neq\bar{y}(t)\}$, $\mathcal{I}(s)$ is an indicator function that returns the index of the disjoint subset containing the $s$-th sample, and $\mathbf{p}^{\mathcal{I}(s)}_{s}$ denotes the $s$-th sample probability vector generated by the softmax layer of the subnetwork trained on the $\mathcal{I}(s)$-th subset. This loss forces the distance between two nodes of different classes on the graph to be large.

$\mathcal{L}_{G}$ has an additional weighting factor $\sqrt{\bar{w}_{s}\bar{w}_{t}}$ which gives an instance pair with a lower confidence level a lower weight to alleviate possible negative effects of instances with lower confidence levels. $\mathcal{L}_{G}$ is different from typical entropy loss and triplet loss [31] designs because $\mathcal{L}_{G}$ aims to map instances of different classes into feature clusters far away from each other. On the contrary, both cross-entropy loss and triplet loss encourage the network to map instances of the same class into the similar features. Consequently, $\mathcal{L}_{G}$ can assist our model to produce distinguishable features for label correction tasks. Finally, the total loss of the proposed model is

We evaluate the proposed method on i) simulated noisy datasets based on CIFAR-10/100 and ii) two other real-world noisy datasets: Clothing1M [32] and Food101-N [33]. The properties of Clothing1M and Food101-N are described below.

First, Clothing1M contains images with 14 categories of fashion clothes. The data in Clothing1M are collected from the Internet, and their category labels are generated based on the surrounding texts, which leads to a label noise rate of 38.46%. Clothing1M also provides a small clean set with labels corrected by human annotators. The numbers of images in Clothing1M’s label-corrected training, validation, testing sets, and the noisy labeled set are about 47.5k, 14.3k, 10.5k, and 1M, respectively. In the four sets, only the 1M noisy labeled set contains noisy labels. We use the noisy labeled set to train our model to evaluate its performance, and then we conduct the ablation study jointly on the noisy labeled set and the human-annotated clean testing set.

Second, Food101-N contains about 310K food images of 101 categories. The images in Food101-N are collected from the Internet with a noisy label rate of 20%. To evaluate the performance of our method, we apply it on Food101-N and derive a label correction result first, and then we train a classifier on the label-corrected dataset and conduct testing on Food-101’s testing set [34]. Note that Food-101’s testing set contains only 55k images with clean labels.

We summarize our experiment settings as the following four points. First, after the data splitting phase of each training epoch, we resize training images $x^{m}_{i}\in\mathcal{S}_{m}$ to $256\times 256$, and then cropped them randomly into $224\times 224$ patches. Hence, training samples for each training epoch are different variants of the source data, which improves the diversity of training samples for each epoch. Second, the optimizer is SGD, and the momentum value is $0.9$. The initial learning rate is $0.01$, and it is reduced by a factor of $\frac{1}{10}$ every $5$ epochs. The maximal training epoch is set to be $15$, and the batch size is $64$. We also adopt ridge (L2) regularization that is weighted by $5\times 10^{-3}$. Third, the number of branches is $M=5$, and the number of data packages per class in each branch is $B=4$. Fourth, the features of all training samples are extracted initially by a ResNet-50 [35] pretrained on ImageNet. After the first training epoch, the features used for our data (re-)splitting phase are derived by the latest network model trained in the noisy label branch.

We evaluate our model’s performance on Clothing1M through two different scenarios. The first scenario is to train our model and other state-of-the-arts on only noisy datasets without any additional supervision. In the second scenario, all models are trained on the same hybrid set consisting of i) the noisy set and ii) the clean set consisting of 50K human-annotated images, and then the obtained models are further fine-tuned on the 50K-image clean dataset via the cross-entropy loss. The results of the first and the second parts are shown respectively in Tables I and II. Note that in the second scenario, the labels of the clean dataset are kept unchanged, and their certainty weight are set to be $1$. Table I demonstrates that our method achieves state-of-the-art performance, i.e., 75.17% accuracy, compared with the other methods. Meanwhile, in the second scenario, our method still outperforms all compared state-of-the-arts without any fine-tuning and is comparable to Self-Learning [10] after fine-tuning with the 50k clean set.

The classifier, trained on the data corrected by our method, achieves an second best accuracy of 85.13% on Food101-N, comparable to the best performance of 85.79% achieved by NoiseRank [9], as demonstrated in Table III. Although the noise rate of Food101-N is about 20%, much lower than that of Clothing1M, the noisy labels in Food101-N are much more complicated. Incorrect labels in Food101-N involve three different types: i) photos with confusing contents, ii) multi-label images, e.g. photos with multiple foods, and iii) anomalies, e.g. a shopfront photo labeled as “fish and chips”. Fig. 4 shows four samples of the second and the third types of mis-annotations. Because our method is designed for confusing label noises, i.e., only one of the three types of mis-annotations in Food101-N, the performance of our method is just comparable with typical data cleaning methods like NoiseRank [9]. Nevertheless, data cleaning methods tend to remove hard samples simultaneously, which leads to irreversible information loss. Consequently, our method works much better than NoiseRank on Clothing1M, a dataset containing no anomalies, but not so good as NoiseRank in fighting against multi-label samples and anomalies in Food101-N.

In this subsection, we discuss the effect of each component in our method, including i) the number of local manifolds $B$, ii) the number of ensemble branches $M$, iii) the settings of $\mathcal{L}_{G}$, and iv) the re-splitting process.

Moreover, Fig. 6 provides another interpretation. By checking the average cosine similarities in an adjacency matrices derived by different ensemble branches during training, we find that the final ensemble model performs the best when the average cosine similarity is in between $0.7$ and $0.8$. When the average cosine similarity reaches $0.9$, the model becomes overfit because the testing accuracy of the ensemble model decreases, slightly though.

We validate the data re-splitting process in the following two steps. First, we extract the feature vectors of images in the to-be-corrected noisy dataset by using a ResNet-50 pretrained on Clothing1M via cross entropy. Second, with the obtained features, we split the to-be-corrected noisy dataset into $M$ disjoint subsets and then apply our method for label correction without data re-splitting.

Fig. 8 illustrates how certainty weight varies with training epochs. The certainty weight described in (6) represents the probability of the correctness of a corrected label. These three figures jointly show that our system design is valid because the number of samples with larger certainty weights increases with the number of training epochs.

In this section, we examine our system architecture design and clarify the effects brought by different combinations of loss terms and system components for all variants of the proposed method. We first examine the performance of each of our model variants on CIFAR-10 with man-made symmetric noises. Fig. 9 shows the t-SNE maps of datasets used for this part of ablation study.

In this experiment set, what we examine include i) the necessity of noisy label branch in our default architecture, ii) the impact of Mixup [39] used for data augmentation, and iii) the effects brought by different loss terms used in the ensemble branches. Mixup is a procedure developed for data augmentation for semi-supervised learning, that enables the obtained model to produce linear prediction results between any pair of samples. Hence, it can yield smooth and robust pseudo labels and also avoid overfitting problems. We summarize the comparison among different loss term designs as follows:

$\bullet$ Ensemble branch:
Although the graph smoothness loss described in (13) is beneficial for deriving a better adjacency matrix, this loss may overfit wrongly-labelled samples easily. Hence, in our default setting, we measure this loss only on those samples whose labels remain unchanged after our correction procedure. However, such setting may reduces the effectiveness of this loss, because, when the loss rate is too high, the amount of uncorrected samples participating in the computation of the graph smoothness loss would be too small. As a result, we measure the weighted sum of i) the MAE (mean absolute error) $\mathcal{L}_{en\_mae}$ between $1$ and the classification probability of corrected samples, and ii) the cross entropy $\mathcal{L}_{en\_ce}$ of uncorrected samples as an alternative of the loss term for ensemble branches. We then compare this alternative design with our default setting $\mathcal{L}_{G}$ described in (13).
$\bullet$ Corrected label branch:
Similarly, we adopt the sum of i) MAE $\mathcal{L}_{cor\_mae}$ and ii) cross entropy $\mathcal{L}_{cor\_ce}$ as an alternative loss design for the corrected label branch. We use the soft-label, which is estimated as a convex combination, weighted by the certainty weight $\bar{\omega}$ depicted in (6), of the prediction result derived by the corrected label branch and that derived the ensemble branches, to compute both the $\mathcal{L}_{cor\_mae}$ and the $\mathcal{L}_{cor\_ce}$. A larger $\bar{\omega}$ will produce a soft-label closer to the label suggested by the corrected label branch. In addition, the weighting factors for loss terms used in our ablation study are listed in Table V.

Finally, Tables VII, VIII, and VI show that by removing the noisy label branch, by changing the loss functions, or by using soft-labels, the proposed system can become suitable for correcting CIFAR-10/CIFAR-100 datasets with man-made random label noises. However, such modifications degrade our system performance for real-world applications, e.g. Clothing1M, whose data are contaminated by confusing label noises, as shown in Fig. VI. Hence, we finally choose to adopt our default architecture and loss settings because this combination is most beneficial for real-world applications.