Group Benefits Instances Selection for Data Purification

Noise-robust learning approaches directly train models using noisy labeled data. They aim to become insensitive to the presence of noisy labels⁵². One typical branch is to develop robust loss functions to overcome the problem that cross-entropy loss is sensitive to samples with corrupted labels⁵³. For example, Wang et al. proposed to leverage a noise-robust reverse cross-entropy³⁴ to symmetrically boost cross-entropy loss. Ma et al. ³⁵ proposed a framework to build new loss functions by combining active and passive robust loss functions. Zhang et al. ³⁶ proposed a generalization of mean absolute error and cross-entropy loss. However, since these robust loss functions typically aim to deal with noisy labels through under-learning, they inevitably underfit clean samples. Another branch is to employ regularization to improve robustness. For example, LS³⁸ built soft labels by combining a one-hot and uniform distribution to provide regularization. OLS³⁹ further improved LS by replacing the uniform distribution with a more reasonable probability on non-target categories. However, these approaches do not explicitly tackle noisy labels, leading to a suboptimal performance.

Noise-cleaning methods aim to tackle noisy labels by discarding or relabeling them. An intuitive type of research is label correction that corrects noisy labels. For example, several works⁴⁰ tried to correct noisy labels by estimating the noise transition matrix. However, it is difficult to estimate the accurate noise transition matrix. Furthermore, these label correction methods are unable to deal with out-of-distribution (OOD) instances²⁸ whose true labels do not belong to the training set. This drawback restricts their application in the real-world scenario. Another typical idea of combating noisy labels is to perform sample selection that identifies and removes corrupted data through proper criteria^{46, 47, 54}. For example, Co-teaching⁴³ utilized the small-loss principle and let peer networks select noisy samples for each other. JoCoR⁴⁴ claimed that different models tend to disagree on false labeled samples and leveraged this principle for noise identification. Nevertheless, the above approaches typically performed sample selection within each mini-batch with a fixed drop rate. Jo-SRC⁵⁰ and Zhang et al. ⁵⁵ claimed that noise ratios in different mini-batches inevitably fluctuate in the training process. To overcome this problem, they replaced the widely-used mini-batch noise identifying strategy with a global one for the purpose of stabilizing the selection results. A growing number of methods such as SELFIE⁴⁹ and Co-learning⁵⁶ combined label correction and sample selection to further boost the performance. Some state-of-the-art noise-cleaning approaches also adopted noise-robust techniques. For example, Jo-SRC⁵⁰ and DivideMix⁴¹ leveraged LS trick and mix-up⁵⁷, respectively.

Generally, we train a DNN on a noisy dataset $\mathcal{D}=\{(x_{i},y_{i})|1\leq i\leq N\}$ with $C$ classes, where $x_{i}$ and $y_{i}$ denote the $i$-th training sample and the corresponding label, respectively. We define the label distribution $q$ of the one-hot label $y_{i}$ as $q(c=y_{i}|x_{i})=1$ and $q(c\neq y_{i}|x_{i})=0$. The DNN model takes each sample $x_{i}$ as input and predicts a probability $p(c|x_{i})$ for each class $c$ using the softmax function. The cross-entropy training loss between the predicted probability distributions $p$ of training images and their corresponding label $q$ is written as

Since commonly-used cross-entropy loss is proved to be sensitive to noisy labels⁵³ and DNNs can perfectly memorize noisy samples^{23, 24}, deep models tend to suffer from label noise when trained using the noisy dataset $\mathcal{D}$ with $\mathcal{L}_{CE}$.

Our learning framework is illustrated in Fig. 2. Motivated by OLS³⁹, we adopt a noise-robust strategy through regularizing the training process at the category level. We define $S=\{S^{0},S^{1},\cdots,S^{t},\cdots,S^{T-1}\}$ as the collection of soft labels of each class for $T$ training epochs. For each epoch $t$, $S^{t}$ is a $C\times C$ matrix, whose columns correspond to soft labels and are initialized to zero. Given an input image $x_{i}$, if the classification is in accord with its label $y_{i}$ , the soft label $S_{y_{i}}^{t}$ of the target class $y_{i}$ will be updated using the predicted probability $p(x_{i})$ through

where $c\in\{1,2,\cdots,C\}$ denotes the indexes of $S_{y_{i}}^{t}$ and $M$ indicates the number of correctly predicted samples with label $y_{i}$. According to Eq. (2), each soft label is the average predicted probability of correctly classified samples belonging to one class.

To stabilize the estimated class soft labels $S^{t}$, we further apply an exponential moving average (EMA) strategy through

where $m$ denotes the momentum that controls the weight on the previous result. EMA smoothes out $S^{t}$ by alleviating the issue that probabilities predicted by the model can be unstable in the training process. After obtaining the soft label, we utilize $S^{t-1}$ to supervise the training process in each epoch $t$. Then the soft training loss can be formulated as

Similar to LS, $\mathcal{L}_{Soft}$ assigns weights to non-target categories. Consequently, it reduces overfitting and improves the robustness against noisy labels. Moreover, it encourages DNNS to learn inter-class similarities and benefits challenging image recognition tasks, e.g. fine-grained classification³⁹.

However, we find that $S_{y_{i}}^{t}$ tends to approach the one-hot label distribution where $y_{i}$ has a large value while other categories only share tiny weights as shown in Fig. 3 (a). This behavior may derive from the strong fitting ability of cross-entropy loss. To address this issue, we utilize the maximum entropy principle⁵⁸ as a strong regularization. It forces the model prediction to be less confident. The maximum entropy loss can be formulated as

Since $\mathcal{L}_{ME}$ aims to increase the entropy of prediction $p(x_{i})$, it leads to a more reasonable soft label as illustrated in Fig. 3 (b). More importantly, since $\mathcal{L}_{ME}$ makes predictions less confident, it further reduces the risk of overfitting noisy labels and guides DNNs to become more noise-robust.

Finally, both the hard and soft labels are leveraged as supervision with the maximum-entropy principle applied as regularization. The total training loss of our group regularization strategy can be represented by

where $w$ and $\gamma$ are weights to balance contributions of $\mathcal{L}_{CE}$, $\mathcal{L}_{Soft}$, and $\mathcal{L}_{ME}$. Owing to $\mathcal{L}_{Soft}$ and $\mathcal{L}_{ME}$, our group regularization strategy guides DNNs to become less sensitive to noisy labels and boosts their robustness. Furthermore, class soft labels generated by group regularization strategy can be leveraged for instance purification.

After obtaining class soft labels, we can perform data purification from the instance aspect. According to the memorization effect^{23, 24}, DNNs learn the clean and easy pattern in the initial epochs. Accordingly, clean instances tend to have more contributions to the estimation of class soft labels at the early stage. Moreover, our group regularization strategy reduces the contribution of noisy labels in model optimization by significantly improving noise-robustness. Therefore, generated class soft labels should be closer to predictions of clean samples than that of noisy ones. Based on this phenomenon, we can build a noise identification criterion.

Inspired by Jo-SRC⁵⁰, we adopt the Jensen-Shannon (JS) divergence between each probability $p(x_{i})$ and its class soft label $S_{y_{i}}^{t-1}$ as our noise identification criterion through

where $D_{KL}$ is the Kullback-Leibler (KL) divergence function and $p_{i}$ denotes the simplified form of $p(x_{i})$. In this formula, $d_{i}$ is a measure of the difference between predicted probability $p_{i}$ and corresponding class soft label $S_{y_{i}}^{t}$, where a larger value indicates a more significant difference. Moreover, $d_{i}$ is a symmetric measure and bounded in $[0,1]$ when using a base $2$ logarithm⁵⁹.

Since clean samples tend to be closer to their class soft labels, they should have smaller values of $d_{i}$ than noisy instances. Accordingly, we can utilize a threshold to separate clean instances from noisy ones. We define the threshold $thr$ for each epoch $t$ as

where $d=\{d_{1},d_{1},\cdots,d_{i},\cdots,d_{N}\}$ indicates the collection of $d_{i}$ for the entire dataset, $\textbf{mean}(\cdot)$ and $\textbf{std}(\cdot)$ denote computing average value and standard deviation respectively, and $\alpha$ is a hyperparameter. Specifically, for each training batch $\mathcal{B}$, we divide it into a clean set $\mathcal{B}_{clean}$ and a noisy set $\mathcal{B}_{noisy}$ after a warm-up period $t_{m}$ by

Once noisy samples are identified, we further perform label re-assignment on the noisy samples $\mathcal{B}_{noisy}$ to leverage revisable ones. Specifically, we first calculate the JS divergence between the probability $p(x_{i})$ of each instance in $\mathcal{B}_{noisy}$ and soft label $S_{\hat{y}_{i}}^{t-1}$ of its predicted class $\hat{y}_{i}$ through

Then, since $\hat{d}_{i}$ is bounded in $[0,1]$, we can utilize a hard threshold $\tau$ to select revisable samples in $\mathcal{B}_{noisy}$ through

Eq. (11) indicates that we regard a noisy sample as a revisable one when its probability $p(x_{i})$ is close enough to the soft label of its predicted category $S_{\hat{y}_{i}}^{t-1}$. Its predicted class $\hat{y}_{i}$ is assigned as pseudo label.

Finally, we jointly utilize clean and relabeled samples for training by Eq. (6). Furthermore, we apply $\mathcal{L}_{ME}$ in Eq. (5) on discarded images to encourage the model to generate even probabilities for them. This design guides predicted probabilities of noisy samples to become more different from estimated class soft labels and therefore improves the reliability of noise-cleaning in the following epochs. Details of our proposed GRIP are demonstrated in Algorithm 1.

We follow the common experimental settings as in recent works^{43, 44} and manually corrupt CIFAR-10 and CIFAR-100⁵¹ to create synthetic noisy datasets. Specifically, we adopt symmetric and asymmetric noisy labels with noise transition matrices with a transition matrix $Q$.

We generate symmetric noisy labels by uniformly flipping a given ratio of training labels to other classes. Asymmetric label noise simulates the fine-grained recognition task with noisy labels, where very similar classes may confuse annotators. In this setting, we only flip a specific set of classes, e.g. flipping CAT to DOG in CIFAR-10. The definition of transition matrix $Q$ is as follows:

where $n$ and $\epsilon$ denote the number of the class and noise ratio, respectively. For evaluating the model classification performance, we take Average Classification Accuracy (ACA) as the evaluation metric.

Following JoCoR⁴⁴, we adopt a 7-layer CNN network architecture and leverage Adam optimizer with momentum set to $0.9$ for training. We train the network for $200$ epochs with batch size set to $128$. The learning rate is initialized to $0.001$ and starts to linearly decrease to $0$ after $80$ epochs. As for hyper-parameters, we set momentum $m$ for EMA and weight $w$, $\gamma$ to $0.5$, $0.5$, $1$, respectively. Warm-up epoch $t_{m}$ and hyper-parameter $\tau$ are empirically set to $10$ and $0.03$, respectively. As for $\alpha$, we adopt a linear decrease from $1$ to $0.2$ in $5$ epochs to smooth the discarding process, which is similar to the increasing drop rate trick in Co-teaching⁴³. All experiments are conducted on one NVIDIA Tesla V100 GPU.

To evaluate our approach on synthetic datasets, we compare our approach with the following state-of-the-art algorithms: Decoupling⁴², Co-teaching⁴³, Co-teaching+⁶¹, JoCoR⁴⁴, SELC⁶² and the backbone network straightforwardly trained on noisy datasets. As the work of JoCoR⁴⁴ has reproduced most other baselines^{42, 43, 61}, we directly copy results from it for simplicity. For SELC⁶², we re-execute their code and report the experimental results. Our method is trained using the same experimental settings as the above methods, and therefore the comparisons are fair.

In this subsection, we visualize the noise transition matrix $Q$ and soft labels $S$ estimated by GRIP to illustrate the noise-robustness of our approach. To be detailed, we investigate the Symmetry-$50\%$ and Asymmetry-$40\%$ cases and demonstrate soft labels estimated after the warm-up period in Fig. 5.

From Fig. 5, we can observe that in the Symmetry-$50\%$ case, all non-target classes of soft labels share similar weights (Fig. 5 (a) and (c)), and target categories have lower weights in soft labels than in noise transition matrices. This phenomenon indicates that our approach becomes less confident on the noisy datasets and shows robustness against label noise.

While in the Asymmetry-$40\%$ case, flipped noisy classes have much lower weights in soft labels than in noise transition matrices (Fig. 5 (b) and (d)), which indicates that our approach is less likely to overfit noisy labels. The great noise-robustness results from the maximum entropy regularization, which guides the model to become less confident on potential noisy labels.

We evaluate our approach on WebFG-496²⁰, which is designed for research on webly supervised fine-grained classification tasks. It contains three real-world sub-datasets, Web-bird, Web-aircraft, and Web-car. They utilize the fine-grained category labels of benchmark datasets CUB200-2011⁶⁹, FGVC-aircraft⁷⁰, and Cars-196⁷¹ as target categories to crawl web images from Bing Image Search Engine. Compared with benchmark datasets, they have a larger number of training samples (18388, 13503, and 21448, respectively) but potential noisy labels. The test sets are directly taken from three benchmark datasets (CUB200-2011, FGVC-aircraft, and Stanford Cars). Details of three sub-datasets are as follows and summarized in Table 3.

Web-bird consists of 200 different subcategories of birds. Its web training set contains 18388 images and the test set has 5794 clean samples. The training label accuracy estimated by random sampling is about 65%.

Web-aircraft covers 100 variants of aircraft. The number of web training samples and clean test images are 13503 and 3333, respectively. Its estimated web label accuracy is approximately 73%.

Web-car contains 196 types of vehicles. Its training set consists of 21448 web samples with a roughly estimated label accuracy of 67%. The test set contains 8041 manually labeled images.

We follow the evaluation metric in section 4.1 and utilize ACA to evaluate model performance.

We adopt a ImageNet pre-trained ResNet-50⁷² model as our backbone network and resize input images to $(448,448)$ with random horizontal flip as weak data augmentation. The network is trained for $100$ epochs with batch size set to $32$. The learning rate is initialized to $0.01$ and decreases in a cosine annealing manner⁷³. The momentum and weight decay of stochastic gradient descent (SGD)⁷⁴ optimizer are set to $0.9$ and $10^{-5}$, respectively.

As for hyper-parameters, we set momentum $m$ for EMA and weight $w$, $\gamma$ to $0.5$. Warm-up epoch $t_{m}$ is empirically set to $5$. We set $\alpha$ to $0.5$ on Web-bird and adopt a linear decreasing $\alpha$ on Web-aircraft and Web-car. Specifically, $\alpha$ linearly decreases from $1$ to $0.8$ and $0.3$ in $5$ epochs on Web-aircraft and Web-car, respectively, which is motivated by the increasing drop rate trick⁴³. Threshold $\tau$ for relabeling is set to $0.04,0.02,0.04$ on Web-bird, Web-aircraft, and Web-car, respectively. Our experiments are all performed on one NVIDIA Tesla V100 GPU.

On real-world datasets, our baselines contain the following state-of-the-art methods: Decoupling⁴², Co-teaching⁴³, Sub-center⁶³, Co-teaching+⁶¹, Self-Adaptive⁶⁴, JoCoR⁴⁴, DivideMix⁴¹, Jo-SRC³⁹, OLS³⁹, PLC⁶⁵, Peer-learning²⁰, SELC⁶², Co-LDL⁶⁶, AGCE⁶⁷ and CMW-Net-SL⁶⁸. We reproduce most of the above baselines^{42, 43, 63, 61, 64, 44, 41, 39, 65, 20, 62, 67} using the same backbone network for fair comparisons. In addition, we also train a ResNet-50 network using the cross-entropy loss function for comparison (Backbone).

We demonstrate ACA performances of baseline approaches and GRIP on WebFG-496 in Table 4. From Table 4, we can observe that GRIP significantly outperforms baselines on all three datasets. Specifically, it surpasses the best results of baselines by $1.01\%$, $0.56\%$, and $1.36\%$ on Web-bird, Web-aircraft, Web-car, respectively. For clearer illustration, we also present test accuracy trending of GRIP and compare it with some representative noise-cleaning baselines in Fig. 6. As illustrated in Fig. 6, GRIP shows higher accuracies and faster training speeds than other noise-cleaning approaches^{42, 44} on all benchmark datasets. This superiority owes to our group regularization strategy, which significantly improves the noise-robustness. In addition, compared with noise-robust approaches^{63, 39, 62, 67, 68} in Table 4, our approach achieves higher test accuracies by leveraging the instance purification strategy to specifically tackling noisy labels. The significant improvements over baselines demonstrate the effectiveness of simultaneously leveraging both noise-robust and noise-cleaning strategies for combating noisy labels.

Particularly, GRIP shows superior performance than DivideMix⁴¹, which utilizes noise-cleaning and noise-robust algorithms simultaneously. DivideMix has large gaps to GRIP on Web-bird ($8.13\%$) and Web-car ($5.22\%$). The reason may lie in the Gaussian Mixture Model (GMM)⁷⁵ used for dividing training samples in DivideMix. It can be difficult to fit a GMM on the real-world noisy data distribution. If GMM fails, the performance will inevitably decline.

Furthermore, we can observe from Table 4 that some baselines (Decoupling⁴², Co-teaching+⁶¹, Self-Adaptive⁶⁴, PLC⁶⁵, CMW-Net-SL⁶⁸) only show slight improvements or even inferior performances to Backbone. The reason can be that they are designed and tested on coarse-grained synthetic noisy datasets. As a result, they tend to be less practical for fine-grained tasks in a real-world scenario. Compared with them, our approach is more effective in practical application.

In this experiment, we investigate the parameter sensitivities of weights $w$ and $\gamma$ for loss functions, momentum $m$ for EMA, $\alpha$ and $\tau$ for noise-cleaning, and warm-up epoch $t_{m}$. Although our approach seems to have many parameters, we will show that half of them are robust and easy to adjust. The experimental results are shown in Fig. 7.

We set $\gamma$ to $0.5$ and changes $w$ from $0$ to $0.9$ for investigation. From Fig. 7 (a), we can observe that the performance first steadily increases to the optimal value as $w$ rises. This phenomenon indicates that $\mathcal{L}_{soft}$ boosts the robustness with a proper $w$. However, if $w$ further increases, the supervision of $\mathcal{L}_{CE}$ is weakened, which results in a performance decline. To achieve the best performance, we set $w$ to $0.5$ on all datasets, which is a balanced weight between $\mathcal{L}_{soft}$ and $\mathcal{L}_{CE}$. In addition, we can also see that $w$ is robust in $[0.3,0.6]$, which indicates a relatively balanced $w$ can work well.

Similar to the analysis on $w$, we set $w$ to $0.5$ and changes $\gamma$ from $0$ to $1$. We can observe from Fig. 7 (b) that the performance climbs as $\gamma$ rises from $0$ to $0.5$, then slightly decreases when $\gamma$ further increases. Supported by this result, we simply set $\gamma$ to $0.5$ on all datasets. We can also find that $\gamma$ is robust in $[0.4,1]$.

We analyze the effect of $m$ in Fig. 7 (c). It can be observed from Fig. 7 (c) that unless $m$ is too large (over 0.8), applying EMA can boost the performance over the baseline ($m$=$0$). Furthermore, EMA can work well and achieve close test accuracies in $[0.3,0.7]$.

We do not perform label re-assignment and only analyze the effect of $\alpha$ on sample selection in Fig. 7 (d). When $\alpha$ increases from $0$ to $0.5$, the performance steadily climbs because more training samples are utilized. However, when $\alpha$ is larger than $0.5$, the performance declines with some fluctuations. The reason is that fewer noisy images are discarded and thus noise-cleaning becomes less effective.

The analysis of $\tau$ is illustrated in Fig. 7 (e). From Fig. 7 (e), we can see that performance climbs as $\tau$ increases from $0.03$ to $0.05$. Then it drops fast when $\tau$ becomes larger ($\tau=0.07$). The reason can be that too many noisy samples are relabeled and reused. Some of them may still have false labels or even be out-of-distribution (OOD). From this result, we believe that a small value of $\tau$ is safe and can boost performance.

The analysis of $t_{m}$ is illustrated in Fig. 7 (f). We can observe from Fig. 7 (f) that if the warm-up period is too short (less than $3$), the model cannot learn reliable soft labels for noise identification, which results in an unsatisfying performance. A proper $t_{m}$ lies in $[5,10]$. If $t_{m}$ further increases, the model will be affected by noisy labels in the warm-up period and noise-cleaning becomes less useful, resulting in declining test accuracies.

From the experimental results in Fig. 7, we can conclude that weights $w$, $\gamma$, and momentum $m$ are robust and easy to adjust. Parameters $\alpha$ and $\tau$ for noise-cleaning should be changed on each dataset because they are concerned with noise ratios. As for $t_{m}$, a relatively small value is recommended, e.g. in $[5,10]$.

In this experiment, we investigate the applicability of GRIP on small datasets by changing the number of web images used for each category on Web-bird²⁰. To be detailed, we construct sub-datasets from Web-bird by randomly selecting $10$ to $90$ samples for each class. Then we compare the performance of the backbone network, our group regularization strategy (GR), and our proposed GRIP on each sub-dataset in Fig. 8. It can be observed from Fig. 8 that our group regularization strategy shows significant and consistent improvements across different dataset sizes over the baseline. After applying the instance purification strategy, GRIP further boosts the performance on each sub-dataset. It shows remarkable noise-robustness even when the dataset is rather small, e.g. $20$ images per class. From the experimental results, we can conclude that our approach is insensitive to the dataset size.

We can also see from Fig. 8 that the ACA performance climbs steadily when more training samples are utilized. Therefore, leveraging free web images is a promising research direction as it allows boosting model performance and robustness through enlarging datasets.

We test our approach using different backbone networks on Web-bird to analyze the applicability in Table 5. We can observe from Table 5 that our approach boosts the performance by $9.53\%$, $7.81\%$, and $9.23\%$ on VGG-16⁷⁶, ResNet-18, and ResNet-50, respectively. The experimental results indicate that our approach is robust and shows remarkable improvements across different CNN architectures.

In this subsection, we gradually add components in our method to the baseline model and present the contribution of each component in Table 6. From lines $1$ to $6$ in Table 6, we can observe that all components contribute to the performance improvements. To be detailed, leveraging $\mathcal{L}_{Soft}$ surpasses the baseline by around $2\%$. Then we further makes remarkable improvements through applying $\mathcal{L}_{ME}$ ($3.35\%$) and EMA ($0.44\%$). Owing to our group regularization strategy, the ACA performance reaches $76.9\%$ and significantly surpasses the baseline. After further applying instance purification, the final performance reaches $78.91\%$ by employing JS divergence noise identification ($1.42\%$) and relabeling ($0.59\%$). Resorting to the contribution of each component, our approach boosts the final performance by $7.81\%$ over the baseline.

In this experiment, we demonstrate the superiority of our proposed approach over other similar strategies. We first compare the proposed group regularization with the widely-used LS trick. Then we compare JS divergence noise identification with the small-loss principle. We also compare the global and mini-batch selection based on our JS divergence principle.

The experimental results are presented in Table 6 (lines $7$ to $11$). We can observe from Table 6 that LS shows a slightly inferior performance to simply leveraging soft label supervision $\mathcal{L}_{Soft}$. Applying LS and maximum entropy strategy $\mathcal{L}_{ME}$ together shows nearly no improvement over utilizing $\mathcal{L}_{ME}$ alone. On the contrary, combining $\mathcal{L}_{Soft}$ and $\mathcal{L}_{ME}$ boosts the performance significantly. This result demonstrates the advantages of utilizing estimated soft labels over LS: higher performance and better flexibility. Owing to $\mathcal{L}_{Soft}$ and $\mathcal{L}_{ME}$, our group regularization strategy surpasses LS remarkably.

Comparing lines $10$ and $11$ in Table 6, we can find that noise identification within mini-batch based on JS divergence is superior to that based on loss. This result supports our argument that utilizing class soft labels is more effective than simply relying on loss values in noise identification. Comparing ling $5$ and $11$, we can observe that global selection further boosts the performance. The improvement derives from that the noise rate imbalance problem is alleviated.

Note that we apply $\mathcal{L}_{ME}$ on discarded images. In order to demonstrate the contribution of this design, we remove it for comparison. Then we find that the performance drops from $78.32\%$ to $77.66\%$. Since noisy samples are utilized for training in the warm-up period, the network memorizes them to some extent. They potentially misguide noise identification and degrade performance. To solve this problem, $\mathcal{L}_{ME}$ provides a strong regularization to guide the network to forget noisy samples and make them farther from class soft labels to guarantee more reliable noise identification.

We sampled noise-cleaning results on WebFG-496 and visualize them in Fig. 9. We can observe from Fig. 9 that our method can effectively divide clean, revisable, and discarded samples. We can also notice that web dataset inevitably contain noisy labels. For example, searching web images for cars has a risk of getting some images of tires and steering wheels. This phenomenon reminds us that noise-cleaning operation is necessary for training robust models in learning with noisy labels tasks.