Towards Robustness to Label Noise in Text Classification via Noise Modeling

It has been empirically observed that classifiers that capture input semantics do not fit the noise before significantly learning from the clean samples (Arazo et al., 2019). For a classifier trained using a cross entropy loss($\mathcal{L}_{CE}$) on the noisy dataset, this can be exploited to cluster the input samples as being clean/noisy in an unsupervised manner. Initially the training loss on both clean and noisy samples is large, and after a few training epochs, the loss of majority of the clean samples reduces. Since the loss of the noisy samples is still large, this segregates the samples into two clusters with different loss values. On further training, the model over-fits on the noisy samples and the training loss on both samples reduces. We illustrate this in Fig. 2(a)$-$(c). We fit a 2-component Beta mixture model (BMM) over the normalized training losses ($\mathcal{L}_{CE}(\hat{y}^{(c)},\cdot)\in[0,1]$) obtained after training the model for some warmup epochs $T_{0}$. Using a Beta mixture model works better than using a Gaussian mixture model as it allows for asymmetric distributions and can capture the short left-tails of the clean sample losses. For a sample $(x,y)$ with normalized loss $\mathcal{L}_{CE}(\hat{y}^{(c)},y)=\ell$, the BMM is given by:

where $\Gamma$ denotes the gamma distribution and $\alpha_{c/n},\beta_{c/n}$ are the parameters corresponding to the individual clean/noisy Beta distributions. The mixture coefficients $\lambda_{c}$ and $\lambda_{n}$, and parameters ($\alpha_{c/n},\beta_{c/n}$) are learnt using the EM algorithm. On fitting the BMM $\mathcal{B}$, for a given input $x$ with a normalized loss $\mathcal{L}_{CE}(\hat{y}^{(c)},y)=\ell$, we denote the posterior probability of $x$ having a clean label by:

The BMM $\mathcal{B}$ learnt from Fig. 2b is shown in Fig. 2d.

We aim to train $M,N_{M}$ such that when given an input, $M$ predicts the clean label and $N_{M}$ predicts the noisy label for this input (if $\mathcal{F}$ retains the original clean label for this input, then both $M,N_{M}$ predict the clean label). Thus for an input $(x,y)$ having a clean label, we want $\hat{y}^{(c)}{=}\hat{y}^{(n)}{=}y$; and for an input $(x,y)$ having a noisy label, we want $\hat{y}^{(n)}{=}y$ and $\hat{y}^{(c)}$ to be the clean label for $x$. We jointly train $M,N_{M}$ using the de-noising loss proposed below:

By jointly training $M,N_{M}$ with $\mathcal{L}_{DN}$, we implicitly constrain the label noise in $N_{M}$. We use an alternative loss formulation by replacing the Bernoulli $\mathcal{B}(x)$ with the indicator $\mathbbm{1}[\mathcal{B}(x){>}0.5]$. For ease of notation, we refer the former (using $\mathcal{B}(x)$) as the soft de-noising loss $\mathcal{L}_{DN{-}S}$ and the latter as the hard de-noising loss $\mathcal{L}_{DN{-}H}$.

Thus we use the following 3-step approach to learn $M$ and $N_{M}$:

1. (1)

   Warmup: Train $M$ using $\mathcal{L}_{CE}(\hat{y}^{(c)},y)$.

2. (2)

   Fitting BMM: Fit a 2-component BMM $\mathcal{B}$ on the $\mathcal{L}_{CE}(\hat{y}^{(c)},y)$ for all $(x,y)\in(X,Y^{(n)})$.

3. (3)

   Training with $\mathcal{L}_{DN}$: Jointly train $M$ and $N_{M}$ end-to-end using $\mathcal{L}_{DN{-}S/H}$.

We summarize our methodology in Algorithm 1, when using $\mathcal{L}_{DN-H}$.

For a specific Noise %, we randomly change the original labels of this percentage of samples. Since the noise function is independent of the input, we use logits from $M$ as the input $R(x)$ to $N_{M}$. We report the Best and (Last - Best) test accuracies in Table 1. From the experiments, we observe that:

(i) $L_{DN-S}$ and $L_{DN-H}$ almost always outperforms the baseline across different noise levels. The performance of $L_{DN-S}$ and $L_{DN-H}$ are similar. We observe that training with $L_{DN{-}S}$ tends to be better at low noise %, whereas $L_{DN{-}H}$ tends to be better at higher noise %. Our method is more effective for TREC than AG-News, since even the baseline can learn robustly on AG-News.

(ii) Our approach using $L_{DN{-}S}$ and $L_{DN{-}H}$ drastically reduces over-fitting on noisy samples (visible from small gaps between Best and Last accuracies). For the baseline, this gap is significantly larger, especially at high noise levels, indicating over-fitting to the label noise. For example, consider word-LSTM on TREC at 30% noise: while the baseline suffers a sharp drop of 24.8 points from 79.6%, the accuracy of the $L_{DN{-}S}$ model drops just 1.0% from 83.4%.

We further demonstrate that our approach avoids over-fitting, thereby stabilizing the model training by plotting the test accuracies across training epochs in Fig. 3. We observe that the baseline model over-fits the label noise with more training epochs, thereby degrading test accuracy. The degree of over-fitting is greater at higher levels of noise (Fig. 3(b) vs Fig. 3(a)). In comparison, our de-noising approach using both $L_{DN-S}$ and $L_{DN-H}$ does not over-fit on the noisy labels as demonstrated by stable test accuracies across epochs. This is particularly beneficial when operating in a few-shot setting where one does not have access to a validation split that is representative of the test split for early stopping.

We heuristically condition the noise function $\mathcal{F}$ on lexical and syntactic input features. We are the first to study such label noise for text inputs, to our knowledge. For both the TREC and AG-News, we condition $\mathcal{F}$ on syntactic features of the input: (i) The TREC dataset contains different types of questions. We selectively corrupt the labels of inputs that contain the question words ‘How’ or ‘What’ (chosen based on occurrence frequency). For texts starting with ‘How’ or ‘What’, we insert random label noise (at different levels). We also consider $\mathcal{F}$ conditional on the text length (a lexical feature). More specifically, we inject random label noise for the longest x% inputs in the dataset. (ii) The AG-News dataset contains news articles from different news agency sources. We insert random label noise for inputs containing the token ‘AP’, ‘Reuters’ or either one of them. We concatenate the contextualised input embedding from the penultimate layer of $M$ and the logits corresponding to $\hat{y}^{(c)}$ as the input $R_{M}(x)$ to $N_{M}$. We present the results in Tables 3 and 4.

On TREC, our method outperforms the baseline for both the noise patterns we consider. For the question-length based noise, we observe the same trend of $L_{DN{-}H}$ outperforming $L_{DN{-}S}$ at high noise levels, and vice-versa. On AG-News, the noise % for inputs having the specific tokens ’AP’ and ’Reuters’ are relatively low, and our method performs at par or marginally improves over the baseline performance. Interestingly, the input–conditional noise we consider makes effective learning very challenging, as demonstrated by significantly lower Best accuracies for the baseline model than for random noise. As the classifier appears to overfit to the noise very early during training, we observe relatively smaller gaps between Best and Last accuracies. Compared to random noise, our approach is less efficient at alleviating the (Best-Last) accuracy gap for input-conditional noise. These experiments however reveal promising preliminary results on learning with input-conditional noise.