Learning Image Labels On-the-fly
for Training Robust Classification Models

Supervised deep learning methods, although proven to be effective on many tasks, rely heavily on the quality of the data and its corresponding annotations. Some tasks enjoy almost error-free annotation, such as handwritten numbers and simple natural images. However, for other applications including most medical image analysis tasks, the inherent ambiguity of the task leads to unavoidable noise and fuzziness within the annotations themselves, no matter how experienced the expert labelers are. Meanwhile, under a multi-labeler setting for quality control purpose, the significant intra- and inter-observer variability injects even more uncertainties into the resulting labels. Beyond the above challenges, for the specific task of chest X-ray image classification, due to the fact that most labels of the available large-scale open datasets are automatically mined by Natural Language Processing (NLP) algorithms, there will be yet another layer of error-prone operation on top of existing variability. Ideally, we would prefer multiple manually and reliably labelled close-to-truth annotations, while in reality, most of the data only have a single annotation from an algorithm with relatively low accuracy.

Learning to learn from a variety of data (labels) falls within the scope of meta-learning, which is popular in many machine learning application, e.g., domain adaptation/generalization (Finn et al., 2017; Dou et al., 2019) and few-shot learning (Snell et al., 2017; Liu et al., 2019). Those previous meta-learner models (as illustrated in Fig. 1(b)) often focus on learning the distribution of data (inputs of tasks) and specifying the update strategy of learner model parameters. Indeed, data from different sets (distributions) will contribute to the final learner model. On the contrary, we do not want the model to learn from erroneous labels (from less-experienced labelers) but learn only from “true” labels. We utilize the learner model parameters (via a meta-training process) to sample “true” labels for training a single learner model (as shown in Fig. 1(c)).

To address these challenges, we proposed an attention-on-label strategy to benefit the training from multiple labels on the same subject. Instead of the resource-demanding process of asking several human annotators to label the same data, we choose to utilize annotations from different algorithm-based labelers, which only add little overhead beyond the single-labeler scenario. A meta-training scheme is adopted and integrated in our proposed label-sampling module to attend the labels that benefit the model learning the most through additional back-propagation processes.

Our contributions in this work are three-fold: 1) We proposed a training framework to compute the image labels on-the-fly in the classification tasks. A meta-training based approach is introduced to attend and sample the labels from multiple annotators; 2) Gradient flows towards the label are investigated and implemented. Indeed, the multiple sets of labels are inputs to the training framework. Learnable operations of the labels will require additional updates of label-related model parameters; 3) We not only prove the concept on CIFAR-10 but also perform experiments on two chest X-ray datasets with both image-only and image-text classification tasks. In all datasets, superior performance of the proposed method is demonstrated in the image classification tasks compared to baseline methods.

Meta learning: Meta learning aims to learn a generalizable model by situating itself at a higher level than conventional learning. This can be achieved in several ways such as finding weights that can be easily adapted to other models (Finn et al., 2017) or domains (Li et al., 2017) during the training process. Meta learning results in models that can converge quickly with a few examples (Ravi & Larochelle, 2017). They all share a similar meta-training process while the various goals of meta-training can divide them to different routes as examples shown in Fig. 1. In this work, we target at weighting the importance of each label set based on its meta-training feedback and learning from the most effective labels.

Learning from noisy labels: Learning from noisy labels (Natarajan et al., 2013; Li et al., 2020; Zhang et al., 2020b) has been a popular topic in deep learning due to its prevalence in many existing datasets with intra- and inter-observer variability, and the inherent uncertainties of both data and task themselves. For medical imaging applications with a high degree of ambiguity, this issue is even more significant. Recent works attempt to address this challenge via a consistency loss with a teacher model  (Li et al., 2019), loss weighting with 2nd order derivatives (Zhang et al., 2020b), and for medical image specifically, an online uncertainty sample mining strategy (Xue et al., 2019). Please note that they all focus on noise labels from a single annotator, while we attempt to design an attention mechanism to utilize labels from different sources together.

Multi-label classification in chest X-ray: Because of its wide application and easy accessibility, chest X-ray is one of the major research areas in the field of medical image analysis. Among the pioneering works (Wang et al., 2017; Rajpurkar et al., 2017; Yao et al., 2017; Li et al., 2018; Tang et al., 2018; Hwang et al., 2019; Tang et al., 2020) in this area of deep learning, TieNet (Wang et al., 2018) first introduces an end-to-end trainable CNN-RNN architecture to extract distinctive text representations in addition to image features for improving label quality. More recently, a graph model was incorporated to enhance the learning accuracy (Zhang et al., 2020a).

Multi-observer studies: To ensure the annotation quality, especially for medical images where high expertise is required, it is common to have multiple set of labels on the same set of data (Rasch et al., 1999; Cherian et al., 2005). In a sense, each annotation can be regarded as an estimation with uncertainty, good or bad, for the underlying “true label”. Thus, algorithms taking this uncertainty factor into consideration are needed in order to make better use of such multi-observer data. Kohl et al. (2018) attempt to learn a distribution from a set of diverse but plausible segmentation from multiple graders. A recent work (Tanno et al., 2019) proposed to model annotators by a confusion matrix which is jointly estimated during classification. In our work, instead of human annotators with different skill-levels, we employed several “algorithmic labelers” to generate multiple annotations from the same raw data. Comparing with their human counterparts, there exists less limitations and cost to increase the number of labelers, while the resulting labels can be more noisy. Hence, we choose a different strategy of attention model and meta-learning to benefit the model learning process.

The accuracy of NLP algorithm-based labelers has been studied (Peng et al., 2018; Irvin et al., 2019) and verified by a small set of hand-labeled data (based on associated report texts). The noises in annotations could be traced from many sources, e.g., algorithmic errors, incomplete information in reports, and misjudgement from the clinicians. All of these could elevate the uncertainty and impair the reliability of the publish data and associated ground-truth labels, in terms of their utilization in modern machine learning paradigms. “Who to believe?” becomes a fundamental question to answer, which will ultimately have a significant impact on the performance of trained model.

MIMIC-CXR dataset (Johnson et al., 2019) provides the labels sets from two independent algorithm based annotators with positive, negative, and uncertain cases. The availability of all these different sourced labels enables the observation of the uncertainty inherent to some data sample (with different values in multiple label sets). As shown in Table 1, there are many uncertain cases in each kind of finding while the uncertainty / positive ratio may vary in diseases (ranged from $10\%$ to $110\%$).

Here, we try to tackle this problem by training a single multi-label classification model while considering all the available label sets. A novel attention-on-label training process is introduced. For each set of labels, we perform individual back-propagation as a form of meta-training and then compute the new image/image+text feature using the individual updated model. Based on the new features, we attended to the label set with more representative features (as a weight) and sample the weighted summary of labels for the final update of the model in each iteration. Fig. 2 illustrates the overall architecture and learning processes for both image and label model parameters.

CIFAR-10: We simulate 5 different types of annotators (with different experience) in a similar manner as Tanno et al. (2019) by injecting label noises into the training set of CIFAR-10, namely 1) hammer-spammer (HS), 2) structured-flips (SF), 3) ordered-confusion (OC), 4) Adversarial (AD), and 5) average (AVG) of 1-4. Each set of noisy labels are generated based on the defined confusion matrices for each type (as shown in Fig. 3). Whether each sample would have a noisy label is randomly selected while the overall noisy distribution should correspond to each confusion matrix, individually. Within all the noisy training data, we randomly select $20\%$ as the validation set.

MIMIC-CXR: The MIMIC Chest X-ray (Johnson et al., 2019) Database is a large publicly available dataset of chest radiographs with labels mined from image-associated text radiology reports using two different NLP based annotation tools, i.e., negbio (Peng et al., 2018) and chexpert (Irvin et al., 2019). The uncertain findings are marked as -1 in the original datasheet. Here, uncertainties are set to either 0 or 1 to form 4 different label sets, i.e., negbio_u0, negbio_u1, chexpert_u0, and chexpert_u1. The dataset contains 377,110 radiographs and labels from the 227,827 free-text radiology reports. Totally, 14 disease findings are listed ($N=14$). In our experiments, only the frontal view images are adopted, the number of which is equal to the number of reports. We utilize the official data split for training, validation and testing.

MIMIC-CXR 1K hand-labeled Test: Following the same labeling protocol proposed by Demner-Fushman et al. (2012), we randomly selected 1000 images and associated textual reports from the testing set of MIMIC-CXR and one of our staff (trained by a board-certified radiologist) hand-labeled the 1000 images by assigning the 14 labels manually to each image based on the reports, which will be released publicly. We believe it will also benefit the research in chest x-ray disease classification.

OpenI: OpenI (Demner-Fushman et al., 2015) is a public dataset of chest X-rays collected from multiple institutes by Indiana University. In total, we fetch 3,851 unique radiology reports and 7,784 associated frontal/lateral images. To keep the consistency with MIMIC-CXR dataset, we use the same 14 categories of findings as mentioned above in the experiments. In our experiments, only 3,643 unique front view images and corresponding reports are evaluated.

The following methods in addition to the proposed method (Ours) are included in the comparison:

ResNet-50 (R50, He et al., 2016): We take the network based on ResNet-50 as a baseline. It adopts an ImageNet pre-trained ResNet-50 (from Conv1 to Res5c) as the backbone, followed by a GAP layer and a fully-connect layer for the final classification. Optionally, BioBert embedded text features will be concatenated with the output of GAP before the classification.

CM (Tanno et al., 2019): This method multiplies a confusion matrix with the probability that the model produces for each class. The basic assumption is that this confusion matrix can correct the missed labeled data and return the probability for the truth using the learned confusion matrices. We carefully implement it according to the code snapshot provided by the authors.

NG (Zhang et al., 2020a): This method utilizes the prior knowledge of the disease relations as a form of knowledge graph. By injecting such prior knowledge and employing a graph convolutional network, it learns the underlying info for the final classification and report generation task. It is worthy to note that the results we report in the experiments are produced by a model that is both trained and evaluated on the OpenI dataset.

TieNet (Wang et al., 2018): It focuses more on how to learn the image and text embedding together using a CNN+RNN framework. Its LSTM based text embedding is relatively more complicated but also more representative through learning. We only adopt the text embedding from a pre-trained BioBERT model for our comparison, which is less customized.

Evaluation Metric: Receiver Operating Characteristic (ROC) curve is the standard metric to evaluate the performance of multi-label classification tasks. Here, Area Under the Curve (AUC) values are computed for all the experiments on MIMIC-CXR and OpenI. We compute the multi-class classification accuracy (using sklearn.metrics.accuracy_score) for all the experiments on CIFAR-10.

In this paper, we introduced a novel learning framework (meta-training with the attention-on-label module) for handling data with multiple noisy label sets. The variability that the multiple label sets bring could in fact benefit the learning of a more accurate and robust model. We demonstrated the application of our proposed method on both multi-label and multi-class classification tasks and we believe it will be quite straight-forward to adapt it for other task, e.g., image segmentation. Indeed, the proposed method provides a viable way to handle the challenges of learning large-scale data with algorithm generated labels, where the label is a huge burden, especially for medical image analysis.

The proposed label sampling module convert the hard labels ($0$ or $1$) to soft labels (a value between $0$ and $1$) and it also reduces the overfit towards erroneous labels, which is similar to the idea of label smoothing (Müller et al., 2019). Different to hard label smoothing, we assigned the new label on-the-fly (based on the network feedback) instead of instantly replacing 0 and 1 with $0+\epsilon$ and $1-\epsilon$. Such setting is extremely effective to handle the partial label errors in multi-label classification. Indeed, label smoothing is a form of loss-correction (Lukasik et al., 2020) and we also prove the proposed label weighting is equivalent to directly performing the weighted average of the losses (generated from using different label sets) to form the final loss (see the proof in A.1).