Learn like a Pathologist: Curriculum Learning by Annotator Agreement
for Histopathology Image Classification

Curriculum learning [1] is an elegant idea inspired by human learning that proposes that neural networks should be trained on examples in a specified order based on difficulty (typically easy to hard), as opposed to the random ordering that is currently common in practice. As such, curriculum learning requires both that there exists some range of difficulty among training examples and that we define a method for ranking examples. In most cases, however, it is unclear whether a range of difficulty exists among the examples, and even when a range of difficulty exists, an ideal ranking function is rarely available. In this paper, we try to answer the question: are there tasks with domain-specific properties that are naturally appropriate for addressing these challenges of curriculum learning?

Interest in using deep learning to analyze histopathology images (stained tissues and cells that are typically manually examined under a microscope) has increased in recent years, with neural networks achieving pathologist-level performance on a variety of tasks [2, 3, 4, 5, 6, 7, 8]. In this paper, we propose that histopathology image analysis is a suitable scenario for curriculum learning for two reasons. First, due to the nature of histopathology images, we know that a range of difficulty in examples exists for many tasks. Second, medical image datasets typically have annotations from multiple clinicians—these annotations can be leveraged as a natural proxy for ranking example difficulty. Specifically, our paper makes the following contributions:

1. 1.

   We contend that histopathology image classification is a natural scenario for applying curriculum learning, and we propose a curriculum learning approach that leverages annotator agreement as a proxy for difficulty.

2. 2.

   We evaluate our proposed approach on a colorectal polyp classification dataset, for which a baseline model achieved an AUC of 83.7% and our best single-stage baseline achieved an AUC of 84.6%. When trained with curriculum learning, our model’s AUC improves to 88.2%, outperforming the average pathologist annotator on our test set in terms of Cohen’s $\kappa$ [9].

The rest of our paper is outlined as follows. $\S$2 analyzes the challenges of curriculum learning and presents our intuitions on why histopathology image classification is a suitable context for curriculum learning. $\S$3 describes our task and dataset. $\S$4 presents the main results of our proposed curriculum learning approach. $\S$5 compares the value of annotator agreement with two previously-proposed methods of scoring difficulty. $\S$6 shows how the increase in AUC from curriculum learning translates to improvements in performance relative to pathologist performance. $\S$7 discusses the implications of our work. $\S$8 puts our study in the context of prior work on curriculum learning for medical imaging and concludes our paper.

As such, a diverse set of approaches has been explored in this area of research. These approaches generally first score examples by difficulty and then train models using a schedule based on example difficulty, where easier examples are typically seen first and harder examples are seen later. For instance, Bengio et al.’s original work [1] explored a noising-based curriculum for shape detection and a vocabulary-size based task for language modeling. As popular recent examples, Weinshall et al. [10] use the confidence of a pre-trained classifier as an estimator for difficulty; Korbar et al. [11] use a schedule with self-defined easy and hard examples for learning of audio-visual temporal synchronization; Ganesh and Corso [12] propose to incrementally learn labels instead of learning difficult examples; and various teacher-student frameworks have been proposed in the context of curriculum learning [13, 14].

First, curriculum learning assumes that a range of easy and hard examples exists. Although it could be argued that this is a true statement for any given dataset for at least some definition of easy and hard, the distribution of example difficulties likely varies based on the nature of the task and the dataset. Since the added value of curriculum learning comes from utilizing the varying degrees of difficulty in a task, tasks with a smaller range of example difficulty are less conducive to effective curriculum learning. Empirically, Weinshall et al., present some evidence related to this claim, showing that curriculum learning had a larger improvement compared with regular training when applied to tasks that were more difficult and likely included challenging examples (e.g., distinguishing small mammals in CIFAR-100), than when applied to tasks that were easier and did not have examples that were difficult to classify (e.g., discriminating between 5 well-separated classes in CIFAR-100) [10].

Second, a curriculum learning approach must somehow categorize examples as easy or hard. Prior work has tried to address this challenge in many ways, including trying to discover inherent patterns in the data [10], using hand-picked heuristics [15], and creating custom training progressions [11]. Though sometimes effective, these methods can be difficult to implement, and it is often unclear whether an approach that works on one dataset will also work on another. Indeed, scoring images by difficulty is often the core problem addressed in many curriculum learning papers.

In this paper, we focus on the task of colorectal polyp classification, a challenging and clinically-important task in pathology. As shown in Table 1, our dataset contains 3,152 images in total, each annotated with a binary label of either hyperplastic polyp (HP) or sessile serrated adenoma (SSA).

We propose a curriculum learning framework that leverages annotator agreement to rank images by difficulty. Specifically, we define images with high annotator agreement to be easy and images with low annotator agreement to be hard. For our dataset, which was labeled by seven annotators, we partition our images into four discrete levels of difficulty: very easy (7/7 agreement among annotators), easy (6/7 agreement among annotators), hard (5/7 agreement among annotators), and very hard (4/7 agreement among annotators). An overview schematic of this setup can be seen in Figure 1. For our training schedule, we train our network on progressively harder images in four stages:

• •

  Stage 1: Very easy images only

• •

  Stage 2: Very easy + easy images

• •

  Stage 3: Very easy + easy + hard images

• •

  Stage 4: Very easy + easy + hard + very hard images

At each stage, we make sure to include images from the previous stages to prevent catastrophic forgetting [39]. Though our current model uses four levels of difficulty and four stages of training, our general framework could be used in any scenario where annotator agreement data is available.

For more-robust evaluation, for each model we consider the five highest AUCs on the test set, which are evaluated at every epoch. We report the mean and standard deviation of these values calculated over 20 different random seeds.

As shown in the first block of Table 3, our vanilla baseline that uses all images achieves an AUC of 83.7%. Interestingly, the network trained on only very easy and easy images achieved an overall AUC of 84.6%, an almost 1% improvement over the vanilla baseline. As shown by the performances on the test set when stratified by difficulty, this network trained on very easy and easy images does better on very easy and easy images in the test set while doing worse on hard and very hard images, leading to a higher overall AUC because there are more very easy and easy images than hard and very hard image in the testing set.

Perhaps strikingly, model performance actually decreases in the fourth stage of training that includes very hard examples, as performance on very hard images in the test set increases but performance on other images in the test set decreases. One explanation for this slight dip in performance is that very hard images, which have only 4/7 pathologist agreement, could be too challenging to analyze accurately (even for expert humans), so their features might not be beneficial for training machine learning models either.

In terms of statistical significance, our curriculum learning model at the second, third, and fourth stages outperforms the vanilla-training model with $p\leq 0.001$, based on a two-sample t-test for means.

This section compares various proxies for difficulty in terms of their usefulness for curriculum learning. Whereas our model so far has used annotator agreement as a proxy for example difficulty, prior work has proposed that the output confidence of a machine learning classifier can also be a proxy for example difficulty [42, 10]. First, we perform a sanity check to see whether classifier output confidence correlates with annotator agreement. As we might expect, they do—predicted confidence distributions of a model pre-trained on ImageNet and fine-tuned on our dataset appeared substantially different for different levels of annotator agreement (Figure 8 in the Supplementary Materials).

We conduct a simple ablation study to compare annotator agreement and model output confidence as proxies for image difficulty in curriculum learning. For simplicity, we use a two-step curriculum learning scheme—where training is done in one stage containing only easy images and a following stage containing a mixture of easy and hard images—and use a single-step pacing schedule [42].

We evaluate the following three proxies for difficulty:

1. 1.

   Self-taught scoring, where the classifier with randomly initialized weights is pre-trained on our dataset, and output confidences are used to sort examples by difficulty.

2. 2.

   Transfer learning, where a classifier pre-trained on ImageNet is fine-tuned on our dataset, and output confidences are used to sort examples by difficulty.

3. 3.

   Coarse annotator agreement, a simplified version of our curriculum learning scheme above, where images are divided into two categories of either easy (6/7 or 7/7 agreement) or hard (4/7 or 5/7 agreement), instead of the four categories used in $\S$4.

The self-taught scoring and transfer learning proxies assign each image with a confidence score: images with confidence score greater than a threshold $\tau$ are classified as easy and images with confidence score less than $\tau$ are classified as hard. We choose $\tau$ such that the proportion of easy and hard images was the same as the natural distribution of easy and hard images from coarse annotator agreement. Then, a new classifier is trained in two stages: (1) easy images only, and (2) easy images and hard images combined at various ratios. Our coarse annotator agreement method in this section follows this same training scheme of two stages.

Figure 6 shows the results for the various ratios of hard images we used in the second stage of training. Without making any general claims, we see that on our dataset, annotator agreement appears to be a more useful proxy for difficulty than model output confidence. One possible explanation for this result is that although the transfer learning and self-taught scoring approaches work (marginally, in our case), much of the information provided by the pre-trained classifier is shared with the resulting classifier (i.e., much of the added value of the pre-trained classifier can already be discovered by the resulting classifier), whereas using annotator agreement as a proxy provides new information about difficulty that the model would not have had access to otherwise. Moreover, we find that including a greater proportion of easy images than hard images in the second stage of training is important for preventing catastrophic forgetting, a finding consistent with prior work [11, 10, 42]. Using only hard images in the second stage was especially problematic for the self-taught scoring and transfer learning models, possibly because the images most challenging for these proxy models to learn will also be challenging for the learner model to optimize.

For more context on the significance of the improvement that curriculum learning brings to model performance, in this section, we compare the performance of our models with that of pathologists. Specifically, we frame the predictions of each model as the annotations of an additional pathologist, and we compare these predictions with the annotations of actual pathologists in terms of Cohen’s $\kappa$ [9], a common measure of inter-annotator agreement.

For our models, we select the best-performing curriculum learning model (Stage 3) and compare it with the vanilla-baseline model that was trained on all images in a single stage. As our models output a continuous distribution of probabilities for HP and SSA, we evaluate each model at several different confidence thresholds (a lower threshold results in higher recall, whereas a higher threshold results in higher precision). For average pathologist performance, we compute the Cohen’s $\kappa$ between all pairs of pathologists, and for each of the seven pathologists we show the mean Cohen’s $\kappa$ of all six pairs involving that pathologist.

Figure 7 shows these results comparing our models with both individual pathologists and the average of all pathologists. We see that there is a wide range of Cohen’s $\kappa$ for individual pathologists—the mean of our pathologists’ Cohen’s $\kappa$ scores was 0.450, which is in the moderate range of 0.41-0.60 [43] (a similar study found a Cohen’s $\kappa$ of 0.380 found among four pathologists [35]). Our curriculum learning model (AUC = 88.2%) outperforms the pathologist average for multiple thresholds and the baseline model (AUC = 83.7%) for all thresholds. In particular, adding a curriculum schedule increases performance from the baseline’s maximum $\kappa$ = $0.384$ to the curriculum learning model’s maximum $\kappa$ = $0.473$, the difference needed to outperform the pathologists’ average ($\kappa$ = $0.450$).

Some prior work has demonstrated that what makes an image difficult for neural networks to classify might not always match what makes it difficult for human annotators, an observation that recent work on adversarial examples takes advantage of [44]. Our study explores a converse idea for histopathology image classification, finding that machine notions of difficulty do correlate with human annotator agreement and contending that annotator agreement can be a useful proxy for facilitating curriculum learning.

Certainly, not every scenario in medical imaging is ideal for applying curriculum learning. For instance, both pathologists and deep learning models have achieved high performance in distinguishing high-grade lung cancers and normal tissue, which is considered a relatively easy task that is less likely to exhibit a range of difficulty among examples. Another negative example could be detecting bone fractures: since bone fractures are typically caused by instantaneous traumatic events, there is no progression of disease development, so we are less certain that a range of example difficulty exists. Potential scenarios conducive to curriculum learning by annotator agreement should ideally involve a progression of disease development and be challenging problems where even specialized pathologists might disagree. In particular, we believe that cancer datasets can benefit from curriculum learning because of the inherent natural progression of cancers (i.e., cancer develops over time and is not sudden). This could include tasks such as distinguishing among subtypes of lung adenocarcinoma or assessing tumor proliferation in breast cancer tissue samples.

Furthermore, our dataset contained annotations from seven pathologist annotators, allowing us to categorize images into four discrete levels of difficulty. For medical image datasets that have fewer annotators and therefore fewer categorized levels of annotation difficulty, we suspect that the benefits of our approach could be slightly diminished. For example, our coarse annotator agreement method in $\S$5, which only used two levels of annotator agreement, achieved a smaller performance improvement (1.5%) than our four-level annotator agreement method (4.5%). Moreover, our dataset size is modest in the medical imaging domain, and so whether these curriculum methods work for other dataset sizes is still relatively unexplored. We believe, however, that our curriculum learning methodology might still be worth exploring for these datasets with fewer annotators, as even small improvements in performance are important given the high cost of data annotation and the importance of accuracy toward patient outcomes.

Although we argue that histopathology imaging is a promising context for curriculum learning, we are not the first to explore curriculum learning in the medical imaging domains. Oksuz et al. [45] used curriculum learning for artifact detection by first training on heavily-corrupted images and then introducing less-corrupted images, thereby improving performance on borderline cases. Maicas et al. [46] used a meta-training approach called teacher-student curriculum learning to improve breast screening classification on a weakly-labeled dataset. Moreover, Jesson et al. [47] used adaptive curriculum sampling to better detect lung nodules in extreme class-imbalance scenarios, and Oksuz et al. [45] applied a curriculum based on disease severity levels in radiology reports (e.g., mild, moderate, severe).

These prior studies have demonstrated empirical evaluation of curriculum learning and helped inspire our work, but their methods tend to be complicated and use inductive biases specific to certain datasets. For instance, the approach of Oksuz et al. [45] only applies to artifact detection, the approach of Jesson et al. [47] is specific to segmentation tasks, and disease-severity information from radiology reports used in Oksuz et al. [45] might not exist for many datasets and can be challenging to parse. Our method, on the other hand, is easy to implement, presents no modifications to network architecture, and only requires annotator agreement data, which is often readily available.

Based on a thoughtful analysis of the assumptions in curriculum learning, we have presented a simple yet effective curriculum learning framework which leverages easily-obtained annotator agreement data. In histopathology image analysis, where data collection and annotation can be especially costly, it is important to combine the natural properties of classification tasks with the most-appropriate inductive biases. We aim to have provided a well-motivated argument for more intentional application of curriculum learning to readers from both computer vision and medical imaging analysis backgrounds.

Figure 8 on the next page shows the correlation between difficulty as perceived by a pre-trained model and pathologist annotators. Table 4 on the next page lists examples of medical image classification tasks from prior work where we believe annotator agreement data to be accessible.