Characterizing Continual Learning Scenarios for Tumor Classification in Histopathology Images

CRC: We characterized CL model performance with a colorectal cancer (CRC) dataset from [13], whose training and test set are composed of stain-normalized [18] 224x224 patches at 20x from hematoxylin and eosin (H&E) stained whole-slide images (136 patients) with tile-wise class labels for 9 tissue types - tumor, stroma, normal, lymphocyte-rich, mucin, muscle, adipose, debris, background. To establish a class-balanced dataset, we randomly selected 8700 images from each class for training (7000 train; 2700 validation) and the entire test set (7150 images) for testing. All the selected images were used for stain augmentation (See Section 3.2). PatchCam: To characterize the effectiveness of CL methods on learning from multiple tissue types, PatchCam from [5] was used for breast cancer classification, which comprises 96x96 patches with class labels of “normal” and “tumor” from 400 breast cancer H&E whole-slide images at 20x resolution. We stain-normalized [18] the patches to be consist with colorectal cancer dataset.

Simulating Domain Shifted Data Streams To systematically evaluate the CL algorithms in DP settings, we need a sequence of data to simulate CL scenarios.

To the best of our knowledge, no such benchmark has been proposed for DP-based analysis. We thus generated an augmented CRC dataset comprised of multiple subsets, each with distinct colors and stain intensities, simulating the commonly observed variations of stain appearance (i.e. domain shifts) from multiple data sources. Except for the images provided by CRC (Domain 1), we simulated 4 additional data sources (Domain 2-5) (Fig. 1 Left). For each data split, We first randomly divided images in each of the 9 classes into 5 splits, kept 1 split unchanged for Domain 1 and performed stain unmixing [21] for the remaining splits. We further applied the following augmentation to remix the stain-unmixed intensity images into augmented H&E and refer to this augmented dataset as augmented CRC. Pseudocode for creating this augmented dataset is at https://github.com/kaustabanv/miccai2022-cl-in-dp.

• •

  Domain 2 (increased stain intensity, simulating, for example, concentration increase of the eosin and/or hemotoxylin solutions, each with a different extent of change): eosin intensity was randomly increased with a scaling factor sampled from [1.75, 2.75], hematoxylin from [1.5, 2.0].

• •

  Domain 3 (decreased eosin stain intensity, simulating, for example, slides prepared from many years ago with fading stain): eosin intensity scaling factor sampled from [0.4, 2.75].

• •

  Domain 4 (change of hue, simulating, for example, change of reagent manufacturer, scanner or stainer): eosin hue changed by a scaling factor sampled from [-0.05, -0.03], hematoxylin from [0.05,0.08].

• •

  Domain 5 (change of hue and saturation, simulating, for example, change of reagent manufacturer, scanner or stainer): eosin hue changed by [0.03, 0.05]; saturation increased for eosin by [1.2, 1.4] and hematoxylin by [1.1, 1.3].

Qualitative Assessment of Augmented CRC We presented four randomly selected images from each domain to two pathologists. Both of them confirmed that images of Domain 1-4 look realistic and one pathologist thought the Domain 5 is slightly deviating from majority of DP images from our organization.
CL Setup with Augmented CRC For Domain-IL, the 5 domains were used as individual batches of data with domain shifts, where each domain contains examples from all 9 classes (e.g. for training, 8700/5 = 1740 examples per class per domain). In Class-IL, each data batch contains all examples from the new classes and thus the same number of examples from each domain for each class. In Data-IL, each data batch was randomly selected and roughly contained the same number of examples from each domain.
CL Methods: Recent proposed CL methods can be largely classified into three categories: replay, regularization and parameter isolation [28]. With replay methods, original images [20], deep representations [27] or model-generated pseudo samples [25] are selected via various heuristics, stored in memory and replayed at later learning stages to overcome forgetting. Regularization-based methods [14, 23, 30] avoid storing examples and instead include a regularization term in the loss function to penalize model updates that could lead to large deviation from an existing model, thus avoiding forgetting of learned knowledge. Parameter isolation methods assign different model parameters to each task. Note that while replay and regularization apply to all scenarios, parameter isolation only applies to Task-IL, because such methods are designed to either pre-allocate different parts of the network for different tasks [24], or freezing previous model parameters [22]: both require multi-headed designs and use task IDs to select a task-specific head during inference. Here we evaluate the following CL methods.

• •

  EWC/Online EWC: Regularization-based methods that use model parameters as prior [14] to learn new data. The loss function includes a regularization term, which penalizes large changes to network weights that are important for previous tasks (knowledge from previous data) and is controlled by a weighting hyperparameter. The magnitude of penalty for each model weight is proportional to its importance for previous tasks. In EWC, the number of quadratic penalty terms grows linearly with the number of tasks. Online EWC [23] uses a single quadratic penalty term on parameters whose strength is determined by a running sum of parameter importance from previous tasks. Online EWC includes an additional hyperparameter to determine the magnitude of decay of each previous task’s contribution.

• •

  LwF: Data-focused regularization method [16] which distills knowledge from a previous model to an updated model trained on new data. The loss function has an additional distillation loss for replayed data. Each input is replayed with a soft target obtained using a copy of the model stored after finishing training on the most recent task.

• •

  iCaRL: Replay-based method using stored original images from previous data streams [20]. This method stores a subset of most representative examples per class in memory (GPU, CPU, disk, etc), which are chosen according to approximate class means in the learned feature space. The memory size can be determined by grid search and/or availability of computing resources.

• •

  A-GEM: In this online constrained replay-based method [8], model updates are constrained to prevent forgetting by projecting the estimated gradient on the direction determined by randomly selected samples from a replay buffer. Buffer size and sampling size from the buffer are both tunable hyperparameters. In the literature, A-GEM was tested only in Class-IL and Task-IL scenarios as reported in [8].

• •

  CoPE: An online rehearsal based approach [10], which enables rapidly evolving prototypes with balanced memory and a loss function that updates the representations or pseudo prototypes (i.e. from model-generated pseudo samples), thus alleviating catastrophic forgetting. We tested CoPE in Data-IL, Domain-IL and Class-IL scenarios.

Model Evaluation Strategies We refer to learning from a batch of data for CL as an experience. Within each experience, we split data into train, validation, test (see Section 3.1; e.g. images from the training set of augmented CRC were kept as training). We report (1) accuracy averaged over test sets at the end of training the last experience from multiple repeats (Fig. 1 Right, Table S1) and (2) accuracy over multiple repeats at each experience (Fig. 2-4). (3) While evaluation on test sets from past experiences indicates how well the model retains past knowledge without forgetting (backward transfer), accuracy on current/future test sets reflects new knowledge assimilation (forward transfer) (Fig. S1-S4).
Baselines All the methods were compared against two baselines: 1) joint training (upper bound), where a model was trained on all the data available so far; 2) finetuning (i.e transfer learning; lower bound), where a model was trained sequentially with exposure to only the data from the current experience. For A-GEM and CoPE we also compared the online versions of baselines for fair comparison: the A-GEM baselines were trained in an online setup for 1 epoch and for the CoPE baselines, the experiences were divided into mini experiences (128 samples each) in a data incremental fashion and trained for 1 epoch per mini experience, similar to its set-up in Data-IL, Domain-IL and Class-IL scenarios.

We used a ResNet-18 [11] feature extraction layers followed by a single-layer nearest mean classifier [20], where the feature vectors generated from the feature extractor were used to assign the class label with the nearest prototype vector. Training was conducted with 15 epochs and a batch size of 16 on 1 NVIDIA V100 or A100 GPU for 3-5 repeats. For Task-IL, in the multi-headed designs, each head was trained for a different task. Models were optimized with Stochastic Gradient Descent starting with a learning rate of 0.1 and momentum of 0.9 and a weight decay of 0.00001 applied after epochs 10 and 13. In all scenarios we ran grid search for best hyperparameters (See details at https://github.com/kaustabanv/miccai2022-cl-in-dp).
Comparison of Offline CL Methods on Augmented CRC Overall, the accuracy of joint baseline was comparable across scenarios, while the performance of finetuning varies among scenarios with best performance on Data-IL and worst on Class-IL (Fig. 1 Right and Table S1).

Of the offline CL methods tested, LwF and iCaRL performed much better than finetuning in all scenarios except for Data-IL. Accuracy of EWC/online EWC was comparable to finetuning in all scenarios except for Domain-IL, where online EWC was 6.09% higher than EWC. It should be noted that the computation time for joint baseline was much longer (250 ± 18 minutes) compared to that of for CL methods across scenarios (70 ± 8 minutes). In Data-IL, though CL methods were expected to outperform finetuning, we found that only LwF (accuracy: 91.32%) outperformed finetuning and even 2.87% higher than the upper bound joint training. While iCaRL was the best method for other scenarios, it was 7.52% lower than finetuning. iCaRL was designed and tested for Class-IL in [20] and the replay memory was divided equally to store examples from each class. In our Data-IL setting, the allocated memory for each class was split further for each of the five domains. We hypothesized a bigger memory with more examples for each class per domain could have produced better results. In Domain-IL, iCaRL achieved the best accuracy of 73.41%. An interesting observation is that while other methods show a reduction in performance after learning Domain 2 and 4, the performance of iCaRL continued to improve (Fig. 2). Domain 1 (original stain-normalized images) and Domain 4 (change of hue and saturation) showed minimal forgetting (Fig. S2), while Domains 2 and 3 with eosin intensity changes were much harder for CL, suggesting that intensity changes in practice due to, for example, scanner differences, may lead to larger challenges for CL than hue/saturation changes due to, for example, changes in staining reagent/condition. These 2 domains were the farthest from baseline DP settings in appearance. Note that here the 5 domains were designed to have relatively large domain gaps with non-overlapping augmentation settings and in practice by performing stain normalization during model training or for datasets with smaller domain shifts, the domain gaps would be reduced and thus potentially improving CL performance. In Task-IL, finetuning accuracy was much higher than that of Class-IL, likely due to the presence of task-specific network modules and additional information leveraged (taks IDs) during inference. However, unlike [19, 28, 27] have reported, LwF was the only method that performed better than finetuning (Fig. 1 Right and S3). This could be because in our network design task identities were only used in the output layer, while it was reported to be more effective to embed task identities in hidden layers [10]. Class-IL is the most difficult learning scenario and as expected most methods suffered from severe forgetting of previous knowledge. iCaRL had the best overall accuracy of 72.06% when tested with a random ordering of classes (Fig. S4). This is consistent with [27] that replay methods are more effective in Class-IL than regularization methods. We also studied the impact of class order, class grouping and number of classes with iCaRL. The results (Table S2) were consistent with the hypothesis of curriculum learning that knowledge is better captured if harder tasks follow easier tasks [6].
Comparison of Online CL Methods on Augmented CRC A-GEM outperformed the online joint baseline for A-GEM (Fig. 3 Left) in Task-IL by 6%, but suffered from catastrophic forgetting without task identity. CoPE was worse than finetuning in Data-IL and Class-IL, but was 6% better in Domain-IL (Fig. 3 Right), demonstrating that online CL poses challenges, which calls for more attention in research. Despite lower performance compared to offline methods, knowledge retention (backward transfer) with CoPE was much better in Data-IL and Domain-IL (Fig. S1 and S2). We note that CoPE was sensitive to softmax temperature and further finetuning could help improve forward transfer.

Continually Learning from Multiple Tumor Types We tested a dramatic domain shift in data streams by training a model with the original CRC images in one experience and PatchCam dataset in another experience. We experimented with LwF for continually learning the two datasets in a Domain-IL setting, where each tumor type is considered as one domain. We experimented with ordering of the domains as well as the volume of PatchCam data used in training. As shown in Fig. 4, joint baseline accuracy was around 80%. Interestingly, LwF performed better than finetuning when PatchCam was introduced first during training followed by CRC in the subsequent experience. As expected, including 3 times more data in training improved accuracy by over 11% (Table S2).

We present a systematic study that characterizes the performance of recently proposed CL methods for different scenarios using augmented DP images with domain shifts. We found that though regularization-based methods performed well in Data-IL and Domain-IL, only iCaRL, a rehearsal method, is effective in the most challenging Class-IL. Surprisingly, Task-IL scenarios may not be as easy to learn for DP as for other domains. CL methods were also computationally efficient, taking only about 28% of the runtime as joint training. Though patient data evolve quickly nowadays, FDA has not approved algorithms based on CL [28] and extensive research is needed to establish regulations for safely incorporating CL in clinical settings. Our evaluation approaches and proposed method to generate domain-shifted datasets can potentially serve as the first step towards this goal.