HistoKT: Cross Knowledge Transfer in Computational Pathology

Currently in the United States, there are a reported $3.94$ pathologists per $100,000$ people. In Canada, this number rises slightly to $4.81$ pathologists per $100,000$ people [1]. This severe scarcity of pathologists, combined with a rigorous set of duties that involves patient care and extraneous specimen diagnoses, results in decreased diagnosis quality and diminished patient experience [1]. To alleviate these burdens, the computational pathology (CPath) field has created numerous computer-aided diagnosis (CAD) tools to assist pathologist diagnoses [2]. These CAD tools utilize computer vision techniques and neural network architectures to solve a plethora of tasks, including classification, segmentation, and localization [2].

However, challenges in implementing CAD systems arise in part due to limitations in pathology datasets. Namely, despite the presence of large pathology datasets, the lack of proper annotations or labels hinders development of supervised neural networks [2]. Additionally, different standards for staining histology slides and varying optical configurations introduce further complications when creating CAD systems [3, 4, 5]. Combined, these factors result in a valuable data landscape comprised of sparse and non-comprehensive datasets.

Transfer learning is widely used in machine learning to compensate for the absence of comprehensive annotated datasets. Using transfer learning, knowledge gained from one source domain can be applied to problems on another target domain. In CPath, transfer learning using either natural image datasets or other pathology datasets enables networks to generalize to specific target domains where labeled data is scarce [6, 7, 8, 9].

The merits of transfer learning in CPath are clear: models achieve higher metrics on a target domain when pretrained on a relevant source domain dataset [6, 7, 8, 9]. These benefits are explored in previous works through model-centric approaches. While these approaches are useful in exploring quality in various model architectures, they ignore key issues introduced when these models are applied to other datasets. Due to a lack of standardized data preparation when transferring knowledge between datasets, models trained on two distinct datasets are likely to learn at two distinct biological scales. Previous work has shown that models trained from a dataset gathered by one pathology laboratory can underperform when applied to images gathered by a separate laboratory [5]. Furthermore, dataset choice for the source domain is not well explored, with little known about what makes a “good” dataset. These issues can only be resolved through a data-centric approach towards exploring the interactions of source domain data with target domain data under transfer learning.

In this paper we introduce the following contributions: i) we propose a standardized platform to aggregate learned histopathological knowledge, including an image standardization workflow, training, and a tuning pipeline ; ii) we examine the potential for aggregation of learned knowledge between multiple pathological datasets. Using cross transfer over nine classification task datasets, we evaluate both the quantity and quality of transferable information between datasets; iii) we propose weight distillation: a method for combining learned information from encoders trained on separate datasets. iv) we assess the utility of large natural image domains (ImageNet) as a source domain with two stage transfer learning; v) we visualize the transferred knowledge using t-SNE plots and Grad-CAM images.

We introduce our pipeline for knowledge transfer, which includes dataset preprocessing, model training, and evaluation. In our evaluation, we considered the databases summarized in Tab. 1. The overall HistoKT workflow is summarized in Fig. 1.

Single Stage Transfer Results. Tab. 2 shows the results with a single stage transfer learning process. The models with the highest top-1 test accuracies, shown on the main diagonal, are used to deep tune on the applied dataset. If a deep tuned model outperforms the baseline, the result is highlighted green, and if it underperforms compared to the baseline, the result is highlighted in red.

From the results, noticeable improvement from the baseline is seen when large datasets are used as a source domain: ADP, CRC, OS, and BCSS all display the ability to transfer knowledge to other, usually smaller, datasets. Training on smaller datasets like GlaS or AJ-Lymph consistently decreased performance compared to the baseline. Of the large source domains, ADP performs poorly in tasks focused on cancer detection, such as MHIST and PCam, likely due to these task being out of domain. Note that ADP is focused only on healthy tissue, while CRC, OS, and BCSS all have diseased classes. These results show that proper choice of a source domain can affect performance, and consideration of what classes in the source domain are shared with the target domain is vital.

Two Stage Transfer Results. Tab. 3 shows the top-1 test accuracy of all cross transferred datasets with a two-stage deep tuning process. Pretrained ImageNet weights are deep tuned using each dataset in the trained column, shown along the main diagonal, and these models are then deep tuned again with the applied dataset.

ImageNet pretraining improves overall performance for most datasets, and produces a higher peak performance in all datasets except AJ-Lymph, which is still within margin of error. Interestingly, datasets that negatively impacted most results in single stage transfer learning, such as GlaS, BACH, and AJ-Lymph, had many more positive interactions. We posit that this is due to ImageNet pretraining providing necessary low level features that were hard for models to learn from random initialization, due to the small size of the source domain dataset. Datasets that are multi-class and are initially hard to learn (low baseline top-1 accuracy) appear to benefit most from a transfer learning process, as shown in datasets such as ADP and AJ-Lymph.

The above results describe a preliminary analysis of how much knowledge is transferable between histopathological datasets. An increased accuracy could be obtained with an experimental search of the optimal neural architecture and hyperparameters for each case. However, such analysis would be outside the scope of the paper.

t-SNE and Grad-CAM Analysis. We chose to show t-SNE results to visualize changes to latent space representation, along with Grad-CAM to show an intuitive view on how a deep-tuned model classifies images. In Fig. 2, t-SNE plots for training and test sets for the BACH dataset are shown. Deep-tuning on the best performing dataset, CRC, we see that the representation for the test set becomes more disentangled for both single stage and two stage ImageNet training procedures.

Using Grad-CAM, visualizations of the baseline model and transfer learning models are shown in Fig. 3. During generation of Grad-CAM visualizations, both augmentation smoothing and eigen smoothing were used. All activations shown are of the model output prediction, which matched with ground truth for all classes. According to expert pathologists, the ADP source domain model initialized with ImageNet weights and tuned on BACH had the best correlating activation heat map with ground truth for Benign, Invasive, and Normal classes. In contrast, the BACH model initialized with ImageNet weights (baseline model) had poor heatmap correlation with ground truth. However, both the ADP source domain model and BACH baseline model yielded the same, correct predictions for Benign, Invasive, and Normal classes.

Weight Distillation. Fig. 4 displays top-1 accuracies of top performing models evaluated on the BACH dataset. All single dataset results, other than the baseline, are first trained on the specified dataset, and deep tuned on the BACH dataset. For the models created by weight distillation, we first apply our weight distillation workflow to combine two or more baseline weights (no tuning), and then deep tune the resulting combined model on BACH. T-SNE and Grad-CAM visualizations for weight distilled models can be found in the supplementary material.

Here, all models outperformed the BACH baseline, and all weight distilled models perform better than or equal to the one stage deep-tuning models, other than ADP + CRC + OS. We posit that this effect is due to different CPath datasets offering knowledge from different domains, shown through our Grad-CAM analysis where models trained on different source domains learned different approaches to the same task. This enables our weight distillation model to outperform even the highest performing ImageNet tuned model, demonstrating that CPath datasets hold valuable domain specific knowledge that cannot be seen in natural image datasets.

In this work, we proposed and tested a cross domain knowledge transfer pipeline consisting of dataset standardization, data augmentation, and training procedures over nine histopathological datasets. To assess transferred knowledge, we conducted experiments comparing source domain viability for each of the nine datasets and two stage transfer viability using ImageNet pretrained weights. To demonstrate the validity of our transfer learning framework and to visualize the learned knowledge from one dataset to another, we use t-SNE and Grad-CAM to show the change in latent space representation and class activations, respectively. Additionally, we apply weight distillation to top performing models to aggregate knowledge across datasets. We find that knowledge is transferred between histopathological datasets, and that hard to learn, multi-class datasets benefit most from transfer learning. Datasets that share a common organ class or common tasks tend to also share knowledge more effectively, especially when the constraint of learning low level features, governed by dataset size, is removed through ImageNet pretraining. These effects are also displayed through the t-SNE and Grad-CAM analysis, with more disentangled latent representations and more meaningful class activations, respectively. Weight distillation harnesses the different learned approaches by models trained on different source domains, allowing combined models to reach higher than ImageNet pretraining accuracies, with much less computational cost compared to training on ImageNet. We present these finding in an effort to push for a more data-driven approach to transfer learning in CPath, and to create a future where CPath knowledge can be shared between any number of datasets.