Comparative Analysis of Diffusion Generative Models in Computational Pathology

Denoising Diffusion Probabilistic Models (DDPM) [18] follow the same training and sampling methods explained above, with both the forward and reverse processes occurring in the pixel space. In the forward process, Gaussian noise is added to the image $x_{0}$ at each timestep $t$ according to:

where $\beta_{t}$ is the variance schedule. The reverse process aims to denoise the image step-by-step using a learned model $p_{\theta}$, represented as:

The objective function to train the model is:

Latent Diffusion Models (LDM) [32] operate in a lower-dimensional latent space, which is learned by an autoencoder. The training and sampling stages follow the same principles as the general diffusion models described above, but with processes occurring in the latent space. An extra layer is added to encode images into this latent space before the diffusion processes begin. The forward process is defined as:

where $\mathbf{z}_{t}$ represents the latent variables. The reverse process denoises the latent variables step-by-step:

LDMs also incorporate cross-attention mechanisms within the architecture, enhancing conditional image synthesis. After the latent space processing, the autoencoder transforms the latent variables back into images. The objective function for training remains similar:

Recent advancements in DPMs have introduced class-conditional generation, where additional class-related information is incorporated to guide the generation process. Our findings revealed that class-conditional generation significantly enhances the fidelity of the generated images to specific class characteristics. As the guidance scale increased, the generated images exhibited more precise and accurate representations of the target classes.

Classifier-Free Guidance (CFG) [33] is a technique that enables the generation of high-quality samples without relying on a classifier, addressing the limitations associated with classifier guidance. CFG modifies the score function in a way that emulates the effects of classifier guidance, but without using an explicit classifier. The approach involves training an unconditional denoising diffusion model alongside the conditional model, using a single neural network to parameterize both. The sampling process utilizes a combination of the conditional and unconditional score estimates, allowing for effective guidance without a classifier. This results in the production of high-quality synthetic images that are both varied and representative of the original dataset, enhancing the model’s performance in generating realistic images.

Here, $\epsilon_{\theta}(x_{t},c)$ is conditional model and $\epsilon_{\theta}(x_{t})$ is unconditional model. $w$ is used as a guidance scale.

To explore different sampling methods and their effects on image generation and model performance, we incorporated two additional techniques: DDIM and Epsilon Scaling. DDIM accelerates image generation by introducing a non-Markovian process that redefines the diffusion process, utilizing a subset sampling strategy for faster sampling without compromising model performance. The reverse diffusion process in DDIM is defined as:

where the variance $\sigma_{t}$ is given by:

In DDIM, setting $\eta=0$ eliminates noise, making it equivalent to DDPM, while $\eta=1$ maintains the standard diffusion model, allowing interpolation between DDIM and DDPM. To address exposure bias, we employed Epsilon Scaling [34], which scales the epsilon value using a linear function, ensuring consistency between training and sampling, thereby reducing bias and improving sample quality. The epsilon value is scaled as:

where $\lambda_{t}=\lambda_{t}k+b$. The scaled $\epsilon_{\theta}$ is then used in the DDPM equation:

KGH (Kingston General Hospital) datasets is made of 1037 WSI of healthy colon and four types of colonic polyps. The pathological slides are annotated on the Region of Interest (ROI) level. The WSIs are rectangular and of different sizes but their size is around 60k $\times$ 100k pixels and they take approximately 1 GB of memory. Because of the size of the WSIs and our computational resouces, we cannot process the slides as-is. Therefore, we have to extract some tiles of these images, called patches. To extract patches, we have to precise the FOV of the patches. We have used FOV as 224 from magnification 20X. We resized these patches to 128X128.
KGH datasets presents healthy colon tissue and four different types of colon polyps: Tubular Adenoma (TA), Sessile Serrated Lesions (SSL), Hyperplastic Polyps (HP), Tubulovillous Adenoma (TVA). HP is a non-neoplastic polyp; whereas the other 3 types SSA, TA and TVA have clonal abnormality at the molecular levels, and hence are considered neoplastic. All the 3 polyps have the potential to progress to adenocarcinoma.

We can visualize patches from these classes in Figure 3. Despite some visual differentiation in certain regions among these classes, distinguishing them can be challenging without a pathologist’s expertise. Understanding these classes is crucial for making accurate analogies with our generated images. The visual distinctions, although subtle, provide necessary context for training our models. To manage the high-dimensional nature of these pathology images, we extract smaller sections referred to as patches. To extract these patches, we specify the FOV. We also eliminated noisy images from the original set.

Patches were extracted from the annotated WSIs at specified FOVs. For our experiments, we maintained a consistent patch resolution of 1.75. This approach allowed us to observe how different FOVs provide various perspectives of the same tissue at the same magnification. Specifically, we used FOVs of 224 and 336 for creating the PKGH dataset. These FOVs were selected to ensure that the patches offered a comprehensive view of the cellular and tissue structures necessary for training generative models. The chosen FOVs are not excessively large to avoid having a limited number of patches available for training and classification.

We used a UNet model based on [21] for our experiments. We converted 128 $\times$ 128 images to noise, which were then fed into the UNet network. The network predicted the amount of noise added to these images and outputted the noise prediction. We employed a loss function that compares the predicted noise with the actual noise from the stochastic process. In Diffusion Models, Gaussian noise is iteratively added to the original image according to a variance schedule with a large total number of steps (T = 1000). The UNet model also incorporates timestep and class embeddings to learn class-specific features.

We implemented the LDM architecture from rombach2022high [32], which consists of the Variational Autoencoder (VAE), the U-Net denoiser, and added the class embedding. We used an autoencoder based on vector quantization, VQ-autoencoder, referred to as VQF4-DM [35].

A classic approach for generative model’s evaluations is to compare the log-likelihoods of models. This approach, however, has several shortcomings. A model can achieve high likelihood, but low image quality, and conversely, low likelihood and high image quality. The two most common GAN [12] evaluation measures are Inception Score (IS) [36] and Fréchet Inception Distance (FID) [37].

Inception Score (IS) rely on a pre-existing classifier (InceptionNet) [38] trained on ImageNet. IS computes the KL divergence between the conditional class distribution and the marginal class distribution over the generated data. IS does not capture intra-class diversity, is insensitive to the prior distribution over labels (hence is biased towards ImageNet dataset and Inception model. Therfore, we would not be using it for our dataset as it has completely different data than Imagenet. However, we would be using FID and KID.

FID [37] calculates the Wasserstein-2 (a.k.a Fréchet) distance between multivariate Gaussians fitted to the embedding space of the Inception-v3 network of generated and real images. The Kernel Inception Distance (KID) [39] aims to improve on FID by relaxing the Gaussian assumption. KID measures the squared Maximum Mean Discrepancy (MMD) between the Inception representations of the real and generated samples using a polynomial kernel. This is a non-parametric test so it does not have the strict Gaussian assumption, only assuming that the kernel is a good similarity measure.

We further use deep learning based classifiers approach to compare original vs generated data to show how synthetic data is actually applicable in real life. This is separately explained in Section.

The analysis focuses on comparing the performance of DDPM and LDM across two datasets: PKGH_224 and PKGH_336.

The Table 3 summarizes the results of all methods.

In our study, the PKGH_224 dataset revealed that the DDPM model, with DDPM sampling, achieved an FID score of 19.08 and a KID score of 0.0134, indicating a strong balance of image quality and diversity. In comparison, the LDM with DDPM sampling produced an FID score of 24.43 and a KID score of 0.0185, showing that while LDM’s latent space representation is effective, DDPM slightly outperformed it in this case. When using DDIM sampling, DDPM scored 22.66 (FID) and 0.0154 (KID), while LDM scored 25.56 (FID) and 0.0161 (KID), further demonstrating DDPM’s superior performance across sampling methods.

For the larger PKGH_336 dataset, DDPM again excelled with an FID of 18.45 and a KID of 0.0129 using DDPM sampling, maintaining consistent high performance across different dataset sizes. LDM with DDPM sampling scored 23.10 (FID) and 0.0160 (KID), showing good results but not surpassing DDPM. The DDIM sampling method reinforced this trend, with DDPM scoring 21.34 (FID) and 0.0159 (KID), while LDM scored 26.41 (FID) and 0.0199 (KID).

These results highlight that DDPM consistently outperforms LDM in generating detailed and realistic images, especially in smaller datasets like PKGH_224. This suggests that DDPM is more effective in controlling pixel space and producing high-quality images, making it a robust choice for synthetic pathology image generation across different dataset sizes.

Figure 6 shows a visual comparison of the results of the DDPM and Epsilon sampling methods applied to histopathology images. The upper row illustrates output from DDPM, while the lower row showcases Epsilon sampling results. Both methods can capture larger tissue structure from pathology images, but the DDPM images display more consistent structural details across different samples. In contrast, Epsilon sampling produces slightly varied textures, which could indicate greater flexibility in capturing diverse tissue characteristics. These observations suggest that while both methods are effective, DDPM might be more reliable for producing uniformly detailed images, whereas Epsilon sampling can capture a broader range of textural variations. This could be a reason for the higher FID score of the epsilon scaling method.

The results indicate that the original patch size with a FOV of 224 and a patch resolution of 1.75 mpp achieved the lowest FID score of 19.08, reflecting the highest fidelity in generated images, which is expected as the images were trained on this model. However, examining the FID score for other patch sizes, especially those not used during training, provides insights into the model’s ability to generate detailed images from different patch sizes.

This model is trained with a patch size of 128, and we experimented with generating images using different patch sizes, both smaller and larger. The table highlights the variation in image fidelity across different patch sizes and FOVs in generated pathology images. Smaller patch sizes, such as 64x64 and 96x96, exhibit higher FID scores of 161.01 and 33.71 respectively, indicating lower image fidelity. As the patch size increases to 160x160 and 192x192, the FID scores improve to 25.71 and 38.41 respectively, suggesting better image quality but with some variability. The 224x224 patch size, while larger, shows a slight decrease in fidelity with an FID score of 41.37. The visual representations in Figure 8 demonstrate that larger patch sizes generally capture more detailed and high-level cellular structures across various classes, enhancing the potential utility of synthetic data in diagnostic workflows. The observed high FID score for the 64x64 patch size can be attributed to the fact that images generated at this resolution are often blurred and lack sufficient detail as shown in Fig 9. At such a low resolution, the model struggles to accurately generate fine patterns and intricate structures that are critical in pathology images. This inability to capture detailed cellular patterns results in poor-quality images, leading to the significantly higher FID score. In contrast, the 128x128 patch size allows the model to generate clearer images with more recognizable features and patterns, which contributes to a lower FID score. The blurriness for Patch size 64x64 reflects the model’s limitation in generating complex, high-fidelity images at this lower resolution.

Interestingly, even without training on certain pixel levels, the model can still perform well and generate quality images, demonstrating its robustness and versatility. This suggests that the generative model has the capacity to generalize beyond its training data to some extent, making it a valuable tool in scenarios where training data are scarce or diverse. This experiment highlights the model’s adaptability and versatility.

The table below summarizes the classification performance on the KGH dataset under different scenarios, providing a clear understanding of how the use of generated data impacts model accuracy.

The results reveal notable differences in model accuracy between real and generated datasets for both PKGH_224 and PKGH_336. When trained on the real PKGH_224 dataset, the ResNet-50 model achieved an accuracy of 89.95%. However, training solely on the generated PKGH_224 dataset resulted in a slight drop in accuracy to 88.62%. Interestingly, augmenting the real dataset with generated data led to an improvement in test accuracy to 90.75%, indicating the added value of synthetic data in enhancing model performance.

For the PKGH_336 dataset, the model reached an accuracy of 94.06% on the real dataset, and 92.44% when trained on the generated dataset alone. However, combining real and generated data slightly decreased the accuracy to 90.76%. Despite this, the PKGH_336 dataset consistently demonstrated high performance, underscoring the robustness of the data, especially when leveraging the generated samples.

The variations in accuracy between PKGH_224 and PKGH_336 can be attributed to differences in the Field of View (FOV) of the datasets. The larger FOV in PKGH_336 captures more contextual information and intricate details, which likely enhances the model’s ability to learn and generalize from the data, contributing to its superior performance.

These findings highlight the potential benefits of incorporating synthetic data to boost model performance, particularly when real data is scarce or challenging to obtain. The overall accuracy scores affirm the effectiveness of DGM-generated synthetic samples in capturing essential features and improving the classification task.