What Appears Appealing May Not be Significant! - A Clinical Perspective of Diffusion Models

In recent years, creating a pictorial view of what we imagine has become possible with the advent of diffusion models. This capability is coupled with text-based control, allowing us to type in a text prompt that describes our desired image output. These models can be fine-tuned to downstream tasks, expanding their viability to different applications. Generally, in most such applications, the visual outcomes during each fine-tuning iteration can be used to correlate them with improvement or overfitting. However, does this concept extend to medical images? Does solely relying on appealing visuals provide enough evidence for the clinical significance of generated medical images? These are some crucial questions that need attention for critical medical analysis tasks. ^††Accepted in WiCV (CVPR 2024) under poster category.

In this work, we investigate the feasibility of synthetic medical data in both qualitative terms and clinical relevance to provide insights into the above questions. This study aims to generate colonoscopy images featuring polyps, which are precursors to colorectal cancer, the third most common malignancy. These polyps can be adenomatous (AD), which means malignancy potential, or non-adenomatous (Non-AD), which means benign. Additionally, we emphasized high-quality image generation. The synthetic images are obtained using a stable diffusion model [7] and are further examined for pathological relevance by performing a binary classification (AD/Non-AD). The existing literature lacks such pathology-based studies and mainly focuses on polyp generation using binary masks [4, 6]. Our contributions are summarized below:

• •

  We investigate the correlation between diffusion models’ qualitative and quantitative results to determine their alignment or disparity in clinical settings.

• •

  We explore the strategies that can be adopted to conclude the clinical relevance of synthetically generated images.

The overview of the proposed framework is shown in Fig. 1. Our approach adopts a stable diffusion based model with a denoising U-Net module. This module undergoes a two-step training process. Initially, the model learns from a large number of polyp and non-polyp colonoscopy images with additional quality-specific text prompts. This step allows it to learn polyp-focused patterns that could assist in further training it on a small-scale polyp classification dataset. Subsequently, we freeze some layers of the U-Net encoder and train it using pathology (AD/Non-AD) based high-quality images. To enhance the model’s performance, we also included a few samples of the previous dataset and some high-quality images for which pathological annotations are unavailable. The generated images of AD and Non-AD classes are further evaluated using three strategies to assess their clinical relevance. These strategies include plotting t-SNE embeddings, using the Kernel Inception Distance (KID) metric, and augmenting the real images with generated images for a binary classification. Additionally, our approach incorporates iteration-wise analysis to ascertain whether a relation between qualitative outcomes and their clinical viability could be established.

The experiments are conducted using three publicly available datasets, namely, SUN Database [5] ($1,09,554$ non-polyp and $49,136$ polyp frames), CVC-ClinicHDSegment [10, 2, 1] ($164$ images), and CVC-ClinicHDClassif [3, 8] (train set: $536$ adenoma, $252$ non-adenoma, test set: $77$ adenoma, $36$ non-adenoma). We used the official validation set of CVC-ClinicHDClassif for testing due to the unavailability of the annotations in the official test set. The initial training of our diffusion model is conducted using the SUN Database, which is fine-tuned using the other two datasets. The pathological and clinical relevance evaluation is performed using CVC-ClinicHDClassif. The implementation uses the PyTorch framework, and execution is performed on NVIDIA A100 and NVIDIA Titan-Xp GPUs.

First, we conducted an iteration-wise analysis using t-SNE plots and the KID metric. Simultaneously, we tried to relate them with qualitative outcomes. It can be observed from Fig. 2, Fig. 3, and Table 2 that despite visually stunning outcomes in the initial iterations, the corresponding t-SNE plots and KID metrics signify clinical irrelevance of the synthetic images. This observation contradicts general images, where qualitative results often show correlations with quantitative outcomes. Moreover, it is difficult to visually conclude the pathology of such anomalies without clinical expertise. Furthermore, the best outcomes of t-SNE plots and KID metrics, i.e., iteration 8k, could not outperform real images when used for augmentation (see Table 1). The reason for underperformance can be inferred from Table 2, where iteration 8k is selected based on comparing synthetic AD/Non-AD with their real counterparts. However, it is important to consider the comparison between synthetic AD/Non-AD and real Non-AD/AD, respectively. This comparison signifies how distinct one class is from another, which is unfavorable in the 8k iteration.

In this work, we proposed a diffusion-based approach to generate polyp images with different pathologies and investigated different strategies to assess their clinical relevance. We demonstrated that, unlike general images, it is difficult to establish a direct relation between qualitative outcomes and their clinical significance without some additional investigations. Our study provides pathways for future research using generative techniques with clinical images.