MM-Retinal: Knowledge-Enhanced Foundational Pretraining with Fundus Image-Text Expertise

To construct MM-retinal with high-quality image-text pairs covering CFP, FFA, and OCT modalities, as shown in Fig. 1(a), our designed semi-automatic pipeline of dataset construction contains four steps. For further details, please refer to the supplementary material. A six-person team took four weeks to get MM-Retinal completed.

Step 1: Image-Text Pair Collection: First, image-text pairs are captured from the books. Note that we keep the resolution of each image in the pair no less than 800×800. Afterwards, Adobe API and OCR techniques are used for raw image and text extraction. Step 2: Image-Text Alignment: As mentioned earlier, not all the initially extracted images and texts by the program are very well aligned, as there are some sub-figures correspond to one caption within a screenshot. Thus, we use regular expression matching for sub-figure caption separation to align image-text pairs. Step 3: Modality Classification: Since the fundus images in the books include multiple modalities, we categorize images into CFP, FFA, and OCT using K-means and color histogram analysis, and excluding unusual modalities with scarce samples. Step 4: Text Cleaning and Bilingual Translation: After manually correcting OCR errors and irrelevant texts, we translate the texts to provide bilingual (English and Chinese) versions, aiming to standardize the language and enhance the influence of MM-Retinal since the diagram books we used include both Chinese and English.

Current version of MM-Retinal dataset includes 2,169 CFP cases, 1,947 FFA cases and 233 OCT cases. Each case is provided with an image and texts in both English and Chinese. Due to the small scale of OCT modality, we do not explore it for now and will extend this part in future. Detailed statistical analysis of the CFP and FFA data is provided in the Fig.  1, focusing on aspects of frequently used words, text length, vocabulary size, and comparison with other datasets.

Frequently Used Words: Since MM-Retinal dataset is built based on comprehensive ocular fundus diagram books, it covers a wide range of disease categories, such as macular, retinal vascular, and optic nerve diseases. In Fig.  1(b), we only show a small part of the words frequently appeared in CFP and FFA modalities. More retinal abnormal fundus changes and disease categories can be found in the supplementary material.

Text Length: Fig. 1(c) illustrates the text length distribution of our dataset. About 75% of English texts and 45% of Chinese texts range from 1 to 40 words, while 19% of English texts and 43% of Chinese texts contain between 41 and 80 words. Since our dataset is sourced from ophthalmologists’ diagram books, it features much longer texts compared to FLAIR which simply maps class names of public datasets into fixed texts.

Vocabulary Size: MM-Retinal dataset contains diverse textual descriptions, such as disease diagnosis, lesion characteristics (e.g. color, shape, appearance), clinical manifestations, and post-treatment efficacy information. Fig. 1(d) showcases the average vocabulary size and lexical diversity across different modalities and languages, where average vocabulary size represents the total number of unique words contained in the whole texts within the dataset and lexical diversity refers to the vocabulary size averaged over each text.

Discussion: Compared to public fundus image datasets, our MM-Retinal stands out from four aspects: 1) Multi-modality: MM-Retinal is the pioneering fundus dataset that includes high-quality image-text expertise data for CFP, FFA, and OCT. 2) Data Quality. In contrast to other medical image datasets that often contain low-quality images and captions sourced from websites or academic papers, MM-Retinal provides high-quality images with a resolution over $800\times 800$, accompanied by accurate and pertinent text descriptions. The images closely match clinical data with minimal domain shift. 3) Category Variety. According to the contents of four diagram books, MM-Retinal covers a broad range of categories with over 96 abnormalities and diseases. 4) Text Diversity. MM-Retinal encompasses diverse vocabulary and long texts, containing extensive expert knowledge, and can be explored to enhance data-driven fundus image analysis models.

As shown in Fig.  2, we propose KeepFIT, a knowledge-enhanced foundational model, pretrained on public and MM-Retinal datasets. In this paper, we define public datasets with only category-level labels as unimodal dataset and we follow FLAIR[25] to fill category-level labels into a prompt template to create texts. We utilize ResNet50[11] as image encoder $E_{v}$ and BioClinicalBert[3] as text encoder $E_{t}$, followed by projection heads $P_{v}$ and $P_{t}$ to match image and text feature dimensions $d$. Thus, given a image $x_{i}$ and a text $y_{j}$, the extracted image features $v_{i}$ and text features $t_{j}$ are:

Given that texts detail the associated fundus images, our goal is to maximize similarity between paired image-text and minimize similarity for unpaired ones in a multimodal space. For MM-retinal, which are genuine texts rather than textual prompts, we follow CLIP[22] to implement the contrastive loss as shown in Eq. 2, and the matching labels $G_{v2t}^{m}$ and $G_{t2v}^{m}$ are two $\mid\mathcal{B}\mid\times\mid\mathcal{B}\mid$ identity matrices:

For public datasets that only have prompt texts mapped from category-level labels, we follow FLAIR to calculate the category co-occurrence relationships among samples within a batch and construct a target matrix to encourage the pairs belonging to the same category closer. Thus, the objective function Eq. 2 is converted into:

As our proposed MM-Retinal features high-quality image-text pairs, long text description, rich vocabulary, and comprehensive ocular diseases categories, it encapsulates extensive fundus expert knowledge. Inspired by TipAdapter[31], we propose a lightweight expert knowledge integration method, called Image Similarity-Guided Text Revision along with Mixed Training Strategy, to boost the expert knowledge from our MM-Retinal dataset.

Image Similarity-Guided Text Revision: The descriptions within MM-Retinal are more comprehensive and professional than the simple texts mapped from class names in public datasets. Despite this, the images from both sources share a notable similarity. Hence, we start from identifying visual features in the public datasets that resemble those in MM-Retinal. Visual similarities are used as guidance to extract relevant prior knowledge from MM-Retinal’s text features to refine and enhance the textual prompts of the public datasets.

Specifically, given an input image-text pair $[x_{p},y_{p}]$ from public datasets and $[x_{m},y_{m}]$ from MM-Retinal, the extracted features are $(v_{p},t_{p})$ and $(v_{m},t_{m})$. A multi-head cross-attention[27] is applied to exploit prior expert knowledge as:

Mixed Training Strategy: Since the public datasets has relatively homogeneous text prompts and almost no expert knowledge, which is vastly different from the texts of our dataset, we propose a mixed training strategy to avoid model optimization bias during the training process. To detail, the samples from public datasets and our dataset are in a 1:1 ratio in each batch.

Overall Training Objective: The overall training objective comprises three parts, which are public datasets contrastive loss $\mathcal{L}_{p}$, MM-Retinal contrastive loss $\mathcal{L}_{m}$ and expert knowledge revision loss $\mathcal{L}_{EK}$:

Pre-training Data: 1) flair[25] (CFP) compiles 37 open-access fundus image datasets covering 96 categories with up to 284,660 images. These datasets provide category-level labels for classification. 2) SYNFUNDUS-1M[24] (CFP) is a synthetic dataset with 1 million images for 14 diseases, created by a diffusion model[12] trained on 1.3 million private fundus images. 3) FFA-IR[17] (FFA) provides 10,790 reports along with 1,048,584 images from clinical practice. It includes a schema of 46 categories of lesion and bilingual reports. 4) MM-Retinal (CFP+FFA+OCT) contains over 4.3K high-quality image-text pairs from professional fundus diagram books, covering 96 categories of abnormalities and diseases.

Foundation Model Baselines: 1) MoE[28] (MTL) uses a multi-task approach with mixture-of-experts for fundus, macula, and optic disc images. 2) RETFound[35] (MIM) is a masked autoencoder with Transformers[7], trained for retinal image reconstruction. 3) FLAIR[25] (CLIP) utilizes a CLIP model and brief textual prompts that mapped from category labels for pre-training, but the texts are so limited and brief that fail to fully describe the images.

Evaluation Data and Metrics: For CFP modality, we evaluate on five datasets across finetuning, few-shot, and zero-shot settings, using classification accuracy (ACA)[32] for REFUGE[20], FIVES[13], ODIR200$\times$3[1] and TAOP[2], receiving-operative-curve(AUC) for REFUGE and AMD[9], F1-score for AMD. For FFA, we evaluate on image captioning task using the FFA-IR test split, applying BLUE 1-4, Meter, Rouge, and Cider metrics for consistency with FFA-IR.

Implementation Summary: In CFP and FFA, image and text encoders are initialized from ImageNet-1K[23] and BioClinicalBERT, respectively. Attention mechanisms are trained from scratch, with 512 feature dimensions. Images are adjusted to $512\times 512$ size. All the texts we used are in English version. Text tokens length is set at 256. Evaluation uses five-fold cross-validation averaging. We employ AdamW (lr=1e-4, decay=1e-5) optimizer and cosine scheduler with initial warm-up for the first epoch. Training is conducted on 4 RTX 3090 GPUs with batches of 24.

To compare the foundational performance of different models, we conducted experiments in zero-shot and few-shot scenarios with unseen categories. We adopted 1, 5, 10 shots and adapter tuning scheme in few-shot, including ClipAdapter[10], TipAdapter[31] that adds a few task-specific parameters.

In Tab 1, KeepFIT trained by MM-Retinal and 50% flair achieves competitive performance across the board. 50% flair is obtained by randomly sampling 50% of each dataset in the flair. For example, it outperforms FLAIR trained by flair by 0.268 and KeepFIT trained by flair by 0.061 in average, and even shows superior performance to KeepFIT trained by MM-Retinal and flair with an improvement of 0.012. The results demonstrate several key insights: 1) The best performance comes from using 50% flair with MM-Retinal, which is even higher than using 100% flair with MM-Retinal. This indicates that large datasets may introduce noise and diminish transfer effectiveness since they lack expert knowledge. 2) KeepFIT performs better when trained by MM-Retinal and flair than by synfundus-1M and fliar, underscoring MM-Retinal’s superior expert knowledge over large datasets like synfundus-1M for model generalization ability and transferability.

To assess KeepFIT’s transferability, we fine-tuned it on six unseen datasets. Specifically for CFP, we added and fine-tuned a fully connected layer to the image encoder and keep the other parts frozen. We conducted five fine-tuning settings, including 20%, 40%, 60%, and 80% of the data for training, the rest 20% for testing or following the official dataset partition. For FFA, due to dataset scarcity, we utilized image captioning for assessment following FFA-IR[17], using ResNet to extract images features and Transformer-based decoder for caption generation.

As shown in Tab 2(a), KeepFIT trained on MM-Retinal and flair achieves SOTA in almost all the unseen downstream datasets. Especially, it outperforms KeepFIT trained on flair by 0.035 and FLAIR trained on flair by 0.053. Similarly in Tab 2(b), the performance of FFA modality is improved when trained on both MM-Retinal and FFA-IR by 0.017. More results are provided in the supplementary material.

We ablated KeepFIT on all flair and MM-Retinal from three aspects. 1) Image similarity-guided text revision module: assess the necessity of revising the text features of public datasets using the knowledge extracted from MM-Retinal by the guidance of image similarity. 2) Mixed training strategy: test the necessity of including data from two sources in one batch. 3) Text fusion module: we substituted text revision with a text fusion module. It integrates knowledge extracted from MM-Retinal by multi-head cross attention into public dataset text features via residual connections to test the necessity of augmenting text features.

Table 2(c) shows that image similarity-guided text revision module and mixed training strategy are essential for the performance improvement. This is because revision module injects the appropriate expert knowledge into the texts and mixed training strategy significantly boosts the performance by reducing conflicts. However, the text fusion module makes minimal contribution, suggesting that text revision is more effective at injecting knowledge than text fusion.