Deep Learning Ensemble for Predicting Diabetic Macular Edema Onset Using Ultra-Wide Field Color Fundus Image

The DIAMOND Challenge provided data sourced from 14 French hospitals as part of the EVIRED project, which aims to predict ci-DME development and anticipate the onset of DR complications. For evaluation, the DIAMOND Challenge incorporated independent datasets from Algeria alongside the French hospital data. This approach promoted universally applicable solutions, capable of serving diverse population groups and settings. During the coding period, a synthetic dataset generated using images from the Deep Diabetic Retinopathy Image Dataset (DeepDRiD) was provided to assist in the development and local testing of the algorithms. DeepDRiD is available under the Creative Commons Attribution Share Alike 4.0 International license, and served as a development and testing resource with ci-DME labels generated based on DR severity. This dataset includes 204 UWF-CFP in total, with 154 images partitioned for training and 50 images partitioned for validation. Also a one-patient dataset is provided, including real image data from one patient, with images taken by different UWF-CFP device (OPTOS and CLARUS) from each eye to enhance model development with actual ci-DME labels.

We adopted four CNN architectures to develop a generalisable and robust model: Resnet, Densenet, EfficientNet and VGG.

ResNets (Residual Networks) are renowned for high performance in classification tasks, particularly in medical image analysis. A deep residual learning framework is employed to address the vanishing gradient problem and the degradation of network performance as depth increases. This approach enables the training of very deep networks by introducing residual mapping, which allows gradients to propagate through the network more effectively, enhancing training efficiency and performance [7].

DenseNet (Densely Connected Convolutional Networks) excels in efficient feature usage and parameter reduction. It introduces dense blocks where each layer connects to every other layer in a feed-forward manner, promoting feature reuse and mitigating the vanishing gradient problem. DenseNet is often preferred for its ability to achieve high performance with fewer parameters than with ResNets [8].

EfficientNet scales network dimensions (depth, width, and resolution) uniformly using a compound scaling method. This model achieves superior performance on the ImageNet dataset and transfers well to other datasets, offering a balance of high accuracy and computational efficiency. EfficientNet’s ability to maintain small model sizes while ensuring fast computation speeds makes it suitable for local applications such as hospital-based retinal image classification [9].

VGG (Visual Geometry Group Network) is an earlier convolutional neural network (CNN) architecture which uses small convolutional filters and deep networks (16-19 layers) to attain strong performance on large-scale image recognition tasks like ImageNet. VGG is known for its large model size and lengthy computation times relative to newer architectures, but remains a useful baseline model for image classification tasks such as object recognition and medical image classification [10]

Model ensembling combines the outputs produced by multiple models into a single prediction process, which can overcome shortcomings associated with individual estimators such as high variance, noise, and bias. In this work, three ensemble strategies were used.

Plurality Voting: The first was plurality voting: where the class with the highest number of votes is used as the final prediction. All models contribute equally to the decision-making process, preventing dominance by any single model. Plurality voting treats all votes equally, ignoring the confidence levels of individual model [11].

Averaging: The second was averaging: where the final outputted probability is the unweighted average of the probabilities estimated by each model. By averaging probabilities instead of class labels, this method incorporates the confidence of each model’s predictions, often leading to better-calibrated outputs. Averaging can produce smoother predictions, especially in cases of highly imbalanced data, as it reduces the influence of extreme or outlier predictions from individual models [12].

Label Fusion: The third technique was label fusion using a three-layer simple neural network, which was trained using the predictions from several individual models as inputs, thereby learning to assign appropriate weights to each individual model. An example of these three strategies using five models and two classes is shown in Fig. 1. Final prediction outputs can vary widely with different ensemble strategies.

Image Preprocessing. In all experiments, the images were resized to 224 $\times$ 224 pixels from their original size. Other image augmentation techniques were also employed, including random horizontal reflection, random vertical reflection, and random rotation. Example images are shown in Fig. 2.

Training Hyper-parameters. Throughout the training process, the adam optimizer was used with a learning rate scheduler with exponential decay. Networks were trained using 200 epochs and the models with the best performance were saved. Performance was gauged with the metrics suggested by the DIAMOND Challenge: Area under the receiver operating characteristic curve (AUC), F1-score (F1) and Expected Calibration Error (ECE) [13].

For evaluation, we used the following metrics provided by the challenge organizer:

Area under the curve (AUC): The AUC is computed as the area under the receiver operating Characteristic (ROC) curve. AUC values range between 0 to 1; with 1 representing perfect classification performance across all model thresholds. The use of AUC aligns with the goal of achieving high sensitivity and specificity in predictions and was therefore weighted highest during evaluation and ranking.

F1 score and expected calibration error (ECE): The F1 score and ECE were considered as secondary metrics (lower weighted) to further assess the relevance of computed probabilities. The F1 score is the harmonic mean of precision and recall, which is used to balance these two indicators. F1 helped ensures that binary predictions (when using a probability cutoff of 0.5) were relevant. F1 also assesses precision (or positive predictive value) which is not captured by AUC [14]. ECE measures the calibration of predicted probabilities, indicating the difference between predicted confidence and actual correctness. The calculation method of ECE is to divide the prediction distribution of the model output into multiple intervals, calculate the difference between the average prediction probability and the actual accuracy in each interval, and then sum them up to get the ECE value, lower ECE values indicate better calibration of the model. The use of ECE helped calibrate model properties for the context in which it will be used, such as by ensuring that its predicted probabilities matched the actual probabilities of the ground truth distribution [13]. The following performance function was used to determine which model was saved.

The results in Table. 1 compare the performance of 5 baseline models, with the DenseNet-121 achieving the highest overall score (1.5262) due to its strong AUC (0.7617), F1 score (0.7347), and low ECE (0.2057), reflecting its efficiency in classification and calibration. ResNet-152 also performed well, with the highest AUC (0.7633) and a solid overall score (1.47875), although its lower F1 score (0.6275) limited its effectiveness. EfficientNet-b7 demonstrated a balance between metrics, with a respectable overall score of 1.46325, but was outperformed by DenseNet-121. ResNet-50 and VGG-19, while adequate, lagged behind in overall scores (1.4449 and 1.4231, respectively), showing limitations in calibration and predictive accuracy compared to the more modern architectures.

Table. 2 shows the performances of the ensemble methods, evaluated on the internal testing dataset. For the individual evaluation, label fusion obtained the best performances (overall score of 1.4343) comparing to the other models ensemble methods. The results in Table. 2 highlight the strengths and weaknesses of three ensemble methods for ci-DME prediction. Plurality voting, while simple and well-calibrated (ECE = 0.1023), achieved the lowest AUC (0.6517), F1 score (0.5714), and overall score (1.3862), indicating limited predictive capability. Averaging improved AUC (0.71885) and F1 score (0.6862) significantly, showcasing better classification performance, but it suffered from poor calibration (ECE = 0.5361), which lowered its overall score to 1.2939. Label fusion outperformed the other methods with the highest overall score (1.4343), offering a balanced performance with competitive AUC (0.7017), F1 score (0.6512), and improved calibration (ECE = 0.2860), demonstrating its ability to effectively integrate model outputs.