Dual-Branch Network with Dual-Sampling Modulated Dice Loss
for Hard Exudate Segmentation from Colour Fundus Images

Our goal is to pursuit deep network parameters that best fit the hard exudate ground-truth in different sizes with given training images. For hard exudates in different sizes, learning representation and classifier in different manners is desired. To this end, we propose to adopt two branches to separately learn representation and classifier. One branch, named large hard exudate biased segmentation branch, is mainly responsible for hard exudates in large size. The other, named small hard exudate biased segmentation branch, is responsible for hard exudates in small size. To achieve the size bias of each branch adaptively, we design a dual-sampling modulated Dice loss, termed DSM. Fig. 2 illustrates our proposed dual-branch network.

Formally, let $X\in\mathcal{R}^{H\times W\times 3}$ denote a training colour fundus image size of $H\times W$ and $Y\in\mathcal{R}^{H\times W}$ is the corresponding ground-truth, which is a binary map within the context of hard exudate segmentation. From training set, we randomly fetch two images $\{X_{L},Y_{L}\}$ and $\{X_{S},Y_{S}\}$ and feed them into large hard exudate biased segmentation branch and small hard exudate biased segmentation branch respectively to obtain the final predictions $\hat{Y}_{L}$ and $\hat{Y}_{S}$. Next we elaborate the architecture of our dual-branch network and training details with our dual-sampling modulated Dice loss.

We let both branches economically share the same segmentation network structure, as illustrated in Fig. 2. Our dual-branch segmentation network can adopt arbitrary segmentation network. In this paper, we take three state-of-the-art segmentation networks, i.e. PSPNet (Zhao et al. 2017), Deeplabv3 (Chen et al. 2017b) and HED (Xie and Tu 2015) as examples to introduce our dual-branch segmentation network. PSPNet (Zhao et al. 2017) and Deeplabv3 (Chen et al. 2017b) adopt ResNet50 (He et al. 2016) equipped with dilation convolution (Chen et al. 2017a) in the last two stages as the backbone while HED (Xie and Tu 2015) adopts the five-stage VGG16 (Simonyan and Zisserman 2015) as backbone. Both ResNet50 (He et al. 2016) and VGG16 (Simonyan and Zisserman 2015) contain five stages of convolutions. Additionally, in PSPNet (Zhao et al. 2017), a pyramid pooling module is attached in the last convolutional stage, then a classifier is followed to make dense predictions. An auxiliary classifier is attached on the second convolution stage and auxiliary loss is generated to help optimise the learning process. In Deeplabv3 (Chen et al. 2017b), an atrous spatial pyramid pooling module is attached on the last convolutional stage to generate multiscale feature maps. In HED (Xie and Tu 2015), five side-output layers are stitched on five convolutional stages and finally a fusion layer is used to aggregate the side-output predictions. To reduce the model complexity and speed up the inference, for PSPNet (Zhao et al. 2017) and Deeplabv3 (Chen et al. 2017b) as segmentation branch, weights in first four stages of backbone networks are shared while rest weights are learned separately. For HED (Xie and Tu 2015), only the first three stages of backbone networks are shared. In this way, the representations for final classifiers are specific to hard exudate in different sizes. The loss items in PSPNet (Zhao et al. 2017), Deeplabv3 (Chen et al. 2017b) and HED (Xie and Tu 2015) are replaced with our proposed dual-sampling modulated Dice loss.

In each training iteration, two pairs of fundus images and their ground truth denoted by $\{X_{L},Y_{L}\}$ and $\{X_{S},Y_{S}\}$ are randomly fetched. $X_{L}$ is fed into the large hard exudate biased segmentation branch and predictions are obtained and denoted by $\hat{Y}_{L}$. Similarly, $X_{S}$ is fed into the small hard exudate biased segmentation branch and predictions are obtained and denoted by $\hat{Y}_{S}$. As hard exudate segmentation suffers from the issues of extreme class imbalance and large variation in size, rather than using the class balanced cross-entropy loss (Guo et al. 2019a, 2020), we propose the dual-sampling modulated Dice loss (DSM loss). As shown in Fig. 2, the total loss can be expressed as:

$$\mathcal{L}_{total}=\mathcal{L}_{DSM}\;.$$

(1)

Dual-Sampling Modulated Dice Loss. In our design of DSM loss, two different samplers are used to sample pixels from predictions by two branches separately. Then Dice loss is used to measure the dissimilarity between set of sampled pixels and their ground truths.

For large hard exudate biased segmentation branch, with predicted segmentation mask $\hat{Y}_{L}$, we use a uniform pixel sampler which samples hard exudate pixels and background pixels with equal probability. We denote the sampled pixel set with $\hat{\mathcal{S}}_{L}=\{\hat{y}_{L,n}\}_{n=1}^{N}$ where $N=H\times W$ and the corresponding set of ground truth with $\mathcal{S}_{L}=\{y_{L,n}\}_{n=1}^{N}$ where $y_{L,n}$ is ground truth of $\hat{y}_{L,n}$. We use Dice loss (Milletari, Navab, and Ahmadi 2016) to calculate the dissimilarity between sampled pixel set and the corresponding ground truth labels:

$$\mathcal{L}_{L}(\hat{\mathcal{S}}_{L},\mathcal{S}_{L})=1-\mathcal{D}(\hat{\mathcal{S}}_{L},\mathcal{S}_{L})\;,$$

(2)

where $\mathcal{D}(\hat{\mathcal{S}}_{L},\mathcal{S}_{L})$ measures the overlapping degree between two sets:

$$\mathcal{D}(\hat{\mathcal{S}}_{L},\mathcal{S}_{L})=\frac{\sum_{\hat{y}_{L,n}\in\hat{\mathcal{S}}_{L}}2\hat{y}_{L,n}y_{L,n}}{\sum_{\hat{y}_{L,n}\in\hat{\mathcal{S}_{L}}}\hat{y}_{L,n}+\sum_{y_{L,n}\in\mathcal{S}_{L}}y_{L,n}}\;.$$

(3)

Obviously, the loss defined by Eq. (2) and (3) focuses on overlapping error, which greatly alleviates the bias towards majority class like cross-entropy loss as well as minority class like CBCE loss. Dice loss produces serious penalty on misidentification of large hard exudate regions while slight penalty on misidentification of small hard exudate regions, which results in bias towards large hard exudate regions.

To remedy the misidentification on small hard exudate regions, from prediction by small hard exudate biased segmentation branch, we use a re-balanced pixel sampler to sample pixels involving loss calculation. Particularly, we oversample the exudate pixels and undersample background pixels. Thus hard exudate pixels in small regions are sampled multiple times with a high confidence, which increases the penalty of misidentification on small hard exudate regions. As a result, the learning focus is shifted into small hard exudate regions. In formulation, we randomly sample $N_{1}$ hard exudate pixels and $N-N_{1}$ background pixels with replacement. We denote the sampled pixel set with $\hat{\mathcal{S}}_{S}=\{\hat{y}_{S,n}\}_{n=1}^{N}$ and the corresponding set of ground truth with $\mathcal{S}_{S}=\{y_{S,n}\}_{n=1}^{N}$ where $y_{S,n}$ is ground truth of $\hat{y}_{S,n}$. Similarly, the loss for small hard exudate biased segmentation branch is calculated as follows:

$$\mathcal{L}_{S}(\hat{\mathcal{S}}_{S},\mathcal{S}_{S})=1-\mathcal{D}(\hat{\mathcal{S}}_{S},\mathcal{S}_{S})\;,$$

(4)

where $\mathcal{D}(\hat{\mathcal{S}}_{S},\mathcal{S}_{S})$ is the Dice coefficient which is computed same with Eq. (3).

As the segmentation of hard exudates in large size is much easier than in small size, we propose an easy-to-difficult learning strategy. We adaptively modulate the losses of two branches such that the dual-branch network first learns to handle the easy task, then focuses on difficult task. At the beginning of learning, we multiply the loss of large hard exudate biased segmentation branch by a large weight $\alpha$ while the loss of small hard exudate biased segmentation branch by a small weight $1-\alpha$ to enforce dual-branch network learn to segment hard exudate in large size. As the large hard exudate biased segmentation branch becomes more and more sophisticated in segmentation of large hard exudates, we gradually decrease the loss weight $\alpha$ and increase $1-\alpha$. In this way, the focus of dual-branch network is shifted to segmentation of small hard exudates gradually. Formally, we express our proposed dual-sampling modulated Dice loss as

$$\mathcal{L}_{DSM}=(1-\alpha)\cdot\mathcal{L}_{L}(\hat{\mathcal{S}}_{L},\mathcal{S}_{L})+\alpha\cdot\mathcal{L}_{S}(\hat{\mathcal{S}}_{S},\mathcal{S}_{S})\;,$$

(5)

where $\alpha$ is relative to learning epoch

$$\alpha=1-\left(\frac{epoch}{epoch_{max}}\right)^{2}\;.$$

(6)

Inference. In inference phase, the test fundus image is fed into both branches and two predictions are obtained. As both branches are equally important, we simply perform element-wise average on two predictions to obtain the final prediction.

Data. In our experiments, we validate our method on two public datasets for hard exudate segmentation. The first one is the DDR dataset (Li et al. 2019), which is made public for diabetic retinopathy classification, lesion segmentation and detection in 2019. To our best knowledge, DDR (Li et al. 2019) is the largest dataset for hard exudate segmentation. Fundus images in DDR (Li et al. 2019) are collected from 147 hospitals, covering 23 provinces in China and their size ranges from $1088\times 1920$ to $3456\times 5184$. The large variant in image size and large domain gap make the classification and segmentation on DDR (Li et al. 2019) more challenge. Specific to lesion segmentation, DDR (Li et al. 2019) provides 757 fundus images with pixel-level annotation, among which 383 images are for training, 149 for validation and 225 for testing. The other is IDRiD (Porwal et al. 2018, 2020), provided by a grand challenge on “Diabetic Retinopathy – Segmentation and Grading” in 2018. It provides 81 fundus images size of $4288\times 2848$ with pixel-level hard exudate annotations, among which 54 images are used for training and 27 for testing. All of those images are acquired from an eye clinic in India.

Evaluation Metrics. We evaluate segmentation methods at both pixel-level and region-level. With regard to pixel-level metrics, following (Li et al. 2019) and (Porwal et al. 2020), Intersection of Union (IoU) and Area Under Precision-Recall Curve (AUPR) are adopted. We also adopt F-score for evaluation, which is harmonic mean of sensitivity ($SN$) and positive predicted value ($PPV$).

With regard to region-level metrics, we follow (Zhang et al. 2014; Liu et al. 2017; Guo et al. 2020) and re-define true positive (TP), false positive (FP) and false negative (FN). We denote predicted hard exudate connected component set with $\hat{C}=\{\hat{C}_{1},\cdots,\hat{C}_{N}\}$ and ground truth hard exudate connected component set with $C=\{C_{1},\cdots,C_{M}\}$. A pixel is defined as a TP if, and only if, it belongs to:

$$\{\hat{C}\cap C\}\cup\left\{\hat{C}_{i}\bigg{|}\frac{|\hat{C}_{i}\cap C|}{|\hat{C}_{i}|}>\sigma\right\}\cup\left\{C_{i}\bigg{|}\frac{|\hat{C}_{i}\cap C|}{|C_{i}|}>\sigma\right\}\;,$$

(7)

where $|\cdot|$ accounts the cardinality and $\sigma\in[0,1]$ is the overlap ratio threshold. The larger $\sigma$ is, more rigorous the condition that a pixel is treated as a TP is. A pixel is considered as an FP if, and only if it belongs to

$$\{\hat{C}_{i}|\hat{C}_{i}\cap C=\emptyset\}\cup\left\{\hat{C}_{i}\cap\bar{C}\bigg{|}\frac{|\hat{C}_{i}\cap C|}{|\hat{C}_{i}|}\leq\sigma\right\}\;,$$

(8)

where $\bar{C}$ is complementary set to $C$. A pixel is considered as an FN if, and only if it belongs to

$$\{C_{i}|\hat{C}\cap C_{i}=\emptyset\}\cup\left\{C_{i}\cap\bar{\hat{C}}\bigg{|}\frac{|C_{i}\cap\bar{\hat{C}}|}{|C_{i}|}\leq\sigma\right\}\;,$$

(9)

where $\bar{\hat{C}}$ is complementary set to $\hat{C}$. Rest pixels are considered as TNs. The region-level F-score is defined as

$$F_{\sigma}=\frac{2\times SN_{\sigma}\times PPV_{\sigma}}{SN_{\sigma}+PPV_{\sigma}}\;,$$

(10)

where $SN_{\sigma}$ is sensitivity defined as $SN_{\sigma}=\frac{TP}{TP+FN}$ and $PPV_{\sigma}$ is positive predictive value defined as $PPV_{\sigma}=\frac{TP}{TP+FP}$. In our experiment, $F_{0.2}$, $F_{0.35}$, $F_{0.5}$, $F_{0.65}$ and $F_{0.8}$ are reported.

Data Preprocessing and Augmentation. For images in DDR (Li et al. 2019), we first crop the bounding box of field of view, then for cropped boxes whose short side is less than 1024, we enlarge the short side to equal length of long side via zero padding. Finally, we resize them to $1024\times 1024$. For images in IDRiD (Porwal et al. 2018, 2020), we directly resize images to $1440\times 960$. Following (Guo et al. 2019a, 2020), on both datasets, two tricks are adopted to augment the training data: rotation (90^∘, 180^∘ and 270^∘) and flipping (horizontal and vertical).

Experimental Setting. We build three variants of dual-branch segmentation network based on PSPNet (Zhao et al. 2017), Deeplabv3 (Chen et al. 2017b) and HED (Xie and Tu 2015). We call them dual-PSPNet, dual-Deeplabv3 and dual-HED respectively and implement them within PyTorch framework. We initialise the parameters in backbones with the pre-trained model on ImageNet (Russakovsky et al. 2015) and the parameters associated with classifiers with Gaussian distribution with zero mean and standard deviation of 0.01. SGD is used for parameter optimisation. Hyper-parameters include: initial learning rate(0.03 poly policy with power of 0.9), weight decay (0.0005), momentum (0.9), batch size (2) and iteration epoch (100 on DDR (Li et al. 2019) and 40 on IDRiD (Porwal et al. 2018)). Sample rate $N_{1}/N$ in re-balanced pixel sampler is set to 0.5. The models are trained on GeForce RTX 2080 Ti with two GPUs.

Effect of Sample Rate. We take dual-PSPNet as example and first explore the effectiveness of sample rate $N_{1}/N$ of re-balanced pixel sampler. We set it to 0.25, 0.5, 0.75 and reversed class frequency and train dual-PSPNet on DDR (Li et al. 2019) training set separately. Results on test set are reported in Table. 2. They show that our dual-PSPNet network achieves best when sample rate $N_{1}/N$ is set to 0.5. In what follows, except for extra illustration, $N_{1}/N=0.5$ is the default setting.

Ablation Study on Different Losses. To evaluate dual-branch network, we conduct experiments on three state-of-the-art dense classification methods PSPNet (Zhao et al. 2017), Deeplabv3 (Chen et al. 2017b) and HED (Xie and Tu 2015) with several settings, including single branch network with CBCE loss, single branch network with Dice loss (Milletari, Navab, and Ahmadi 2016) and our proposed dual-branch network with DSM loss on both DDR (Li et al. 2019) and IDRiD (Porwal et al. 2018).

Table. 3 reports the results on DDR (Li et al. 2019). Obviously, single branch networks with Dice loss (Milletari, Navab, and Ahmadi 2016) are much more sophisticated in hard exudate segmentation than with CBCE loss. This is because that CBCE loss re-weights costs according to reverse class frequency. It over-weights costs on hard exudate pixels and under-weights costs on background pixels, which makes the network be biased in favour of hard exudates. Thus the pixel-level sensitivity is very high while the PPV is very low. The harmonic mean i.e. pixel-level F-score is low. On the contrary, single network branches trained with Dice loss improve the pixel-level PPV significantly but inferior sensitivity. Nevertheless, final pixel-level F-score has been greatly improved. Compared with single branch networks with Dice loss, our dual branch networks with proposed DSM loss further improve the performance. In terms of IoU, our dual-branch ones with DSM loss are superior to the single branch ones. In terms of AUPR, our dual-branch ones outperform single branch ones except for the dual HED. In terms of region-level evaluation metrics, from Table. 3, we can see that dual Deeplabv3 (Chen et al. 2017b) with DSM loss consistently achieves better than single branch ones. Our dual PSPNet with DSM outperforms the single branch PSPNet (Zhao et al. 2017) with CBCE loss and Dice loss except for $\sigma=0.65$. Our dual HED with DSM outperforms the single branch HED (Xie and Tu 2015) with Dice loss when $\sigma$ is small. As $\sigma$ is larger than 0.5, our dual HED with DSM loss achieve inferior regional F-score to single branch network trained with Dice loss. The possible reason is that our dual HED with DSM segment small sized hard exudates coarsely and background pixels around small sized hard exudates are misidentified. When $\sigma$ is small, those misidentified background pixels are treated as true positives. Thus our dual HED with DSM loss achieve high regional F-scores than single branch network with Dice loss. When $\sigma$ increases, those misidentified background pixels are considered as false positives, thus the regional F-scores of ours are lower than single branch one with Dice loss. On IDRiD (Porwal et al. 2018), from Table. 4 we can see that our dual-branch networks with DSM outperform single branch networks with CBCE and Dice loss except for Dual HED with DSM when $\sigma$ is 0.5.

In terms of number of parameters, parameters in two branches with PSPNet (Zhao et al. 2017) and Deeplabv3 (Chen et al. 2017b) are almost 1.5 times of those in single branches while two branches with HED (Xie and Tu 2015) is 1.9 times of that in single branch.

Visual comparisons on DDR (Li et al. 2019) and IDRiD (Porwal et al. 2018) of our dual-branch networks with DSM loss to single branch networks with CBCE and Dice losses are provided in Fig. 3 and Fig. 4 respectively. We can see that single networks with CBCE loss are prone to misidentify background pixels around hard exudates. Single networks with Dice loss are prone to misidentify small size hard exudates. Our dual-branch networks work better than single branch networks.

On DDR (Li et al. 2019), we compare our three dual-branch networks with three deep learning based methods: Deeplabv3+ (Chen et al. 2018), DNL (Yin et al. 2020) and SPNet (Hou et al. 2020). For DNL (Yin et al. 2020) and SPNet (Hou et al. 2020), we provide results with two losses: CBCE and Dice loss. All these methods are originally designed for natural scene image segmentation. Table. 5 reports the results. We note that results of Deeplabv3+ (Chen et al. 2018) in the first row are provided by (Li et al. 2019) and the rest are obtained by fine-tuning on DDR (Li et al. 2019). As shown in Table. 5, our dual-branch networks achieve superior performance in terms of both pixel-level and region-level metrics.

On IDRiD (Porwal et al. 2018), we compare our dual-branch network with five deep learning based methods: DNL (Yin et al. 2020), SPNet (Hou et al. 2020), L-seg (Guo et al. 2019a), LWENet (Guo et al. 2019b) and Bin loss (Guo et al. 2020). For LWENet (Guo et al. 2019b) and L-seg (Guo et al. 2019a), the predicted binary masks are provided by authors. With them, $IoU$, $F_{pixel}$ and the region-level metrics are computed. For the rest methods, results are obtained by fine-tuning. Table. 6 reports the results. We can see that our dual-branch network achieves superior results than compared methods.

Visual comparisons between previous methods and ours on DDR (Li et al. 2019) and IDRiD (Porwal et al. 2018) are performed. In Fig. 5, the segmentation results by our dual-branch networks based on PSPNet (Zhao et al. 2017), Deeplabv3 (Chen et al. 2017b) and HED (Xie and Tu 2015), DNL (Yin et al. 2020) with CBCE loss and Dice loss, SPNet (Hou et al. 2020) with CBCE loss and Dice loss are shown. We can see that (1) the compared DNL (Yin et al. 2020) and SPNet (Hou et al. 2020) with CBCE loss are hard exudate biased and prone to misidentify background pixels as hard exudate pixels; (2) SPNet (Hou et al. 2020) with Dice loss are background biased and prone to misidentify hard exudate pixels as background pixels; (3) Our dual-branch networks achieve better than them. In Fig. 6, the segmentation results by ours, DNL (Yin et al. 2020) SPNet (Hou et al. 2020), L-seg (Guo et al. 2019a), LWENet (Guo et al. 2019b), and Bin loss (Guo et al. 2020) are shown. Similarly, we can see that (1) DNL (Yin et al. 2020) and SPNet (Hou et al. 2020) with CBCE loss are hard exudate biased while with Dice loss are background biased; (2) L-seg (Guo et al. 2019a) is background biased while LWE (Guo et al. 2019b) and Bin loss (Guo et al. 2020) are hard exudate biased; (3) ours achieve better than the compared methods.