Learning With Context Feedback Loop for Robust Medical Image Segmentation

\IEEEPARstart

Medical image segmentation is often profoundly important in clinical image analysis and image-guided interventions. It involves partitioning a medical image into multiple areas, where these areas can be used for better clinical analysis or clinical target visualization. For example, in prostate radiotherapy, accurate clinical target volume segmentation from magnetic resonance (MR), ultrasound (US), or computed tomography (CT) images is often essential in computer-aided diagnosis, therapy, and post-therapy analysis of prostate cancer [1] [2]. Indeed, it is critical to select patients for a specific treatment, guide source delivery during the intervention, and compute the dose distribution using MR, US, and CT images, respectively [1]. Similarly, in cardiac image analysis, accurate segmentation of the heart structures such as the left and right ventricular cavities, and the myocardium is essential to calculate the volume of the cavities at end-diastolic and end-systolic phases and the left ventricular myocardial mass [3] [4]. Inner ear structures segmentation such as the cochlea, vestibule, and semi-circular canals from preoperative CT images can be used in the treatment of patients with hearing impairments. In cochlear implant surgery, preoperative cochlea segmentation from CT images is useful for determining the length of the implanted electrode and improving the insertion guidance [5]. Other important medical image analysis applications using image segmentation includes in 2D echocardiography [6] and brain image segmentation [7] among others.

Despite the necessity of accurate and robust image segmentation in clinical routines, it is often challenging. Indeed, it highly depends on the imaging modality and the target (such as anatomy or tumor area). The main challenges in developing accurate and automatic medical image segmentation methods include a wide range of patient characteristics, low-contrast inherent imaging characteristics of some imaging modalities, artifacts (e.g., metallic artifacts in CT), respiratory motion, limited data in medical image analysis, and significant variation of shape and size of organs among cases [8] [9]. Unfortunately, manual segmentation is frequently prone to subjective errors and is time-consuming. It might not always be easy to analyze multiple sequences of examinations manually. Researchers have applied different methods to address these challenges, thereby better analyzing the medical images and improving case outcomes. For example, several groups have approached medical image segmentation using contour and shape detection, deformable models, conventional machine learning, and deep learning approaches [8] [10].

In the early times, conventional medical image segmentation approaches are often based on edge detection and fitting predefined parametric shapes, level-set methods, and shape models [3] [8]. Many classical machine learning-driven methods (supervised and unsupervised) are also proposed to tackle this challenging problem. Although traditional machine learning approaches like active shape, atlas methods, and statistically supervised and unsupervised methods are still prevalent [10], they often involve careful integration of hand-crafted image features. These hand-crafted features need to represent the input image. However, designing or engineering distinctive image features by human experts can be difficult and sometimes impossible. Moreover, manually designed distinctive features on a given task might not be easy to adapt to new cases, which is the main problem in developing a general method that can be used to extract the distinctive image features [8] [10].

In this regard, deep learning methods based on convolutional neural networks (CNN) have emerged as an alternative and reliable solution [10]. These CNN-based methods can automatically learn to extract the hierarchical distinctive image features. It avoids the need to develop hand-crafted features besides the generic characteristics of these extracted features. In fact, CNN-based approaches have achieved the state of the art (SOTA) performance in various tasks such image classification [11], object detection [12], segmentation [13] [14], registration [15] [16], and other tasks [17] [18]. For example, fully convolutional neural networks are trained on pixel-to-pixel transformations to extract high-level image features and predict the outputs from any arbitrary size inputs [14]. Following the success of these approaches in image and video processing, Simonyan et al. [18] proposed to increase the convolutional network depth via a network called VGG16. In this approach, convolutional layers are stacked one after the other to extract more distinctive image features.

Motivated by the promising results of the VGG16 network (deep encoder like structure), Badrinarayanan et al. [19], proposed to expand it into a deep fully convolutional neural network (FCN) architecture for semantic pixel-wise segmentation. It employs a trainable encoder network for high-level feature extraction and a corresponding decoder network. The decoder network maps the low-resolution encoder feature maps to a full input resolution feature maps for pixel-wise classification using up-sampling. This up-sampling operation is then modified to be a learning-based deconvolution network in the work of Noh et al. [20]. Szegedy et al. [21] also proposed a new design of CNN architecture to increase the depth and the width of CNN-based methods. As CNN architectures go deeper (i.e., large number of convolutional layers are stacked one after the other), residual connections are inherently crucial for the training [22] [23]. It smooths training and avoids loss of high-level features during sampling or pooling operations.

Ronneberger et al. [13] modified the FCN proposed by Long et al. [14] to propagate contextual information from the encoder into the decoder. It is done by connecting the encoder with the decoder through skip connections that creates a U-shaped architecture (and named U-Net). Similarly to the residual networks [22], this skip-connection recovers spatial information (such as localization information) that could be lost during the consecutive striding and convolutional operations. Several groups have modified the U-Net architecture for various medical image analysis applications [10]. Çiçek et al. [7] adopted the 2D U-Net for volumetric (3D) segmentation. Tu et al. [24] introduced auto-context U-Net in which they applied parallel 2D convolutional layers for the axial, coronal, and sagittal planes of brain images.

Most previous works based on encoder-decoder (or U-Net) architectures have been focused on improving encoder’s feature extraction ability either by constraining the segmentation results [25], incorporating the prior knowledge such as shape models [26] [27], or using transfer learning from pre-trained networks [28] [29].

Meanwhile, there has been an increased interest of researchers to incorporate the shape of anatomical structures into U-Net like architectures using multi-task learning [27] [30] [31]. These approaches aimed at solving the common limitations of encoder-decoder architectures in capturing the structural information and interdependence of the output while training from pixel-wise objective functions [10]. Consequently, several groups approached the incorporation of shape prior in image segmentation using statistical shape modeling [27] or learning-based modeling [25] [30] [32] [33]. These approaches generally showed a relative improvement over the methods without the shape prior. However, these approaches often require explicit prior knowledge of the target. It needs the modeling of the prior knowledge, for example, anatomical shapes, and then embed it into the U-Net architectures to constrain the learning process [6] [25] [27]. Other approaches include post-processing methods based on either denoising [34] or variational auto-encoders [35] and showed an improvement in the plausibility of the results. However, they are not free of limitations. Post-processing methods do not see the original input image. Thus, they might not always produce accurate results from a given erroneous segmentation [35].

In encoder-decoder based deep learning methods, although the encoder is essential to extract high-level and distinctive input image features, it could be challenging due to the often inconsistent contrasts and potential artifacts associated with the inherent limitations of medical imaging devices. Thus, similar to the residual [22] and the long skip connection [13] approaches, different alternative methods have been proposed to improve the contextual information propagation from the encoder to the decoder. These methods use either gated attention-based networks [36] or recurrent neural networks [37]. They can be used either to highlight salient image features by capturing richer contextual dependencies [38] [39] or to recurrently recover information loss from the consecutive sampling and re-sampling operations [40] [41] [42] [43] in the feed-forward learning approach. Recurrent based networks use a similar concept to residual networks. Firstly, they are used to accumulate features that lead to better feature representation. Secondly, they are used to smooth the training [42]. Other approaches proposed to apply additional paths to the encoder-decoder architecture [44] [45]. Most of these approaches are based on CNNs in the feed-forward training approach and could be biased towards recognizing the texture features than the contextual or shape features [46].

Nonetheless, capturing the contextual information, the inter-region relationship, and implicitly learning the prior knowledge of a target remained a wide interest of researchers in deep learning-based medical image analysis. Still, encoder-decoder based image segmentation methods do not get the second chance to look at the segmentation results, except through the gradient backpropagation from a defined pixel-wise objective function. Indeed, training from only pixel-wise objective functions can lead to the loss of spatial information, less output pixels interdependence, and less inter-region relationship representation and hence sometimes, produce unrealistic segmentation results [25]. A recent study has also demonstrated that CNNs are strongly biased towards recognizing textures rather than shapes [46]. Thus, capturing the output pixels’ interdependence would enable the network to learn distinctive image features. The network can learn to extract texture and contextual similarity between the same labeled pixels and the difference between differently labeled neighboring pixels, thereby producing realistic segmentation.

In this regard, the decoder should be intelligent enough to improve the classification of each pixel. To do so, the decoder needs to capture the contextual information, inter-region relationship, and distinguish errors that could be introduced during the consecutive striding, convolution, and up-sampling operations. Indeed, feeding back the predicted probabilistic output could help the decoder extrapolate contextual information and correct mistakes over time, thereby improving prediction accuracy. The idea of a feedback loop is indeed used in the control system’s theory, in which the system’s output error signal is used to make adjustments on the input signal. Alternative studies on generative adversarial neural networks (GANs) have also recently gained an interest in using a feedback loop to improve the model’s learning capacity [47] [48].

Therefore, in this study, to address the difficulties of CNNs in capturing the contextual information and the inter-region relationship, and to implicitly integrate the prior knowledge into the feed-forward learning process, we formulated image segmentation as a recurrent framework using two systems (named Learning with context FeedBack system (LFB-Net)). One can notice that we used the term system to refer to the two different interconnected networks. The forward system, a modified U-Net architecture, learns and predicts a probabilistic output from the raw input image. The probabilistic refers to the pixel levels’ softmax assignment at the output of the network. The second system, a fully convolutional network (FCN), named hereafter feedback system, transforms the predicted probabilistic output of the forward system into another spatial higher-level representation. The transformed high-level representation is then integrated back into the feed-forward learning path of the forward system. This feedback strategy, as we will show, would allow the main segmentation module to improve the performance over time. More importantly, the feedback system mitigates the conditions in which the forward system can potentially fail.

In this study, we present the first deep learning formalism using a feedback loop in encoder-decoder based image segmentation. To this end, we designed the image segmentation problem as a recurrent framework. The image segmentation process of a network is conditioned by its previous segmentation results’ latent space and a spatial response map of another network that enables it to improve the performance over time.

Our main contributions can be summarized as follows:

1. 1.

   We introduce a feedback looping system to enhance the feed-forward learning process of encoder-decoder CNNs for medical image segmentation.

2. 2.

   We model the image segmentation problem as a two systems task, from system one to system two approach. We integrated the proposed feedback looping system with a modified U-Net architecture (in the forward system) as a regularizer that enables the network to attend and fix segmentation uncertainties over time.

3. 3.

   The proposed method, LFB-Net, is validated on different medical image segmentation applications. Specifically, we evaluated it on short-axis cardiac-MRI, long-axis echocardiographic images, CT of the prostate, and CT of the inner ear image segmentation applications. Our extensive experiments and ablation studies on clinical databases support that our method consistently outperforms the SOTA methods at a reduced computational cost. As will be shown in the result section, it yielded accurate and plausible results without the need to incorporate shape prior or apply post-processing methods.

The rest of this paper is organized as follows. Section II introduces the proposed method (LFB-Net). In section III, we present the experimental results and discussions. Finally, we draw our conclusions in section IV.

We formulate the main segmentation network $S$ (forward system in Fig. 1) as two main parts: a segmentation encoder $S_{e}$ which encodes the raw input image into high feature space, and a decoder $S_{d}$ which then decodes the encoded features into the target labels. Specifically, the encoder network $S_{e}$ transforms the input image $x$ into the encoded features space $h$, i.e., $S_{e}:x\mapsto h$, and the decoder transforms the intermediate representation $h$ into the desired label $y$, i.e., $S_{d}:h\mapsto y$. This consecutive stage is equivalent to

$$\hat{y}=S(x)=S_{d}(S_{e}(x)),$$

(1)

where $x\in\mathbb{R}^{W\times H}$ is the input image and $\hat{y}\in\mathbb{R}^{W\times H}$ is the predicted output label. Here, $W$ and $H$ indicate the original image width and height respectively. Thus, the latent space dimension of this network is $S_{e}(x)=h_{s}\in\mathbb{R}^{W/d\times H/d\times C}$, where $C$ indicates the number of feature channels at the latent space while $d$ is the depth of the network.

In order to not lose spatial information during down-sampling, this network is composed of skip connections. It is typically similar to the U-Net architecture [13], except the encoder and decoder are designed so that they can be trained jointly and separately, as will be shown in section 2.3.

The feedback system aims to provide a second chance for the forward system’s decoder network to look back on its predicted output. This approach could be made by recurrently formulating the network and merging their full-resolution outputs at the last layer of the decoder. However, providing a more high-level representation of the predicated output with small correction using the feedback system (i.e., spatial feedback) would enhance the decoder’s learning capacity [48]. Thus, the forward system’s probabilistic output is fed to a new FCN ($F$) architecture (the feedback system), which transforms the predicted probabilistic output into another high-level feature space. This approach of correcting and feeding back the output can be considered as an analogy to the teacher correcting the exam of a student and then providing back the correct answers. Therefore, this strategy allows the forward system to attend to specific regions of its previous results and improve prediction accuracy.

Lets divide the feedback system $F$ into two: the encoder $F_{e}$ which encodes the forward system’s probabilistic output $\hat{y}$ into a high-feature space $h_{f}$ ( i.e., $F_{e}:\hat{y}\mapsto h_{f}$) and the decoder $F_{d}$ then transforms the high-feature space $h_{f}$ into desired label $y$ (i.e., $F_{d}:h_{f}\mapsto\hat{\hat{y}}$). As in (1), this process can be written as:

$$\hat{\hat{y}}=F(\hat{y})=F_{d}(F_{e}(\hat{y})),$$

(2)

where $\hat{\hat{y}}$ is the predicted probabilistic output label from the feedback system, and $\hat{y}$ is the predicted probabilistic output label from the forward system.

We need to integrate the feedback system $F$ with the forward system $S$. Thus, we designed the integration of the two systems as a recurrent process. This recurrent process is formulated as follows. Firstly, the segmentation module produces a label map $\hat{y_{i}}$ (at iteration $i$) using (1). Secondly, the feedback system predicts a label map $\hat{\hat{y_{i}}}$ taking $\hat{y_{i}}$ as input from the output of the forward system. Now, we can write the feedback system’s encoder response map $F_{e}$ as:

$$h_{f_{i}}=F_{e}(\hat{y_{i}})$$

(3)

where $h_{f_{i}}\in\mathbb{R}^{W/d\times H/d\times C}$ is high-feature space representation or latent space of the feedback system at iteration $i$. It has the same size as the forward system’s encoder output, $h_{s_{i}}\in\mathbb{R}^{W/d\times H/d\times C}$. The decoder ($F_{d}$) then transforms $h_{f_{i}}$ into the output label map $\hat{\hat{y_{i}}}$.

Thirdly, we use the high-feature space of feedback information $h_{f_{i}}$ along the encoded input from forward system $h_{s_{i}}$ as input to the decoder $S_{d}$, at iteration $i+1$. Therefore, now we can redefine (1) at $i+1$ as:

$$\hat{y}_{i+1}=S_{d_{i+1}}\big{(}h_{s_{i}},F_{e_{i}}(\hat{y}_{i})\big{)}$$

(4)

which can be written as

$$\hat{y}_{i+1}=S_{d_{i+1}}\big{(}h_{s_{i}},h_{f_{i}}\big{)}\$$

(5)

As our system aims to incorporate the context spatial feedback in the feed-forward learning process, at the first stage we set the feedback latent space $h_{f_{0}}=0$ (i.e., $h_{f_{0}}=h_{0}$). Then in the second stage (i.e., at $i=i+1$), we replace $h_{f_{0}}$ by the feedback latent space $h_{f_{i}}$. This is indicated in Fig. 1 as switch between $h_{0}$ and $h_{f}$.

The integration and training strategy of the forward system and the feedback system can be summarized as follows:

1. []

2. 1.

   Train the neural network weights of the forward system, $w^{s}_{i}$, considering the raw input image $x$ and zero feedback latent space (i.e., $h_{f_{i}}=0$) as inputs, and the ground truth labels $y$ as output.

3. 2.

   Train the neural network weights of the feedback system, $w^{f}_{i}$, considering the input from the predicted output of the forward system’s decoder network $\hat{y}$, and the ground truth label $y$ as output as in (2).

4. 3.

   Train the neural network weights of the forward system’s decoder part only $w^{s}_{d_{i+1}}$, taking the inputs from previously extracted high-level features from the raw input image in step 1 (i.e., $h_{s_{i}}$) and the feedback latent space $h_{f_{i}}$ from the feedback system in step 2 as in (5). Here, the forward system’s and the feedback system’s encoder are designed to predict (i.e., freeze) from previously learned and updated weights during step 1 and step 2, respectively.

5. 4.

   While not converged, repeat the above steps. The convergence is determined by the change in the validation loss at the output of the forward system. These steps are also shown in Fig. 2.

Note that each step is trained at a time until the network sees the whole training dataset. During the training phase, the feedback system provides feedback to the forward system, but the decoder part is discarded during the testing phase.

To train our model, we used an average of binary cross-entropy and Dice coefficient loss functions as:

$$L_{total}=\frac{1}{2}\times(L_{1}+L_{2})$$

(6)

where $L_{1}$ and $L_{2}$ are the cross-entropy and the Dice loss functions, respectively. The cross-entropy $L_{1}$ is computed as :

$$L_{1}=-\sum_{k=1}^{c}\sum_{i=1}^{I}\log{\bigg{[}\frac{e^{s(k,i)}}{\sum_{i}e^{s(k,i)}}\bigg{]}}$$

(7)

where $s(k,i)$ is the probabilistic feature maps at pixel $i\in I$, belonging to the pixel class $k\in 1,2,3,...c$ ($c$ number of classes), and $I$ being the number of pixels in the training batch. The Dice coefficient loss is calculated as:

$$L_{2}=1-\frac{1}{\sum_{k}\gamma_{k}}\bigg{[}\sum_{k}{\gamma_{k}}\frac{2\times\sum_{i\in I}u_{i}^{k}v_{i}^{k}}{\sum_{i\in I}u_{i}^{k}+\sum_{i\in I}v_{i}^{k}}\bigg{]}$$

(8)

where $u$ is the predicted output of the network, $v$ is a one-hot encoding of the ground truth segmentation map, $\gamma$ is the weight associated to class $k\in{0,1,2,...}$, being the pixel class. Both $u$ and $v$ have shape $I$ with $i\in I$ being the number of pixels in the training batch. The loss function was the same for both the forward and the feedback systems.

The complete network architecture is shown in Fig. 1. In the forward system, the encoder’s blocks consist of repeated applications of $3\times 3$ convolutions, exponential linear unit (ELU) activation, batch normalization, and followed by a squeeze-and-excitation network [49] (Fig. 1 (B)). Moreover, we applied a $2\times 2$ max pooling operation with stride 2 for downsampling. Contrarily, the decoder is composed of a $2\times 2$ up-convolution and concatenation layer, followed by the same block as in the encoder (Fig. 1 (B)).

The feedback system is a fully convolutional network architecture with a learning deconvolution network [20]. It consists of a repeated application of $3\times 3$ convolutions, ELU activation, batch normalization in both the encoder and decoder blocks. We selected a concatenation layer to merge the latent spaces from the two systems. Please refer to the supplementary material for a detailed description of the architectures and hyper-parameter values used in this work.

The system was implemented and developed in Python, Keras API, with a Tensorflow backend. The weights of the model were updated by using (6), and an ADAM optimizer with a learning rate of $10^{-3}$ until convergence [50]. All networks, including the baseline models, were trained from scratch with an early stop of 100 and batch-size of 10. We mostly divided the dataset into 70% training and the rest 30% for testing. For each clinical datasets which involve volumetric computation, the division was done based on the patient cases. The model is trained on the given training dataset and then tested on unseen new image cases. The model’s hyper-parameters were optimized only from the validation dataset.

The proposed network’s parameters were the same for all segmentation applications, except for the output activation function and the number of output feature channels. For the single label segmentation (i.e., for prostate and inner ear), it was a sigmoid activation function. For the multi-label segmentation (i.e., for cardiac cine-MRI and ultrasound with four channel outputs), it was a softmax activation function. In all experiments, the intensity values were normalized according to each dataset’s mean and standard deviation. It allows the model to learn the optimal model parameters quickly.

The proposed method was compared with other methods such as U-Net [13], residually connected U-Net (ResU-Net) [22], FCN [20], Post-DAE [34], and attention gated network (AGN) [38]. We chose these networks as they have a similar architecture to our method. For the public datasets, we evaluated our method on the testing data. We also compared the proposed method with other methods such as [25], [35], and [51] on the public data. We extensively experimented during our method’s ablation study with and without a feedback loop such as using only the forward system (FS). All settings were then the same for the baseline methods. Indeed, the baseline method’s hyper-parameters were optimized to yield the best results.

We considered both volume and distance metrics to evaluate the methods, such as the Dice similarity coefficient (DSC), Hausdorff distance (HD), and relative volume difference (RVD). For the volumetric datasets, the evaluation was in 3D. Unless specified, we adopted these metric’s notations throughout the paper. We also ran a statistical test using the Wilcoxon test [52] and we consider that the results are significantly different when the p-value is less than 0.05 (i.e., $p<0.05$). Moreover, we visually verify the plausibility of the results. In this context, plausibility refers to the complete, realistic, and anatomically possible segmentation. Therefore, a plausible or realistic output of a method contains no hole, no segment from unexpected regions of the image, and segmented areas are similar in shape to the targets’ structure.

Precise prostate gland segmentation from CT images is critical in the treatment of prostate cancer. However, it is not easy due to the inherently low contrast characteristics of CT for soft tissues and the appearance of high metal artifacts. For example, in CT exams of patients who have received low dose rate brachytherapy treatment for prostate cancer, in which tiny radioactive elements are permanently implanted into the gland, accurate prostate segmentation is difficult. The quality of the CT image is worsened by the high metal artifacts coming from the implanted radioactive elements. In our study, a clinical database of 78 CT prostate image cases was collected. All cases had received a prostate cancer treatment using a low dose rate brachytherapy technique. The in-plane resolution varies from $0.4$ $mm$ x $0.4$ $mm$ to $0.58$ $mm$ x $0.58$ $mm$ with a slice thickness between $1.5$ $mm$ and $2.5$ $mm$. We changed the datasets into the same voxel size of $0.5$ $mm$ x $0.5$ $mm$ x $1.25$ $mm$. A radiation oncologist with an experience of more than ten years ($>100$ implants/year) manually delineated the prostate. These delineation procedures are routinely done in the clinical treatment where it involves permanent brachytherapy with ${}^{125}I$ for localized prostate cancer [1] [2]. The dataset was randomly divided into 70% for training, and the rest 30% for testing and validation (20% for testing and 10% for validation).

From the experimental result shown in Table 1, our method can segment the prostate gland with an average volumetric Dice index of 0.91 and with a 3D HD of 6.1 mm. It appeared to outperform the other methods. For example, it produced a 2% increased Dice index and decreased the 3D HD value by 11 mm over U-Net [13]. Our method without the feedback loop, i.e., only the forward system (FS), produces a Dice index of 0.90 and a 3D HD value of 10.3 mm. It improved both metrics over the other methods. Using the feedback loop reduces the 3D HD error value by 4.3 mm over not using it (only FS). Attention gated-based method [38] has only reduced the 3D HD metric over the U-net [13] by 0.9 mm, while the Post-DAE method [34] reduced it by 6.2 mm. However, the Post-DAE did not improve the average Dice index. The proposed LFB-Net method significantly outperformed $(p<0.05)$ the other methods for the Dice index value and 3D HD.

Inner ear segmentation is essential for 3D visualization and modeling of the inner ear for surgeries. We chose the Hear-EU cochlear data descriptor public dataset consisting of $\mu$CT-scans of 17 dry temporal bone specimens [5]. The ground truth labels include the cochlear scala, semicircular canals and the vestibule. The images were acquired at 16.3 and 19.5 $\mu$m voxel resolutions for 13 and 4 $\mu$CTs, respectively. The original volume size of the $\mu$CTs ranged from 618 x 892 x 600 to 1500 x 1500 x 1500. The $\mu$CTs were standardized to a fixed size of 256 x 256 x 256 voxels for computational and memory requirements. The ground truth labels of 5 patients were not aligned with the $\mu$CT images. The data for these patients were manually corrected to align with the $\mu$CT data perfectly. To compare all methods, a two-fold cross-validation technique was performed by randomly dividing the data at each fold into 76% for the training and 24% for the validation.

As can be seen from Table 1, LFB-Net achieved an increase in 4% of the average Dice index and reduced relative volume difference by 6% over the U-Net [13]. It also yielded accurate segmentation results without missing the unconnected small parts of the inner ear on CT images, which can then be used to model the inner ear structures for ear surgeries [53]. In contrast, the other methods yielded reduced segmentation results. In particular, the post-processing method, Post-DAE [34], appeared to degrade the segmentation results of the U-Net [13], in particular the Dice index by 5%.

The third application of our method, which aims to demonstrate the performance of our model for multi-class segmentation applications, is on cardiac cine-MRI segmentation. Although short-axis cardiac cine-MRI is essential to study the cardiac function of the left and right ventricles, accurate segmentation of both ventricles remains challenging. The segmentation of the endocardial border in diastole and systole is required to evaluate the cardiac function (i.e., cavity volume and ejection fraction) for the left and right ventricles, and the epicardial border of the left ventricle is mandatory to evaluate the myocardial mass and thickness. The main challenge is the variability in the shape of the left and right ventricular cavities. In this regard, we chose the Automated Cardiac Diagnosis challenge (ACDC) [4] to evaluate the proposed method. It contains 100 patients for the training and 50 for the testing. These datasets are available for download at ¹¹1https://acdc.creatis.insa-lyon.fr/. To analyze how the baseline methods would work in the testing sets, we have access to the ground truth upon request to the organizers. We randomly divided the 100 cases into 75 patients (75%) for training and 25 patients (25%) for validation (all the cases with diastole and systole phases). Thus, to further inspect the performance of the method, we evaluated the end-diastolic and end-systolic phases separately on the 50 testing cases.

In all 3D experimental measurements on the validation sets shown in Table 2, our method appeared to improve segmentation accuracy. It yielded an average Dice and HD values, respectively of $0.96\pm 0.02$ and $6.7\pm 3.95$ mm for the left ventricular cavity (LV), of $0.91\pm 0.05$ and $13.2\pm 7.90$ mm for the right ventricular cavity (RV), and of $0.90\pm 0.03$ and $9.4\pm 5.21$ mm for the myocardium (MYO) on the validation datasets. The 3D HD value of the myocardium refers to the largest distance error from either the endocardium or the epicardium. More importantly, we observed that the improvement is significant in cases where the segmentation task is difficult, such as in the right ventricular cavity and the myocardium.

As shown in Table 3, on the ACDC testing set, our method yielded an average Dice index and 3D HD values, respectively of $0.94\pm 0.06$ and $7.5\pm 5.54$ mm in the LV, of $0.92\pm 0.06$ and $11.9\pm 6.49$ mm in the RV, and $0.90\pm 0.03$ and $9.5\pm 5.58$ mm in the MYO. We observed that LFB-Net significantly outperforms the other methods ($p<0.05$). To further analyze the method’s generalizability property, results from the unseen testing set and validation set are shown in Table 4. Although we have different sample sizes for the validation set, 25 cases (each with end-diastolic and end-systolic phases), and the testing set, 50 cases (each with end-diastolic and end-systolic phases), these average values in Table 4 can be considered to infer the methods’ performance for each set. Indeed, the proposed method showed almost no differences between the unseen testing and the validation sets by achieving a total Dice index of 0.92 and a 3D HD value of around 10 $mm$. In contrast, the other methods showed a large difference between the validation and testing sets. For example, LFB-Net has significantly outperformed the other methods in the heart segmentation on the testing data (shown in Table 4). However, this is not always the case on the validation data.

Accurate cardiac structure segmentation from echocardiography images is profoundly vital in cardiac diagnosis. For this, we chose Multi-structure Ultrasound Segmentation (CAMUS) dataset to evaluate our method [51]. It contains two and four-chamber acquisition from 500 patients, with end-diastolic and end-systolic phases. Thus, a given patient has four images (two in end-diastole and two in end-systole). The segmentation reference and raw data of 450 patients are available for download at ²²2https://camus.creatis.insa-lyon.fr/challenge/. We randomly divided this data into 396 patients for training and 54 patients for validation.

As shown in Table 2, on the 54 validation exams, our method has improved both the Dice and the HD values. For the two view acquisition (two-chambers and four-chambers) based segmentation results, our method yielded an average Dice index of $0.94\pm 0.03$, $0.92\pm 0.04$, and $0.86\pm 0.06$ for the 4-chamber, and $0.94\pm 0.03$, $0.92\pm 0.05$, and $0.88\pm 0.04$ for the 2-chamber respectively in the left ventricular cavity (LV), left atrium (LA), and myocardium (MYO). The average HD values were $5.0\pm 2.83$ mm, $5.2\pm 3.48$ mm, and $6.7\pm 3.04$ mm for the 4-chambers, and $5.6\pm 3.22$ mm, $4.8\pm 2.79$ mm, and $7.1\pm 3.86$ mm for the 2-chambers respectively in the LV, LA, and MYO. Although it produces similar results for the different view acquisitions in LV and LA, it improved the myocardium segmentation on the 2-chambers over the 4-chambers by 2% in the Dice index.

As shown in Table 5, on the 50 CAMUS testing exams, LFB-Net achieved an average Dice index of $0.96\pm 0.02$ for the LV epicardium, $0.94\pm 0.03$ for the LV endocardium, and $0.91\pm 0.07$ for the left atrium. It outperformed the other CAMUS challengers [25] [51]. LFB-Net improves the segmentation in all multi-label structures with less accuracy variability among the test data. Moreover, it notably improved the segmentation by large value on the left atrium in the end-diastolic phase. However, as the results were obtained by submitting the predicted images to the challenge website, we could not perform a statistical comparison [51]. The proposed method also achieved comparable results with the intra-observer values. Mainly, it yielded better results in the Dice index except for the endocardium in the systolic phase. Thus, segmentation with the context feedback loop yields consistent results.

As one can observe from the qualitative segmentation results in Fig. 3 for the single label and Fig. 4 for the multi-label segmentation, our model produces more plausible results than the other methods. From a careful visual checking of the results, if it is with holes for a given single structure segmentation and between structures for multi-structure segmentation and comparing the shape, our method produces more plausible results. Whereas the other methods produce holes in a given structure or between structures and sometimes produce atypical results which are type of errors that could not be made by manual segmentation. Moreover, as seen in the ear segmentation [Row 4-6], the other methods appeared to fail in scenarios when they segment multiple unconnected small structures. In contrast, our method produces a more realistic segmentation of all structures. We observed similar scenarios throughout the testing data. Indeed, anatomical plausibly is a prerequisite for the experts to use the segmented structures for clinical assessments. With the proposed method, the reliability of the segmentation renders trustworthy the clinical information extracted from these segmented structures.