Decomposing and Coupling Saliency Map for Lesion Segmentation in Ultrasound Images

Many studies have recently demonstrated the potential of the cues provided by saliency prior to help improve model’s generalization performance [17, 18, 19], especially on medical image segmentation tasks [20, 21]. These methods commonly cluster and highlight visually salient regions to assist downstream segmentation models. For example, Luo et al. [22] considered the exponentialized geodesic distance as saliency map and took it as additional input of the network. Feng et al. [20] utilized the saliency cue from interactive manual correction on the prediction map in an iterative manner. Vakanski et al. [12] generated the visual saliency map and integrated it into the model via attention mechanism for accurate lesion segmentation. However, most methods only focus on foreground saliency prior and ignore the structural characteristics contained by background, which may result in sub-optimal solutions especially in the complex scenario of ultrasound images. Actually, several studies have manifested that the introduction of background prior also contributes to the performance gain [23, 11, 24, 13]. For instance, Xu et al. [23] transformed user-provided positive and negative clicks into foreground and background saliency maps, for boosting the network’s capability of learning target-related representation. Wang et al. [11] proposed to form the geodesic distance maps of foreground and background based on the initial prediction map and concatenate them with the original image as model’s input for refining segmentation results. Ning et al. [13] tried to generate hierarchical foreground and background saliency maps that were hinted by three random seeds to assist downstream lesion segmentation model. However, existing methods generally neglect the potential associations between foreground and background, which may lead to inadequate exploitation of complementary information. Additionally, most methods adopt standalone generation algorithms that usually require interactive hand-clicked seeds to produce saliency prior, which, however, is time-consuming and experience-dependent. In other word, the learning of saliency map and segmentation model is carried out in two separated processes and it may lead to a sub-optimal result, even though each of these two processes could achieve their individual optimization. An appealing way is to integrate saliency map generation and lesion segmentation into a unified framework without manual intervention, and exploit complementary information derived from background representation based on its dependency on foreground to reinforce foreground-salient representation learning for accurate lesion segmentation.

Convolutional neural networks have revolutionized ultrasound lesion segmentation over the last few years [25, 26, 27], most of which belong to the variants of FCN and U-Net. For instance, Wang [28] et al. stacked two DeeplabV3plus to perform coarse-to-fine thyroid nodule segmentation and obtained promising results. Yap et al. [29] developed several FCN-based variants for the semantic segmentation of breast lesion. However, existing methods are still confronted with some difficulties in feature fusion during dimension reduction and resolution restoration. For the former, pooling operator in the encoder is commonly used to aggregate neighborhood features for reducing spatial resolution and enlarging receptive field. For lesion segmentation networks, default pooling operators include average pooling [30], max pooling [31, 32] and stride convolution due to their widespread adoption and proven efficacy [33]. Although these pooling operators are straightforward to aggregate regional features and alleviate the overfitting issue, they may also result in detail information dropout especially on relatively small lesions. Currently, some works have suggested that local detail preservation during the pooling stage is beneficial to model’s generalization [34, 35]. For example, Saeedan et al. [35] proposed a detail-preserving pooling strategy to magnify spatial changes and reserve structural patterns. Gao et al. [34] devised a local importance-based pooling operator to aggregate features within the window for detail preservation. However, these methods only concentrate on feature aggregation at the specified pooling window with a fixed receptive field, neglecting the significant contextual details inherent in neighborhoods (naturally with larger receptive field). As to the feature fusion during resolution restoration, skip connectivity mechanism [36] (commonly with channel-wise concatenation [13, 37, 12, 38, 39, 40, 41, 30] or pixel-wise addition [42]) is introduced to transmit the morphological context features from encoding blocks to decoding ones. To promote the efficiency of feature transmission and fusion, some works have ameliorated the skip connection strategy [43, 44]. For example, Liu et al. [43] modified U-Net with the redesigned skip connection operator to merge multi-scale semantic features for thyroid nodule stratification. Zhu et al. [44] introduced a multi-skip-connection strategy that cooperated with the squeeze-and-excitation block to fuse the feature maps from encoder and decoder. Nevertheless, most of them ignore the inner relationship between low-level visual information and high-level semantic representation, and previous studies have demonstrated that feature fusion based on correlation would reduce feature redundancy and is conducive to feature propagation [45, 46, 47].

As another challenge for ultrasound image segmentation, the hard or low-confidence samples/pixels (around blur edge and appearance) generally occupy a smaller proportion than the easy ones, which means the easy samples would dominate gradient descent during the training stage and might result in sub-optimal performance [48]. To this end, some researches have designed supervision functions based on focal loss for emphasizing on training a sparse set of hard samples [49, 50, 51]. For example, Li et al. [50] proposed to combine focal loss and dice loss to promote the model’s capability of segmenting hard samples. Yeung et al. [51] presented a unified focal loss that generalized dice loss and cross-entropy loss for coping with hard samples. Compared with cross-entropy (CE) loss, focal loss introduces a factor $(1-p)^{\gamma}$ to adjust the loss assigned to samples, which relatively raises the network’s concern for hard samples. However, as $\gamma$ increases, focal loss further down-weights easy samples, but concurrently overwhelms some relatively low-confidence ones (e.g., $p=.4\sim.75$). Actually, those samples still have positive effects on network training, which might be one of the reasons for the best choice of small $\gamma$ in most methods [49, 50]. Therefore, we advocate to enforce the network to simultaneously focus on hard and low-confidence samples.

We aim to distinguish lesion region (foreground) from its surrounding tissues (background) in ultrasound images. Denote $\mathcal{D}=\{\textbf{I}_{i},\textbf{Y}_{i}\}_{i=1}^{n}$ as a dataset containing $n$ instances, where I_i and Y_i represent the $i$-th original image and its corresponding pixel-wise annotation, respectively. An automated segmentation model $\mathbb{C}$ can be formulated as

where $\hat{\textbf{Y}}_{i}$ denotes the estimated label for the input image.

In practice, it is difficult to construct a satisfactory segmentation model only based on original ultrasound images due mainly to blur boundary and appearance encountered in the complex scenario. Thus, recent studies have sought help from foreground and background saliency maps [11, 13, 24]. Accordingly, given $\mathcal{S}^{f}=\{\textbf{S}_{i}^{f}\}_{i=1}^{n}$ and $\mathcal{S}^{b}=\{\textbf{S}_{i}^{b}\}_{i=1}^{n}$ as foreground and background saliency maps, respectively, the segmentation model $\mathbb{C}$ is learned based on the saliency maps in addition to the original images, which is formulated as

where $\mathcal{L}$ denotes the loss function. Generally, the generation of saliency maps requires the interactive hand-clicked seeds, which is time-consuming and experience-dependent. To address this issue, we first try to conduct lesion segmentation in a saliency map disentanglement-fusion manner and integrate saliency map generation and lesion segmentation into a collaborative framework without manual intervention. Let $\mathbb{D}_{f}$ and $\mathbb{D}_{b}$ represent the mapping functions that transform the original image into saliency maps, i.e., $\textbf{S}^{f}_{i}=\mathbb{D}_{f}(\textbf{I}_{i})$ and $\textbf{S}^{b}_{i}=\mathbb{D}_{b}(\textbf{I}_{i})$. Then, Eq.(2) can be rewritten as

where $*$ denotes $f$ and $b$. Based on Eq.(3), we propose DC-Net which includes two major components, namely a decomposition subnet (i.e., $\mathbb{D}$) for saliency map generation and a coupling subnet (i.e., $\mathbb{C}$) for lesion segmentation with the assistance of saliency prior fusion, as illustrated in Fig. 2(a). Notably, the decomposition subnet $\mathbb{D}$ is utilized to produce foreground and background saliency maps simultaneously.

Different from previous methods that utilize manual intervention to generate saliency prior [11, 13], the decomposition subnet $\mathbb{D}$ (Fig. 2(a)) disentangles the original image into foreground and background saliency maps in a data-driven fashion. It is primarily composed of an attention backbone which contains a convolutional unit and five attention blocks (Fig. 2(b)), and a saliency decomposition module (Fig. 2(c)).

The coupling subnet $\mathbb{C}$ is proposed for accurate lesion segmentation under the assistance of saliency maps, consisting of an encoder, a decoder and a coupler. Unlike prior ultrasound lesion segmentation networks [13, 37, 38, 40, 41, 30], the coupling subnet involves three fusion modifications: 1) A differentiable context pooling (DCP) operator is devised and inserted in the encoder to aggregate regional features and adaptively preserve local contextual details during dimension reduction; 2) The decoder is armed with a cross-correlation fusion (CCF) module to model the relation between low-level and high-level features and fuse them during resolution restoration; 3) A coupler is embedded to reinforce foreground-salient representation learning by incorporating complementary information derived from background-salient cues based on its dependency on foreground.

We design a hybrid loss to effectively train the proposed DC-Net, in which the decomposition subnet $\mathbb{D}$ and coupling subnet $\mathbb{C}$ are optimized jointly. According to Eq.(6) and Eq.(12), the total loss $\mathcal{L}$ can be formulated as

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are weight coefficients. Insufficient attention to hard samples encountered during the training stage is still a challenge for accurate ultrasound lesion segmentation, since these samples (i.e., pixels around blur edge and appearance) generally occupy a smaller proportion compared to the easy ones. Although focal loss [50, 53] has shown its efficacy on increasing the network’s attention to hard samples by reshaping CE loss such that it down-weights the loss assigned to well-segmented samples, it also reduces the loss amplitude of all samples and suppresses relatively low-confidence samples (e.g., $p=.4\sim.75$) especially when $\gamma$ increases. It is expected that, in the scenario of easy samples being down-weighted, the loss margin and gradient of hard samples can still be preserved and even increased to achieve continuous attention, while the range of suppression can be narrowed so as to guarantee the network to focus on low-confidence samples. Therefore, we introduce a harmonic loss function for ultrasound lesion segmentation.
Definition 2 (Harmonic Loss Function $\mathcal{L}_{h}$). The harmonic loss function is defined as

where $\sigma\geq 0$ and $\gamma$ are harmonic parameters, $p_{t}$ equals to $p$ for $y=1$ and $1-p$ for $y=0$, and $p$ and $y$ denote the estimated probability and label of a pixel, respectively. We can derive several interesting properties as follows:
P-1. The function is non-negative, continuous and differentiable.
P-2. $\mathcal{L}_{h}\left(p_{t}\right)=-\left(1-p_{t}\right)^{-1}\log\left(p_{t}\right)$, if $\sigma\rightarrow 0$.
P-3. $\mathcal{L}_{h}\left(p_{t}\right)=-\left(1-p_{t}\right)^{\gamma-1}\log\left(p_{t}\right)$, if $\sigma\rightarrow+\infty$.
P-4. $\mathcal{L}_{h}(p_{t})=-\left(1+\sigma)(1-p_{t}\right)^{-1}\log\left(p_{t}\right)$, if $\sigma\ll(1-p_{t})^{\gamma}$.
P-5. $\mathcal{L}_{h}(p_{t})=-\frac{(1+\sigma)}{\sigma}(1-p_{t})^{\gamma-1}\log(p_{t})$, if $\sigma\gg(1-p_{t})^{\gamma}$.

Obviously, the P-1 holds, thus it is suitable as a loss function. The P-2 suggests that $\mathcal{L}_{h}$ degenerates into CE loss $\mathcal{L}_{c}$ with the modulating factor $(1-p_{t})^{-1}$ when $\sigma$ approaches zero. Inspired by [54], we rewrite $\mathcal{L}_{c}$ and $\mathcal{L}_{h}|_{\sigma\rightarrow 0}$ in a Taylor expansion form:

Accordingly, their gradients can be computed by summating series polynomials as follows:

Intuitively, $\mathcal{L}_{h}|_{\sigma\rightarrow 0}$ provides a smaller constant gradient than $\mathcal{L}_{c}$, which can prevent the model from emphasizing the majority class (i.e., easy samples) [54]. Thus, it can be regarded as an improved CE loss. Similarly, P-3 indicates $\mathcal{L}_{h}$ is an equivalent variant of focal loss $\mathcal{L}_{f}$ if $\sigma$ approaches the infinite, and their gradients can be calculated by

It is apparent that $\mathcal{L}_{h}|_{\sigma\rightarrow+\infty}$ is equivalent to shifting all the polynomial coefficients of $\mathcal{L}_{f}$ by 1. As to P-4 and P-5, they suggest that $\mathcal{L}_{h}$ additionally introduces the amplification factors $1+\sigma$ and $\frac{1+\sigma}{\sigma}$ for $\mathcal{L}_{h}|_{\sigma\rightarrow 0}$ and $\mathcal{L}_{h}|_{\sigma\rightarrow+\infty}$, respectively, when the conditions of $\sigma\ll(1-p_{t})^{\gamma}$ and $\sigma\gg(1-p_{t})^{\gamma}$ are satisfied.

Furthermore, we integrative different settings of $\sigma$ and $\gamma$ for $\mathcal{L}_{h}$, and provide an intuitive comparison among $\mathcal{L}_{h}$, $\mathcal{L}_{f}$ and $\mathcal{L}_{c}$ in Fig. 3. Several key points can be observed as follows: 1) Compared with $\mathcal{L}_{c}$, $\mathcal{L}_{h}$ down-weights easy samples as $\mathcal{L}_{f}$ does; 2) $\mathcal{L}_{h}$ almost preserves and even elevates the loss margin and gradient of hard samples; 3) $\mathcal{L}_{h}$ narrows down the suppression range to prevent low-confidence samples from being overwhelmed, and the range would shrinkage as either $\gamma$ or $\sigma$ decreases. Additionally, we introduce a weighting factor $\alpha_{t}\in[0,1]$ to alleviate imbalance between foreground and background, and the Eq.(19) can be rewritten as

where $\alpha_{t}=\alpha$ if $y=1$, otherwise $\alpha_{t}=1-\alpha$ ($\alpha$ is a hyper-parameter).

Considering that compound loss function would help obtain a robust and accurate model for medical image segmentation [55], we design $\mathcal{L}_{\mathbb{C}}$ by incorporating $\mathcal{L}_{h}$ with dice similarity coefficient (Dice) loss $\mathcal{L}_{d}$ [56]:

where ${\lambda_{4}}$ is a weight coefficient. Since decomposition subnet aims to preliminarily disentangles the original image into saliency maps and it does not require accurate estimation, we thus set ${\mathcal{L}_{\mathbb{D}_{f}}}=\mathcal{L}_{d}$ and ${\mathcal{L}_{\mathbb{D}_{b}}}=\mathcal{L}_{d}$ to reduce the computational cost.

We evaluated our proposed method on two different ultrasound lesion segmentation tasks, i.e., breast ultrasound (BUS) segmentation and thyroid ultrasound (TUS) segmentation. Five multi-site BUS datasets that contains 2862 images and a TUS dataset which enrolls 646 images were used to develop and assess the proposed method. Brief sample distribution of BUS and TUS datasets is summarized in TABLE I.
BUS. S1: Dataset1 collects 1812 BUS images offered by the SonoSkills and Hitachi Medical Systems Europe⁴⁴4https://www.ultrasoundcases.info/; S2: Dataset2 has 200 BUS images acquired by the Ultrasonix SonixTouch Research ultrasound scanner with a L14-5/38 linear array transducer [57]; S3: Dataset3 contains 163 BUS images provided by the UDIAT Diagnostic Center with a Siemens ACUSON Sequoia C512 system and a 17L5 HD linear array transducer ($8.5$ MHZ) [29]. S4: Dataset4 enrolls 506 BUS images derived from the Baheya Hospital with a LOGIQ E9 ultrasound system or LOGIQ E9 Agile ultrasound system [58]. S5: Dataset5 includes 181 BUS images obtained by a Philips iU22 ultrasound scanner at Thammasat University Hospital [59]. The ground-truths of all datasets have been provided by the organizers except for Dataset3. Thus, the annotations of Dataset3 have been manually delineated by a radiologist with two years of clinical experience and confirmed by another radiologist with seven years of clinical experience. Also, they have reviewed the annotations of other datasets and removed partial poor-quality images that may result in inconsistent suggestions.
TUS. The dataset recruits 646 B-mode TUS images and their ground-truths, which is provided by the IDIME Ultrasound Department in Colombia [60]. The radiologists have been also invited to review and confirm image’s quality as well as annotation.

For BUS segmentation task, we trained the model on two sites (i.e., S1 and S2 [57], 2012 images in total) via inner 5-fold cross-validation strategy and tested on three sites (i.e., S3 [29], S4 [58] and S5 [59], 850 images in total) to assess model’s generalization capability on multiple sites. As for TUS segmentation task, we utilized 5-fold cross-validation strategy to evaluate the proposed method in view of the limited sample size [60]. All images were resized into a fixed size of $256\times 256$ and the intensity values in each image were normalized into the range of [0, 1] by Min-Max normalization. The performance of model was assessed in terms of dice similarity coefficient (Dice), mean intersection over union (m-IoU), precision and 95% hausdorff distance (95HD). The parameters $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, $\sigma$, $\gamma$ and $\alpha$ were experimentally set to 0.5, 0.5, 1, 10, 0.001, 5 and 0.25, respectively. Notably, the influence of key parameters on model’s performance has been discussed in the section of Discussion. We initialized all parameters with "he_normal" and adopted Adam solver with the momentum of 0.9 for network training. The batch size, learning rate and epoch number were set to 4, $3\times 10^{-4}$ and 200, respectively. All intensive calculations were offloaded to two 12 GB NVIDIA Pascal Titan X GPU.

We first compared the proposed method with the state-of-the-art deep learning approaches in ultrasound lesion segmentation. The competing methods include: 1) Classical medical image segmentation models: DeepLab V3+ [61], U-Net [16], U-Net++ [45], and AU-Net [62]; 2) Breast lesion segmentation models: RDAU-Net [37], AE-Net [12], ConvEDNet [63], SK-Net [38], CF²-Net [39], MNFE-Net [64], and CSwin-PNet [65]; 3) Thyroid nodule segmentation models: TNSCUI-R1 [28], TNSCUI-R2 [66], resDU-Net [40], JU-Net [41], and SGU-Net [30], FDE-Net [67], and BPAT-UNet [68]. As not all comparison methods have been evaluated on the same datasets, we reimplemented them based on the details and source codes provided by their work. Table II shows the experimental results.

We have several key observations. First, all U-Net variants outperform DeepLab V3+ on both ultrasound lesion segmentation tasks, which might benefit from the skip connectivity operator in U-Net variants. Second, AU-Net obtains remarkable performance gain than most U-Net variants, especially on TUS segmentation task. It further indicates that attention mechanism can help boost model’s performance for lesion segmentation [62]. Third, multi-scale strategy is competitive across model’s capability of learning features for lesions in complex-scenario images. For example, SK-Net produces higher segmentation accuracy by expanding receptive field to extract multi-scale contexts via the dilated convolution, while TNSCUI-R2 adopts the ensemble refinement mechanism to iteratively optimize the network with multi-resolution images. Fourth, ConvEDNet displays pretty promising in BUS segmentation and is competitive with CF²-Net. It is likely that the utilization of edge constraint advances edge recognition in low-contrast and blurry-boundary images.

Fifth, well pre-trained backbones may help to improve model performance. For example, CSwin-PNet and BPAT-UNet introduced pre-trained transformer blocks to improve the network’s capability of extracting long-range dependency and achieved better segmentation results. Finally, the proposed DC-Net which we find to reach the noticeably higher performance in terms of all evaluation metrics when compared with other methods. Several potential advantages exist in the proposed method: 1) It integrates saliency map generation and lesion segmentation into a unified framework, and introduces a coupler to reinforce foreground-salient representation learning by incorporating complementary information derived from background-salient cues. 2) It performs regional feature aggregation via DCP operator to alleviate the issue of information dropout by adaptively preserving local important contextual details and enlarging receptive field. 3) It strengthens the connection of information from encoding and decoding streams (via CCF) to efficiently conduct relation-aware representation fusion of low-level and high-level features. 4) The harmonic loss encourages the network to focus more attention on hard and low-confidence samples during the training stage, which helps improve generalization ability. In Fig. 4, we also provide the visualization results of all competing methods on some representative cases for further comparison. From Fig. 4, we can find that DC-Net can more accurately locate the lesion than others, which mainly credits to the utilization of the cues provided by saliency maps. Also, DC-Net can work well in the scenario where lesion region shares similar intensity with surrounding tissues, especially on the aspect of edge’s integrity.

The saliency map is one of core components of DC-Net and DaR module helps couple the saliency maps of background and foreground. In this part, we aim to validate their efficacy by developing and evaluating various variants. For convenience, we name these variants as follows. 1) DC-Net_woB (DC-Net_woF): It removes coupler and only uses the concatenation of original image and foreground (background) saliency map as the input of coupling subnet; 2) DC-Net_woC: It removes both decomposition subnet and coupler, and only takes the original image as the input of coupling subnet. 3) DC-Net_woD: It abandons DaR module and links foreground-salient and background-salient representations by channel-wise concatenation operation. The experimental results are shown in Fig. 5. The DC-Net_woC is inferior to others on two segmentation tasks, which verifies the conclusion that saliency map is a significant step towards complex-scenario ultrasound lesion segmentation [23, 11, 13]. And DC-Net significantly overpasses both DC-Net_woB and DC-Net_woF, owing to the coupler that can extract discriminative information derived from background to complement foreground-salient representation, especially when background contains richer texture patterns than foreground. Also, the DC-Net_woD is found to suffer from a profound performance degradation, and this degradation suggests DaR module can better help reinforce foreground-salient representation compared to simple concatenation operation that may introduce more irrelevant information. Fig. 8 shows the attention map (i.e., $\psi(\tilde{\textbf{F}}^{f}\times{\textbf{K}}^{T})$) in the DaR module on some representative cases from BUS and TUS datasets. From the Fig. 8, we can find that the attention map mainly concentrates on the foreground (i.e., lesion region), which can be utilized to mine foreground-dependent complementary information derived from background and reinforce the foreground-salient representation learning. To further investigate the effectiveness of supervision strategy for decomposition subnet, we implement DC-Net with only the minor modification of using different supervision schemes. 1) DC-Net_woS_f (DC-Net_woS_b): It discards the supervision of foreground (background) mask on saliency map generation; 2) DC-Net_woS: It trains decomposition subnet in an unsupervised manner; 3) DC-Net_wSB: It replaces the precise mask with bounding box to optimize saliency map generation. In Fig. 7, as we expected, DC-Net_woS underperforms other variants, since supervision signal can restrain the uncertainty of parameters and accelerate the convergence of model. Additionally, DC-Net surpasses DC-Net_woS_f and DC-Net_woS_b, which indicates that the cooperation of foreground and background masks can help to efficiently disentangle the original image into foreground and background saliency maps and strengthen model’s performance. Interestingly, DC-Net_wSB is also likely to achieve promising performance, implying proxy annotation may be an alternative choice for training saliency map generation. Fig. 6 displays some representative cases from BUS and TUS datasets for further comparison. From Fig. 6, we can see that the model can show better segmentation performance by simultaneously learning foreground and background representation under the assistance of foreground and background saliency maps.

We additionally experimented with equipping the network with the following pooling approaches to help specify the efficacy of DCP operator. 1) Classical pooling methods in lesion segmentation network: average pooling (AveP), max pooling (MaxP) and mix-pooling (MixP); 2) Local detail-based pooling approaches: stride convolution (StrC), polynomial pooling (PolyP) [69], local importance-based pooling (LIP) [34] and detail-preserving pooling (DPP) [35]; 3) Context aggregation strategies: atrous spatial pyramid pooling (ASPP) [61] and switchable ASPP (SAC) [70]; 4) The variant of DCP (denoted as $\overline{\mbox{DCP}}$) that neglects context aggregation. From Fig. 9, we can observe some key points. First, the model with MixP achieves the best performance among three classical pooling methods, perhaps because a compound pooling mode can remedy the pattern dropout caused by single one. Second, the models with local detail-based pooling approaches (also including DCP and $\overline{\mbox{DCP}}$) work better than those with three classical pooling methods and context aggregation-based methods in general. A potential reason is that these methods aggregate local features via the learnable estimation rather than the prior statistics (e.g., local mean or maximum value). Finally, the DCP method shows more promising results than others, which may owe to three aspects, including 1) learning the adaptive pooling kernel; 2) preserving local contextual details; 3) enlarging the receptive field. Fig. 10 gives the visualization results of variants with various pooling strategies on some representative cases from BUS and TUS datasets. Obviously, our proposed DCP can reduce the error in separating the lesion region from surrounding tissues that shares similar intensity. To analyze the complexity of different pooling strategies, we also calculated the trainable parameters, floating point operations (FLOPs) and the inference time of each image for the model with different pooling operators. All results are reported in Table  III. From Table  III,we can observe that the increasing model parameters with DCP operator is marginal when compared with other operators, which suggests the proposed DCP is relatively efficient.

This ablation study is to validate the effectiveness of CCF module by comparing it with other methods. Concretely, these methods include: 1) Baseline: it removes the CCF from DC-Net; 2) Concat/Add/RMSC: it replaces the CCF with the channel-wise concatenation operation, channel-wise addition operation and residual multi-scale connection strategy [44], respectively. From Fig. 11, we can observe that the model with Concat or Add performs better than Baseline. It demonstrates the skip connection strategy that transmits low-level features from encoder to supplement morphological information for decoder contributes to the performance improvement by a large amount. It is worthwhile mentioning that the model with RMSC overpasses those with Concat and Add on two tasks. The performance gain may benefit from the multi-scale skip fusion strategy and attention mechanism. In addition, CCF is superior to other competing methods, which implies that fusing low-level visual information with high-level semantic representations under the prior of correlation can reduce redundant information and strengthen information connection. Fig. 12 displays the integrated (via pixel-wise addition operation) low-level and high-level feature maps (i.e., $\textbf{X}_{E}$, $\hat{\textbf{X}}_{D}$, $\textbf{X}_{E}\oplus\hat{\textbf{X}}_{D}$, $\bar{\textbf{X}}_{E}$, $\bar{\textbf{X}}_{D}$ and $\bar{\textbf{X}}_{E}\oplus\bar{\textbf{X}}_{D}$) from CCF module. As shown in Fig. 12, low-level feature maps contain rich textural information (see Fig. 12(a)(d)), while high-level ones have better capability of lesion localization (see Fig. 12(b)(e)). Also, the prior of channel correlation can enhance the network’s ability of capturing detail information, such as shape and boundary (as shown in Fig. 12(a)(b) and (d)(e)). Therefore, feature integration under the prior of channel correlation can strengthen the efficiency of feature propagation. Fig. 13 displays the visualization results to qualitatively compare various skip connection strategies. Though all methods can accurately locate the lesion, the model with CCF shows better ability in fitting the lesion shape, regardless of it with large-size or small-size.

Here, we mainly make a comparison the proposed optimization loss (i.g., harmonic loss $\mathcal{L}_{h}$ and its hybrid version with Dice loss $\mathcal{L}_{h+d}$) with the following losses: 1) Single function: CE loss $\mathcal{L}_{c}$, Dice loss $\mathcal{L}_{d}$, focal loss $\mathcal{L}_{f}$, topk loss $\mathcal{L}_{t}$ [71]; 2) Hybrid function: joint CE loss and Dice loss $\mathcal{L}_{c+d}$, generalized Dice and CE-based loss $\mathcal{L}_{u}$ [51], joint focal loss and Dice loss $\mathcal{L}_{f+d}$, joint topk loss and Dice loss $\mathcal{L}_{t+d}$. Fig. 14 shows the experimental results. Generally speaking, the models trained with hybrid loss functions work better compared to those trained with single loss functions, which further verifies the conclusion stated by [55]. As expected, $\mathcal{L}_{c}$ is inferior to its generalized variants (i.e., $\mathcal{L}_{f}$, $\mathcal{L}_{u}$, $\mathcal{L}_{h}$, and $\mathcal{L}_{t}$), as the latter compels the network to give more attention to hard samples during the training stage.

The models with $\mathcal{L}_{h}$ and $\mathcal{L}_{h+d}$ achieve the highest metric scores among those models with single functions and hybrid functions, respectively. Unlike $\mathcal{L}_{f}$, $\mathcal{L}_{u}$ and $\mathcal{L}_{t}$, the proposed $\mathcal{L}_{h}$ can down-weight easy samples, preserve and even lift the loss margin and gradient of hard samples. More importantly, it can also narrow down the suppression range to prevent low-confidence samples from being overwhelmed. Fig. 15 visualizes some representative cases from BUS and TUS datasets for qualitative comparison of various loss functions. It can be found that the output of model trained with $\mathcal{L}_{h+d}$ can approximately coincide with the ground truth. Meanwhile, we also notice $\mathcal{L}_{h+d}$ helps model work in the blurry-boundary scenario, especially on TUS datasets. Fig. 16 shows the gradient maps of several representative losses on the same prediction map from previous training epoch. We mainly focus on the comparison of cross-entropy loss $\mathcal{L}_{c}$ and its advanced variants (i.e., focal loss $\mathcal{L}_{f}$, topk loss $\mathcal{L}_{t}$ and harmonic loss $\mathcal{L}_{h}$) to analyze their influence on easy, low-confidence and hard samples. From Fig. 16, we have the following findings. 1) $\mathcal{L}_{c}$ still produces gradients for easy samples/pixels (e.g., purple pixels in $\mathcal{L}^{\prime}_{c}$). Although these gradients are weak, they may dominant and affect the direction of network optimization due to the large proportion of easy samples in the whole image. 2) Compared with $\mathcal{L}_{c}$, $\mathcal{L}_{f}$ can obviously suppress the gradient of easy samples (e.g., those pixels changes from purple to black), but it also suppresses the low-confidence ones (e.g., $\mathcal{L}^{\prime}_{f}$ has more incomplete contours of some lesions in BUS images than $\mathcal{L}^{\prime}_{c}$). 3) $\mathcal{L}_{t}$ relatively alleviates the aforementioned issues, but it still pays insufficient attention to low-confidence samples, which results in flawy boundaries of some lesions. 4) $\mathcal{L}_{h}$ can not only suppress the gradient of easy samples, but also focus on the hard and low-confidence samples around the boundary (e.g., yellow and purple pixels in $\mathcal{L}^{\prime}_{h}$), so as to obtain promising segmentation performance.

In this subsection, we compare the proposed DC-Net with several saliency map-based methods, including iFCN [23], DeepIGeoS [11], and SMU-Net [13]. All of them require interactive saliency map generation. For deeper comparison, we run the DC-Net by using the interactively-generated saliency maps from other methods as input, and also feed the automatically-generated saliency map from DC-Net into other methods. For convenience, we mark those methods based on interactively-generated and automatically-generated saliency maps with the tails of "interac" and "automac", respectively. Experimental results are shown in Table IV. From Table IV, we have several observations as follows. 1) DC-Net demonstrates enhanced performance when utilizing interactively-generated saliency maps from SMU-Net. But, it experiences significant performance degradation when employing interactively-generated saliency maps from DeepIGeoS. The saliency maps from SMU-Net cover low-level and high-level image structures that help distinguish lesion from background. However, subtle gray difference between lesion and background leads to the blurry boundary in the geodesic distance map (i.e., interactively-generated saliency maps from DeepIGeoS), which may give wrong prompt and hinders the performance improvement. Even so, both of them require interactive hand-clicked seeds, thus are time-consuming and experience-dependent. It is worth mentioning that the proposed method has smaller model parameters than SMU-Net (54.4M vs. 103.3M). 2) When the same saliency maps are used, DC-Net can achieve comparable and even higher results than others, which demonstrates its efficacy.

In this part, we discuss the influence of key parameters in our proposed method, including $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ in Eq.(18), $\lambda_{4}$ in Eq.(27), and $\gamma$, $\sigma$, $\alpha$ in Eq.(26). 1) To determine the ratio of $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, we tested the model under different settings on two tasks. As shown in Fig. 17 (a)-(b), the model gets the best performance when the ratio is set to 0.5: 0.5: 1. It is not surprising about that as focusing more attention on final precise segmentation (via the couple subnet) is beneficial to performance improvement. We further investigate the ratio of $\lambda_{3}$ and $\lambda_{4}$. From Fig. 17 (c)-(d), as we expected, a large $\lambda_{4}$ is a better choice, which assigns a large weight to the harmonic loss. And the best results are obtained when the ratio equals to 1:10. 2) We conducted experiments by varying the value of $\gamma$, $\sigma$ and $\alpha$ in the range of $\{3,5,7,9\}$, $\{10^{-4},10^{-3},10^{-2},10^{-1}\}$ and $\{0.1,0.25,0.5,0.75\}$, respectively, for having an insight into their influence on the proposed model. In addition, we also designed two unique settings for $\alpha$, including 1) removing $\alpha$ from the harmonic loss (denoted as "none") and 2) adaptively determining $\alpha$ according to the proportion of the foreground to the whole image (denoted as "adp"). From Fig. 18, we can observe that the model with $\gamma=5$, $\sigma=10^{-3}$ and $\alpha=0.25$ outperforms those with other settings, for both tasks. As illustrated in Fig. 18, the harmonic loss with $\gamma=5$ and $\sigma=10^{-3}$ can suppress easy samples (with $p_{t}$ in the range of (0.75, 1]) and enhance the gradient of low-confident samples (with $p_{t}$ in the range of [0.4, 0.75]).

To discuss the model’s complexity, we calculated the number of model parameters, FLOPs and the inference time of each image for several representative algorithms, including DeepLabV3+, U-Net, U-Net++, AU-Net, DeepIGeoS, iFCN, and SMU-Net. Table V shows all comparison results. The complexity of DC-Net approximates that of DeepLab V3+, but is higher than most of other methods. It is worth mentioning that both SMU-Net and DC-Net contain special streams or branches to deal with foreground and background representation, and they experience a significant performance improvement at the cost of a relatively high complexity. But, compared to SMU-Net, DC-Net seems to be more efficient and is free of manual interaction.