SAM-FNet: SAM-Guided Fusion Network for Laryngo-Pharyngeal Tumor Detection

Motivated by DLGNet [15], a dual-branch network can capture both global and local (lesion) information, enabling accurate detection of lesions. In this study, we propose a SAM-guided fusion network for laryngo-pharyngeal tumor detection, following dual-branch architecture [15], which is shown in Fig. 1. Specifically, it consists of five parts: a SLL (see Section II-B), a GFE (see Section II-C), a LFE (see Section II-D), a GFO (see Section II-E), and a classifier (see Section II-F). Unlike DLGNet, we introduce the SAM to accurately locate the lesion region. Additionally, we design a GAN-like feature optimisation module (GFO) to enhance global and local feature representations, thereby improving the model’s ability to extract more discriminative features.

Formally, the dataset is denoted as $\mathcal{D}=\{(x_{g}^{(i)},y^{(i)})\}_{i=1}^{N}$, where $N$ is the number of total laryngo-pharyngeal endoscopic images, $x_{g}^{(i)}$ represents a holistic laryngoscopic image, and $y^{(i)}\in\{0,1,2\}$ represents the labels for the three categories: normal laryngo-pharyngeal tissues (normal for short), benign tumors (benign for short), and malignant tumors (malignant for short). The lesion area image $x_{l}^{(i)}$ corresponding to $x_{g}^{(i)}$, is obtained through the SLL module and share the same label $y^{(i)}$ as $x_{g}^{(i)}$. Following the paradigm of DLGNet, we adopt a multi-task learning framework with several loss functions tailored for each branch, which can leverage domain-specific information from complementary tasks, enhancing prediction accuracy and generalization for each task [20]. Specifically, global, local, and fused feature representations are fed into their respective classifiers for final prediction. The discrepancy between these predictions and the ground truth labels is measured by cross-entropy losses denoted as $\mathcal{L}_{g}$, $\mathcal{L}_{l}$, and $\mathcal{L}_{f}$, respectively. Additionally, a GAN-like loss $(\mathcal{L}_{s}+\mathcal{L}_{d})$, comprising a similarity loss $\mathcal{L}_{s}$ (to align global and local features) and a binary cross-entropy loss $\mathcal{L}_{d}$ (to differentiate between global and local feature distributions), is introduced to further refine the feature representations, ensuring the capture of complementary and distinctive characteristics. In this study, we employ a joint learning scheme, where the total objective loss is defined as:

where the weights $\alpha$, $\beta$, and $\gamma$ are trade-off hyperparameters of each component loss.

To improve the model’s capability in extracting lesion features, recent studies have concentrated on dual-branch networks [15, 16, 21]. Specifically, these networks typically use segmentation methods, such as Mask R-CNN [22], to identify and crop the lesion region as the local input [15]. However, these approaches often face challenges in accurately locating lesions because of the high similarity between the lesions and the surrounding tissues. To address this issue, we employ the SAM, which has demonstrated excellent performance in detecting pixel-level objects. However, the SAM is primarily trained on annotations for natural images, which differ significantly from medical images in domain characteristics. This domain discrepancy can lead to a substantial drop in performance when directly applied to medical images [23, 24]. Therefore, we utilize the low-rank-based (LoRA) fine-tuning strategy to make the SAM adapted to the laryngo-pharyngeal endoscopic images. Specifically, we freeze the SAM’s image encoder and apply the LoRA layers to the query and value projection layers of the multi-head attention mechanism within each transformer block of the encoder, while keeping all parameters in the mask decoder and prompt encoder trainable. Furthermore, the rank of the LoRA layers is empirically set to 4 to balance efficiency and performance.

After the fine-tuning process, we employ the LoRA-based SAM model to segment the entire image $x_{g}^{(i)}$, obtaining the corresponding lesion mask $x_{s}^{(i)}$, which can be defined to be

where $\mathcal{F}_{SAM}(\cdot)$ represents the function of the LoRA-based SAM that generates the lesion mask from the input image. Next, we apply pixel-wise multiplication between the entire image $x_{g}^{(i)}$ and the corresponding lesion mask $x_{s}^{(i)}$. This process isolates the lesions, effectively removing background information. The resulting image, containing only the lesions, is then used for lesion-based cropping to extract the lesion area image $x_{l}^{(i)}$. The entire process can be defined as follows:

where $Crop(\cdot)$ denotes lesion-based cropping function, and $\odot$ indicates the pixel-wise multiplication.

To capture comprehensive context information, we propose the GFE, designed to extract global features from the whole LPC image. The GFE includes three components of an encoder ($E_{g}$), a Feature Pyramid Network ($FPN_{g}$), and a Convolutional Block Attention Module ($CBAM_{g}$). Specifically, we utilize the ResNet-50 [25] with the initial weights transferred from the model zoo trained on ImageNet as the backbone of the encoder $E_{g}$. Here, we remove the final fully connected layer. Then, we utilize the Feature Pyramid Network [26] to further learn the fine-to-coarse granular information from feature maps output by $E_{g}$. The FPN architecture, as proposed in the work by Lin et al. [26], requires four-scale feature maps as inputs. In this study, we leverage the feature representations extracted from the last four blocks of the ResNet-50 model as the four-scale inputs to the FPN. As the FPN is initially proposed for object detection task, it derives five projection heads from the top-down pathway for further processing. However, this study is a classification task which merely requires one projection head for subsequent process. Thereby, we retains the bottom projection head which preserves the rich semantic information of images. To reduce the model’s focus on irrelevant information among feature maps, we employ the CBAM [27] with a channel attention module and a spatial attention module to highlight the inter-channel and inter-spatial relationships of features pertinent to laryngo-pharyngeal tumors. Finally, a global average pooling is utilized to map the feature maps into a global feature vector $F_{g}^{(i)}$, which can be defined to be

where $\mathcal{F}_{GFE}(\cdot)$ represents the global feature extractor.

As shown in the middle bottom part of Fig. 1, the Local Feature Extractor (LFE) employs the identical network architecture as the GFE described in Section II-C to learn the local feature information, that is, the feature information of the lesion region. This allows us to effectively capture the relevant characteristics of the localized lesion area, which can be important for accurately identifying and analyzing the medical condition. Specifically, the LFE consists of an encoder ($E_{l}$), a FPN ($FPN_{l}$), and a CBAM ($CBAM_{l}$). Importantly, the LFE does not share the parameters with the GFE, thereby enhancing its ability to learn the lesion-specific information. Formally, given a lesion area image $x_{l}^{(i)}$, passing it through the LFE processing, a local feature vector $F_{l}^{(i)}$ is obtained, which is defined to be

where $\mathcal{F}_{LFE}(\cdot)$ represents the local feature extractor.

The GAN-Like Feature Optimization (GFO) module is proposed to improve the learning of complementary features between the global and local branches of the SAM-FNet architecture. Inspired by the Generative Adversarial Networks (GAN) framework [28], the GFO module introduces an adversarial training process to better align the features learned by the global and local branches. Concretely, the GFO utilizes two loss functions of the similarity loss $\mathcal{L}_{s}$ and the discrimination loss $\mathcal{L}_{d}$, which is shown in the middle part of Fig. 1. The similarity loss aims to align the global feature vector $F_{g}^{(i)}$ (see Eq. 4) and the local feature vector $F_{l}^{(i)}$ (see Eq. 5), promoting the GFE and the LFE to better capture the coherent feature information. On the contrary, the discrimination loss encourages the model to learn the complementary feature information between the GFE and LFE. In this context, the similarity loss is calculated specifically for laryngo-pharyngeal endoscopic images that contain tumors. The labels for these images are either benign (with a label value of 1) or malignant (with a label value of 2). The similarity loss is calculated by cosine function, which is defined to be

where $N_{t}$ is the number of laryngo-pharyngeal endoscopic images with tumor labels, and $\|F_{g}^{(i)}\|$ and $\|F_{l}^{(i)}\|$ are the Euclidean norms of the global and local feature vectors, respectively. However, simply reducing the distance between $F_{g}^{(i)}$ and $F_{l}^{(i)}$ in the feature space may lead to the loss of their specific advantages. Therefore, inspired by the adversarial approach in GAN, we introduce a discriminator network. The objective of this discriminator is to ensure that $F_{g}^{(i)}$ and $F_{g}^{(i)}$ maintain their unique characteristics: the global features should retain their broad contextual understanding, and the local features should focus on detailed information of tumors. Specifically, the discriminator, denoted as $FC_{d}$, is a fully connected layer that processes either the global feature vector $F_{g}^{(i)}$ or the local feature vector $F_{l}^{(i)}$. The output of the discriminator is given by:

where $||$ denotes ”or”. The binary cross-entropy loss $\mathcal{L}_{d}$ is then computed by comparing this predicted probability $\hat{y}^{(i)}_{d}$ with the ground truth label $y^{(i)}_{d}$, which is defined as:

where $y^{(i)}_{d}=0$ indicates a global feature, and $y^{(i)}_{d}=1$ indicates a local feature. This loss function optimizes the discriminator’s ability to correctly classify the features as either global or local, ensuring that each feature retains its distinctive characteristics. With this adversarial optimization strategy, SAM-FNet can effectively extracts discriminative features between the global and local branches, enhancing the richness and complementarity of the fused feature representation.

In the training phase, following the DLGNet [15], we adopt the multi-task learning framework that incorporates three cross-entropy losses from the global, local, and fused branches. This framework has the potential to enhance prediction accuracy and generalization across each task [20]. As shown in the right part of Fig. 1, the fused feature vector $F_{f}^{(i)}$ is obtained by concatenating the global feature vector $F_{g}^{(i)}$ (see Eq. (4)) and the local feature vector $F_{l}^{(i)}$ (see Eq. (5)). The vectors $F_{g}^{(i)}$, $F_{l}^{(i)}$, and $F_{f}^{(i)}$ are then passed through their respective classifiers, denoted as fully connected layers $FC_{g}$, $FC_{l}$, and $FC_{f}$. This produces the probability distributions $\hat{y}_{g}^{(i)}$, $\hat{y}_{l}^{(i)}$, and $\hat{y}_{f}^{(i)}$, respectively, which can be defined as follows:

To optimize the SAM-FNet model, this study employs three separate cross-entropy loss functions, denoted as $\mathcal{L}_{g}$, $\mathcal{L}_{l}$, and $\mathcal{L}_{f}$, corresponding to the respective fully connected layers of $FC_{g}$, $FC_{l}$, and $FC_{f}$. The formulation of the loss functions can be expressed as follows:

In the inference phase, inspired by ensemble learning, we generate the final output by averaging the values of $\hat{y}_{g}^{(i)}$, $\hat{y}_{l}^{(i)}$, and $\hat{y}_{f}^{(i)}$. This approach leverages the strengths of different branches, helping to mitigate individual prediction errors and enhance overall model robustness and accuracy.

In order to demonstrate the effectiveness of our proposed architecture intuitively, Gradient-weighted Class Activation Mapping (Grad-CAM) [30] was applied to show which parts the model pay attention to. Fig. 3 shows examples of Grad-CAM visualizations on laryngoscopic images. It is observed that SAM-FNet correctly identifies tumors, and is able to focus on and highlight effective tumor characteristics more precisely than other state-of-the-art counterparts.

Furthermore, we conducted experiments and visualizations using the LoRA-based SAM on the FAHSYSU dataset to validate its effectiveness in lesion localization. Specifically, the LoRA-based SAM achieved a mean Dice coefficient of 0.5918 for benign tumors and 0.7966 for malignant tumors. In particular, the LoRA-based SAM demonstrates promising results in the segmentation of malignant tumors, potentially enhancing the lesion feature extraction capabilities of the LFE. To visually demonstrate the effectiveness of the LoRA-based SAM, we also present several segmentation results in Fig. 4. The results indicates that the LoRA-based SAM exhibits strong segmentation performance on tumors, even when tumors are very small.