Deep Multimodal Collaborative Learning for Polyp Re-Identification

Traditional research works (Prosser et al, 2010; Zhao et al, 2013, 2014) related to hand-crafted systems for image retrieval task aim to design or learn discriminative representation or pedestrian features. For example, Prosser et al (2010) propose a reformulation of the person re-identification problem as a learning to rank problem. Zhao et al (2013) exploit the pairwise salience distribution relationship between pedestrian images, and solve the person re-identification problem by proposing a salience matching strategy. Besides directly using mid-level color and texture features, some methods (Zhao et al, 2014; Xiang et al, 2020a) also explore different discriminative abilities of local patches for robust generalization. Unfortunately, these handcrafted feature based approaches always fail to produce competitive results on large-scale datasets. The main reason is that these early works are mostly based on heuristic design, and thus they could not learn optimal discriminative features on current large-scale datasets.

Recently, there has been a significant research interest in the design of deep learning based approaches for image or video retrieval (Shao et al, 2021; Xiang et al, 2023a; Ma et al, 2020; Xiang et al, 2020a). For example, Shao et al (2021) propose temporal context aggregation which incorporates long-range temporal information for content-based video retrieval. Xiang et al (2020a) propose a feature fusion strategy based on traditional convolutional neural network for pedestrian retrieval. Ma et al (2020) explore a method for performing video moment retrieval in a weakly-supervised manner. Lin et al (2020) propose a semantic completion network including the proposal generation module to score all candidate proposals in a single pass. Besides, Kordopatis-Zilos et al (2022) propose a video retrieval framework based on knowledge distillation that addresses the problem of performance-efficiency trade-off. As for the self-supervised learning, Wu et al (2018) present an unsupervised feature learning approach called instance-wise contrastive learning by maximizing distinction between instances via a non-parametric softmax formulation. In addition, Chen et al (2023) propose a self-supervised contrastive representation learning scheme named Colo-SCRL to learn spatial representations from colonoscopic video dataset. However, on the one hand, all above approaches divide the learning process into multiple stages, each requiring independent optimization. On the other hand, despite the tremendous successes achieved by deep learning-based approach, they have been left largely unexplored in terms of semantic feature embedding.

In this work, we propose a novel Deep Multimodal Collaborative Learning framework named DMCL for multimodal polyp re-identification, which remarkably encourage modality collaboration and reinforce generalization capability in medical scenarios. On the basis of it, a dynamic multimodal feature fusion strategy is proposed to fully exploit the prior knowledge of visual-text feature, thereby improving the accuracy of polyp ReID in complex textured regions of unencoded subimages. To the best of our knowledge, this is the first attempt to employ the visual-text feature with collaborative learning mechanism for colonoscopic polyp re-identification task. To this end, the deep multimodal collaborative learning system proposed in this study not only improves the polyp ReID performance to a certain extent but also effectively maintains the social logical consistency of the extracted feature, which ensures the reliability and usability of the paradigm in various application scenarios, providing strong support for its application.

Assuming that we are given a training dataset set $\mathcal{D}$, which contains its own label space $\mathcal{D}=\left\{\left(\boldsymbol{x}_{i},y_{i}\right)\right\}_{i=1}^{N}$ in terms of visual and textual feature, where $N$ is the number of images in the training dataset set $\mathcal{D}$. Each sample $\boldsymbol{x}_{i}\in\mathcal{X}$ is associated with an identity label $y_{i}\in\mathcal{Y}=\left\{1,2,\ldots,M\right\}$, where $M$ is the number of identities in the training dataset set $D$, along with the corresponding text description $\boldsymbol{x}_{t}$ of colonoscopic polyp dataset. The main goal of this work is to learn a robust polyp ReID model on the basis of visual-text representation.

In this work, we design a DMCL framework for polyp re-identification in medical scenarios, which includes a visual feature extractor, a textual feature encoder and dynamic multimodal feature fusion strategy.

Image Encoder. Image encoder is adopted as the visual feature extractor in our DMCL framework. To be more specific, we adopt the ResNet-50 (He et al, 2016) as the backbone for the image encoder module. Benefit from the merit of residual connection design, ResNet-50 network can effectively address the problem of gradient vanishing, enabling the model to deeply learn image features. On the other hand, our image encoder can extract rich details and global structures to form efficient feature encoding, which exhibits powerful performance and generalization ability in feature extraction. To be more specific, given a 224 $\times$ 224 $\times$ 3-sized polyp image, the main goal of the image encoder is to encode the input into a $I^{1\times 2048}$-sized vector for subsequent feature fusion and processing, which provides a strong foundation for downstream polyp ReID task.

Text Encoder. As for the text encoder module, we employ the BERT model (Devlin et al, 2018) to encode the text description of corresponding polyp. In essence, BERT model is a pre-trained language model based on the Transformer architecture, which learns language representations through masked language modeling and next sentence prediction tasks on a large amount of unlabeled text. In this work, given a text description of the polyp with $n$ characters, BERT will encode it into a $n$ $\times$ 768-dimensions vector matrix for subsequent feature fusion and processing, formulated as $[T_{1},T_{2},T_{3},\cdots,T_{n}]^{1\times 768}$. The bidirectional encoder design of BERT enables it to capture contextual information of words, generating more accurate text representations.

Dynamic Multimodal Feature Fusion. In the general scenarios, there exists the obvious data discrepancy between image and text in our dataset, which inevitably has a negative impacts on the model’s performance. To overcome this issue, we creatively introduce a multimodal fusion strategy based on the self-attention mechanism, which is a crucial part of our proposed deep collaborative training framework. To be more specific, we obtain the image encoding of $I$ after passing through the image encoder, whose size is then dimensionally reduced from the original 2048-dimension to 768-dimension. Similarly, we can also obtain the text encoding through the text encoder, which is denoted as:

where $T$ denotes the text encoding, and $T_{n}$ represents the number index of character in the polyp’s text description. Subsequently, we concatenate the image encoding $I$ and text encoding $T$ to formulate a new encoding of fusion feature, which can be formalized as:

where $K$ represents the features generated by the multimodal fusion component. Remarkably, multimodal fusion component consists of multiple self-attention layers to accomplish the fusion of visual feature and text feature. After passing through three linear layers, these three vectors are then inputted into the self-attention module. The output from the self-attention module is:

where $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ represents the Query, Key, and Value of self-attention mechanism respectively, and $d$ denotes the dimension of the input vector. After modal fusion, we can obtain the new updated features:

Generally speaking, visual feature is of paramount importance for the task of polyp re-identification. Inspired by this, we choose the feature of $I_{f0}$ at the corresponding position of the original image feature, which can be adopted as the testing feature of the corresponding polyp in the testing phase for downstream polyp ReID task. The whole procedure of our collaborative multimodal training system is illustrated in Algorithm 1.

In essence, many previous works (Luo et al, 2019; Xiang et al, 2023a; Luo et al, 2019; Xiang et al, 2020b) have been found that performing training with multiple losses has great potential to learn a robust and generalizable ReID model. During the training process, we adopt a identification loss (Zheng et al, 2017a) and triplet loss (Hermans et al, 2017) as the optimization metric for our DMCL model. Specially, the triplet-loss function aims to reduce the feature distance between similar polyp images while increasing the feature distance between different polyp images. The identification loss function, on the other hand, is to enhance the model’s ability to recognize and classify polyp categories.

To be more specific, for a single-label $N$ classification task, the identification loss (cross-entropy loss) is written as,

where $M_{batch}$ is the number of labeled training images in a batch, $\hat{y_{ij}}$ is the predicted probability of the input belonging to ground-truth class $y_{ij}$. For a triplet couple {$x_{a}$, $x_{p}$, $x_{n}$}, the soft-marginal triplet loss $\mathcal{L}_{triplet}$ can be calculated as:

where $x_{a}$, $x_{p}$ and $x_{n}$ denote the anchor image, positive sample and negative sample respectively. As for the triplet selection, we randomly select different polyps from $16$ patients and sample $4$ images or texts from image or text modality for each patient to form the triplet couple, which can greatly help to construct the discriminative embedding for multimodal representation learning.

Finally, the overall objective loss function in a training batch is expressed as:

To this end, our deep multimodal collaborative learning scheme fully considers the information from both images and texts, providing stronger support for clinical colonoscopy polyp examinations and potentially improving model’s performance with a clear margin.

We conduct experiments on several large-scale public datasets, which include Colo-Pair (Chen et al, 2023), Market-1501 (Zheng et al, 2015), DukeMTMC-reID (Zheng et al, 2017b) and CUHK03 dataset (Li et al, 2014). We follow the standard evaluation protocol (Zheng et al, 2015) used in the ReID task and adopt mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at Rank-1, Rank-5, and Rank-10 for performance evaluation on downstream ReID task.

In our experiment, ResNet-50 (He et al, 2016) is regarded as the backbone with no bells and whistles during visual feature extraction. Following the training procedure in (Tong et al, 2022), we adopt the common methods such as random flipping and random cropping for data augmentation and employ the Adam optimizer with a weight decay co-efficient of $1\times 10^{-5}$ and $1\times 10^{-7}$ for parameter optimization. Besides, we adopt the ID loss and triplet loss functions to train the model for 180 iterations, and the cosine distance is also adopted to calculate the similarity of polyp features in the dataset for the task of polyp re-identification. Please note that the text information is also employed during testing phase. In addition, the batch size $M_{batch}$ for training is set to 64. All the experiments are performed on PyTorch framework with one Nvidia GeForce RTX 2080Ti GPU on a server equipped with an Intel Xeon Gold 6130T CPU.

Colonoscopic Polyp Re-Identification. In this section, we compare our proposed method with the state-of-the-art algorithms, including: (1) transformer based (soft attention) models ViT (Caron et al, 2021), Colo-SCRL (Chen et al, 2023) and VT-ReID (Xiang et al, 2024b); (2) knowledge distillation-based methods, such as CgS^c, FgAttS${}^{f}_{A}$ and FgBinS${}^{f}_{B}$ (Kordopatis-Zilos et al, 2022); (3) feature level based methods, such as ViSiL (Kordopatis-Zilos et al, 2019), CoCLR (Han et al, 2020), TCA (Shao et al, 2021), CVRL (Qian et al, 2021). According to the results in Table 1, we can easily observe that our method shows the clear performance superiority over other state-of-the-arts with significant Rank-1 and mAP advantages. For instance, when compared to the knowledge distillation-based network FgAttS${}^{f}_{A}$ (Kordopatis-Zilos et al, 2022), our model improves the Rank-1 by +44.6% (54.3 vs. 9.7). Besides, VT-ReID also surpasses recent transformer based (soft attention) models ViT (Caron et al, 2021), Colo-SCRL (Chen et al, 2023) and VT-ReID (Xiang et al, 2024b). Specially, our method outperforms the second best model VT-ReID (Xiang et al, 2024b) by +8.5% (46.4 vs. 37.9) and +30.9% (54.3 vs. 23.4) in terms of mAP and Rank-1 accuracy, respectively. The superiority of our proposed method can be largely contributed to the visual-text representation mined by our DMCL during multiple collaborative training, as well as the dynamic multimodal feature fusion strategy, which is beneficial to learn a more robust and discriminative model in polyp ReID tasks.

Person Re-Identification. To further prove the effectiveness of our method on other related object ReID tasks, we also compare our DMCL with existing methods in Table 2. we can easily observe that our method can achieve the state-of-the-art performance on Market-1501, DukeMTMC-reID and CUHK03 datasets with considerable advantages respectively. For example, our DMCL method can achieve a mAP/Rank-1 performance of 92.1% and 96.3% respectively on Market-1501 dataset, leading +1.0% and +0.6% improvement of mAP and Rank-1 accuracy when compared to the second best method NFormer (Wang et al, 2022) and MGN (Wang et al, 2018). Besides, our DMCL method can also obtain the improvement of +1.2% and +1.4% in terms of mAP and Rank-1 accuracy on CUHK03 dataset when compared to the VT-ReID (Xiang et al, 2024b). These experiments strongly demonstrates the priority of our proposed deep multimodal collaborative learning framework.

In this experiments, to verify the effectiveness of our deep multimodal collaborate learning framework, we perform ablation study with multimodal feature from both the qualitative and quantitative perspectives.

Effectiveness of deep multimodal collaborative learning: Firstly, from the quantitative aspect, we evaluate the effectiveness of deep multimodal collaborative learning DMCL framework. As illustrated in Table 3, when adopting our deep multimodal collaborative learning framework, results show that mAP accuracy improves significantly from 25.9% to 46.4% on Colo-Pair dataset with visual-text induction. Additionally, similar improvements can also be easily observed in terms of Rank-1, Rank-5 evaluation metrics, leading to +24.3% and +9.1% respectively. These results prove that multimodal collaborative learning paradigm has a direct impact on downstream polyp ReID task.

Secondly, from the qualitative aspect, we also give some qualitative results of our proposed deep multimodal collaborative learning framework named DMCL. Fig. 3 provides some qualitative visualization of ranking results of our proposed approach DMCL with collaborative training mechanism on Colo-ReID dataset. To be more specific, we can obviously observe that our model attends to relevant image regions or discriminative parts for making polyp predictions, indicating that DMCL can greatly help the model learn more global context information and meaningful visual features with better semantic understanding, which significantly makes our collaborative training method DMCL model more robust to perturbations.

Effectiveness of dynamic multimodal training strategy: In this section, we proceed to evaluate the effectiveness of dynamic multimodal training strategy by testing whether text modality or image modality matters. According to Table 3, our dynamic multimodal training strategy DMCL with text representation (DMCL w/ text) can lead to a significant improvement in Rank-1 of +5.6% on Colo-Pair dataset when compared with baseline setting. Furthermore, when adopting image representation, our method (DMCL w/ image) can obtain a remarkable performance of 36.6% in terms of mAP accuracy, leadig a significant improvement of +8.1% when compared to DMCL w/ text. The effectiveness of the dynamic multimodal training strategy can be largely attributed to that it enhances the discrimination capability of other collaborative networks during multimodal representation learning, which is vital for polyp domain adaptation in general domain where the target supervision is not available.

Visualization of Feature Response: To further explain why our deep multimodal collaborative learning strategy works, we perform in-depth analysis of feature response in DMCL method, and also show some qualitative examples of EigenGradCAM (Muhammad and Yeasin, 2020) visualizations in Fig.4. In fact, the Grad-CAM is a package with some methods for Explainable AI for computer vision, which provides attributions to both the inputs and the neurons of intermediate layers, so as to make CNN-based models more transparent by producing visual explanations. Specifically, EigenGradCAM decodes the importance of each feature map to a specific class by analyzing the gradients within the last convolutional layer of a CNN, generating a heatmap that highlights the image regions that contribute most significantly to the prediction result. This heatmap visually represents which parts of the image are most influential in the model’s final decision. As illustrated in Fig.4, compared with CNN-based training method without text information, we observe that our model attends to relevant image regions or discriminative parts for making decison, indicating that our DMCL model can effectively explore more global context information and meaningful visual features with better semantic understanding, which significantly make our model more robust to perturbations.