A Prompt-driven Universal Model for View-Agnostic Echocardiography Analysis

Echocardiography is the most frequently used imaging modality performed in cardiology, which helps assessment of cardiac function by examining different views by multiple standard views scanning. Given the complexity in image analysis and workload of sonographers, there is a growing interest in developing automated methods for segmentation in Echo [8, 9, 11]. Existing methods have demonstrated superior performance in accurately delineating anatomical structures within specific views when they are trained on corresponding datasets. This process involves a step of identifying the desired views in the patient study before conducting the analysis, which requires an additional steps to select the appropriate files in the scan files [3, 6]. The exploration of a general model capable of independently performing echocardiography segmentation tasks across multiple standard views has not been explored.
Currently, the general solution is training N models on N standard views. However, this solutions have a limitation as the number of models are increases with the number of standard views. One naive approach to creating a universal model is to train the same network on data from various standard view scans. This approach may result in a performance degradation of the model, as each standard view manifests its distinct visual characteristics [9, 15]. Echocardiography poses distinct challenges, including view domain shift among scan views and sparse annotation across frames. While there may not be identical solutions to these problems, similar universal models have been developed [23, 2, 13, 22]. DoDNet[23] introduces a dynamic head with an encoder-decoder architecture. Different task information is encoded to one-hot vector, and it controls the task-specific controller. CLIP-drvien universal model[13] extends this idea by utilizing a pre-trained text model and managing segmentation heads using semantically embedded class features. Although this CLIP-based strategy has demonstrated success in CT organ segmentation tasks, it has limitations in the medical framework due to disparities between natural and medical texts. UniSeg[22] approaches with learnable prompts have been developed to address various segmentation tasks in CT,MR, and PET. However, it introduces a universal method using three anatomically similar datasets, where the images may differ only in texture while the underlying anatomy remains the same. Consequently, approach is likely to be less suitable for addressing view shifts in echocardiography, potentially resulting in suboptimal performance as in Table 2.
To address above problems, we propose a prompt-driven universal model that allows superior segmentation of cardiac structures with state-of-the-art performance. Our model incorporates a prompt learning with prompt pool and pre-trained language model’s knowledge by pixel-text alignment. We first employ a prompt pool-based prompt learning approach enables the development of a universal model capable of handling various scan view data. Accordingly, it effectively tackles the aforementioned issue by enabling dynamic adaptation to diverse inputs. Second, The score maps facilitates pixel-text alignment, which allows our model to fully leverage language information for medical segmentation tasks. It effectively address the above problem by enabling input video to select particular view specific prompt and focus on particular semantic features with language model. To the best of our knowledge, this is the first work that approaches unified model segmentation in echocardiography. Our method simplifies current cardiac analysis by removing the necessity for a view identification step to acquire the desired view from various DICOM scans of the patient. Our method is evaluated on three standard views from three different datasets and show promising performance compared to other universal methods.
Our contributions can be summarized as follows:
• We present a prompt-driven universal model, comprising a prompt pool to accommodate different standard views, and leveraging pixel-text alignment with the prior knowledge of a pre-trained text model for view-agnostic echocardiography segmentation.
• The proposed method streamlines cardiac analysis by minimizing the requirement for a view identification step during the retrieval of the desired view from patient scans.
• We demonstrate that our model achieves SOTA performance for cardiac segmentation tasks compared to the previous universal approach through extensive experiments on various datasets.

As illustrated in Figure 1, our proposed approach comprises the following components: text, video encoder, a prompt pool consisting of trainable key and value, an MLP layer, and a video decoder. We utilize a clinicalBert [1] for enhanced extraction of medical text representations. Our goal is to segment objects in all frames across different scan views. In order to achieve this goal, we introduce two key components in our model: 1) a pixel-text dense alignment mechanism that effectively bridges the gap between a pre-trained language model and pixel-level representations for dense prediction tasks, and 2) a prompt matching technique that leverages a prompt pool to adaptively select the optimal view-specific prompt for each task.

Materials. We evaluated the proposed method using three publicly available datasets [12][19][16]. These datasets consist of 2D B-mode scan data annotated different cardiac chambers at end-diastole (ED) and end-systole (ES). The annotation includes the left ventricle endocardium (LV${}_{\text{endo}}$) and the left ventricle epicardium (LV${}_{\text{epi}}$) in apical two-chamber (A2C), apical four-chamber (A4C), and parasternal short-axis (PSAX) views. We followed a predefined split set as shown in Table 1.
Implementation and Evaluation Metric To guarantee a fair comparison across all experiments, we standardized the training and testing settings. The experiments were conducted using PyTorch, with a consistent batch size of 5 over 100 epochs with Nvidia A100. We utilized the Unet architecture [20] as the backbone to incorporate our key components. For optimization, we employed the MADGRAD optimizer [4], setting the learning rate to 1e-4. Images are resized to 224$\times$224 pixels with 16 frames and normalized to ensure a zero mean and unit variance. To enhance the robustness of our model, we apply various augmentation technique, including random flip, rotation within a range of $-$30 to +30, and shearing along x-y dimensions. We utilize the Dice Similarity Coefficient (DSC) for the evaluation of our model’s performance. We compared our method across three scan views: A4C, A2C, and PSAX. These scan views were selected based on the currently available echo dataset. The performance of the models was evaluated on ED and ES cardiac phases, where annotations were available.
Comparison study We present the performance of the proposed method for cardiac segmentation across different scan views. To the best of our knowledge, our method is the first approach capable of performing cardiac segmentation with view-agnostic input that can perform universal segmentation. We compared our proposed method under two different settings: 1) trained and tested on the same views noted as view-specific model approach, and 2) trained on all views and tested on all views noted as view-integrated model approach. We chose two methods, SwinUNETR [5] and U-transformer [17] for baseline segmentation model based on previous study [7]. We also compare the performance of proposed method against universal models, including DoDNet [19], CLIP-driven universal model [13], UniSeg [22], and UniverSeg [2] for cardiac segmentation tasks on three datasets. The CLIP-driven universal model [13] substitutes one-hot embeddings [23] with CLIP text embeddings. UniSeg [22] employs a learnable prompt for generating task-specific embeddings. Additionally, our comparision includes a few-shot based universal segmentation model called UniverSeg [2].
As depicted in Table 2, our method exhibits the capability to generate excellent segmentation results even with view-agnostic input, as illustrated in Fig. 2. In comparison to baseline models with view-integrated approaches, our model demonstrates better performance compared to the view-specific based U-transfomrer except on LV_endo (93.2 vs. 93.3) and LV_epi (88.3 vs. 88.5) in A2C and A4C, respectively. Moreover, by incorporating prompts to adaptively integrate input view information, our model outperforms all universal models in delineating regions of interest (ROI) across all views. Since, our model adapts the prompt to match input view types, resulting in enhanced segmentation results and superior performance. Furthermore, our method improves mean segmentation performance compared to employing a few-shot segmentation method (89.64 vs 81.7). These experimental findings demonstrate that our approach effectively leverages adaptable prompts to produce superior segmentation results.
Ablation study To evaluate the effectiveness of the each component, we conducted an ablation study to quantify the impact of different elements in our model on segmentation performance. First, we evaluated performance metrics without text-encoder path, and then with text-encoder path employing various designs including one-hot encoding, as well as leveraging prior knowledge from language models including CLIP and ClinicalBert. In the absence of text-encoder path, which removes pixel-text alignment for auxiliary loss, we observed a degradation in performance (89.6 to 85.6) showing that an pixel-text alignment is crucial for our model. Additionally, we compared different types of text encoders trained on both natural and medical text. The findings indicate that the CLIP language model is not as effective in representing medical text compared to the ClinicalBert (88.8 vs. 89.6). Secondly, we assessed the classification performance based on selected prompt keys. We examined the assigned prompt keys with majority voting determined by $\arg\max_{G\in{A,B,C}}\sum_{i=1}^{3}\mathbb{I}(x_{i}\in G)$. T-SNE visualization [14] is presented in Figure 3. We observed that the accuracy for distinguishing between apical views and parasternal views was 0.96. However, the accuracy for distinguishing between A2C and A4C was 0.54 and 0.6, respectively. Secondly, there is variability between view classes annotated by human readers due to ambiguous scan angles between A2C and A4C. In fact, the probe, guided by manual manipulation, often fails to accurately capture both the A2C and A4C angles when rotated from a single position. This is evident in Table 4, where providing view information instead of selecting keys from the prompt pool resulted in a model performance of 89.4, lower than the performance of 89.6 achieved when view information was not provided.

In this study, we introduce an innovative prompt-driven universal echocardiography segmentation model that is capable of learning cardiac segmentation across different standard views using partially labeled data. This model incorporates the knowledge of pre-trained language models by aligning text representation with visual pixel data. We also suggest a prompt matching technique through a prompt pool to achieve view-agnostic echocardiography segmentation. Our research utilized three standard views to demonstrate the feasibility of our proposed model, highlighting its potential to be expanded to additional standard views to create a universal model for echocardiography. This method simplifies the process by eliminating the need for a separate step to identify views, thereby reducing the variability introduced by humans in selecting views for analysis in a patient’s study. Extensive experiments on the echocardiography segmentation benchmark across various scan views have shown that our approach not only performs superiorly but also proves to be highly effective.