Textual Inversion and Self-supervised Refinement for Radiology Report Generation

Radiology report generation is an image-to-text cross-modal task, as medical images and radiology reports are in two different feature spaces. Existing methods are more tend to improve the overall performance by extracting refined image features [30, 19, 32] or improving the network structure of the text decoder [34, 15, 11, 35] and ignoring the gap between modalities. Therefore, we propose textual inversion to reconstruct image representation within the text embedding space to eliminate the spatial gap. In this module, we map image embeddings $\mathcal{I}$ to pseudo words $\mathcal{P}\in\mathbb{R}^{B\times M\times D}$. via feeding image features into a three-layered full-connected network, which can be formulated as:

After obtaining the pseudo words, we input them into the text decoder after a series of operations to obtain $\mathcal{O^{\prime}}$. We explicitly constrain that the generated pseudo words should be able to represent the image features sufficiently by calculating the contrastive loss between $\mathcal{O^{\prime}}$ and the image feature $\mathcal{I^{\prime}}$. This optimization process guides the network to generate reports that are faithful to images.

We first perform cross-modal interaction by employing cross attention mechanism [31] between pseudo words $\mathcal{P}$ and text embeddings $\mathcal{T}$. Since pseudo words are derived directly from image features, it is beneficial to align visual and linguistic features through this interaction. This process can be expressed as:

We assume that the pseudo words can compensate for the missing information or correct the redundant information for the text feature $\mathcal{T}$. Based on this intuition, we concatenate aligned text feature $\mathcal{T^{\prime}}$ with aligned pseudo words $\mathcal{P^{\prime}}$. The concatenated features can be fused well through multi-layer perceptron (MLP), and thus we obtained the processed pseudo words $\mathcal{P^{\prime\prime}}\in\mathbb{R}^{B\times S\times D}$, where $S=M+N$. It can be expressed using a formula:

The log probability $\mathcal{O^{\prime}}\in\mathbb{R}^{B\times S\times D}$ is obtained by decoding $\mathcal{P^{\prime\prime}}$. We then implement self-supervised refinement by calculating contrastive loss between text embeddings $\mathcal{O^{\prime}}$ and image features $\mathcal{I^{\prime}}$. By minimizing the contrastive loss, we encourage the network to generate $\mathcal{P^{\prime\prime}}$ that closely resemble the expression of $\mathcal{I^{\prime}}$. After continuous back-propagation and optimization, the generated pseudo words $\mathcal{P}$ can adequately represent the image semantics, which is beneficial for the generation of reports faithful to the original images.

We utilize $\mathcal{L}_{rrg}$ to quantify the difference between the generated report and the ground truth [6], thus guiding the model to generate reports that are close to the ground truth. The formulation of $\mathcal{L}_{rrg}$ is as follows:

The log probability of the output against the target sequence $T$ is obtained by $O_{bs,T_{bs}}$ for the position $s$ of the $b$-th sample. To ensure consistent input sequence length, all sequences are filled to the same length. The mask $M_{bs}$ indicates whether a real word exists at the position: if present, it is 1; otherwise, it is 0. The log probability of the filled part is set to 0 by multiplying with the mask $M_{bs}$ to prevent the filled part from affecting $\mathcal{L}_{rrg}$. Finally, we normalize $\mathcal{L}_{rrg}$ by dividing it with the sum of all mask values $\sum M$ to ensure that the loss value is not affected by changes in sequence length.

In addition to optimizing the network to generate more accurate reports through $\mathcal{L}_{rrg}$, we also constrain the textual inversion to generate pseudo words that are close to the image representation through $\mathcal{L}_{sr}$. We obtained the score matrix $\mathcal{S}$ by calculating the correlation between image features and text features via dot product, which can be expressed as $\mathcal{S}=\mathcal{I^{\prime}}\times\mathcal{O^{\prime}}^{T}$, where $\times$ denotes matrix multiplication. we evaluate the cosine similarity between image features and text features and get the score matrix $\mathcal{S}$ of size $B\times B$. We optimize the network by constructing a symmetric cross-entropy loss to maximize the cosine similarity between $B$ real image-text pairs while minimizing the cosine similarity between $B^{2}-B$ unpaired image-text pairs [27].

$\mathcal{M}$ is a matrix of size $B\times B$, where the elements on the diagonal are 1, indicating positive samples, while the off-diagonal elements are 0, indicating negative samples. The overall loss function $\mathcal{L}$ of our network is defined as: $\mathcal{L}=\mathcal{L}_{rrg}+\mathcal{L}_{sr}$. Instead of relying on manually labeled datasets, we leverage contrastive learning to measure the similarity between text and image, guiding the network to optimize parameters for generating reports faithful to the visual content.

Dataset. We conducted experiments on two widely-used datasets: the small dataset IU X-ray^{†\dagger†\dagger}$\dagger$https://openi.nlm.nih.gov/. [8] (containing 7,470 chest X-ray images and 3,955 corresponding reports) and the large dataset MIMIC-CXR^{†\dagger†\dagger}$\dagger$https://physionet.org/content/mimic-cxr/2.0.0/. [13] (containing 377,110 images and 227,835 corresponding reports). To ensure consistency and fairness in comparisons, we followed the data processing methods utilized by the three baselines [4, 5, 6]. After excluding samples without corresponding radiology reports, IU X-ray is divided into training, validation and testing sets with a proportion of 7:1:2 [20] while MIMIC-CXR is divided according to the official splits [6].

Evaluation Metrics. We evaluate TISR on natural language generation (NLG) metrics including BLEU [26], METEOR [1] and ROUGE-L [21], which are widely used to assess the fluency and accuracy of generated reports. We not only focus on the quality of the generated reports but also on their ability to accurately capture lesions in the images. Therefore, we employ clinical efficacy (CE) metrics to evaluate the detection accuracy of generated reports. CheXbert [29] is applied to extract labels of 14 medical observations from reports. Precision, recall and F1 are calculated by comparing the labels of the generated reports with ground truth.

Comparison with Baselines. To verify the generalization and effectiveness of TISR, we use R2Gen [6], R2GenCMN [5] and VLCI [4] as the baseline models in our experiments. These baseline models are improved with TISR, and the results are compared with the original baselines, as shown in Table 1. Experimental results show that all metrics are enhanced by improving networks with TISR, which indicates that TISR can eliminate the gap between modalities and generate more accurate reports. It is remarkable that our approach does not require additional data and can seamlessly integrate into these baselines [4, 5, 25], which is of great importance for network migration and practical applications. What’s more, we can recognize from the results that prior methods have overlooked the impact of the gap between modalities on radiology report generation. Hence, future research should focus more on cross-modal interactions.

Ablation Study. To explore the effectiveness of each component in TISR and the rationality of the network structure, we conducted various ablation experiments. First of all, we explored the effectiveness of textual inversion and self-supervised refinement, as shown in Table 2. The significance of textual inversion is investigated by calculating the contrastive loss between image features $\mathcal{I}$ and text embeddings $\mathcal{T}$, while the role of self-supervised refinement is explored through the calculation of the contrastive loss between image features $\mathcal{I}$ and pseudo words $\mathcal{P}$.

Secondly, we carried out experiments to explore the structure of TISR. We replace the three-layer linear structure with a three-layer transformer encoder to explore the structure of the textual inversion module. It’s easy to see that the result is worse than MLP with the same dimension of the hidden layer from Table 3. We hypothesize that this is because the medical image features are not complex, so using a transformer may lead to overfitting, and it will also increase computational costs. We investigate the significance of cross-modal interaction and MLP by removing cross attention and MLP from the complete self-supervised refinement network respectively. Pseudo words’ significance is investigated by directly incorporating $\mathcal{T}$ into the self-supervised refinement network. Furthermore, the importance of decoding text embeddings is explored by computing the contrastive loss between $\mathcal{P^{\prime\prime}}$ and $\mathcal{I^{\prime}}$. We can speculate that each module plays an important role in generating more refined pseudo words from Table 4 since removing any one of them the performance of the network is degraded.

Quantitative Analysis. We draw attention maps to explore the region of the medical image that the word of the generated report is interested in. Fig. 3 illustrates that the model improved by TISR is more sensitive to the correct regions and can generate reports that are closer to the ground truth. This demonstrates that our model can eliminate the cross-modal gap and thus generate reports faithful to images.