Learning Visual-Semantic Embeddings for
Reporting Abnormal Findings on Chest X-rays

Jing et al. (2017) proposed a co-attention based Hierarchical CNN-RNN model that jointly trains two tasks: report generation and Medical Text Indexer (MTI) prediction. The model first predicts MTI tags and the semantic embeddings of the predictions are fed into the cascaded decoder for generation. Similarly, Yuan et al. (2019) extracted medical concepts from the CXR reports using SemRep²²2https://semrep.nlm.nih.gov/ as alternatives to MTI tags. To address data bias, Harzig et al. (2019) proposed a CNN-RNN model with dual word-level decoders: one for abnormal findings and the other for normal findings. It jointly predicts whether the next sentence is a normal or abnormal finding, and uses the corresponding decoder to generate the next sentence. However, it still formulates the task as text generation and has the limitations of such models.

There has been increasing interest in studying hybrid retrieval-generation models to complement generation. Li et al. (2018) introduced a hybrid retrieval-generation framework which decides at each step whether it retrieves a template or generates a sentence. Li et al. (2019) proposed a model based on abnormality graphs, which first predicts existing abnormalities on the radiology images, then retrieves and paraphrases the templates of that abnormality. However, such models usually require non-trivial human effort to construct high quality prior knowledge (e.g. sentence templates, abnormality terms). Unlike previous work, we leverage unsupervised methods and minimal rules to group sentences into different abnormality clusters, seeking to minimize human effort.

Learning visually grounded semantics to facilitate cross-modal retrieval (i.e., image-to-text and text-to-image) is a challenging task for cross-modal learning (Faghri et al., 2018; Wu et al., 2019). Different from image captioning tasks, radiology reports are often longer and consist of multiple sentences, each related to different abnormal findings; meanwhile, there are fewer distinct objects in radiology images and the differences among images are more subtle.

Assume each report $R_{a}=\{a_{1},a_{2},\ldots,a_{M}\}$ includes $M$ abnormal findings (i.e., sentences). $R_{a}$ is a subset of the complete report $R=\{s_{1},s_{2},\ldots,s_{N}\}$, where $s_{i}$ can either be an abnormal sentence $a_{i}$ or not.

Let $v\in\mathbb{R}^{d1}$ be the semantic embedding of an abnormal finding $a$ of this report, and $E=\{m_{j}\in\mathbb{R}^{d2}\}_{j=1}^{w\times h}$ be the feature maps of the radiology image $I$ associated with $R_{a}$, where $j$ means the $j$-th region of the feature map. We first transform them into the joint embedding space $\mathbb{R}^{d}$ with separate linear projection layers:

$$\mathbf{v}=\mathrm{norm}(\mathrm{linear}(v));\mathbf{m}_{j}=\mathrm{norm}(\mathrm{linear}(m_{j})),$$

where we apply $l_{2}$ normalization on the joint embeddings to improve training stability, following work in visual-semantic embeddings (Faghri et al., 2018).

Next, we need to measure the similarity between the semantic and visual embeddings. As different regions may include details about different abnormal findings, we propose Conditional Visual-Semantic Embeddings (CVSE) to learn the fine-grained matching between regions and a target abnormal finding:

	$\displaystyle d(a,I)$	$\displaystyle=-\sum_{1\leq j\leq w\times h}{\alpha_{j}||\mathbf{m}_{j}-\mathbf{v}||^{2}},$		(2)
	$\displaystyle\hat{\alpha_{j}}$	$\displaystyle={\mathbf{v}_{\alpha}}^{\top}(W_{\alpha}[\mathbf{m}_{j};\mathbf{v}]+\mathbf{b}_{\alpha}),$
	$\displaystyle\bm{\alpha}$	$\displaystyle=\mathrm{softmax}(\bm{\hat{\alpha}}),$

where $\alpha_{j}$ is the attention score that represents the relevance between the region $\mathbf{m}_{j}$ and the abnormal finding $\mathbf{v}$, $d(a,I)$ is the similarity score between image $I$ and the abnormal finding $a$, which is calculated as an attention-weighted sum over the similarity scores of each region with the abnormal finding. We use the (negative) squared $l_{2}$ distance to measure similarity. Since each report has both frontal and lateral views, the final similarity score is calculated as the average:

$$d_{*}(a,I)=\frac{1}{2}(d(a,I_{f})+d(a,I_{l})).$$

(3)

Finally, we optimize the hinge-based triplet ranking loss to learn the visual-semantic embeddings:

	$\displaystyle\mathcal{L}$	$\displaystyle=\sum_{I}[d_{*}(a^{-},I)-d_{*}(a^{+},I)+\delta]_{+}$		(4)
		$\displaystyle+\sum_{a}[d_{*}(a,I^{-})-d_{*}(a,I^{+})+\delta]_{+},$

where $\delta$ is the margin, $[x]_{+}=\mathit{max}(x,0)$ is the hinge loss, $a^{+}$ ($I^{+}$) denotes a matched abnormal finding (image) from the training set while $a^{-}$($I^{-}$) denotes an unmatched abnormal finding (image) sampled during training.

To identify abnormal findings in radiology reports, we train a sentence-level classifier which determines whether a sentence includes abnormal findings or not. We fine-tuned BERT (Devlin et al., 2019) on an annotated sentence-level dataset released by Harzig et al. (2019), which is a labeled subset of the Open-I dataset (Demner-Fushman et al., 2016). We achieve an F1-score of 98.3 on the held-out test set. We then use it to distantly label the reports from the MIMIC-CXR dataset (Johnson et al., 2019), which is the largest public CXR imaging report dataset.

Given that most medical reports are written following certain templates, many abnormal findings are often paraphrases of each other. We obtain the sentence embeddings via pre-trained models and apply K-Means to cluster the sentences about similar abnormal findings into 500 groups. We also design several simple mutual exclusivity rules to refine the groupings. We consider critical attributes such as position (e.g. left, right), severity (e.g. mild, severe) which often are not present at the same time. Then we apply these rules to separate each group formed by K-Means. Ultimately, we obtained 1,306 groups of abnormal findings.

We consider (1) the Hier-CNN-RNN model (Jing et al., 2017; Liu et al., 2019), as denoted in eq. 1; (2) Hier-CNN-RNN + co-attention (Jing et al., 2017) with co-attention on both the images and the predicted medical concepts; (3) Hier-CNN-RNN + dual, with the dual word-level decoders (Harzig et al., 2019). We also implement two simple variants: (4) Hier-CNN-RNN + complete, which considers the complete medical reports (i.e., both normal and abnormal findings) as input; (5) Hier-CNN-RNN + shuffle, whose input reports have a shuffled sentence order. Vinyals et al. (2015) has shown that input order affects the performance for encoder-decoder models and (5) could potentially address the training issue due to the static input order.

In all experiments, the abnormal set and complete set consist of the same (image, report) pairs. As discussed in Section 3.1, the abnormal set only considers the abnormal finding sentences of the report, which is a subset of sentences of the complete report. We compare these two sets to show that models trained on the abnormal sentences would achieve substantial improvement than those trained on the complete reports, which has not been studied before.

We use the CheXpert labeler to evaluate the clinical accuracy of the abnormal findings reported by each model, which is the state-of-the-art medical report labeling system (Irvin et al., 2019; Johnson et al., 2019). Given sentences of abnormal findings, CheXpert will give a positive and negative label for 14 diseases. We then calculate the Precision, Recall and Accuracy for each disease based on the labels obtained from each model’s output and from the ground-truth reports.

We consider CXRs from the MIMIC-CXR dataset with both frontal and lateral views which include at least one abnormal finding. Ultimately, we obtain 26,946/3,801/7,804 CXRs for the train/dev/test sets, respectively. For the CVSE model, we set $\alpha$ to 0.2 and for each sample we randomly pick 8 negative samples. We use the pre-trained DenseNet-121 to obtain the feature maps of the CXR images. We use the pre-trained biomedical sentence embeddings (Zhang et al., 2019) to obtain initial embeddings for the abnormal findings.³³3https://github.com/ncbi-nlp/BioSentVec The final dimension of the joint embedding $d$ is set to 512. We take the top 3 retrieval results as the predicted abnormal findings. For all CNN-RNN based models, we use a VGG-19 model as the encoder, a 1-layer LSTM as the sentence decoder and a 2-layer LSTM as the word decoder. All dimensions are set to 512. Greedy search is applied during the decoding stage, following Jing et al. (2017). Our code are available online.⁴⁴4https://github.com/nijianmo/chest-xray-cvse

We conduct experiments on both the abnormal and complete set of the MIMIC-CXR dataset which consider the abnormal findings in reports and the complete reports, respectively. As shown in Table 1, adding co-attention over medical concepts and dual decoders both improve the vanilla Hier-CNN-RNN model’s clinical accuracy on the complete dataset. However, simply training the Hier-CNN-RNN model on the abnormal set would achieve better clinical accuracy. This shows the importance of addressing dataset bias. We also observe that the Hier-CNN-RNN model with a shuffled sentence order doesn’t improve performance, which indicates the difficulty of addressing order bias during training of encoder-decoder models.

Our CVSE model outperforms all baselines on clinical accuracy metrics, which demonstrates its capability to accurately report abnormal findings. Notably, CVSE achieves significant improvements on precision and recall. On the other hand, the baseline models will always miss abnormal findings thus leading to 0 precision and recall for many disease classes. More detailed results are included in the appendices.

Refining the groups with mutual exclusivity rules further improves the performance of CVSE. We also report the automatic evaluation of NLG metrics. As shown in Table 1, CVSE achieves higher scores than other baselines on the abnormal set.⁵⁵5Models trained on the complete set can match the predominant normal findings thus leading to higher NLG metrics.

We performed a human evaluation in which we sampled 20 images and asked a board-certified radiologist to give Likert scores (1 to 10) based on how closely the results generated by the model relate to the input images. The ground-truth obtained an average score of 7.85; our CVSE achieved a score of 6.35, higher than Hier-CNN-RNN trained on the abnormal set which obtained 6.15. The radiologist commented that Hier-CNN-RNN’s outputs were simpler predictions, with less details; meanwhile, CVSE covered more abnormalities but may included false information sometimes.

In Figure 2, we visualize the attended regions on CXRs to investigate what part is important for reporting abnormal findings. We observe that our attention mechanism is able to detect relevant regions (e.g. heart, left opacity, wires) to determine which abnormal findings reside in the CXRs.