Unifying Relational Sentence Generation and Retrieval for Medical Image Report Composition

Previous works [2, 10] have shown that visual attention can perform fairly well for localizing objects and aiding captions. However, visual attention is usually not sufficient to encode high-level semantic information to recognize abnormalities. To this end, we explore the relationship between the medical terms which can be cooperative with image features for the robust relational topic generation.

We treat the abnormality classification as a multi-label image classification task. Specifically, given a medical image $I$, we first extract its feature maps through a deep CNN and apply a fully connected layer to compute a distribution over all of the abnormal medical terms. To solve multi-label classification, we calculate binary cross entropy for each category. Moreover, according to [41, 48], labels co-occur in images with priors. And relationships among all abnormal medical terms are also crucial for classification as some terms are often entangled with others and co-occurred. For example, “copd” and “pulmonary disease chronic obstructive” often appear together. Therefore, the predicted scores of these two abnormalities should be closer. To exploit these explicit relationships, beyond average binary cross entropy loss, we add another relationship constraint loss. Finally, the abnormality multi-label classification loss is composed of two terms, which is formulated as:

where the first term is an average binary cross-entropy loss for each category, $a_{i}$ denotes probability of abnormal medical term $i$ and $y_{i}$ is the ground truth label; the second term is the relationship constraint loss, $r(i,j)$ represents the relevance between two abnormal medical terms and $R^{\ast}$ denotes the number of non-zeros in relationship matrix $r$. For a pair of abnormality probabilities, namely $a_{i}$ and $a_{j}$, if $r(i,j)$ is larger, the relationship constraint loss can guide $a_{i}$ and $a_{j}$ to be closer. Similarly, $a_{i}$ and $a_{j}$ do not influence each other when $r(i,j)$ is smaller.

The static relationship matrix is computed according to the frequency of co-occurrences for each pair of abnormalities all over the training set [49], which is defined as follows:

where $f(i,j)$ denotes the frequency of co-occurrences, $f(i)$ is the frequency of abnormality $i$, and $F$ denotes the total count of abnormalities. Here we do not consider the extreme case when $f(i,j)=f(i)=f(j)$, and we only make the statistics for a rough prior knowledge.

Sentences in the report are based on the image as well as the previous generated sentence. In order to utilize the relationships among abnormal medical terms and adjacent sentences, we propose a Relational-Topic Encoder to generate more discriminative topics, as illustrated in Figure 3. Our Relational-Topic Encoder has two critical inputs. The first one is the image features which are enhanced by the abnormal medical terms learning. Its visual information can help the encoder focus on the salient regions. Furthermore, we also feed the embedding vector of the previous sentence into the decoder, which imposes the encoder to memorize the previous topic and sentence features. Formally, the Relational-Topic Encoder is composed of an LSTM layer and an attention [19] layer. At each time step, the LSTM layer concatenates the previous state of the Adaptive Generator $\bm{h}_{i-1}^{2}$ and embedding vector of the previous generated sentence $\mathbb{E}_{i-1}$ to form the input vector $\bm{x}_{i}^{1}$:

Supposing a sentence consists of $n$ tokens, a bi-directional LSTM is adapted to encode the sentence as a 2-D matrix [50]. The proposed Relational-Topic Encoder generates a hidden state $\bm{h}_{i}^{1}$ by an LSTM as follows,

This hidden state $\bm{h}_{i}^{1}$ is used to produce three signals. First, $\bm{h}_{i}^{1}$ is linearly projected into a stop control $\bm{z}_{i}$, which determines whether to stop the topic generation process or not,

where $\bm{W}_{z}$ an $\bm{b}_{z}$ are trainable parameters. Second, given a hidden state $\bm{h}_{i}^{1}$ together with image features $\bm{v}$, an attended context vector $\bm{c}_{i}$ is produced with an attention operation att,

Third, the hidden state $\bm{h}_{i}^{1}$ and the context vector $\bm{c}_{i}$ are fed into a fully-connected layer with weights $\bm{W}_{q,h}$ and $\bm{W}_{q,c}$ to produce the topic vector $\bm{q}_{i}$,

In this way, a relational sentence topic vector $\bm{q}_{i}$ is generated for the word decoder to predict words sequentially. Simultaneously, an encoded contextual topic vector $\bm{c}_{i}$ is produced to retrieve templates adaptively.

As the frequency of normal sentences is much higher than that of abnormal sentences, state-of-the-art methods on report composition [2] tend to generate normal sentences such as “The heart is normal in size”, “The lungs are clear” and “No acute bony abnormality”. As for the abnormal sentences like “Scattered thoracic spine spurring”, these models cannot write accurately due to the dataset bias. Different from the previous methods, our proposed Adaptive Generator produces the abnormal sentences from template database and employs the Word Decoder for generating normal sentences, which combines retrieval and generation for automatic report composition, as shown in Figure 3. During training, if the module needs to select a template, the Word Decoder will be masked and vice versa.

In particular, the Adaptive Generator takes encoded contextual topic vector $\bm{c}_{i}$ as input to determine the choice that performances either template retrieval or sentence writing for the current sentence generation. It consists of an LSTM layer and a Softmax classifier. Given a hidden state $\bm{h}_{i}^{1}$ from the Relational-Topic Encoder and contextual topic vector $\bm{c}_{i}$ predicted by the attention layer, the Adaptive Generator produces the adaptive decision $\bm{d}_{i}$ for sentence $i$. Since sentence writing is also one of the decisions, the size of the template database is $N$ and the decision space is $N+1$. The formulation can be summarized as:

From the above Equation 4 and 8, in which two LSTM layers are used to selectively attend to spatial image features, we can see that the bottom-up and top-down attention mechanism [10] is integrated into our Relational-Topic Encoder and Adaptive Generator.

As shown in Figure 2, the relational topic vector $\bm{q}_{i}$ represents the overall feature of the generated sentence. The topic vector and a special START token are regarded as the inputs to the Word Decoder following the previous work [2, 1]. The subsequent inputs are the word embedding sequence.

For each word, the last hidden state $\bm{h}_{w}$of the Word Decoder is used to predict a distribution over the words in the vocabulary. Finally, all sentences from retrieval or writing are concatenated to form the medical report.

Training data consists of tuples $(I,\bm{y},\bm{r})$, where $I$ is an image, $\bm{y}$ denotes the ground-truth abnormalities and $\bm{r}$ is the ground-truth report description, which has $m$ sentences. To perform adaptive generation, the $i$-th sentence has a template index ($t_{i}\in[0,N]$) as well as $n$ word indexes ($w_{i_{1}},w_{i_{2}},...,w_{i_{n}}$).

Given a tuple of $(I,\bm{y},\bm{r})$, we perform multi-label abnormality classification and unroll the Sentence Decoder $m$ timesteps, receiving abnormality distribution $a_{i}$, stop distribution $\bm{z}_{i}$ over the $\{\rm CONTINUE,\rm STOP\}$ states, and template distribution $\bm{d}_{i}$ over the template database. Then we unroll the Word Decoder $n$ timesteps if the adaptive generation mode changes to sentence generation, receiving word probability $\bm{p}_{i,j}$.

Formally, we define a multi-task loss similar to [27]:

where $\rm(B)CE$ denotes (binary) cross entropy function, $\mathcal{L}_{cls}$ is the abnormality multi-label classification loss consisting of binary cross-entropy loss and relationship constraint loss defined in Equation 1, $\mathcal{L}_{stop}$ denotes the stop signal prediction loss, $\mathcal{L}_{tem}$ and $\mathcal{L}_{word}$ are the template classification loss for each sentence and the word prediction loss over word distributions respectively, $m$ denotes the number of sentences that are generated by Word Decoder. If our model needs to select a template, the masks $\mathcal{M}_{tem}$ and $\mathcal{M}_{word}$ are set to $1$ and $0$ respectively, and vice versa. All the components are jointly trained to minimize $\mathcal{L}$.

Training. For two medical datasets, we set the same hyper-parameters, due to our robust captioning network. Our model accepts a $224\times 224$ image as input and yields $(H,W,C)$ feature maps from the last convolution layer. $(H,W,C)$ is $(14,14,512)$ and $(7,7,1024)$ when using VGG-19 [5] or DenseNet-121 [4], respectively. We use 512 as the dimension of all hidden states, topic vectors, word embedding and so on. To minimize $\mathcal{L}$ in Equation 10, we train on a single GPU with an initial learning rate of $5\times 10^{-4}$ and a batch size of 16 for 30 epochs. We employ ADAM [55] optimizer and decrease the learning rate by the factor of 0.8 every three epochs, following [53].

Inference. We execute our model to select a template or write sentence through Word Decoder at each time step until a stop control appears or the number of generated sentences reaches the maximum value $S_{Max}$. Here, we terminate the process as long as the score of stop control exceeds a threshold of 0.5. In detail, predicting the template index $``\textbf{0}"$ means we should believe the written sentence, otherwise, copy the corresponding template to the report directly. In the same way, we stop Word Decoder if we predict a END token or obtain $W_{Max}$ words. We set $S_{Max}$ and $W_{Max}$ as 10 and 15 for IU X-Ray, 24 and 18 for CX-CHR dataset, respectively.

Evaluation metrics. We conduct experiments to evaluate our method from three aspects: 1) Area Under the Curve (AUC) measures the performance of abnormality classification; 2) Automatic evaluation metrics consist of BLEU [56], ROUGE [57] and CIDEr [58]. BLEU is defined as the geometric mean of n-gram precision scores multiplied by a brevity penalty for short sentences. ROUGE-L basically measures the longest common subsequences between a pair of sentences and CIDEr measures the consensus between candidate image description and the reference sentences; 3) Human evaluation is performed by the radiologists with professional experiments. We invite 5 doctors to select the better generated reports between the baseline method CoAtt [2] and ours’ proposed model. The selection criteria consists of abnormal findings, language fluency, and content coverage.

Baselines. We compare our Relation-paraNet with 4 state-of-the-art image captioning methods. They are CNN-RNN [18], LRCN [51], AdaAtt [52], and Att2in [53] respectively. Besides the above methods, we also compare with state-of-the-art methods of medical report composition, including AoA [40], Transformer [35], BUTD [10], CoAtt [2], HRGR-Agent [1], and KERP [29]. For the fair comparison, we employ VGG-19 [5] and DenseNet121 [4] to extract visual features on IU X-ray and CX-CHR dataset respectively, in step with baselines. Further, we conduct additional experiments on both datasets to illustrate how each component contributes to the final results.

Abnormality classification. The comparison of multi-label abnormality classification in terms of AUC metric is shown in Table I. Superior to the baselines that simply apply VGG-19[5] or DenseNet-121[4] as backbone network with binary cross entropy loss, our method adds relationship constraint loss to exploit the relationships among abnormal medical terms, which outperforms baselines by 0.9% of AUC on IU X-Ray dataset and 1.9% on CX-CHR dataset. This demonstrates that our method is able to obtain useful visual features for correct abnormality classification via relationship constraint of medical terms. Its prominent capacity of detecting abnormalities is helpful for radiologists.

Automatic evaluation. The automatic evaluation results under several metrics are shown in the Table II. Obviously, our Relation-paraNet improves all metrics by large margins, demonstrating its effectiveness and extensiveness. CIDEr score represents inverse document frequency (IDF) of each vocabulary in the whole evaluated dataset, while BLEU-n matches n-grams within the evaluated sentences and ground truth sentences for each testing sample. In fact, improving the performance of BLEU-n with shorter n-gram (e.g., BLEU-1) is easier through generating common and seemingly correct words. On the contrary, CIDEr metric pays more attention to the critical but rare words under the consideration of IDF. Thus the CIDEr metric is more important on medical report composition, especially for the small dataset with unbalanced data.

On IU X-Ray dataset (i.e., a relatively small dataset including unbalanced data), the Relation-paraNet outperforms all baselines models (based on VGG-19 [5]) on BLEU-1,2,3,4 scores, showing that the Relational-Topic Encoder and the Adaptive Generator contribute to generating more professional reports. HRGR-Agent [1] combines both the retrieval method and the generative model with reinforcement learning. Thus, it is reasonable that HRGR-Agent achieves the higher CIDEr score. However, our Ralation-paraNet still achieves the best CIDEr score, outperforming HRGR-Agent by 2.2% due to the semantic consistency of abnormal medical terms and final reports encourage to generate rare abnormal descriptions. In addition, when employing the DenseNet-121 [4] backbone as the recent work KERP [29], we can achieve much higher performance on all automatic metrics.

On CX-CHR dataset, Relation-paraNet achieves state-of-the-art performance. Our model improves CIDEr score by 0.399, ROUGE-L score by 5.7%, BLEU-1 score by 3.8%, BLEU-2 score by 4.9%, BLEU-3 score by 5.4% and BLEU-4 score by 7.5%, compared to KERP [29]. Incorporating the semantic consistency of medical terms and considering abnormal sentences as templates, our Relation-paraNet encourages to produce reasonable and meaningful reports and achieves the best performance on automatic evaluation metrics.

To verify the effectiveness of our Relation-paraNet for medical report generation, we conduct extensive comparisons against several state-of-the-art visual captioning methods, such as AoA [40], Transformer [35] and BUTD [10]. As we discussed in Sec. II, the above attention-based methods are inapplicable to medical reports, which consists of multiple sentences or paragraphs. And the results show that our Relation-paraNet outperforms these methods on all automatic evaluation metrics, demonstrating the superiority in retrieving abnormal sentences and generation normal ones.

Human evaluation. Since Hybrid-agent [1] and KERP [29] have not released codes and models, our method is also compared with CoAtt [2] in human evaluation, similar to them. We randomly select 100 generated medical reports of baseline model CoAtt [2] and our methods in the test set. We invite five doctors to choose which one is more similar to ground truth under the consideration of abnormal findings, language fluency, and content coverage. We calculate the average preference percentages by excluding default choices (A default choice is provided in case of no or both reports are preferred), which is shown in Table IV. Obviously, our Relation-paraNet outperforms baseline a lot, indicating that it is capable of generating more accurate and reasonable reports.

Conclusion. Compared with Hybrid-agent [1], the proposed Relation-paraNet firstly preserves the superiority of bottom-up and top-down attention [10] for better semantic alignment between visual features and report descriptions. In addition, it is easier to generate abnormal descriptions by establishing the abnormal template datasets. Based on multi-layer and multi-head attention(Transformer) [35], KERP [29] built abnormality graph to learn relationships between abnormalities implicitly, the attention weights in which are lack of consistency and explanation [59]. In contrast, the Relation-paraNet contains a relationship constraint loss function to learn label co-occurrence dependencies directly. Since the popular visual captioning methods [40, 35, 10] is limited for paragraph generation, our Relation-paraNet derived the hierarchical structure from [27] and achieved excellent performance in terms of paragraph generation.

As shown in Sec. III-D, every term is necessary for the report generation task and any coefficient set to zero will lead to significant performance degradation. In this case, we paid more attention to illustrate the effectiveness of each module in our Relation-paraNet rather than adjusting these coefficients. As manifested in Table III, all those modules contribute to better performance, compared with the final Relation-paraNet scores. Additional visualization results of ablation experiments are provided in Sec. IV-D.

Sentence relation. Our model considers the sequence relationship between adjacent sentences, and encodes the last generated sentence information to guide in the next time step. This suggests once a sentence is generated correctly, it is useful to predict the next one. It encourages the Relation-paraNet to improve performance on all evaluation metrics, especially on BLEU-3 and BLEU-4.

Normal templates vs. Abnormal templates. The previous method HRGR-Agent [1] first extracts normal sentences as templates and writes a new sentence for abnormal findings. In fact, it is easy for current generative model [2] to fit common normal sentences. However, the rare abnormal sentences are more difficult to generate due to the unbalanced data. Instead, our Relation-paraNet regards the rare abnormal sentences as templates. The results show that this change can make the most improvement to our approach on the IU X-Ray dataset.

Abnormality relation. Our Relation-paraNet enforces the semantic consistency of medical terms to be incorporated into the final reports, as discussed in Sec. III-A. Under the relationship constraint, more reasonable visual features are provided for the downstream modules, especially for the attention module. As shown in Figure 4, templates retrieved by our Relation-paraNet are closely related to the abnormalities. Moreover, we can observe our model improves BLEU-n metrics by a larger margin on the IU X-Ray dataset but improves CIDEr and ROUGH-L metrics on the CX-CHR dataset. This is due to the small-scale IU X-Ray dataset, the abnormality is necessary for the report composition. Therefore, the appropriate templates related to the abnormalities can promote BLEU-n metrics. On the other hand, without abnormality relation, our model can learn abnormal features along with downstream tasks adaptively due to the large-scale CX-CHR dataset. However, uncorrected abnormalities may cause unrelated templates. Therefore, the CIDEr and ROUGH-L metrics will be lower.

Figure 4 provides the visualized results generated by the baseline method and our Relation-paraNet on both IU X-Ray and CX-CHR dataset. It shows that CoAtt [2] generates the most common sentences due to the dataset bias but gains not bad in terms of evaluation metrics. In the human evaluation, however, the performance of CoAtt is far worse than our Relation-paraNet (see Table II), which is consistent with the visualization results. As shown in the first example, our system can produce a sentence “there are streaky bibasilar opacities” corresponding to the ground truth report. Specifically, our Relation-paraNet detects abnormality “opacity” first. Then the relation between “opacity” and “there are streaky bibasilar opacities” guides the system to choose this template. Moreover, the abnormality “lung/hypoinflated” is also related to “opacity”, so our report contains “the lungs are hypoinflated”. On CX-CHR dataset, our method also captures the critical topic accurately and generates more reasonable reports. Overall, unifying relational-topic driven retrieval and generation, our Relation-paraNet is able to write abnormal templates precisely and provide additional symptomatic references.

Additional visualization of results are shown in Figure 5 - 9. In the abnormality relation graph, orange digits represent the classification scores. Black digits on the edges represent the relevance among abnormal medical terms. The underlined text expresses alignment between the generated text and ground truth reports. Bold text indicates the correspondence of the retrieved text. And the highlighted section illustrates inferred by abnormality relation and abnormal-template relation.

Figure 5 provides another visualization of results generated by Relation-paraNet on IU X-Ray[54]. Notice that our Relation-paraNet correctly produces “prior granulomatous disease” and “low lung volumes” in two examples,respectively.

Figure 6- 9 present qualitative results of ablation study discussed in our paper. Obviously, our Relation-paraNet can retrieve templates according to the detected abnormalities exactly, compared with other models.