GREN: Graph-Regularized Embedding Network for Weakly-Supervised Disease Localization in X-ray Images

The left and right lung regions in chest X-ray image show some symmetric structures, and the lesions of most diseases rarely appear symmetrically in both sides, such as pneumonia, infiltration and consolidation [37]. It is very useful to compare and distinguish the differences (e.g., textures and shadows) between the two lung regions for radiologists.

The structural relationship of X-ray regions is modeled as a graph $G_{Intra}=(N,E)$, where the nodes $N$ denote the two regions of left lung and right lung. The edges $e_{i}^{lr}\in E$ denote the similarity of the graph nodes $N$,

where $h_{i}^{l}$ and $h_{i}^{r}$ are the representations of the Hash coding of the left lung regions $x_{i}^{l}$ and the right lung regions $x_{i}^{r}$ in $i_{th}$ image, respectively. It is obtained by using perceptual hashing algorithm. $H(h_{i}^{l},h_{i}^{r})$ calculates the Hamming distance of $h_{i}^{l}$ and $h_{i}^{r}$. $D_{e}$ is a normalized parameter, which makes $e_{i}^{lr}\in(0,1)$. The value of it is the product of the length of the Hash coding ($D_{e}$ = 64 in our experiments). The graph regularization term $D(\theta)_{Intra}$ is obtained,

where $e_{i}^{lr}$ denotes the similarity of the left lung region $x_{i}^{l}$ and the right lung region $x_{i}^{r}$, $d(*)$ is the distance metric function, which is the Euclidean distance, $f_{i}^{l}$ and $f_{i}^{r}$ mean the feature map of the left lung region $x_{i}^{l}$ and the right lung $x_{i}^{r}$, respectively. Please note that the $f_{i}^{l}$ and $f_{i}^{r}$ are obtained by taking the masks of the left lung region $x_{i}^{l}$ and the right lung region $x_{i}^{r}$ in the feature map $f_{i}$.

There are phenomena of “different X-ray images of the similar diseases” and “different diseases of the similar X-ray images”, so it is difficult for the deep learning models to locate and classify diseases with limited box-level labels. Therefore, we propose a inter-image knowledge learning module to dig out the structural differences between different X-ray images, which provides a large difference for the similar diseases and similar X-ray images.

A pair of images $x_{u}$ and $x_{v}$ are randomly selected from the input images $X$. The structural relationship information of different X-ray images is modeled to get the graph $G_{Inter}=(N,E)$, where the graph nodes $N$ denote the images in a mini-batch of the network ($N=4$ in our experiments), the edges $e_{uv}\in E$ denote the similarity of graph nodes $N$. Since there are left and right lung regions in an image, the Hamming distance of two lung regions in all images should be calculated,

where $h_{u}^{l}$, $h_{u}^{r}$, $h_{v}^{l}$ and $h_{v}^{r}$ are the Hash codings of left and right lung regions of the image $x_{u}$ and the image $x_{v}$, respectively. $H(*)$ is the Hamming distance calculation. Note that the Hash coding generation is training-free. $D_{e}$ is a normalization parameter, which denotes the product of the length of the Hash coding ($D_{e}$ = 64 in our experiments). The graph regularization term $D(\theta)_{Inter}$ is written as

where $e_{uv}$ denotes the similarity of $x_{u}$ and $x_{v}$, and $d(*)$ is the Euclidean distance, and $f_{u}$ and $f_{v}$ mean the feature maps of image $x_{u}$ and image $x_{v}$, respectively.

The final objective $Q(\theta)$ is the summation of the loss $L(\theta)$ of the localization network and the graph regularization terms $D(\theta)_{Intra}$ and $D(\theta)_{Inter}$,

where the hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ control the balance between the three losses and are set to 0.11 and 0.15 by searching from 0 to 1 with a step of 0.05 and 0.01 for coarse-to-fine searching. For example, it is assumed that the hyper-parameter in coarse searching is 0.10 (with the step of 0.05), and the interval in fine searching is [0.96, 0.99] and [0.11, 0.14] (with the step of 0.01). When $\lambda_{1}$ = 0 and $\lambda_{2}$ = 0, the network reduces to a localization model without the graph regularization.

To calculate the similarity between regions, it is most intuitive to use CNN features. To investigate on this, we replace the similarity computing with CNN features to compare with the Hash coding with Hamming distance. The feature map $F_{Intra}=\{f_{1}^{l},f_{1}^{r},f_{2}^{l},f_{2}^{r},...,f_{n}^{l},f_{n}^{r}\}$ is extracted using the same method as described previously. The similarities between the feature maps are calculated according to

where $f_{u}^{l(r)}$ and $f_{v}^{l(r)}$ are the feature maps of the left (right) regions of the image $x_{u}$ and image $x_{v}$. $s_{uv}$ is the similarity of feature maps $f_{u}$ and $f_{v}$. $s^{lr}_{i}$ is the similarity of the right regions and left regions in $i_{th}$ feature maps. $d_{cos}(*)$ is the cosine distance of the feature maps. The graph regularization term $D(\theta)_{Inter}^{s}$ and $D(\theta)_{Intra}^{s}$ are written as

where $d(*)$ is the Euclidean distance metric. $f_{u}$ and $f_{v}$ mean the feature maps of image $x_{u}$ and image $x_{v}$, respectively. The final objective $Q(\theta)^{s}$ is the summation of the localization loss $L(\theta)$ and the graph regularization loss $D(\theta)_{Inter}^{s}$ and $D(\theta)_{Intra}^{s}$,

where the hyper-parameter $\lambda_{3}$ and $\lambda_{4}$ control the balance between the three losses and are set to 0.15 with the same method as described previously.

We compare the localization results of the baseline model (B) with three models under different configurations, including the model with the intra-image knowledge learning module (B+Intra), the model with the inter-image knowledge learning module (B+Inter), and the model with both knowledge learning modules (B+Intra+Inter). The experiment results in Table 1, Table 2 and Table 3 show that the model (B+Intra) and the model (B+Inter) demonstrate more advantages over the baseline method, and the model (B+Intra+Inter) achieves the best localization result. For example, in Table 1, when T(IoU) = 0.5, the mean accuracy of the model (B+Intra) and the model (B+Inter) are 0.49 and 0.50, and outperform the baseline (B) by 0.21 and 0.22, respectively. The model (B+Intra+Inter) achieves the best localization results, which is 0.54, and outperforms the baseline (B) by 0.26. Additionally, the localization results in Table 1 and Table 2 demonstrate that when there is more annotated data in the training process, the model using the inter-image knowledge learning module has more advantages; when there is no annotated data in the training process, the model using the intra-image knowledge learning module has more advantages. For example, the model (B+Inter) performs better in most cases compared with the model (B+Intra) in Table 1, and the model (B+Intra) performs better in most cases compared with the model (B+Inter) in Table 2. Overall, the experiment results demonstrate that using intra-image and inter-image structural relationship information can improve the performance of models for disease localization. Moreover, the experiment results show that the combined effect of the inter and intra construction can further boost the performance of weakly supervised disease localization.

To investigate the similarity computing method, we compare the Hash coding using Hamming distance with CNN features using cosine similarity (B+Cosine), which is shown in Figure 4(a). The B+Cosine model is competitive to the baseline model but is inferior to GREN for all the three data settings. There are two possible reasons. First, it is not easy to obtain a CNN-based model that needs to not only perform as a feature extractor but also learn class-aware similarity. Hash coding solved this problem via projecting features to a low frequency signature. Second, as a comparison, Hash coding is data-independent and training-free. Overall, the structural relationship (similarity) can be used to regularize the training of deep neural networks, so as to achieve more accurate and integral localization results with few careful annotations. Moreover, using Hash coding and Hamming distance instead of the cosine distance of CNN features can achieve better performance, which demonstrates that the image features extracted directly using Hash coding can to some extent compensate for the information lost in CNNs, making the proposed method reasonable.

We explore the influence of different number of graph nodes $N$ on the results, which is shown in Figure 4(b). The value is equal to the batch size of the model, including 2, 4, 8, and 16. The data of 100% unannotated images and 40% annotated images are used to evaluate the performance. The mean accuracies improve with the increase of batch size, whereas the trend is not an unbounded growth. For example, when T(IoU) = 0.7, the mean accuracy of the model with batch size of 16 is 0.47, outperforming the model with batch size of 2 by 0.03. However, when T(IoU) = 0.5, the mean accuracy of the model with batch size of 16 is 0.54, which is the same as the model with the batch size of 4. Using more images in each graph is good to effective regularization, but the growth is bounded (slow after the batch size exceeds 4). Therefore, the batch size is set to 4 in our experiment by considering the trade-off between memory cost burden and performance.

Graph is a concise and effective way to characterize inter- and intra-image relations. There are several works to investigate the relationship and achieve a superior performance in multi-label natural image recognition, such as [5]. In the X-ray image analysis, the inter- and intra-image relations can be used to simulate the physician’s practice of comparing multiple images and symmetric regions of the same image for diagnosis respectively. Existing literatures [35], [34] imply that the relationship between the left and right lung lobes is useful in automatic diagnosis. However, these methods do not explore the underlying reasons why the inter- and intra- relationship modeling is effective. Here, we construct relation graphs using the left and right lung lobes as the nodes of graphs, and then incorporate the graph as a regularizer to facilitate the representation learning. Specially, we try to explore the underlying reasons of this.

We explore the influence of the inter-image and intra-image similarity as shown in Figure 5. We randomly choose 10 images from the dataset. In Figure 5(a), the horizontal and longitudinal axis are the same lung region images, and the two adjacent images have the same classes. We calculate the similarity of lung regions of these images using the Hash coding and Hamming distance. The two adjacent images have higher similarity compared with other images, where the darker color denotes higher similarity. For example, the diagonal rectangles are the similarities of the same images, so the values are 1. Additionally, we calculate the similarity of the left lung and right lung regions in an image, which is diagonal rectangle, and other rectangles are filled with the same color. In Figure 5(b), the horizontal and longitudinal axis are the corresponding left lung and right lung regions in an image. The darker-colored rectangle denotes higher similarity of the left lung and right lung in an image. Particularly, the sixth and ninth darker color rectangles (from upper left to low right) imply that the sixth and ninth images have highly similar left and right lung, and their categories are actually “No Finding” (The categories of the second, sixth, and ninth images are “No Finding”). The first, fourth, and eighth light color rectangles imply that the left and right lung regions of the first, fourth, and eighth images have low similarity, and their categories are “Cardiomegaly”, “Cardiomegaly”, and “Pneumonia”, respectively. To some degree, the inter-image similarity can indicate the categories of images without annotation data, and the intra-image similarity can indicate the information of differences of left and right lung regions, which is helpful to distinguish the unilateral disease of X-ray images.

We compare the localization results of our model with the attention-based method, which integrates the attention mechanism into the baseline to learn the difference of lung regions. In Table 4, the B+Attention model outperforms the baseline by 0.21 and 0.26 at T(IoU) = 0.5 and 0.7, however, GREN still holds advantages. It ourperforms the B+Attention model by 0.26 and 0.29 under the two thresholds. Overall, the attention-based method can indeed improve the accuracy, but GREN using the intra-image and inter-image structural relationship has larger advantages to enhance the weakly-supervised disease localization.

Although we focus on the weakly-supervised localization task, we additionally evaluate the results of disease identification. Table 5 presents the AUC scores (the area under the receiver operating characteristic curve) for all the classes. Following [16] and [18], we use 70% images for training and 20% images for testing. We compare the performance of disease identification of the Baseline (B) with the B+Intra, B+Inter and GREN. The mean AUC scores of the B+Intra, B+Inter and GREN are 80.25%, 80.32% and 80.74%, outperforming the Baseline (79.65%) by 0.60%, 0.67% and 1.09%, respectively. GREN is more effective in disease localization than disease identification because the task of disease identification has more sufficient supervision. Overall, GREN can maintain performance of disease identification, but enjoys better ability for weakly-supervised disease localization. Particularly, GREN has better application value in the task with limited annotation data.

To better demonstrate the final effect of our method on disease localization, we visualize some of typical predictions of both the baseline model and GREN, as shown in Figure 6. The first column shows the original images, the second and the third column show the localization results of the baseline model and GREN. The green bounding box and red area mean the ground truth and prediction, respectively. It can be seen that GREN can predict more accurate regions compared with the baseline model. For example, for the class of “Nodule”, the localization results of the baseline model are completely inconsistent with the ground truth, but the localization results of GREN are consistent with the ground truth.

In order to demonstrate the difference of GREN (B+Intra+Inter) with B+Intra and B+Inter, we visualize some of typical predictions of them, as shown in Figure 7. From left to right, it shows the localization results of the Baseline (B), B+Intra, B+Inter and GREN. GREN plays an important role in integrating the advantages of B+Inter and B+Intra to make the predictions better. For a sample (Figure 7), some of B+Inter results are better than B+Intra results, such as (a), (b), (d), some of B+Intra results are better than B+Inter results, such as (e), (f). It can be seen that GREN can modify and complement the predicted results of B+Inter and B+Intra to make better predictions. The underlying reason may be that the functions of Inter-image and Intra-image knowledge learning module are not only complementary but also share common mechanisms. For example, the lungs are highly structured. For an X-ray image, there are not only similarities but also differences between left and right lungs. For different X-ray images, there are not only differences but also similarities in ipsilateral lung. Therefore, the functions of Inter-image and Intra-image knowledge learning module cannot be completely separated, and some overlap and crossover inevitably exist between them.

We also visualize some of typical predictions of both the CAM-based method [30] and GREN, as shown in Figure 8. The first column shows the original images, the second and the third column show the localization results of [30] and GREN. It can be seen that GREN has greater advantages over [30]. Moreover, compared with [30], GREN uses the regularization of the intra-image and inter-image structural information to pay more attention to the lung regions than other regions (e.g., clavicle and arm regions). It further demonstrates that GREN can effectively improve the performance of weakly supervised disease localization.