Attention-based Saliency Hashing for Ophthalmic Image Retrieval

As shown in Fig.2, we propose a novel attention-based saliency hashing network to jointly learn visual features and the subsequent mapping to a compact hash-code. The learned visual features can effectively catch the information of salient regions. The upper network in Fig.2 (called ASH-U) consists of a convolutional layer and a max-pooling layer followed by a spatial-attention module and terminates in a fully-connected hashing layer for hash-code generation. And the lower network in Fig.2 (called ASH-L) introduces a spatial-attention module after three residual blocks [33], then a convolutional layer and terminates in a fully-connected hashing layer for hash-code generation. All the convolutional and pooling layers use $3\times 3$ filters and are followed by batch normalization [34]. The fully-connected linear layer contains $4096$ nodes, and the fully-connected hashing layer contains $K$-bits. All convolution layers and fully-connected layers are equipped with the ReLU [35]. The image pairs are inputted into the ASH to generate pair hash-codes and share weights of network in training procedure. This procedure exactly utilizes the siamese structure to learn the objective function with pairwise loss for retrieval scenario.

The spatial-attention module used in this work is a improved variant of CBAM [11] to enhance the representation of salient regions. Let $\bm{X}\in\mathbb{R}^{C\times H\times W}$ be the feature maps extracted from a convolutional layer, where $H$ and $W$ denote the height and width of each feature map, and $C$ denotes the number of feature maps (or channels) in that layer. The spatial-attention module aggregates spatial information of feature maps $\bm{X}$ by utilizing max-pooling ($Max$) and average-pooling ($Avg$) operations jointly along the channel axis. Both operations obtain the local maximum and average value of feature maps $\bm{X}$ respectively, then $Max(\bm{X})$ and $Avg(\bm{X})$ multiply element-wise to further weight the local salient regions. The output of the weighting operation is equipped with the sigmoid function, then multiply with the feature maps $\bm{X}$. The spatial-attention module is defined as:

where $Max(\bm{X})$ and $Avg(\bm{X})$ are $H\times W\times 1$ dimensions, and $\sigma$ denotes the sigmoid function.

In the ASH-U, the spatial-attention module is embedded between a max-pooling layer and a convolutional layer followed by an average-pooling layer. After a max-pooling layer amplifying local salient values, the spatial-attention module can easily capture them while the value of salient regions is usually higher than the other regions. Such a spatial-attention module is beneficial to capture the information of salient regions in the high-resolution ophthalmic image. In the view of flexibility and usability, in the ASH-L, we further incorporate the spatial-attention module into the application-specific CNN structures due to their different capability of feature extraction. The learning procedure of our ASH focuses on the feature extraction of salient regions on the raw pixels of input images by using an attention-based CNN. Such hierarchical non-linear function exhibits a powerful learning capacity and encourages the learned feature to capture the information of salient regions by introducing the attention mechanism.

The main idea behind the pairwise loss of siamese structure is that the codes of a pair of samples from the same class should be as close as possible, while the codes of a pair of samples from the different classes being far away. Based on this objective, the loss function is naturally designed to minimize the distance of similar image pairs and maximize the distance of dissimilar image pairs.

Mathematically, giving a set of samples with $2N$ length and corresponding class labels $\bm{L}=\{1,\dots,C\}$ and dividing the samples with labels into two equal parts $\bm{I_{1}}=\{\bm{x_{1,M}},\dots,\bm{x_{N,M}}\}$ and $\bm{I_{2}}=\{\bm{x_{1,M}},\dots,\bm{x_{N,M}}\}$, we typically define similar labels $\bm{Y}=\{y_{1},\dots,y_{N}\}$, where $y_{i}=0$ if $x_{i,M}$ in $\bm{I_{1}}$ and $\bm{I_{2}}$ are the same class label (similar) and $y_{i}=1$ otherwise (dissimilar). ASH aims at learning a mapping from input image pairs to hash-code pairs $\bm{H_{1}}=\{\bm{h_{1,K}},\dots,\bm{h_{N,K}}\}$ and $\bm{H_{2}}=\{\bm{h_{1,K}},\dots,\bm{h_{N,K}}\}$. For scalable retrieval, the hash-code bits $K$ is much smaller than the dimension of ophthalmic image $M$. The pairwise loss with respect to image pairs is defined as:

where $D(\cdot,\cdot)$ denotes $L_{2}$ normalization to measure the distance between hash-codes, 1 is a vector of all ones, and $r\in[0,1]$ is a weighting parameter that controls the punish strength of differentiating degrees between dissimilar images. In Eq. (2), the first term punishes the similar images mapped to far hash-code, and the second term punishes the dissimilar images mapped to close hash-code when their distance falls below the margin threshold $r\cdot K$.

In the loss function of ASH, the contrastive loss form is applied as that only those dissimilar pairs having their distance within a radius are eligible to contribute to the loss function. When $r=0$, there is no punishment of dissimilar images mapped to close hash-code. When $r=1$, the hash-code of dissimilar images is required to be complete difference. If $r=0.5$, this implies that half of the hash-code bits between dissimilar images is required to be different. To be especial, we do not impose an additional regularizer on the loss function so that we can independently observe the contributions of the spatial-attention module.

(1) ASOCT-Cataract. In the ASOCT-Cataract dataset, each image is labeled as coprresponding grade according to the degree turbidity of cortex, nucleus, subcapsular. Among the three types, the size of the subcapsular is smaller than the others. For validating the capability of our ASH in capturing the information of salient regions, Oculus Dextrus (OD) images labeled five-level turbidity of subcapsular are chosen as training samples (7,000) and query images (700). (2) Fundus-iSee. The Fundus-iSee dataset with four disease classification consists of 10,000 high-resolution images labeled by professional doctors with rich clinical experience, having 720 images of Age-related Macular Degeneration (AMD), 270 images of DR, 450 images of glaucoma, 790 images of myopia, 7770 images of normal. We randomly extract ten percentage of each type for query test, 1,000 images. As shown in Fig. 1, the performance of ASOCT images grading and fundus images classification depend heavily on the differentiation of salient regions. The selected two ophthalmic datasets can be used to verify the effectiveness of our ASH in mapping the information of salient regions into the hash-codes.

In our comparative study, we used a data-independent method: LSH [16]; a shallow learning-based hashing method: MHF [20]; three deep learning-based hashing methods: DPSH [23], DSH [24], DRH [9]. In our implementation of Fig.2, the experiment on ASOCT-Cataract used ASH-U, and the experiment on Fundus-iSee used ASH-L. Further, to study the effects of different CNN structures, we embedded the spatial-attention module before the fully-connected layer of DPSH and DSH, called DPSH-A and DSH-A respectively, and our ASH-L is the variant of DRH by introducing the spatial-attention module. To embody the advantage of the spatial-attention module, we used the same pairwise loss for all deep hashing methods. According to the pairwise loss, the weighting parameter $r$ and the hash-code bits $K$ are synchronous to each other. Reasonably, setting $r=0.5$ refers to that half of the hash-code bits between dissimilar images should be different. As shown in Table I, with the hash-code lengthen, $r$ can be correspondingly set higher than the short hash-code, then the performance can correspondingly improve at the cost of storage and search efficiency. As a trade-off between performance and search cost, we set $r=0.5$ in our experiments, then we evaluated the performance over hash-code bits from 12, 24, 36 to 48, and the most similar images from 5, 10, 15 to 20.

Three metrics were adopted to measure the precision and retrieval quality in our experiments. (1) Mean Hit Ratio (mHR). HR is designed to measure how many images in the returned list are similar to the query image. (2) Mean Average Precision (mAP). In the returned list, AP averages the rank positions of images similar to the query image to measure the rank quality. (3) Mean Reciprocal Rank (mRR). RR refers to the reciprocal of the ranking of the first similar image in the returned list. Our ASH is implemented under PyTorch framework and experiments are run on Geforce RTX 2080 Ti. The indexing and similarity calculation for evaluation uses Faiss [36] which is a library for efficient similarity search and clustering of dense vectors. We use the mini-batch stochastic gradient descent with 0.9 momentum. The mini-batch size of images is fixed as 10 and the weight decay parameter as 0.001. The input image pairs were randomly sampled in every iteration. All deep models are trained from scratch, setting the iteration number as 50. The parameters of comparative methods are set according to their implementation details in the corresponding works, and the best performance is reported.

Fig.3 shows the retrieval performance of different hashing methods in terms of mHR, mAP, and mRR for different hash-code bits. Due to the contributions of the spatial-attention module, the performance of our ASH (red line) is more superior to other state-of-the-art hashing methods: (a) data-independent method (LSH-black line) and shallow learning-based hashing method (MHF-cyan line) using hand-crafted features, (b) deep learning-based hashing methods without attention mechanisms (DPSH-solid green line, DSH-solid blue line, DRH-solid yellow line). Although the longer bits of hash-code can improve the performance due to containing more information of images, we have to make a balance between accuracy and efficiency. The hash-code bits can be properly enlarged according to the volume of the dataset increasing. On the one hand, compared to the deep learning-based hashing methods using our spatial-attention module but different network structures, our ASH consistently outperforms both DPSH-A (dotted green line) and DSH-A (dotted blue line) by 13%-31% on mHR, 11%-21% on mAP, 9%-17% on mRR. On the other hand, we can observe that the performance of DPSH (solid green line) outperforms DPSH-A (dotted green line) while the latter embeds our spatial-attention module. This demonstrate that the effectiveness of attention mechanisms relates to the network structure. For the classification dataset (Fundus-iSee), our ASH-L, which is a variant of DRH adding spatial-attention, is applied to extract visual features. And the order of performance is ASH-L$>$DSH-A$>$DRH$>$DSH. But in the grading dataset (ASOCT-Cataract), the order of performance is ASH-U$>$DRH$>$DSH-A$>$DSH while our ASH-U adopts an off-the-shelf CNN structure. Compared to the different diseases (classification), the region feature is more difficult to extract in the different degrees (grading) of the same disease. The above analysis demonstrates that the structure of our ASH-U can further improve the performance for the retrieval task of grading datasets compared to DRH, while the structure of our ASH-L and DRH is more suitable for the retrieval task of classification datasets than DSH. Through experiments on the ASOCT-Cataract and Fundus-iSee datasets, our ASH is proved to be the best performance by embedding the spatial-attention module into the specific network structure.

In Fig.4, to qualitatively show the contribution of the spatial-attention module, the feature maps of ASOCT-Cataract and Fundus-iSee are shown in (A) and (B). In both (A) and (B), O is the original image, BS and SA are the feature map of the convolutional layer before and after the spatial-attention module, respectively. We can observe: (1) the over-bright salient regions of subcapsular are captured in image SA of (A); (2) large optic cup can be suspected from image SA of (B). This demonstrate that our ASH can detect the subtle difference of ophthalmic images by mining the salient regions with the help of the spatial-attention module. The visual information of salient regions are learned in our ASH and are mapped into the hash-code for differentiating ophthalmic images. The higher mAP means more similar images in the returned list are ranked ahead and can help users to find the required disease case quickly. As (C) of Fig.4 shows, for the ASOCT-Cataract dataset, the top-5 most similar images and the query image are all turbidity level 5 of subcapsular. In (D) of Fig.4, for the Fundus-iSee dataset, disturbed by the similar overall performance of different classes, the top-5 most similar images are different classes by querying DR, but the first and third images are DR. Through the classification and grading on two different ophthalmic image modalities, our ASH is proved to be powerfully capable of capturing the information of salient regions and differentiating them. In the feed-warding process, the spatial-attention module catches the spatial information of salient regions. And in the feed-backing process, the class or grade semantic information are learned by the pairwise loss and encoded into the salient regions. Thus, salient regions not only be captured by the spatial-attention module, but also be differentiated by the pairwise loss according to their corresponding class or grade.

Based on the above analysis, the effectiveness of our ASH is verified by observing the quantitative result in Fig. 3, and the contributions of the spatial-attention module is confirmed by observing the qualitative result in Fig. 4. Next, we introduce the performance over different numbers of the returned list different and analyze the efficiency of our ASH. As shown in Table. II, compared to the three deep hashing methods, our ASH all achieve the best performance over varying number of returned lists from 5, 10, 15 to 20. When the returned list lengthens, the performance of all methods declines to some extent. The better performance of longer returned list implies more powerful capability of differentiating similar images based on the similarity calculation of hash-codes. This further confirm the contributions of the spatial-attention module. At the last analysis of experiments, the efficiency of the proposed method will be discussed on four-folds by putting the Fundus-iSee dataset as an example. (1) Feature computation time. Based on the pre-trained ASH model with 12-bits hash-code, the feature extraction of the training set of 9000 images can be completed in 33 seconds by using GPU. (2)Retrieval time. After features mapping into 12-bits hash-code, the index of the training set is built in 1 second by using Faiss. Then the retrieval of the test set of 1000 images can be done in 3000 $ms$ by returning top-10 most similar images. (3) Training time. The training process takes about one hour by setting the iteration number as 50. (4)Memory cost. During model training with a batch size of 10, the memory-consuming is about 2000 Mbps. The online search for the index also consumes about 3000 Mbps. Compared to three deep hashing methods, the complexity of our ASH is slightly higher in training time and memory cost due to the added spatial-attention module and is fair in feature computation time and retrieval time. According to the above analysis of efficiency, our ASH can provide fair retrieval responses with significantly improving the performance, compared to the state-of-the-art deep hashing methods.