Combating Ambiguity for Hash-code Learning in Medical Instance Retrieval

Hashing methods can be divided into data-independent methods and data-dependent methods. The data-independent methods [20, 21] learn hashing functions in a two-stage manner from hand-crafted features such, and the hash-codes learning procedure is independent of the image features, which may lead to sub-optimal performance. The data-dependent methods, also called learning-based hashing methods, can be further categorized into [22]: (1) shallow learning-based hashing methods, like metric hashing forests [23], and kernel sensitive hashing [24]; (2) deep learning-based hashing methods, like image inpainting-based compact hash-code learning [25], and deep hashing network [26]. In contrast to the data-independent methods, they extract global features for hashing in an end-to-end manner. Early works [27, 28] for instance-level retrieval rely on hand-crafted local descriptors such as SIFT [29] and SURF [30]. Prior to deep learning, these works based on local features extraction, then aggregated into a global vector [31, 32]. The instances relevant to a query are discovered in the candidate images for similarity search by matching local descriptors. However, hand-crafted local features are vulnerable to non-rigid deformations and heavy viewpoint changes. Due to the promising performance in computer vision, CNN-based approaches have been introduced to instance-level retrieval. Instead, the global vector is extracted by a single forward-pass through a CNN, in which the extraction and aggregation steps are not separated. Existing deep hashing methods [33, 34] can be grouped into this category using feature embedding tailor features from fully-connected layers for hash-codes generating. The representative methods include deep pairwise-supervised hashing (DPSH) [7], deep supervised hashing (DSH) [35], and deep residual hashing (DRH) [36]. Since convolutional features have been found to be reasonably discriminative [37], recent CNN-based approaches have shifted to concentrate on feature aggregating rather than feature embedding. CNN-based approaches aggregating convolutional feature maps as global image representation can be roughly divided into two categories.

The first category is the works encoding the activations of a convolutional layer by weighted aggregation. These works’ key idea is to assign different weights to different regions’ activations in feature maps after global convolutional layers generate. SPoC [8] showed that a simple spatial pooling on the convolutional layer outperformed fully-connected layers, and the power of this representation could be enhanced by applying the Gaussian center prior scheme to weight the contribution of the activations before aggregation. Following a similar idea, CroW [38] proposed a non-parametric spatial- and channel-wise weighting method for focusing on salient regions. Unlike the spatial weighting scheme, class activation maps (CAMs) [39] are employed for calculating semantic-aware weights of a convolutional feature map. Based on the bags of local convolutional features (BLCF) [40], BLCF-SALGAN [41] build an efficient image representation by saliency weighting.

The second category is the works performing region analysis using convolutional features. Unlike the first category, this category first generates regions’ convolutional features after region proposal, then aggregates them into global features. The representative work is regional maximum activation of convolutions (R-MAC) [42], which generated a set of regional vectors by performing spatial max-pooling within a particular region and aggregates features from several local regions into a single compact feature. Gordo et al. [43] improved over the original R-MAC encoding by explicitly learning a region proposal network [44] and training in an end-to-end framework with a triplet loss. Laskar et al. [45] used a saliency measure directly derived from the convolutional features to weigh the contribution of the regions of R-MAC before aggregation. Similar to R-MAC, Cao et al. [46] proposed a method to derive a set of base regions directly from the convolutional layer, followed by a query adaptive re-ranking strategy. DeepVision [47] extracted region-level features from the bounding boxes generated by the object detection framework. Regional attention [37] proposed a context-aware regional attention network that weighs an attentive score of a region considering global attentiveness.

Recently, deep hashing methods using feature embedding on the fully-connected layer have also been widely proposed for medical instance retrieval, such as deep multiple instances hashing for tumor assessment [11], deep residual hashing for chest X-ray images [36], order-sensitive deep hashing method for multi-morbidity medical image retrieval [22], deep disentangled momentum hashing for Neuroimage Search [16], etc. Although the prior works have facilitated medical instance retrieval’s prosperity, pathologically abnormal regions’ manifestation ambiguity is challenging for current deep hashing methods. Recent studies [46, 39, 37] have shown that using feature aggregating on the convolutional features achieves promising performance in instance-level retrieval. Following this direction, our work improves the current deep hashing method to combat pathologically abnormal regions’ ambiguity in medical instance retrieval.

In this work, the improvement for the current deep hashing methods includes:

• •

  Unlike the current deep hashing methods jointly learning image descriptors and hash-codes, our work first learns convolutional features from image spaces by supervised training, then aggregates them as hash-codes. The learned convolutional features and following generated hash-codes can effectively preserve the differentiating information of pathologically abnormal regions.

• •

  Inspired to CAMs [39] and R-MAC [42], we endow the class-aware information to the R-MAC descriptors by classification training. The R-MAC descriptors related to classes can enhance their differentiating ability and help to avoid the SPDD problem.

• •

  Motivated by regional attention [37], we adopt feature pyramid networks (FPN) [48] to exploit multi-scale pathologically abnormal regions by pixel-wise segmentation training. The subtle differences are encoded into the convolutional features to overcome the DPSD problem.

We detect if pathologically abnormal regions are presented in each image with classification training, and we locate pathologically abnormal regions using activations with the help of segmentation training. In the end, the convolutional features, having learned class-aware information and subtle spatial differences, are mapped into the hash-codes. Based on the class labels and pixel-wise masks, we argue that our work is a beneficial exploration to combat pathologically abnormal regions’ ambiguity in medical instance retrieval.

Each image is represented by an instance-invariant feature vector, i.e., hash-code. As shown in Fig.2, we present a deep learning framework, called Y-net, to generate distinctive hash-codes from convolutional features. Our Y-Net contains three parts, main branch, R-MAC branch (right), and FPN branch (left). In the training stage (double arrow), we input an image into the main branch and feed-forward to the core node (red rectangle). The core node is a convolutional layer, followed by the R-MAC branch and the FPN branch. In the R-MAC branch, the classification loss minimizes intra-class distance and maximize inter-class distance. The inter-class separation can help avoid SPDD problem. But, to overcome the DPSD problem, the intra-class distance needs to be preserved but not minimized. The FPN branch can locate intra-class differences by pixel-wise segmentation training to balance the reduction of intra-class distance in the R-MAC branch. The core node learns the class-aware semantic information of pathological regions from the R-MAC branch for differentiating the same manifestation of different diseases. Simultaneously, the spatially subtle differences of pathological regions from the FPN branch are encoded into the core node to locate the same disease’s subtle differences at different stages. After the core node absorbing the visual cues from the R-MAC branch and the FPN branch in the training stage, we can generate hash-codes from the learned core node by feature aggregation in the test stage (single arrow).

The image encoding pipeline of the main branch is depicted as follows:

• •

  Training Stage. The input image $\bm{I}$ with a resolution $3\times 256\times 256$ is feed-forwarded into the main branch. The main branch computes a feature hierarchy consisting of a bottom-up block at three scales with a scaling step of $2$. At each scale, we use the feature activations output of bottom-up block [49] to get a receptive field. The three bottom-up blocks are merged into the FPN branch by addition. In the core node of the main branch, the convolutional feature maps $\bm{X}\in\mathbb{R}^{C\times H\times W}$ can be arranged in a tensor of the size $C\times H\times W$, where $H$ and $W$ denote the height and width of each feature map, and $C$ denotes the number of feature maps (or channels) in the convolutional layer. In a convolutional layer, the activations at the same spatial location across all feature maps can be composed into a $C$-dimensional local descriptor for a certain image region. Compared to the activations of the fully-connected layer, the convolutional features retain the spatial information of local image descriptors and are essentially similar to the traditional hand-crafted local features [50, 51]. The convolutional feature maps $\bm{X}$ are further feed-forwarded into the R-MAC branch and the FPN branch. Based on the classification and segmentation training, the semantic and spatial information of pathological regions is encoded into the convolutional feature maps in the feedback process.

• •

  Test Stage. Based on the pre-trained Y-Net, an image with a resolution $3\times 236\times 256$ is feed-forwarded into the main branch and terminated in the core node. The core node is the conjunct point of a Y shape and is the core component in the framework of Y-Net. We apply feature aggregation to generate a $k$-bits hash-code from the learned convolutional feature maps $\bm{X}$ with $C\times H\times W$ in the core node. Based on the existing works, the feature aggregation does not take part in the training and has been found to be more capable of preserving the discriminative information than the feature embedding. Considering the convolutional feature maps $\bm{X}$ with $C\times H\times W$ have learned the visual cues of pathological regions effectively, we convolute the size of $C\times H\times W$ into the size of $c\times h\times w$ without any weighting strategy. Then the three dimensions vectors further are squeezed into one dimension; its size equals the hash-code size of $k$-bits. Such a convolution process can aggregate feature maps of various sizes in three dimensions, such as $1\times 8\times 8$ and $128\times 1\times 1$. Lastly, we apply the hyperbolic tangent function to generate the value between $-1$ and $1$, following by signed as binary hash-code. At this step, we do not introduce any weighting strategy on feature aggregation because the convolutional feature maps have learned the visual cues of pathological regions effectively.

The R-Mac branch contains a convolutional layer using $3\times 3$ filters and followed by batch normalization [52], then an R-MAC block generating a feature vector of length $512$. The feature vector is mapped into a linear layer after the $L_{2}$ normalization. The length of the linear layer is the number of classes. The R-MAC block generates a compact representation by aggregating multiple regions at different scales. By classification training, the highly activated regions can correspondingly respond to the semantic information of the belonging class. The pipeline of the R-MAC block is summarized as follows:

• •

  Based on a convolutional layer with $512\times 8\times 8$, we sample square regions with a region size, $R_{s}$, of a specific scale $s$ in a sliding window manner of $0.4$ overlap between neighbor windows, for all $s=0,...,S$. The region size at a specific scale can be calculated as:

  $$\begin{aligned} R_{s}=2\times\min(W_{r},H_{r})/(s+2)\end{aligned},$$

  (1)

  where $W_{r}$ and $H_{r}$ are width and height of the feature map in the convolutional layer. In our Y-Net, with $W_{r}=8$ and $H_{r}=8$, we set $S=3$, then we totally get sample region of $14$.

• •

  After sampling the regional feature maps, we perform a max-pooling for all regional feature maps of $14$. Each regional feature maps generate a feature vector with $512$, the same as the channel’s size. Last, we aggregate all feature vector of sample regions in the whole image as a global feature vector with $512$ dimensions, named R-MAC descriptor used as a discriminative image representation.

In the pipeline of R-MAC, the local features from a certain convolutional layer are max-pooled across several multi-scale overlapping regions, obtained from a rigid grid covering the whole image, similar in spirit to spatial pyramids, producing a single feature vector per region. Then these region descriptors are sum-aggregated and $L_{2}-$ normalized into a global image representation. The discriminative global image representation is a compact vector whose size is independent of the size of the image and the number of regions. The region pooling is different from a spatial pyramid. The latter concatenates the region descriptors, while the former sum-aggregates them. Comparing the R-MAC descriptors of two images with a dot-product can then be interpreted as a many-to-many region matching.

R-MAC has been known for effective and efficient performance in image retrieval. Nonetheless, the main issue of R-MAC is that all sampled regions are equally treated without considering their varying importance. When aggregating their regional feature vectors, all regions construct their equal attentiveness to the last R-MAC descriptor. To overcome this problem, we integrate R-MAC in our Y-Net for supervised training to avoid the class-agnostic problem of the descriptors in R-MAC. We argue that the convolutional layer activations before the R-MAC block can respond to the semantic information during classification training. The class-based semantic information is conveyed to the sample regions in the R-MAC block. Thus the different regions responding to the classification devote their varying contribution to generating the R-MAC descriptor. The learned R-MAC descriptor containing class-based semantic information can help address the SPDD problem by differentiating regions with a similar texture. Put an example of a chest X-ray, although two chest nodules with different sizes are the same texture, they are labeled benign and malignant and diagnosed with different diseases, respectively. As shown in Fig. 3, the two chest nodules that belonged to different classes vary differently in the learned feature maps of the convolutional layer. By training the R-MAC branch, the R-MAC descriptor can exploit the class-based semantic information of chest nodules and feedback to the convolutional layer in the R-MAC block, then the core node of the main branch.

Pixel-wise segmentation help extract features that emphasize the pathological abnormal regions. Beneficial from the segmentation training, the FPN branch explores the multi-scale subtle differences of pathological regions at different stages and then give feedback to the core node in the main branch. FPN leverages the convolutional features from low to high levels to extract multi-scale spatial information by building a pyramidal feature hierarchy. FPN has been a criteria component in the network of object detection and shows its powerful feature extraction capability to achieve higher accuracy [53, 54]. The multi-scale spatial information of pathological regions helps generate differentiating features in medical instance retrieval. In our Y-Net, we leverage the FPN components to extract multi-scale spatial information from medical images for semantic segmentation by setting the label of mask images semantically. The structure of the FPN branch is introduced as follows:

• •

  Following the core node in the main branch, the FPN branch provides two convolutional layers and three top-down blocks. Last, the FPN branch generates a predicted mask image for pixel-wise segmentation training. The segmentation loss is feedback to the FPN branch and the main branch to help exploit the multi-scale subtle differences of pathological regions at different stages.

• •

  In the feed-forwarding process, corresponding to the three bottom-up blocks {$32\times 64\times 64$ as b1, $64\times 32\times 32$ as b2, $128\times 16\times 16$ as b3} in the main branch, three top-down blocks {$32\times 16\times 16$ as t3, $32\times 32\times 32$ as t2, $32\times 64\times 64$ as t1} in the FPN branch merge them by element-wise addition. The outputs of two bottom-up blocks {b2, b3} convolute into $32$-dimensions channel. The output of the convolutional layer before block {t3} and two top-down blocks {t2, t3} are resized into twice times their width and height by bi-linear up-sampling. Then, the convolutional layer before top-down block {t3} and the bottom-up block {b3}, the top-down block {t3} and the bottom-up block {b2}, the top-down block {t2} and the bottom-up block {b1}, these pairs with the same spatial size are merged by element-wise summation. The addition operation of these three pairs generate the top-down blocks {t3, t2, t1} successively. Last, the top-down block {t3} convolutes into the size of the mask image.

In the main branch, the features of bottom-up blocks with lower-level information are more accurately localized by sub-sampling. In the FPN branch, the features of top-down blocks with higher-level information have a stronger spatial resolution by up-sampling. The features of top-down blocks can be enhanced by merging the features from bottom-up blocks. Based on the segmentation training, the learned pyramidal features can learn multi-scale spatial information and are feedback to the main branch’s core node. As shown in Fig. 4, the subtle CDR differences between two glaucoma images can be observed on the learned feature maps. The minor difference reflects the different information on different CDRs of same glaucoma, and the difference is encoded into the core node. Based on the pixel-wise segmentation training, we argue that the multi-scale subtle differences of pathological regions at different stages can be learned by the FPN branch to tackle the DPSD problem.

In Y-Net, we integrate the classification task in the R-MAC branch and the segmentation task in the FPN branch to learn the semantic and spatial information of pathological regions simultaneously. To balance the two tasks’ loss, we design a coupled loss to unify the classification and segmentation learning. In general, the gradient size is different in the convergence process of different tasks, and the sensitivity to different learning rates is also different. Unifying the scale of different loss functions can prevent the loss items with small gradients from being covered by the loss items with large gradients. Unifying the losses to the same order of magnitude can help improve the generalization of the learned features [55]. The coupled loss function is defined as:

where $\mathcal{L}_{r}$ is the circle loss [56] for the classification training, $\mathcal{L}_{l}$ denotes the cross-entropy (CE) loss for the pixel-wise segmentation training, and $\omega$ is the weight factor. In the circle loss, each similarity score is given different penalties according to its distance to the optimal effect. In the R-MAC branch, instead of the CE loss, we adopt the circle loss to preserve the class-aware similarity of pathological regions and help prevent minimizing intra-class distance.

Based on the coupled loss unifying the classification loss and segmentation loss, the main branch’s core node can effectively retain the multi-scale spatial information from segmentation training and the class-aware semantic information from classification training simultaneously. The convolutional feature maps from a certain convolutional layer can be viewed as an array of local features sampled from a dense sampling grid. In Fig. 5, the pathological region is the cup and disk of glaucoma. By observing the core node’s feature maps, the FPN branch focuses on exploring the pathological region (cup and disk), and the R-MAC branch concerns the highly activated region of the whole image (glaucoma). With the help of the coupled loss balancing the two losses, the Y-Net row’s feature maps confirm the effectiveness of preserving the information from the R-MAC branch and the FPN branch. Hence, the learned convolutional features of the core node can be used to generate hash-codes to combat pathological regions’ ambiguous manifestations.

Fundus [57] contains 650 annotated retina images. Each image is tagged with classification information and manually segmented the result of optic disc and cup. This dataset is obtained from a population-based study and is therefore suitable for evaluating glaucoma screening performance. In this dataset, 168 images from glaucomatous eyes and 482 images from normal eyes are classified. Manual CDR computed from the manually segmented disc, and cup boundaries are necessary for segmentation training. Based on the classification and segmentation labels, we split this dataset into the train set and the test set by ratio $9:1$. The test set of 65 consists of 16 glaucoma images and 49 normal images, and the train set of 585 images covers 152 glaucoma images and 433 normal images.

JSRT [58] provides 154 nodule and 93 non-nodule chest X-ray images. Each nodule case contains a nodule only, which is rated as benign or malignant by 20 different radiologists. A detailed delineation of the segmentation’s nodule is publicly available to train a lung segmentation [59]. This dataset annotates the lesion position and responding diagnosis. For example, the lesion region of a malignant image is located on the left lung’s upper lobe and diagnosed as lung cancer. The annotation images for segmentation tasks are binary images in which pixels are either 255 for the foreground or 0 for the background. We sample 138 images containing 89 malignant nodules and 49 benign nodules to form a train set and 16 images containing 11 malignant nodules and 5 benign nodules to form a test set. The ratio of the train set and the test set is $9:1$.

We mainly use mean average precision (mAP) for quantitative evaluation. In the returned list, mAP averages the ranks of images similar to the query image to measure the rank quality. The mAP is usually adopted for evaluating the retrieval performance [2, 1], and is calculated as follows:

where $R$ denotes the number of similar results for the current query image, $P(k)$ denotes the precision of top-$k$ retrieval results, $rel_{k}$ is a binary indicator function equaling 1 when the $k$-th retrieved results is similar to the current query image and 0 otherwise, and $n$ denotes the total number of retrieved results. Based on the class labels and the aim of instance retrieval assisting the clinician’s own decision-making by reviewing similar cases, the success criteria of similar images are defined as that the two images have similar pathological patterns.

Y-Net is compared against several representative approaches of instance-level retrieval. The comparative approaches are categorized as: weight feature aggregating on convolutional features, regional feature aggregating on convolutional features, and feature embedding from a full-connected layer.

• •

  Weight feature aggregating. CroW [38] estimates a spatial weighting of the features as a combination of convolutional feature maps across all channels of the layer. Features at locations with salient visual content are boosted while weights in non-salient locations are decreased. To explicitly leverage semantic information, CAM [39] obtains semantic-aware weights for convolutional features by exploiting the predicted classes. CAM generates a set of spatial maps highlighting the contribution of the regions within an image. Each map is used to weigh the convolutional features and generate a set of class vectors that are aggregated as the region vectors over the fixed region strategy of R-MAC. CAM inspires our R-MAC branch of Y-Net. BLCF [41] builds an efficient image representation by combining saliency weighting over convolutional features aggregated by using a large vocabulary with a bag of words (BoW) model [60]. SOLAR-Local [61] focuses on second-order spatial information to learn local patch descriptors without extra supervision. Based on the feature weighting strategy [62], it combines the second-order spatial attention and the second-order descriptor loss to improve image features for retrieval and matching.

• •

  Regional feature aggregating. R-MAC [42] is an aggregation method for convolutional features to generate a set of regional vectors by performing spatial max-pooling within a particular region. Building on the R-MAC descriptor, R-MAC + RPN [43, 62] can enhance the ability to focus on relevant regions in the image by replacing the rigid grid with a region proposal network (RPN) trained to localize regions of interest in images. Regional Attention [37] presents a context-aware regional attention network for tackling the problem of region-based feature aggregation suffering from the background clutter and varying importance of regions, especially in R-MAC, by weighting an attentive score of a region. Deep Vision + SOLO [47] is trained for instance-level retrieval of image- and region-wise representations pooled from an object detection CNN. In this experiment, we take advantage of the object proposals learned by SOLO [63] and their associated convolutional features to build an instance search pipeline.

• •

  Feature embedding. Three deep hashing methods using feature embedding are used to build benchmarks for our Y-Net, including DPSH [7], DSH [35], DRH [36], DDMH [16]. DPSH performs simultaneous feature learning and hash-code learning with deep neural networks by maximizing pairwise similarities. Inspired by DPSH, DSH proposes a triplet label-based deep hashing method to maximize the given triplet labels’ likelihood. DRH offers good separability of classes in hashing space while preserving semantic similarities in local embedding neighborhoods for supervised hashing of medical images through residual learning. DPSH and DSH use AlexNet [64] as the backbone. Recently, the residual block [49] has been used popularly as the backbone in deep hashing methods such as DRH and shows the advantage of feature extraction. In our Y-Net, the main branch also uses the residual block as the backbone. DDMH proposes a unique disentangled triplet loss to effectively push positive and negative sample pairs by desired Hamming distance discrepancies for hash-codes with different lengths.

Our Y-Net is implemented under the PyTorch framework, and experiments are run on Geforce RTX 2080 Ti. In our work, the indexing and similarity calculation for evaluation uses Faiss [65], a library for efficient similarity search and clustering of dense vectors. We use the mini-batch stochastic gradient descent with $0.9$ momentum. The mini-batch size of images is fixed as $32$, and the weight decay parameter is $0.001$. All deep models are trained from scratch with $500$ epochs. It spends approximately $3$ hours for training our Y-Net. The pixel-wise cross-entropy loss is used in the segmentation task. The circle loss [56] is used for classification training by using cosine similarity and setting a scale of $32$, a margin of $0.25$. The weight factor $\omega$ in the coupled loss is initially set as $0.5$. We use the $5$-fold cross-validation to select the best classification and segmentation model. The parameters of comparative methods are set according to their implementation details in the corresponding papers, and the best performance is reported. Based on top-$10$ retrieval results, we investigate our Y-Net’s performance over hash-code with lengths of $36$, $64$, $128$, $256$, respectively. According to Table 1, with the hash-code lengthen, the performance can correspondingly improve at the cost of storage and search efficiency. As a trade-off between performance and search cost, we report all the performances over $64$-bits hash-code for our Y-Net.

The following research questions will be answered by analyzing experimental results:

RQ1

Does our proposed Y-Net outperform the state-of-the-art methods on retrieval performance in medical instance retrieval?

RQ2

Can our proposed Y-Net help to combat the ambiguity of pathological regions in medical instance retrieval?

RQ3

What are the effectiveness of the R-MAC branch, the FPN branch, and the coupled loss in our proposed Y-Net framework?

RQ4

How is the retrieval efficiency of our proposed Y-Net?