Improved Deep Classwise Hashing With Centers Similarity Learning for Image Retrieval

Provided the label matrix $Y$, the pairwise similarity between any pair of samples $x_{i}$ and $x_{j}$ in $X$ can be represented as $s_{ij}$, where $s_{ij}=y_{i}^{T}y_{j}$. Thus, $s_{ij}=1$ if $x_{i}$ and $x_{j}$ shares at least one common label, while $s_{ij}=0$ otherwise. Most deep supervised hashing is based on pairwise/triplet labels [6, 7], which learns $f(\cdot)$ to map $X$ from the original space into $B$ in Hamming space while maintaining the semantics similarity in $s_{ij}$. Thus, the loss function $L_{p}$ preserving the pairwise similarity is summarized as follow:

The above loss function expects to minimize the Hamming distance between any pair of similar samples involving $x_{i}$ while maximizing the distance between any pair of dissimilar samples involving $x_{i}$. Since the expensive computation cost of (1), the pairwise-based deep supervised hashing usually samples a portion of data from the whole dataset for training. The insufficient utilization of the supervised information leads to the learned binary codes $B$ less discriminative for image retrieval.

Inspired from deep metric learning [12], DCWH [10] first applies a classwise loss for hashing learning, which directly uses the label information as supervision. The classwise loss can be formulated as a cross-entropy based normalized Gaussian distribution:

where $h_{i}=f(\Theta;x_{i})$ is the output of feature learning part, $M=\{\mu_{j}\}_{j=1}^{C}$ are the class centers, and $\sigma^{2}$ is a hyperparameter controlling the class variance. Note the scale parameter $\frac{1}{\sqrt{2\pi}\sigma}$ is omitted for clarity, since it does not affect the final result. $M$ is updated periodically as $\mu_{j}=\left(\sum_{i=1}^{N}\mathrm{y}_{ji}h_{i}\right)/n_{j}$, where $n_{j}$ is the number of samples with the $j$th class label. By introducing class centers, (2) aims to maximize the probability of assigning intra-class samples to the corresponding class center. However, it cannot guarantee the samples belonging to different classes but lying far from the mean of each Gaussian distribution is closer to their intra-class samples. An illustrative example is shown in Fig. 1. We can see, increasing the Hamming distance between different class centers helps to avoid mis-retrieved datapoints lying on the boundary. We elaborate on how to improve this centers discriminability from next section.

To alleviate the aforementioned issue in DCWH, the proposed IDCWH introduces a novel loss function on centers similarity. For simplicity, we adopt the formulation of original classwise loss in (2) as the base loss and regard the newly designed loss as a regularization term. Note that, we now treat class centers as learnable parameters instead of being updated manually and periodically. IDCWH provides an end-to-end framework combining efficient hashing codes and class centers learning. Specially, we aim to learn the network parameters $\Theta$ in the hashing learning part and the class centers $M=\{\mu_{j}\}_{j=1}^{C}$ in the loss layer. The proposed centers similarity learning has two functionalities: firstly, it dynamically attracts the class centers to concentrate on intra-class samples, which interacts with the original classwise loss to minimize the samples-center distance. Secondly, it maximizes the distance between intra-class samples and centers belonging to other classes for discriminative retrieval. The overview of IDCWH is shown in Fig. 2.

Since we train the network in mini-batch iteration, to capture the information of all the intra-class samples in the hashing codes space, we dynamically estimate the binary centers $U=\{u_{i}\}_{i=1}^{Z}$ from the current binary sum and newly sampled data. Recall (6), we denote the current binary sum of all the intra-class samples belonging to $j$th class as $\mathbf{sum}_{j}^{(t)}$. Then, for each iteration when new samples of $j$th class appear, $u_{j}$ is updated as follows:

where $m_{j}^{(t+1)}=\sum_{k=1}^{n_{j}}\mathbf{sgn}(h_{jk})$ represents the sum of binarized hashing outputs belonging to $j$th class in a newly-come mini-batch data.

With each forward pass in the training procedure, we obtain the hashing outputs $\{h_{i}\}$ of the network and the estimated binary centers $U$. Then we take the derivatives of $L$ w.r.t. $h_{i}$ and $\mu_{j}$ individually and update $\mu_{j}$ by gradient descent, shown as:

And ${\partial L_{2}}/{\partial\mu_{j}}$ can be derived as:

where $r_{ij}=\rho(0.5u_{i}^{T}\mu_{j})$. Then, the gradients are backpropagated to the feature learning part to update network parameters $\Theta$. The overall learning procedure of IDCWH is summarized in Algorithm 1.

After completing the training procedure, the learned network can generate binary hashing codes for unseen samples. We obtain the binary code $b_{test,i}$ of a given test sample $x_{i}$ by simply making a forward pass of the network and taking the binary operation, i.e.:

CIFAR-10 [14] is a standard dataset for image recognition with 60,000 images in 10 classes. The official split contains 50,000 training images with 5,000 images per class, and 10,000 testing images with 1,000 per class. We conduct two groups of experiments following different protocols as used in [6, 11, 7]. In the first protocol, we randomly choose 100 images per class (1000 images in total) as query set, and 500 images per class (5000 images in total) as training set. The rest is all used as the database. In the second protocol, all the images in the official training split are used for training and all the images in the test split are used for query.

CIFAR-100 [14] is similar to CIFAR-10 except that it has 100 classes with 600 images per class. We follow the protocol as in DCWH [10], which adopts the official training set and testing set of CIFAR-100. Specifically, we use 100 images per class (10,000 in total) as a query set and 500 images per class (50,000 in total) for training.

MS-COCO [15] is a popular dataset for image recognition, object segmentation and captioning. The official release of ‘train2014’ and ‘val2014’ contains 82,783 training images and 40,504 validation images. Each image is labelled by some of the 80 semantic concepts. Follow the protocol in [9], we combine the training and validation images together and randomly sample 5,000 images as query set and 10,000 images as a training set. The reminder images are all used as the database.

We adopt the standard similarity protocol for evaluation as in prior works. Specifically, for the single-label datasets, i.e. CIFAR-10 and CIFAR-100, images from the same class are regarded as similar, otherwise dissimilar. For the multi-label dataset, i.e. MS-COCO, any pair of images sharing at least one common label are regarded as similar, otherwise dissimilar. In addition to Mean Average Precision (MAP), four other commonly-used evaluation metrics are used in experiments, i.e. Precision-Recall curves (PR), Precision curve within two Hamming distances w.r.t. different code lengths (P@H=2), Precision curves w.r.t. different number of top returned images (P@N) and Recall curves with two Hamming distances w.r.t. different code lengths (R@H=2).

We compare our proposed method with original DCWH [10] and several classical deep supervised hashing methods including DPSH [6], DTSH [11], DSDH [7] and HashNet [9]. Besides, we also compare the performance with traditional supervised methods i.e. SDH [4] and FSDH [5], combined with deep feature learning. The two methods are denoted as SDH+CNN and FSDH+CNN. For the feature learning part, we simply employ GoogLeNet [16] with batch normalization [17] as the backbone network. The network has been pretrained on ImageNet. Since some other works apply the different network architecture with us, for the fair comparison, we re-run the experiments with the codes released by authors and we report performances of both the original works and their counterparts with GoogLeNet. The latter are marked with an asterisk, e.g. DPSH* and DSDH*.

The configurations of proposed IDCWH are shown as follows: we finetune the pretrianed GoogLeNet from the input to the last layer before the fully-connected layer and train the FC hashing layer (as shown in Fig. 2) for 150 epochs. We use mini-batch stochastic gradient descent (SGD) with momentum 0.9 and weight decay 5e-4. The initial learning rates are 1e-2 and 5e-3 for feature learning part and centers learning, respectively. And the learning rates are decayed by 0.1 every 50 epochs. The three hyperparameters adopted by cross-validation are $\sigma^{2}=4$, $\gamma=1$ and $\beta=0.01$. The batch size is fixed to 128 during all the experiments.

We first present the evaluation on CIFAR-10 dataset. Denote the first protocol and second protocol of the experiments as CIFAR-10 (mini) and CIFAR-10 (full), respectively. The MAP results of compared methods are shown in Table I. We can see that the proposed IDCWH outperforms other methods under almost all the cases of two experiment protocols. Among them, the classwise-based deep hashing methods perform better than other pairwise/triplet labels-based methods. And IDCWH surpasses the performance of DCWH by 1-2%. Specifically, in the first protocol, it achieves the best MAP performance with 86.8% under 32-bit codes. And the best MAP result on the second protocol is 96.9% under 24-bit codes. One can find, DPSH* and DTSH* with employed GoogLeNet backbone, perform significantly better than the original works with their counterpart networks. While the performance of the combination methods of traditional supervised hashing and deep feature learning, i.e., SDH+CNN and FSDH+CNN, is really poor. It verifies the superiority of end-to-end training manner of deep supervised hashing. One can also observe, by training with more samples, the performances on the second protocol of all the methods are greatly improved compared to those of the first protocol.

The performance on CIFAR-10 under other evaluation metrics is shown in Fig. 3. It is clear that the proposed IDCWH is robust as it achieves the best result under various evaluation metrics. From Fig. 3, one can observe two common phenomenons from all the compared methods. The first is, the curve of P@H=2 always first increases then drops with the growth of code length. The second is the overall trend of R@H=2 is decreasing with the increase in code length. The reason may be explained as: with the dimension of Hamming space, the learned binary codes become sparser, which leads to fewer data points fallen into the range within two Hamming distances.

We further present the experimental results on CIFAR-100 dataset. The performance of the MAP on CIFAR-100 is presented in Table II. We can see the proposed IDCWH achieves the best results under all the compared code lengths and it surpasses the second place, i.e. DCWH with an at least 4% margin. Specifically, it greatly improves the MAP scores to more than 80% with codes length not less than 24-bit, which is nearly 7% higher than DCWH under all the cases. While for 12-bit codes, it outperforms other methods with 76.42% MAP performance.

For other pairwise/triplet labels-based methods, the performance on CIFAR-100 is much poorer than that on CIFAR-10. However, IDCWH and DCWH, both of which are based on classwise loss, can still achieve promising results. Consider the increase in the number of classes and the less number of training images in CIFAR-100. It verifies the effectiveness of classwise hashing, which can better capture the semantics information for a large-scale dataset with small inter-class variations and significant intra-class variations. The performance on other evaluation metrics, i.e. P@2, P@N and PR curves, are illustrated in Fig. 4. We can see the proposed IDCWH again preforms the best under various evaluation metrics. Observe that, there is a drastic drop on the curve of P@N when N=500, this corresponds with the fact that the number of training images per class in CIFAR-100 is 500. One can also find that, among all the pairwise/triplet labels-based methods, DTSH performs much better than DPSH and DSDH.

Finally, we summarize the evaluation on the multi-labeled MS-COCO dataset. The performance with HashNet, DHN [18] and DNNH [19] are added, which are directly taken in [9] and we also run experiments with DSDH and DPSH methods. The MAP result of all the compared methods on MS-COCO is presented in Table III. We can see IDCWH performs slightly better than DCWH under each code length and it achieves the highest MAP scores among all the methods. Recall the formulation of IDCWH, it treats each class center independently as learnable hyperparameters instead of updating the center manually. Thus, it can be extended to multi-label cases feasibly without computing the average of any new combination of multi-class centers like in DCWH. One can also observe the performances of DPSH and DSDH are poor on MS-COCO. It again validates our expectation that, by leveraging the classwise difference between all the samples, it helps to generate more semantics-preserving binary codes.