Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

Semantic hashing is an extensively researched and applied technique across various domains (Luo et al., 2021b; TONG et al., 2024), particularly in the realm of large-scale image retrieval. They can be broadly classified into two categories: supervised (Chen and Cheung, 2019; Sun et al., 2022; Zhan et al., 2020; Liu et al., 2019; Tu et al., 2021) and unsupervised (Hu et al., 2017; Zhang et al., 2017; Yang et al., 2018; Luo et al., 2021a, b; Li and van Gemert, 2021; Jang and Cho, 2021). The primary distinction between these two approaches is the availability of supervised information. In this paper, we focus on unsupervised hashing methods, as they effectively leverage unlabeled data and enable practical applications.

Unsupervised hashing methods try to preserve similarities of original data in the Hamming space. With the development of deep learning, deep semantic hashing has attracted growing interest in efficient image search. These methods can be roughly classified into two categories. Firstly, weak supervised learning-based approaches aim to reconstruct similar structures using pre-trained models (Hu et al., 2017; Zhang et al., 2017; Shen et al., 2019; Zhang et al., 2020) or cluster methods (Yang et al., 2018; Luo et al., 2021a, b). These reconstructed structures or labels guide hash code learning via deep supervised hashing methods. Some of them (Yang et al., 2018, 2019; Jang et al., 2022) incorporate the idea of knowledge distillation, but their objective is to construct pseudo labels or signals rather than focusing on reducing the inference time. Secondly, self-supervised learning-based techniques incorporate popular self-supervised methods such as auto-encoders (Dai et al., 2017; Shen et al., 2020) and generative adversarial networks (GANs) (Dizaji et al., 2018; Song et al., 2018) into deep unsupervised hashing. Recently, several methods have adopted contrastive learning in unsupervised hashing (Luo et al., 2021b; Jang and Cho, 2021; Qiu et al., 2021; Wang et al., 2022). For example, CIBHash (Qiu et al., 2021) applies contrastive learning to unsupervised hashing from an information bottleneck perspective.

With the development of large backbones like ViT (Dosovitskiy et al., 2020), some advanced works (Gong et al., 2022; Jang et al., 2022) report a great improvement in semantic hashing when using such a powerful backbones. Unfortunately, the computational overhead of large-scale backbones conflicts with the efficiency requirements of semantic hashing, making it arduous to employ them in edge devices or frameworks (Cui et al., 2024). Therefore, there is a need to discover ways to reduce inference time in practice.

Knowledge distillation aims to transfer knowledge from one model (teacher) to another (student). It has gained widespread attention in recent years (Gou et al., 2021; Wang et al., 2021; Lu et al., 2024; Zhao et al., 2023; Liu et al., 2023). The primary challenge is to determine what knowledge the teacher has learned and how to distill it to the student (Hinton et al., 2015). This paper focuses on exploring knowledge distillation techniques suitable for image retrieval tasks.

Analyzing and exploiting the relation between data samples is a popular approach in image retrieval. In knowledge distillation, similarity-preserving knowledge (SP) (Tung and Mori, 2019) and relational knowledge distillation (RKD) (Park et al., 2019) aim to transfer knowledge by preserving the sample relations between input pairs. However, they use shallow relation modeling between features, which may be suboptimal for capturing complex inter-sample relations. To address this issue, some studies have integrated contrastive learning into knowledge distillation (Tian et al., 2019; Xu et al., 2020; Zhu et al., 2021; Yu et al., 2022). For example, CRD (Tian et al., 2019) combines knowledge distillation with contrastive learning to maximize the mutual information between teacher and student representations. Additionally, SSKD (Xu et al., 2020) employs contrastive tasks as self-supervised pretext tasks to enable richer knowledge extraction from the teacher to the student. CRCD (Zhu et al., 2021) also utilizes contrastive loss but focuses on the mutual relations of deep representations instead of the representations themselves. PACKD (Yu et al., 2022) further considers the correlation among intra-class samples. These approaches enable more effective modeling of complex relations for improved knowledge distillation in image retrieval.

However, existing knowledge distillation methods are designed for real-value representation, which does not consider some special problems in semantic hashing or property in hash codes. We modify our knowledge distillation method to cope with these special challenges in the semantic hashing domain.

Consider a database $X=\{x_{1},...,x_{N}\}$ comprising $N$ images. Semantic hashing targets to learn a hash function $f:x\mapsto h$ that maps each image $x$ to a low-dimensional binary vector $h\in\{-1,1\}^{b}$, referred to as a hash code, where $b$ denotes the dimensionality of $h$. This mapping aims to preserve the pairwise similarities between the images $x_{i}$ and $x_{j}$ in the Hamming space, characterized by the Hamming distance $D_{H}(h_{i},h_{j})=\sum_{k=1}^{N}1_{h_{ik}\neq h_{jk}}$.

When applying knowledge distillation methods to semantic hashing, we identify two search paradigms, including the Symmetric Semantic Hashing search Paradigm (SSHP) and the Asymmetric Semantic Hashing search Paradigm (ASHP).

The search paradigm typically involves online and offline stages. When using the KD technology, we have a pre-trained large teacher model $f_{t}:\mapsto h^{t}$ and a lightweight student model $f_{s}:x\mapsto h^{s}$ guided by the teacher model $f_{t}$. In the SSHP, the student model $f_{s}$ is used to precompute hash codes for candidate images, which can be used to construct the semantic hashing index (Norouzi et al., 2012). In the online stage, the student model $f_{s}$ is also used to extract the hash codes of query images, and the index is utilized to locate relevant images quickly. ASHP introduces a modification to the SSHP. ASHP still employs a lightweight student model $f_{s}$ during the online stage but uses the large teacher model $f_{t}$ during the offline stage. Due to this setting, ASHP can rapidly return results in the online stage while obtaining more accurate hash codes in the offline stage.

To achieve valid results in both SSHP and ASHP, the basic objective of knowledge distillation is semantic space alignment, which consists of two targets. Firstly, the student model should learn the individual-space knowledge from the teacher model $f_{t}$. It targets to force hash codes of the same image $x_{i}$ output in student model $f_{s}$ and teacher model $f_{t}$ close to each other. This can be formulated as follows:

where $D_{H}(\cdot,\cdot)$ represents a Hamming distance measurement, and the lower the value, the more similar the compared images. Secondly, the student model should learn the structural-semantic knowledge from the teacher model. The goal is to ensure that image pairs $(x_{i},x_{p})$ with similar semantics are mapped closely in the Hamming space while pairs $(x_{i},x_{n})$ with dissimilar semantics are far apart when they are inputted into student model $f_{s}$ and teacher model $f_{t}$. This objective can be expressed as follows:

where $N(i)$ denotes the negative set for $x_{i}$, $P(i)$ represents the positive set for $x_{i}$. Considering either of the two objects separately is not sufficient. If we only consider individual-space knowledge, it is hard to optimize all $x_{i}\in X$ to achieve the Eq. (1) because of the capacity gap between student and teacher models (Mirzadeh et al., 2020). Conversely, concentrating solely on structural-semantic knowledge, the original space position is ignored and may lead to suboptimal results.

To design a knowledge distillation objective to satisfy both Eq. (1) and Eq. (2), we still lack information on $N(i)$ and $P(i)$ in the unsupervised scenario. Inspired by SimCLR (Chen et al., 2020), we utilize the augmented images as the positive images of the anchor while including other images from a given batch as irrelevant images. As illustrated in Figure 2 (a), given $M$ randomly sampled images from the training set $X$, our contrastive training mini-batch consists of $2M$ images obtained by applying data augmentation on the sampled image. Then we get original sample set $B=\{x_{1},...,x_{M}\}$ and augmented sample set $B^{\prime}=\{x_{1^{\prime}},...,x_{M^{\prime}}\}$. The only relevant (positive) image for $x_{i}$ is its augmentation, denoted as $x_{i^{\prime}}$, while the other $2(M-1)$ images jointly form the set of negative samples $N(i)$. Using the teacher model, we obtain sample hash code set $H_{t}=\{h_{1}^{t},...,h_{M}^{t}\}$ and augmented sample hash code set $H_{t}^{\prime}=\{h_{1^{\prime}}^{t},...,h_{M^{\prime}}^{t}\}$. By inputting training batch $B$ into the student model $f_{s}$, we get $H_{s}=\{h_{1}^{s},...,h_{M}^{s}\}$. Subsequently, we propose a contrastive loss function as follows:

Here, $\phi(\cdot,\cdot)$ is the cosine similarity, $R(i)=\{N(i),i,i^{\prime}\}$ and $\alpha\in[0,1]$ is a hyper-parameter that controls whether the hash code $h_{i}^{s}$ should be closer to $h_{i}^{t}$ or $h_{i^{\prime}}^{t}$. To explore its behaviors, we derive the gradients with respect to the hash code $h_{i}^{s}$ as:

where $\rho_{1},\rho_{2},\rho_{3}$ are the weighted coefficients. Based on their gradients in Eq. (4), we can find the object of Eq. (3) is the generalization of the combination between the knowledge distillation targets described in Eq. (1) and Eq. (2) with coefficients (detailed proof is presented in Appendix A). Thus, it can facilitate the student model’s learning of individual-space knowledge and structural-semantic knowledge from the teacher model in Hamming space.

Although we have proposed the contrastive knowledge distillation Eq. (3) to support the SSHP and ASHP, relying solely on the proposed contrastive loss can not ensure a robust knowledge distillation process. For example, the hash code of anchor image $x_{i}$ and its augmentation $x_{i^{\prime}}$ should ideally be close in Hamming space. However, as depicted in the upper-right part of Figure 2 (a), due to the teacher model may produce inaccurate hash codes for augmentations, the hash codes of image $A$ and its augmentation $A^{\prime}$ may not be close in Hamming space. Consequently, these augmentations serve as noisy data, and the optimization may force the student model in the wrong direction according to Eq. (3). We term these augmentations as “offset positive samples” and evaluate their occurrence probability in Appendix D.

To ensure a robust optimization process, we propose a cluster-based method that explicitly detects and removes offset positive samples to learn the semantic output space of the teacher model effectively. As illustrated in Figure 2 (b), we first perform k-means clustering on the set of hash codes $H_{t}^{all}=\{h_{1}^{t},...,h_{N}^{t}\}$ of all training images and group them into $k$ clusters. In the unsupervised scenario, we can determine the value of $k$ by using the elbow method or silhouette analysis (Rousseeuw, 1987). Subsequently, we assign a pseudo label $y_{i}$ to image $x_{i}$ based on the closest centroid. In each training batch, augmentations $B^{\prime}=\{x_{1^{\prime}},...,x_{M^{\prime}}\}$ are also assigned their respective pseudo label $y_{i^{\prime}}$ based on the closest centroid. If the anchor image $x_{i}$ and its augmentation $x_{i^{\prime}}$ have different pseudo labels, we consider $x_{i^{\prime}}$ as an offset positive sample. Thus, we define a dynamic $\alpha_{i}^{\prime}$ to replace the original $\alpha$ in Eq. (3) as follows:

Additionally, this method can incidentally solve the problem of false negative samples (Saunshi et al., 2019). The negative sample set $N(i)$ is random and includes semantic content similar to the anchor image $x_{i}$. This inclusion can cause a significant distance between one image and its relevant image, thereby adversely affecting the optimization process. If two different images $x_{i}$ and $x_{j}$ belong to the same centroid in one training batch, we consider $x_{j}$ a false negative sample of $x_{i}$. Thus, we modify the set $R$ in Eq. (3) as follows:

Finally, the contrastive loss becomes:

The new contrastive loss is effective in detecting and removing the offset positive sample and false negative sample issues depicted in Figure 2 (c), where image $A$ eliminates the impact of offset positive sample $A^{\prime}$ and does not consider its relation with false negative samples $B$ and $B^{\prime}$. Thus Eq. (7) can alleviate the influence of noisy samples and achieve a robust knowledge distillation process.

To achieve semantic space alignment, we need to measure the similarity between two hash codes, where we apply cosine similarity $\phi(\cdot,\cdot)$ in the learning objective Eq. (7). However, current approaches may overlook the special distribution property of hash codes referred to as “bit independence.” Specifically, bit independence means each bit is independent of the others in the output distribution for all hash codes (He et al., 2011). It can better preserve the original locality structure of the data (Doan and Reddy, 2020) but may render it unsuitable to calculate the similarity as prior knowledge distillation methods directly. To address this property, we first explore the distribution of bits within one relevance set.

Specifically, we obtain $H_{t}^{all}=\{h_{1}^{t},...,h_{N}^{t}\}$ by using the candidate set $X$ and the trained teacher model $f_{t}$. We then generate relevance sets $\{G_{1},...,G_{k}\}$, $G_{m}=\{h_{1}^{m},...,h_{|G_{m}|}^{m}\}$, $h_{i}^{m}=[h_{i1}^{m},...,h_{ib}^{m}]$ based on their ground truth (e.g., class), where $k$ is the number of relevance sets, $|G_{m}|$ is the number of elements in $G_{m}$ and $h_{ij}^{m}\in\{-1,1\}$ is the j-th bit of hash code $h_{i}^{m}$. We randomly choose a relevance set $G_{m}$ and compute the frequency of 1 and -1 in each dimension for all hash codes in $G_{m}$ to plot the frequency histograms. Figure 3 shows the frequency histogram of eight randomly selected dimensions. We can observe two types of bits: (1) in some dimensions, the frequency of 1 and -1 is disparate (e.g., $b_{1},b_{3},b_{4},b_{5},b_{8}$), and (2) in other dimensions, the frequency of 1 and -1 is slightly disparate (e.g., $b_{2},b_{6},b_{7}$). Under the bit independence assumption, if 1 and -1 have an equal probability of occurring in a specific dimension within a relevance set, this dimension does not provide any useful semantic information because it tends to express random semantics. Worse still, these dimensions lead to the wrong optimization direction for structural knowledge in the learning process. As shown in the left part of Figure 2 (e), to stay away from images $C$ and $C^{\prime}$, image $A$ tends to move away from its relevant images $B$. We refer to such bit as “redundancy bit”.

To mitigate the bad effect of redundancy bits, we propose a bit mask mechanism that excludes redundancy bits in similarity calculation, as shown in Figure 2 (d). In the unsupervised scenario, we use the cluster results of all training hash codes $H_{t}^{all}$ obtained in Section 4.2 to get $k$ cluster sets $\{\mathcal{C}_{1},...,\mathcal{C}_{k}\}$ as relevance sets. Here, $\mathcal{C}_{i}=\{h_{1}^{i},...,h_{|\mathcal{C}_{i}|}^{i}\}$ and $|\mathcal{C}_{i}|$ is the number of elements in $\mathcal{C}_{i}$. Next, we calculate the expectation $e_{ir}$ of each bit for each relevance set $\mathcal{C}_{i}$ as follows:

where $h_{jr}^{i}$ is the r-th dimension of $h_{j}^{i}$. Then, we can define the i-th cluster’s bit mask as $e_{i}^{m}=[e_{i1}^{m},...,e_{ib}^{m}]$, where the r-th bit mask $e_{ir}^{m}$ for relevance set $\mathcal{C}_{i}$ is defined as:

Here, $\delta$ is a threshold. Then, we propose the Bit-mask Robust Contrastive knowledge Distillation (BRCD) as follows:

where $\hat{R}(i)=\{k,i^{\prime}|k\in N(i),y_{k}\neq y_{i}\}$ and $\varphi(\cdot,\cdot)$ represents the revised similarity function, which is defined as:

Here, $e_{c(a)}^{m}$ is the bit mask of the relevance set corresponding to image $a$. The motivation behind it is that if the expectation of one dimension in a relevance set is lower than the threshold $\delta$, it should be the redundancy bit. In Eq. (10), when measuring structural-semantic knowledge, the dimensions of redundancy bits are ignored. This setting prevents the optimization process from the suboptimal results. For instance, in the right part of Figure 2 (e), image $A$ avoids choosing the wrong optimization direction after eliminating the redundancy bit. Certainly, the assumption of “each bit is independent of the other” is a relative concept. Many semantic hashing models (Zheng et al., 2020; Li and van Gemert, 2021; Qiu et al., 2021) approach this target, but the complete bit independence is hard to grasp. Therefore, the hyper-parameter $\delta$ in Eq. (9) can be considered a reflection of bit independence. If a semantic hashing model can achieve bit independence well, a high value should be given to $\delta$; otherwise, a small value is appropriate.

In this experiment, we compare the mAP@K of different knowledge distillation methods on three datasets.

In this experiment, we further validate BRCD on different hashing models by using mAP@1000 on the CIFAR-10 dataset.

In this experiment, we validate our methods when using different backbones on the CIFAR-10 dataset. Specifically, we adopt several representative knowledge distillation methods that have been used earlier for comparison. The teacher model uses ViT_B_16 (Dosovitskiy et al., 2020) or ViT_L_16 as backbone, and the student model uses EfficientNetB0 (Tan and Le, 2019), ResNet18 (He et al., 2016), or MobileNetV2 (Sandler et al., 2018) as backbone. In prior experiments, our proposed BRCD achieves a significant advantage within the ASHP and most KD methods can’t get valid performance in this paradigm. Consequently, we select some representative KD methods for comparison under the SSHP in this experiment. We perform this setting in the CIFAR-10 dataset, use CIBHash as the hashing model, and set the code length to 64. Figure 4 shows the performance, from which we can find that BRCD is still better than other KD methods when using different backbones in the teacher model or student model. This outcome validates the generality of our method across diverse backbone scenarios.