KALAHash: Knowledge-Anchored Low-Resource Adaptation for Deep Hashing

Assuming models have been pre-trained on several large source datasets, our goal is to adapt these pre-trained models to learn a hash function that maps images to binary codes while preserving semantic similarity with an extremely small training set $\mathbb{D}=\{\mathbf{x}_{i},\mathbf{y}_{i}\}_{i=1}^{N}$, including a set of $N$ images and their labels.

LoRA (Hu et al. 2022) is an efficient method for fine-tuning large models. It works by introducing small, trainable matrices into layers of a Transformer model (Vaswani et al. 2017). In a standard fully-connected layer, the output is calculated as

$$\mathbf{o}=\mathbf{W}\mathbf{x},$$

(1)

where $\mathbf{W}\in\mathbb{R}^{d\times k}$ is the pre-trained weight matrix; $\mathbf{x}\in\mathbb{R}^{k\times 1}$ is an input vector; $\mathbf{o}\in\mathbb{R}^{d\times 1}$ is the output vector. LoRA modifies Equation (1) by adding a low-rank update:

$$\hat{\mathbf{o}}=\mathbf{W}\mathbf{x}+\Delta\mathbf{W}\mathbf{x}=\mathbf{W}\mathbf{x}+\eta\mathbf{P}\mathbf{Q}\mathbf{x}.$$

(2)

Here, $\mathbf{Q}\in\mathbb{R}^{r\times k}$ and $\mathbf{P}\in\mathbb{R}^{d\times r}$ are small matrices that form the low-rank update. $r\ll\operatorname{min}(k,d)$. $\eta$ is a scale factor. The key is that the number of parameters in $\mathbf{P}$/$\mathbf{Q}$ is much smaller than the original weight matrix $\mathbf{W}$. During fine-tuning, only $\mathbf{Q}$ and $\mathbf{P}$ are updated, while the original model weights remain frozen. This parameter-efficient approach allows for quick adaptation of large models to new tasks with minimal additional parameters.

However, LoRA, while efficient for parameter updates, does not inherently incorporate task-specific knowledge or constraints. Its generic adaptation mechanism lacks the guidance needed to effectively map high-dimensional image features to compact binary hash codes, especially when provided with only a handful of examples per class, as illustrated in Figure 1. For deep hashing, especially with limited data, additional guidance about class relationships or desired hash code properties are crucial for generating discriminative hash codes.

To address these limitations and provide the necessary task-specific guidance, we propose leveraging class-level textual knowledge. This approach aims to inject semantic information directly into the adaptation process, bridging the gap between the limited visual data and the rich semantic understanding required for effective hash code generation. By incorporating textual descriptions of image categories, we can provide additional context and structure to guide the learning process, even in extremely low-resource scenarios. This textual knowledge serves as a form of prior information, helping to constrain the adaptation process and ensure that the resulting hash codes maintain semantic relevance. In the following section, we detail our method for extracting and utilizing this class-level textual knowledge to enhance the deep hashing process.

Figure 2 illustrates the architecture of the proposed method. We build our approach on the pre-trained CLIP model (Radford et al. 2021), including a text and a vision encoder consisting of multiple transformer layers.

The Text Encoder pre-extracts class-level textual knowledge $\mathbf{K}$ using category names. The Vision Encoder splits images into fixed-size patches which are projected into patch embeddings $\mathbf{V}_{0}$ by the Patch Embed module, and encodes $\mathbf{V}_{0}$ to vision tokens $\mathbf{V}_{L}$ by transformer layers, where $L$ denotes the number of transformer layers. During the encoding process, we introduce Class-Calibration LoRA (CLoRA) module to dynamically construct a weight adjustment matrix $\Delta\mathbf{W}$ by incorporating mapped knowledge $\hat{\mathbf{K}}$ and input vision tokens $\mathbf{V}_{i-1}$ from the $i$-th transformer layer to guide the fine-tuning process. For simplicity, we will omit the subscripts of vision tokens in the following sections. Then, the vision tokens $\mathbf{V}$ are mapped into hash features $\mathbf{H}$. To further improve the hash code generation, we employ Knowledge-Guided Discrete Optimization (KIDDO), a framework that injects the mapped textual knowledge $\mathbf{T}$ into the optimization process.

We use the Text Encoder to pre-extract class-level textual knowledge:

$$\mathbf{K}=[\mathbf{k}_{1},\mathbf{k}_{2},\cdots,\mathbf{k}_{C}]^{\top}\in\mathbb{R}^{C\times d_{t}},$$

(3)

where $C$ is the number of categories. Specifically, we create a prompt based on the hand-crafted template “a photo of a [CATEGORY].” For instance, given a category name dog, the prompt is instantiated as “a photo of a dog.” Next, the Word Embed module and Text Encoder map the prompt into a class-level textual knowledge embedding $\mathbf{k}_{i}$. Note that the konwledge generation only needs to be performed once in the whole process.

We observe that the weight adjustment matrix $\Delta\mathbf{W}$ in Equation (2) can be constructed by:

$$\Delta\mathbf{W}=\eta\mathbf{P}\mathbf{Q}=\eta\sum_{i=1}^{r}\mathbf{p}_{i}\mathbf{q}_{i}^{T},$$

(4)

where $\mathbf{q}_{i}\in\mathbb{R}^{k\times 1}$, $\mathbf{p}_{i}\in\mathbb{R}^{d\times 1}$.

As shown in Figure 3, to constrain the weight adjustment matrix space, spanning by $\mathbf{p}_{i}\mathbf{q}_{i}^{T}$, we replace $\mathbf{p}_{i}$ with the class-level textual knowledge $\mathbf{k}_{i}$ defined in Equation  (3) as anchors:

$$\Delta\mathbf{W}=\eta\sum_{i=1}^{r}\hat{\mathbf{k}}_{i}\mathbf{q}_{i}^{T},$$

(5)

where $\hat{\mathbf{k}}_{i}=\mathcal{F}(\mathbf{k}_{i})$, $\mathcal{F}$ is a linear layer and $\mathcal{F}(\cdot)\in\mathbb{R}^{d\times 1}$.

Considering that different inputs need different knowledge, we design a query-based strategy to dynamically select $r$ knowledge vectors from the knowledge pool $\hat{\mathbf{K}}$:

$$\hat{\mathbf{K}}^{v}=\operatorname{Top}_{r}(\operatorname{avg}(\mathbf{V}),\hat{\mathbf{K}}),$$

(6)

where $\mathbf{V}=[\mathbf{v}_{1},\cdots,\mathbf{v}_{t}]$ is the vision tokens; $\operatorname{avg}(\cdot)$ denotes the average pooling operation. $\operatorname{Top}_{r}(\cdot,\cdot)$ selects the top $r$ vectors in $\hat{\mathbf{K}}$ with the largest cosine similarity to $\operatorname{avg}(\mathbf{V})$, and $\hat{\mathbf{K}}^{v}=[\hat{\mathbf{k}}^{v}_{1},\cdots,\hat{\mathbf{k}}^{v}_{r}]$.

Finally, the weight adjustment matrix is constructed by:

$$\Delta\mathbf{W}=\eta\sum_{i=1}^{r}\hat{\mathbf{k}}^{v}_{i}\mathbf{q}_{i}^{T}.$$

(7)

We first employ a similarity loss function $\mathcal{L}_{\textrm{s}}$ and a quantization loss $\mathcal{L}_{\textrm{q}}$ that are widely used in deep hashing methods, which makes the Hamming distance of two similar points as small as possible and vice versa:

	$\displaystyle\mathcal{L}_{\textrm{s}}$	$\displaystyle=-\sum_{s_{ij}\in\mathbf{S}}\left(s_{ij}\theta_{ij}-\log\left(1+e^{\theta_{ij}}\right)\right),$		(8)
	$\displaystyle\mathcal{L}_{\textrm{q}}$	$\displaystyle=\|\mathbf{H}-\mathbf{B}\|_{2}^{2},$

where $s_{ij}$ is $1$ if image $i$ and $j$ belong to the same category otherwise $0$; $\theta_{ij}=\frac{1}{2}\mathbf{h}^{\top}_{i}\mathbf{h}_{j}$; $\mathbf{H}=[\mathbf{h}_{1},\cdots,\mathbf{h}_{n}]^{\top}$ are real-value image features generated from Hashing layer; $\mathbf{B}=[\mathbf{b}_{1},\cdots,\mathbf{b}_{n}]^{\top}$, $\mathbf{b}_{i}\in\{-1,1\}^{b}$ is the learned binary code and is randomly initialized before the training.

In low-resource settings, the limited number of training images may not be sufficient to cover all aspects of visual concepts, thus leading to over-fitting issues. We argue that language can be used as an abstract conceptual representation as anchor points to aid visual feature learning. For instance, while “dog” is visually represented in various ways that cannot all be covered by a limited number of images, they can be abstracted into a single linguistic concept “dog”.

Motivated by this, we add an alignment loss $\mathcal{L}_{a}$ between the learned binary codes $\mathbf{B}$ and the textual knowledge $\mathbf{K}$ to further improve the hash code generation by leveraging the textual knowledge as anchors:

$$\mathcal{L}_{\textrm{a}}=\|\mathbf{Y}-\mathbf{T}^{\top}\mathbf{B}\|_{2}^{2},$$

(9)

where $\mathbf{T}=\mathcal{G}(\mathbf{K})$, $\mathcal{G}(\cdot)$ denotes a fully-connected layer and $\mathbf{Y}=[\mathbf{y}_{1},\cdots,\mathbf{y}_{n}]\in\mathbb{R}^{C\times n}$ are one-hot label vectors where $C$ is the number of categories.

Finally, the loss function is

$$\mathcal{L}=\alpha\mathcal{L}_{\textrm{a}}+\beta\mathcal{L}_{\textrm{q}}+\gamma\mathcal{L}_{\textrm{s}},$$

(10)

where $\alpha$, $\beta$, and $\gamma$ are scalars that balance the loss values.

When optimizing the loss function $\mathcal{L}$ in Equation (10), it not only exploits the knowledge from the text encoder as complementary information to improve the image hash code generation but also enables the possibility of discrete optimization. To optimize the loss function in Equation (10), we use the standard backpropagation algorithm to learn $\mathbf{H}$ and $\mathbf{T}$. To optimize $\mathbf{B}$, we fix all the variables except for $\mathbf{B}$ and rewrite the optimization formula as

		$\displaystyle\min_{\mathbf{B}}\alpha\left\|\mathbf{Y}-\mathbf{T}^{\top}\mathbf{B}\right\|^{2}_{2}+\beta\left\|\mathbf{H}-\mathbf{B}\right\|^{2}_{2}$		(11)
		$\displaystyle\qquad\text{s.t. }\mathbf{B}\in\{-1,1\}^{N\times b}.$

Then, we adopt the discrete cyclic coordinate descent (DCC) method proposed by Shen et al. (2015) to optimize $\mathbf{B}$ column by column. The optimal solution of Equation (11) is

$$\mathbf{B}^{i}=\operatorname{sign}(\mathbf{S}^{i}-\mathbf{B}^{{}^{\prime}\top}\mathbf{T}^{{}^{\prime}}\mathbf{T}^{i}),$$

(12)

where $\mathbf{B}^{i}$ is the $i^{\text{th}}$ column of $\mathbf{B}$, $\mathbf{B}^{{}^{\prime}}$ is the matrix of $\mathbf{B}$ excluding $\mathbf{B}^{i}$; $\mathbf{S}^{i}$ is the $i^{\text{th}}$ row of matrix $\mathbf{S}$, $\mathbf{S}=\beta\mathbf{Y}\mathbf{T}+\gamma\mathbf{H}$, $\mathbf{S}^{{}^{\prime}}$ is the matrix of $\mathbf{S}$ excluding $\mathbf{S}^{i}$; $\mathbf{T}^{i}$ is the $i^{\text{th}}$ row of $\mathbf{T}$, $\mathbf{T}^{{}^{\prime}}$ is the matrix of $\mathbf{T}$ excluding $\mathbf{T}^{i}$.

In Equation (12), textual knowledge is injected into binary code $\mathbf{B}$, further improving the optimization process of hash code generation $\mathbf{H}$. In the following section, we will demonstrate the effectiveness of our proposed method.

We evaluate our proposed method on three standard benchmarks: NUS-WIDE (Chua et al. 2009), MS-COCO (Lin et al. 2014), and CIFAR-10 (Krizhevsky and Hinton 2009).

In our low-rank adaptation setting, we randomly split $N_{K}$ samples per class to create the training set. $N_{K}$ is $1$, $2$, $4$, or $8$ in our experiments. Following the standard evaluation protocol, we report the mean Average Precision at $K$ (mAP@$K$), the mean of average precision scores of the top $K$ retrieved images, to evaluate the retrieval performance. Specifically, we report mAP@$59000$ for CIFAR-10, mAP@$5000$ for NUS-WIDE, and mAP@$5000$ for MS-COCO, respectively. Notably, for multi-label datasets, two images are considered similar if they share at least one common label.

For a fair comparison, all methods, including baselines, use the same backbone model, optimizer, training hyper-parameters, etc.

If not specified, experiments are conducted on the CIFAR-10 dataset with the 1-shot setting. Detailed settings can be found in the code we provided.

We choose several representative deep hashing methods as baselines, including HashNet (Cao et al. 2017), DSDH (Li et al. 2017), DCH (Cao et al. 2018), GreedyHash (Su et al. 2018), CSQ (Yuan et al. 2020), OrthoHash (Hoe et al. 2021), HSWD (Doan, Yang, and Li 2022), and MDSH (Wang et al. 2023a).

Table 1 presents the mAP results for NUS-WIDE, MS-COCO, and CIFAR-10 across different low-resource settings ($1$-$8$ shots). We note that SOTA methods do not show absolute competitiveness in low-resource settings as on full-size datasets, highlighting the need for more sophisticated adaptation strategies. Our proposed KALAHash consistently outperforms all baselines across all datasets and shot settings. The performance improvements are particularly significant in the extreme low-resource scenarios ($1$-shot and $2$-shot). For CIFAR-10, KALAHash achieves $8.91\%$-$18.01\%$ improvements over the baselines in the $1$-shot setting. This performance gap remains substantial even as the number of shots increases, with KALAHash maintaining $3.63\%$-$7.41\%$ improvements in the $8$-shot setting. We can also observe the same trend on the NUS-WIDE and MS-COCO datasets. The multi-label nature of these two datasets highlights KALAHash’s ability to handle complex semantic relationships even with limited data.

Table 2 demonstrates KALAHash’s plug-and-play capability by applying CLoRA to various baseline methods. Across all baselines and datasets, adding CLoRA consistently improves performance. Specifically, the ranges of improvements are from $1.01\%$-$4.18\%$ on NUS-WIDE, $0.93\%$-$2.80\%$ on MS-COCO, and $0.96\%$-$12.52\%$ on CIFAR-10, respectively. The results underscore the versatility and effectiveness of our proposed method in enhancing existing deep hashing approaches in low-resource scenarios.

To understand the contribution of each component in KALAHash, we conduct ablation studies, as shown in Table 3. Removing CLoRA results in a significant performance drop across all datasets, with mAP decreasing by $2.38\%$, $3.83\%$, and $11.16\%$ on NUS-WIDE, MS-COCO, and CIFAR-10 respectively. Similarly, removing KIDDO leads to performance degradation, with mAP decreasing by $3.72\%$, $4.84\%$, and $6.65\%$ on NUS-WIDE, MS-COCO, and CIFAR-10 respectively, demonstrating the effectiveness of injecting textual knowledge into the optimization process.

We also compare CLoRA to other parameter-efficient fine-tuning techniques in Table 4. CLoRA outperforms standard LoRA (Hu et al. 2022), Prompt Tuning (Zhou et al. 2022), and Bias Tuning (Zaken, Goldberg, and Ravfogel 2022) across all datasets, demonstrating its effectiveness.

Figure 4 presents a comprehensive analysis of KALAHash’s performance as the number of shots increases from $1$ to $500$ on CIFAR-10. KALAHash consistently outperforms all baselines with limited training samples ($1$-$16$ shots). As the number of training samples increases, our approach still maintains a performance comparable to SOTA’s. This highlights the method’s effectiveness in extremely low-resource scenarios while also demonstrating its ability to maintain superior performance as more data becomes available. We note that the SOTA methods do not show absolute competitiveness. The reason may be that we lock the backbone and only fine-tune the FC layer, limiting its ability. However, even full fine-tuning of VLMs on full datasets can also lead to serious distribution shift issue. To demonstrate this, we report the result of MDSH-FFT on the full-size CIFAR-10 dataset as an orange cross, illustrated in Figure 4.

Figure 5 illustrates the performance of KALAHash across different backbone models, including SLIP ViT-S (Mu et al. 2022), CLIP ViT-B, and CLIP ViT-L (Radford et al. 2021). The results show that KALAHash’s performance scales well with larger backbone models, achieving higher mAP scores as the number of parameters increases. This demonstrates the method’s ability to leverage more powerful pre-trained models effectively.

Figure 6 provides a t-SNE visualization (Van der Maaten and Hinton 2008) of the learned hash codes for Full Fine-Tuning (FFT), Lock Backbone (LB), and our proposed method. The results show that the embedding space of FFT is collapsed due to the distribution shift issue. While LB improves this, it still cannot perform satisfactorily, as the red points are scattered in the embedding space. The visualization of KALAHash clearly shows that it produces more compact and well-separated clusters than the other methods. This qualitative result supports our quantitative findings and illustrates KALAHash’s ability to learn more discriminative hash codes even in low-resource settings.

Figure 7 examines the sensitivity of KALAHash to its key hyper-parameters, where “#Layers” denotes the number of layers inserted by CLoRA, and “Position” means which attention matrices are adjusted by CLoRA. The results show that KALAHash is robust to parameter changes, maintaining strong performance across a wide range of values.

We conducted a comprehensive analysis of inference times to evaluate the computational efficiency of our proposed CLoRA method across various backbones with and without CLoRA. As shown in Table 5, the integration of CLoRA introduces minimal computational overhead across all tested architectures demonstrating that CLoRA maintains the efficiency of the original models while providing the benefits of knowledge-anchored adaptation.