Modality-Aware Representation Learning for Zero-shot Sketch-based Image Retrieval

Let $\mathcal{X}$ be a paired set of $n$ samples of a sketch and its corresponding real photo, where each pair is annotated with a class. The $i$-th sample is denoted by $(\textbf{S}^{(i)},\textbf{P}^{(i)},c^{(i)})\in\mathcal{X}$, where $\textbf{S}^{(i)}\in\mathbb{R}^{M\times N\times 3}$ is a sketch image of size $M\times N$, $\textbf{P}^{(i)}\in\mathbb{R}^{M^{\prime}\times N^{\prime}\times 3}$ is its corresponding ground truth photo of size $M^{\prime}\times N^{\prime}$. ($\mathbf{S}$ and $\mathbf{P}$ may have different size.) $c^{(i)}\in\mathcal{C}$ is the ground truth category of the pair, where $\mathcal{C}$ is a set of all categories. Each category $c\in\mathcal{C}$ is transformed into a textual caption, formatted as ‘a photo of a $c$’, denoted by $T\in|\mathcal{V}|^{L}$, where $\mathcal{V}$ is the vocabulary set and $L$ is the maximum length of the sentence.

On zero-shot tasks, $\mathcal{C}$ is further split to the seen classes ($\mathcal{C}_{\text{s}}$) and unseen classes ($\mathcal{C}_{\text{u}}$), used for training and testing, respectively, where $\mathcal{C}_{\text{s}}\cap\mathcal{C}_{\text{u}}=\varnothing$ and $\mathcal{C}=\mathcal{C}_{\text{s}}\cup\mathcal{C}_{\text{u}}$. The sketches and photos are split into train and test sets according to their labels; that is, $\mathcal{X}_{\text{train}}=\{(\mathbf{S}^{(i)},\mathbf{P}^{(i)},T^{(i)})\,|\,c^{(i)}\in\mathcal{C_{\text{s}}}\}$, $\mathcal{X}_{\text{test}}=\{(\mathbf{S}^{(i)},\mathbf{P}^{(i)})\,|\,c^{(i)}\in\mathcal{C_{\text{u}}}\}$.

In Sketch-based Image Retrieval (SBIR) task, a query sketch image $\mathbf{S}$ is given, and the model aims to retrieve relevant images from the candidate photos, either within the same category (category-level) or the exactly matched image instance (instance-level or fine-grained). For category-level SBIR, the photos are considered relevant with any sketch within the same category; that is, $\mathcal{X}_{c}=\{(\mathbf{S}^{(i)},\mathbf{P}^{(j)})\ |\ c^{(i)}=c^{(j)}=c\}$ is a set of correct pairs for a class $c\in\mathcal{C}$. In fine-graind SBIR, there always exists exactly one matched photo for each sketch; that is, $\mathcal{X}_{i}=\{(\mathbf{S}^{(i)},\mathbf{P}^{(i)})\}$ for a sample datum $i$. For Generalized ZS-SBIR, randomly sub-sampled examples from $\mathcal{X}_{\text{train}}$ are added to $\mathcal{X}_{\text{test}}$, where $\mathcal{X}_{\text{train}}$ remains unchanged.

Given $\mathbf{z}_{\text{img}}$ and $\mathbf{z}_{\text{txt}}$, we further align the embeddings by minimizing the following loss:

Our approach is distinguished from existing methods in that ours does not have a direct mechanism to align sketches and photos within the image modality, while the subtle differences between them are indirectly aligned through texts. This approach is opposed to the direct alignment via triplet learning in existing works [42, 16, 4, 13]. The biggest advantage of our approach over the previous direct triplet learning is that ours does not require paired training examples of sketches and photos, which are costly to collect. In order to directly train the model to distinguish the two, previous models have relied on positive pairs of a sketch and a photo. With our indirect approach, however, no explicit positive relationship between a sketch and a photo is utilized. Instead, both sketches and photos are embedded via a common image encoder and aligned with the associated text, taking advantage of the intricate representation capacity of the CLIP. This capability of leveraging unpaired datasets addresses the well-known issue of data scarcity in the community, aligning with the efforts of [1, 27] to tackle limited data availability. Our design also simplifies the contrastive loss term, removing the need for negative sampling.

Under this setting, our model is trained to separate its latent space into its constituent semantic and modality components. Once this is achieved, one can convert an embedding from one domain to another simply by

where $\mathbf{x}\in\mathbb{R}^{d}$ are the learned embedding (containing both semantics and modality), $\mathbf{m}\in\mathbb{R}^{d}$ are the modality encodings corresponding to each dataset, either source (src) and target (trg). We expect the converted representation in this way to yield improved performance on the retrieval tasks.

In addition, to reconstruct more precisely, we introduce a modality classifier on the reconstructed embeddings, $\mathbf{z}^{\prime}_{\text{img}}$ and $\mathbf{z}^{\prime}_{\text{txt}}$. For each, we minimize a CE loss $\mathcal{L}_{\text{mc}}$ defined as

where $\hat{m}_{j}\in\mathbb{R}^{C}$ is the predicted logits for the input embedding to belong to each modality, $m_{j}$ is the one-hot encoding of its ground-truth modality, and $C$ is the number of modalities. By minimizing the classification loss, we ensure that the reconstructed $\mathbf{z}^{\prime}_{\text{\{txt,img\}}}$ originate from separate classes recognizable by the classifier, irrespective of their initial modality. This approach helps emphasize the distinctions between different modalities.

Fine-grained SBIR (FG-SBIR) task aims to retrieve precisely matched photos from a sketch query. Unlike the category-level SBIR, merely retrieving images in the same category is insufficient. Intuitively, our proposed approach in Sec. 3.2 is not suitable for this task, as it mainly relies on the indirect connection between sketches and photos through coarse-grained textual descriptions, without direct enforcement based on paired examples. For this reason, our vanilla model is not suitable for finer-grained alignment.

However, we pose a question: can our model still perform well, if it is trained on the paired data? To answer this question, we adapt our model to the instance-level SBIR setting with some modifications guided by the inherent nature of the task. Specifically, we take a photo-sketch-text triplet as input, instead of an image-text pair, although we still use a common image encoder. It is inevitable in this fine-grained setting to take paired photo-sketch examples. Some loss functions are also modified, explained in detail subsequently. However, there is no major change to the model architecture, other than the straightforward extension to use three encoders depicted in Fig. 2.

The biggest change for fine-grained model is introduction of direct sketch-photo alignment. Specifically, we add another contrastive loss designed for sketch-photo alignment, defined as follows:

where CLIP$(\mathbf{a},\mathbf{b})$ is defined in Eq. (2).We balance this direct alignment with the existing text-guided indirect alignment from Eq. (1) by convex combination:

where $\mathcal{L}_{\text{clip-txt}}$ is equivalent as Eq. (1) but we use different notation to ensure that notation $\mathcal{L}_{\text{clip}}$ continues to represent the overall contrastive loss. With a hyperparameter $\lambda_{\text{clip-img}}$, we adjust the relative influence from the two types of contrastive losses.

We set $d$ to 512 and 768 for ZS-SBIR and FG-ZS-SBIR, respectively. We use pretrained CLIP [24] and ViT (ViT-B/32 for categorical and ViT-B/16 for fine-grained) as backbone. For FG-ZS-SBIR, $\textbf{z}_{\text{img}}\in\mathbb{R}^{768}$ is extracted from the $\mathbf{E}_{\text{img}}$ without the last projection layer. The text embedding is projected to the same size by a projection layer. Also, we strategically freeze the text encoder during training to enhance efficiency, supported by our findings that this maintains performance unaffected for FG-ZS-SBIR. Modality classifier is implemented with a FC layer.

We use AdamW optimizer with early stopping. The learning rate for the image/text encoder is set deliberately lower than the rest, to prevent catastrophic forgetting: (5e-8, 5e-3) for ZS-SBIR, and (5e-5, 1e-2) for FG-ZS-SBIR. We use batch size of 256 on Sketch-Ext and TU-Berlin-Ext for ZS-SBIR, while 64 on QuickDraw-Ext for ZS-SBIR and on Sketchy for FG-ZS-SBIR. We grid search the weight coefficients $\lambda_{\text{\{sem, mc, rec, ortho\}}}$ and set to (0.1, 0.1, 0.02, 0) for Sketchy-Ext, (0.1, 0.1, 0, 0) for TU-Berlin-Ext, and (0.2, 0.2, 0.2, 0.2) for QuickDraw-Ext. For FG-ZS-SBIR, we use (0.5, 0.1, 0.02, 0.1, 0.02) for $\lambda_{\text{\{ps, sem, mc, rec, ortho\}}}$. We report mean performance of three independent experiments. We implement our model with PyTorch [23] and experiment on a NVIDIA RTX A6000 GPU.

We also observe that our [original] and [converted] surpass [clip] version, indicating efficacy of the additional loss terms. Furthermore, [converted] surpassing [original] implies our methods effectively address the modality gap.

Interestingly, we observe that our [original] embeddings outperform our [converted] in this setting, unlike the ZS-SBIR task. We speculate that the conversion to the exact position in the target modality space is more challenging in the fine-grained setting, leading to a slight decline in performance with converted embeddings.

In general, a better result is achieved when more loss terms are used. Taking a closer look, however, we observe slightly different patterns across datasets. On Sketchy-Ext [18], most proposed losses, except for the orthogonal loss, demonstrate effectiveness when added. We speculate that the more stringent constraints imposed by the orthogonal loss slightly impede achieving superior outcomes. On QuickDraw-Ext [5], the inclusion of $\mathcal{L}_{\text{rec}}$ and $\mathcal{L}_{\text{mc}}$ leads to a slight decline in performance, while the orthogonal loss significantly improves the performance, unlike the case on Sketchy-Ext. This suggests the potential interactions among these loss terms. Experiments on TU-Berlin-Ext [18] show distinct trends. Introducing more terms brings about a decline in mAP, while the opposite is observed in precision.

As shown in Table 5, seeing more samples even from unrelated classes improves the results. This is aligned with observations in previous studies [1, 26, 27]. We emphasize that other methods assuming paired sketches and photos are not eligible to take advantage of these unimodal datasets.

Taking a deeper look, adding more sketches yields greater advantage than adding more photos. Augmenting sketches is more efficient but expensive. We leave further investigation as a potential future work.

Table 6 evaluates the effect of fine-tuning the text encoder. As expected, the performance slightly drops when the text encoder remains frozen. We interpret that realigning the text encoder leads to a more effective utilization of latent space, perhaps achieving higher uniformity [40] and better adaptation to SBIR datasets, while simultaneously preventing catastrophic forgetting.

For FG-SBIR, we explore several different values for $\lambda_{\text{clip-img}}$ in Table 7. We observe that alignment with the class hinders the exact alignment of identical instances between sketches and photos, even when we unfreeze the text encoder. This is in contrast to [26], which suggests that such alignment facilitates accurate alignment in the latent space.

We qualitatively compare our retrieval results with the strongest baseline, CLIP-AT [26], in Fig. 4 (categorical) and Fig. 5 (fine-grained). Refer to the supplementary material for more examples. Overall, our method retrieves correct photos for most classes.

For the fine-grained task, even incorrectly marked images turn out to be often correct. For instance, for a drawing of a wind turbine (row 2 in Fig. 5), all 5 photos retrieved by our model are indeed wind turbines, although they are not labeled in the ground truth. The baseline also retrieves 3 wind turbine images, but it includes two windmill images as well (at the 3rd and 5th). In row 3 and 4 of Fig. 5, our method ranks the true image lower than the baseline does, but the retrieved photos actually exhibits greater visual similarity with the queried sketches.