OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection

Image/Text Feature Extraction Since the input of our model involves vision and language two different modalities, we wish the features extracted from image and text have smaller modality gap, so that the text can somewhat correlate with image regions of the same semantics. Such a property could help the model find keypoint locations in query image. To this end, we resort to the CLIP [35] which is pre-trained on large-scale image-text pairs. The CLIP includes image and text encoders, where the text encoder is generally a transformer [43] while the image encoder can be CNN [21] or ViT [11]. We empirically found that the CLIP image encoder based on RN50 [35, 17], a CNN model, gives high efficiency in consideration of both performance and cost. Thus, we choose it as the default backbone. However, when extracting the image features, original CLIP image encoder only retains the classification token $\mathbf{x}_{\text{class}}$ and discards image tokens $\mathbf{X}$ by attention pooling as

While it is natural for image-text matching, we require the image tokens to recover the spatial locations of keypoints. Thus, we propose to obtain the image tokens via a projection composed by $\mathbf{W}_{v}$ and $\mathbf{W}_{o}$ as

In this way, we can not only handle the channel inconsistency between raw image tokens $\mathbf{X}$ and text features, but also reuse the projection layers $\mathbf{W}_{v}$ and $\mathbf{W}_{o}$.

Feature Adaptation Despite CLIP can provide general yet transferrable features, we need to adapt them into a new multimodal feature space that is more suitable for keypoint detection. Inspired by residual learning [17], we propose two adaptation nets $\mathcal{A}_{\text{v}}$ and $\mathcal{A}_{\text{t}}$ to respectively refine the obtained image and text features in a residual way:

The adaptation net $\mathcal{A}$ is highly scalable and can choose transformer [43] or bottleneck [17] as refinement blocks. We found that $\mathcal{A}$ can well adapt features.

Keypoint Prototype Set Once we obtain the adapted features of support image and text prompts, i.e., $\mathbf{X}^{\text{s}}$ and $\mathbf{t}_{n}$, we propose to convert them into keypoint prototypes, which are unified in a keypoint prototype set.

Correlation, Decoding and Fusion To discover the corresponding keypoints in query image $\mathbf{I}^{\text{q}}$, each keypoint prototype $\mathbf{\Psi}_{n}$, either VKP $\mathbf{\Psi}^{\text{v}}_{n}$ or TKP $\mathbf{\Psi}^{\text{t}}_{n}$, is required to be correlated with the query feature map $\mathbf{X}^{\text{q}}$. To this end, we leverage a correlation module $\mathcal{C}$ which takes both $\mathbf{\Psi}_{n}$ and $\mathbf{X}^{\text{q}}$ as input to produce the attentive feature maps $\mathbf{A}_{n}=\mathcal{C}(\mathbf{\Psi}_{n},\mathbf{X}^{\text{q}})$, where $\mathbf{A}_{n}\in\mathbb{R}^{l\times l\times d}$ and $n=1,2,\cdots,N$. For the correlation module $\mathcal{C}$, we explored multiple variants and found that the simple cross correlation already yields high performance, i.e., $\mathbf{A}_{n}=\mathbf{X}^{\text{q}}\odot\mathbf{\Psi}_{n}$, where $\odot$ denotes the channel-wise multiplication.

Improve Model by Contrastive Learning Since the keypoint prototype directly impacts performance, we aim for its representation ability to be as strong as possible. The same type of keypoints should have higher invariance across species, e.g., the ear of cat, dog, and cow; while different types are required to be sufficiently discriminative such that our model can distinguish the ambiguous texts or fine-grained texts like “the left ear” vs. “the right ear”. To this end, we propose to introduce contrative loss over TKPs. If randomly sampling two species $(s,s^{\prime})$ at a time and each species pertaining to an episode, we have pairwise sets of TKPs, i.e., $\mathcal{T}^{\text{t}}_{s}\!=\!\{\mathbf{\Psi}_{n}\}$ and $\mathcal{T}^{\text{t}}_{s^{\prime}}\!=\!\{\mathbf{\Psi}^{\prime}_{n}\}$, which can form similarity matrix as

where $\cos(\cdot,\cdot)$ denotes the cosine similarity. Then, the contrastive loss within textual modality $\mathcal{L}_{\text{tt}}$ becomes

where $\mathbb{I}$ is identity matrix and $\langle\cdot,\cdot\rangle$ denotes inner product. Since CLIP is trained via image-level alignment instead of keypoint-level alignment to text, we wish VKPs and TKPs are aligned so that we can exploit their respective advantages to help each other. This motivates us to propose the second contrastive loss between VKPs and TKPs. Similar to Eq. 7, we build $\mathcal{L}_{\text{vt}}$. We find that visual-textual alignment improves 1-shot testing, but unintentionally drops 0-shot performance slightly. We discover the principle behind this is due to the better clustering effects of textual keypoint representations than visual ones. To address this issue, we perform stop gradient to TKPs in $\mathcal{L}_{\text{vt}}$, thus enforcing the VKPs to align towards TKPs. We also find adding $\mathcal{L}_{\text{vv}}$ does not further advance scores. We conjecture the reason is after aligning VKPs to TKPs via $\mathcal{L}_{\text{vt}}$, the visual features will also improve discrimination along with textual features as $\mathcal{L}_{\text{tt}}$ is applied.

Optimization By incorporating the heatmap regression loss $\mathcal{L}_{\text{kp}}$, intra- and inter-modality contrastive loss $\mathcal{L}_{\text{tt}}$ and $\mathcal{L}_{\text{vt}}$, we have the overall loss as

where $\lambda_{i}$ is the loss weight. Note that contrastive losses serve as a regularization for the main task loss. By default, we set $\lambda_{1}=1$, $\lambda_{2}=\lambda_{3}=0.002$.

Visual Interpolation Following [27], we generate the auxiliary keypoints in visual domain via interpolation $\mathcal{I}^{\text{v}}(z;\mathbf{p}_{1},\mathbf{p}_{2})$, where $z\in(0,1)$ is the interpolation node and $\mathbf{p}_{i}\in\mathbb{R}^{2}$ is the end points that form an interpolation path. An off-the-shelf saliency detector [49] is adopted to filter the auxiliary keypoints out of the foreground [27]. Via interpolation $\mathcal{I}^{\text{v}}$ in visual domain, the auxiliary keypoints $\hat{\mathbf{p}}$ can boost the visual diversity of training keypoints.

Text Interpolation and Selection The $\mathcal{I}^{\text{v}}$ can assist the model in handling unseen keypoints in visual prompt, but helps little if prompted unseen texts. Consequently, we propose using text interpolation $\mathcal{I}^{\text{t}}(z;t(\mathbf{p}_{1}),t(\mathbf{p}_{2}),c)$ to mitigate this issue, where $t(\mathbf{p}_{i})$ are the names of end points $\mathbf{p}_{i}$ and $c$ is object category. Thanks to great language processing ability of LLM (e.g., GPT3.5 [4]), we propose to ask LLM a good question to reason the auxiliary keypoint texts given base keypoints (Fig. 4(b)). E.g., if we want to infer the keypoint text between “thigh” and “paw” of a “cat”, we can design a vanilla prompt to LLM as:

Apparently, we expect that the answers returned by LLM should include “knee” or “elbow”. In practice, the keypoint types are usually more complex thus the reasoning is challenging. To enhance reasoning, we improve the above vanilla prompt into a more advanced one using chain of thought (CoT) [48], whose key idea is to firstly provide an example with analysis and then ask the question. We provide the detailed prompts and examples in §0.A of Suppl.

Then, LLM is able to return the parsed texts regarding $\langle$keypoint$\rangle$ and $\langle$object$\rangle$, i.e., the “left eye”, “nose”, and “cat” in this example. Subsequently, we can directly leverage the parsed results to synthesize simple text prompts to instruct our model and predicting all the keypoints corresponding to the text. In this way, the problem of diverse text prompting can be converted to simple text prompting, thus opening the language diversity for keypoint detection.

Datasets and Splits. 1) Animal pose dataset [5] has five mammal species cat, dog, cow, horse, and sheep, with over 6000 instances with keypoint annotations. Each animal species is alternately chosen as unseen species for testing while the remaining four as seen species for training, which yields five subproblems; 2) AwA [2] has 35 diverse animal species with 10064 images. For AwA, 25 species are for training and 10 for testing; 3) CUB [45] consists of 200 species with 15 keypoint annotations. We use 100 species for training, 50 for validation, and 50 for testing; 4) NABird [42] is a larger dataset than CUB with 555 categories, 11 types of annotated body parts, and 48,562 images. The species split is 333, 111, and 111 for training, validation, and testing, respectively. To test model generalization, we follow [27] to split keypoints into base and novel sets, reporting the performance of both base and novel keypoint detection on unseen species.

Metric. The percentage of correct keypoints (PCK) is used. A predicted keypoint is correct if its distance to GT $d\!\leq\!\rho\!\cdot\!\max(w_{\text{bbx}},h_{\text{bbx}})$, where $w_{\text{bbx}}$ and $h_{\text{bbx}}$ are the edges of object bounding box. Following [27], we set $\rho$ to $0.1$.

Implementation Details. The input image size for all models is $384\times 384$. We freeze the CLIP text encoder while finetune last two layers of image encoder. The temperature $\tau$ used in contrastive loss is 0.05. Same to [27], the visual auxiliary keypoints are generated in pre-defined interpolation path and we set $z\!=\!0.5$. For text interpolation, to reduce randomness, we use GPT3.5 to reason $R=3$ times for all datasets except CUB with $R=10$. Our model is trained with 40k episodes and reports results using 1000 test episodes.

Compared Methods. For few-shot keypoint detection (FSKD), as previous works, we compare the few-shot learning models such as ProtoNet [36], RelationNet [38], and LwoF [15], and FSKD-dedicated works FSKD-R/-D [27]. For zero-shot keypoint detection (ZSKD), we adopt the source code of CLAMP [56] and compare it in our experiments for fairness. Moreover, we build a Baseline shared same backbone with our model but no using the specifically proposed auxiliary texts, contrastive loss, etc. We denote our method as OpenKD.

Comparisons on FSKD and ZSKD: We comprehensively conduct the few-shot keypoint detection across four datasets. Table 2 shows our OpenKD model greatly outperforms compared methods in 15 out of 18 tasks in detecting novel or base keypoints. Moreover, if adding the texts during testing, the performance of our model can be further improved, which yields 63.19% in Animal pose dataset (63.19% vs. 44.75% of prior-art FSKD-D) and 79.02% in AwA. We also notice the improvements on CUB and NAB are modest, which might be due to the auxiliary texts are relatively harder to reason, thus bringing less bonus to few-shot model. For ZSKD (Table 3), again, our model outperforms other methods (in 17 out of 18 tasks) and the improvement of CLAMP^† on novel keypoint detection strongly shows the benefits of adding auxiliary texts into training.

Why 0-shot can outperform 1-shot? We may observe the 0-shot testing scores on base keypoints are always higher than 1-shot testing. In this case, the training texts are accurate (cf. auxiliary text is noisy) and have high semantic similarity to testing texts. We dig more to visualize the corresponding text and keypoint features. Fig. 5 shows that textual keypoint features have better clustering effects and lower variance than visual ones, which suggests textual representations of base keypoints are better thus 0-shot yields higher scores.

Contrastive Learning (CL). After applying $\mathcal{L}_{\text{tt}}$, we observe the 0-shot performance significantly improves, e.g. 84.00% vs. 81.67% (2nd row, Table 4(a)), as CL improves the discrimination between text features. However, when adding visual-textual alignment (3rd row), 1-shot testing scores boost but slightly decreasing 0-shot scores. We suspect it is due to the imbalance of representation quality between multimodal features. After adding stop gradient, it mitigates negative impacts from “weak” modality, thus achieving higher scores (4th row). We also observe $\mathcal{L}_{\text{vv}}$ does not further improve scores in our losses (5th row).

Text Interpolation. Table 4(b) shows that using chain of thought (CoT) significantly improves performance, especially on AwA (3rd row), as CoT improves the quality of reasoned texts. Moreover, after applying FTC, the scores can strike a high balance in all datasets (5-th row), as our FTC rejects low-quality texts and further reduces noise after including visual features into decision. We also explore the repetition $R$ for text reasoning. Table 5(a) shows the importance of setting $R$ more than 1.

Diverse Text Prompting. To evaluate our model under diverse text prompting, we randomly sample 1000 diverse texts from each dataset to conduct zero-shot testing. As shown in Table 6, after coupling with GPT3.5, our model strongly keeps the performance even under diverse texts (with less than 3% drop). The strong results compared to no parsing shows leveraging LLM as a parser to handle language diversity is a possible solution. Moreover, we also found coupling with GPT3.5 has higher performance than Vicuna, as GPT3.5 owns stronger keypoint text parsing ability, e.g., 96% vs. 93% in AwA (Table 5(b)).