VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis

Human-object interaction (HOI) detection is an active area of research in computer vision. As noted in the introduction, HOI detection is a high-level task built on top of object detection. Existing HOI detection methods can be categorized into two main approaches: one-stage models and two-stage models.

One-stage models[31, 37] aim to detect human-object interactions in a single feedforward pass through a neural network. These models take an image as input and directly output bounding boxes for humans, objects, and their interactions. While computationally efficient, one-stage models struggle to model complex interactions and scale to large numbers of interaction categories. In contrast, two-stage models[6, 25, 49, 50, 51, 12] separate the tasks of detecting humans and objects from modeling their interactions. In the first stage, off-the-shelf object detectors are used to localize candidate humans and objects in the image. The second stage then classifies their relationship based on appearance and spatial cues. By decomposing the problem, two-stage models are able to achieve higher accuracy but at the cost of reduced speed.

Another content [47, 26, 16, 18, 34, 52] recent work has focused on improving both branches of HOI detection. For one-stage models, newer architectures incorporate attention mechanisms and graphical networks to better model interactions[16, 18, 34]. For two-stage models, progress has been made in embedding space designs and developing robust spatial models[52].

From another perspective, GEN-VLKT[26] proposes a Visual-Linguistic Knowledge Transfer (VLKT) strategy that leverages CLIP to enhance interaction understanding. It uses CLIP text embeddings to initialize the HOI classifiers and mimics CLIP image features. RLIP[47, 48] proposes a pre-training strategy to train a robust backbone network by aligning the image representations of entities and relations with their corresponding text descriptions. Both works are impressive approaches in that leverage VLM and pre-training strategy.

Integrating vision and language has been a long-standing goal in artificial intelligence. Early work focused on image captioning[39], generating textual descriptions of image contents. Recent years have seen rapid progress in visual question answering[2, 1], enabled by large-scale datasets[7] and deep neural encoder-decoder models[43].

More advanced vision-language tasks require a tighter integration between the visual and linguistic modalities. This has led to a surge of interest in unified multimodal representation models that can process both images and text within a single framework[3].

One line of work explores joint embedding models to learn aligned vector representations for image regions and language fragments[19, 15]. However, these approaches do not explicitly model interactions between modalities.

Recently, pretrained language models, such as BERT[11] and GPT[4], have significantly influenced the vision-language domain. Researchers have extended these models to handle multimodal inputs, leveraging their contextual understanding of text. More recent methods like VisualBERT[23] and LXMERT[38] utilize pretrained BERT weights for joint image-text comprehension.

Unified architectures like ViLBERT[30], VL-BERT[36], UNITER[8], BLIP[21] integrate masked language modeling objectives alongside paired image-text prediction tasks within a single Transformer model. These models set new state-of-the-art results on downstream tasks like visual question answering (VQA), visual reasoning, and image retrieval.

Specifically, we utilize BLIP as our vision-language model backbone. BLIP uses a flexible multimodal encoder-decoder model that can handle both understanding and generation tasks. It also improves training data quality through generating new image captions and filtering noise. These advantages lead to strong performance.

Instruction-based VLMs like GPT-4V[33], LLaVA[28], and InstructBLIP[10] are a new type of large vision-language model that are trained to follow natural language instructions and prompts to perform various tasks. A key advantage these models have over traditional fine-tuning is they can adapt to new tasks without needing gradient updates or lots of specific training data. The instruction format allows rapid adaptation. However, still challenges exist around instruction following, ambiguity, high resource usage, and potential biases.

While large models like CLIP[35] excel at overall image-text similarity, their limitations in pinpointing specific objects become apparent. This paper takes a step forward, focusing on BLIP[21, 20], a VLM specifically designed for object-level understanding, and its potential as a powerful teacher model for knowledge distillation in localization tasks. Prior works[26, 32] have utilized CLIP as a teacher model, achieving notable results. However, we argue that BLIP offers distinct advantages for localization due to its inherent strengths.

• •

  Object-centric Design: BLIP learns representations for both images and object-specific descriptions, enabling pinpoint localization beyond CLIP’s image-level approach.

• •

  Fine-grained Matching: Unlike CLIP’s simple cosine distance, BLIP employs sub-word level matching, connecting individual textual words to image regions, and facilitating precise object identification.

• •

  Specialized Training Data: BLIP is pre-trained on object-centric datasets with detailed descriptions, equipping it with nuanced understanding of object features and spatial relationships, crucial for localization.

Figure 1, suggests that the VLM model is able to better capture the semantic relationships between images and text for natural interactions. In contrast, the CLIP similarity scores are not as discriminative between positive and negative triplets. This suggests that CLIP may not be as effective at capturing the more nuanced relationships between images and text.

Human-object interaction (HOI) detection aims to locate human and object instance in an image and classify their interaction relationship. Formally, given an image $I$, the objective is to detect a set of $N$ interacted human-object pairs $\{(h_{i},o_{i},v_{i})\}$ where $h_{i}\text{ and }o_{i}$ represent detected human and object. $v_{i}$ denotes the interaction between $h_{i}\text{ and }o_{i}$.

Each $h_{i}$ and $o_{i}$ is represented by a bounding box $b_{i}^{h}$ and $b_{i}^{o}$ respectively. The overall HOI prediction for the image can be denoted as:

where $H$ is the final output containing $N$ detected HOI triplets. The HOI task can be decomposed into two sub-problems: Human and object detection $H_{det}=\{(b_{i}^{h},b_{i}^{o},o_{i})\}$ and interaction classification given information of human and object, $H_{hoi}=\{(v_{i}|b_{i}^{h},b_{i}^{o},o_{i})\}$.

The objective of the HOI triplet association is to convert these triplets into grounded natural language representations to leverage the language comprehension capabilities of vision-language models (VLMs). Our model first predicts a set of $N$ predicted HOI triplets $T=\{(\hat{h}_{1},\hat{o}_{1},\hat{v}_{1}),...,(\hat{h}_{N},\hat{o}_{N},\hat{v}_{N})\}$ from the input image $I$, where $\hat{h},\hat{o},\hat{v}$ are the detected human, object and predicted interaction verb, respectively.

Given these predictions, we extract the detected object and interaction class names $o_{i}$ and $v_{i}$ from the classifier outputs for each triplet. We exclude any triplets predicted as “no interaction” or “no object”, denoted by $\oslash$, as determined by the Hungarian matching. For simplicity, we set all predicted human classes $\hat{h}$ as “A person”. We then construct a positive sentence $S^{pos}_{i}$ for each triplet using the template:

The examples of these sentences are described in Figure 1. The generated sentences may not be grammatically correct, since the class names in most HOI datasets are not designed to produce fluent phrases. Simply concatenating the human, verb, and object classes can result in unnatural or awkward language. However, these grounded sentences still provide useful context and supervision for knowledge distillation from the pre-trained VLM. The key elements of human, action, and object capture the core semantics of the visual HOI detections in a textual representation. While not linguistically flawless, this allows transferring relevant knowledge about humans, objects, and their interactions from the VLM to the HOI model.

The paired positive and negative sentences provide contrasting language context for the VLM to comprehend the visual HOI concept deeply. This gives us a set of positive sentences $S^{pos}=\{s^{+}_{1},...,s^{+}_{n}\}$ and a set of negative sentences $S^{neg}=\{s^{-}_{1},...,s^{-}_{m}\}$ that are semantically grounded in the visual HOI predictions. By optimizing the VLM representations on these contrasting text sets, we enable the model to learn fine-grained HOI concepts based on its language priors. This facilitates transferring the VLM’s comprehensive language understanding to improve HOI prediction.

To harness the image-text matching capabilities of VLMs, we compute similarity scores between a given image $I$ and paired sentence sets: $S^{\text{pos}}$ for positive (correct) associations and $S^{\text{neg}}$ for negative (incorrect) ones. The calculation is formalized as:

where $\text{sim}(I,S)$ denotes the image-text similarity score obtained from the VLM for the image $I$ and sentence set $S$.

Ideally, the similarity scores of positive sets should be as close to zero as possible, while the similarity scores of negative sets should be as high as possible. However, Figure 1 illustrates that similarity scores cannot be zero for positive triplets. This ensures that the loss is never zero, even when all predictions are correct. We will discuss later how this problem inhibits the optimization of other losses.

To explicitly distill the rich knowledge encoded in the VLM’s parameters, we employ these scores in a contrastive learning loss:

Here, $\alpha$ serves as positive margins that anchor the contrastive loss. This loss function is structured to optimize the pairing of images with their corresponding positive sentences while disassociating them from negative ones.

The key insight is that the pre-trained vision-language model has learned a robust joint distribution over both visual and textual modalities through extensive pre-training. By optimizing our HOI network to align with the VLM’s image-text similarities via contrastive loss, we essentially distill this strong prior knowledge into our model. This guides the HOI network to find an optimal parameter configuration that encodes visual-linguistic concepts effectively. This distillation leads to more efficient optimization and substantial gains for rare HOI categories that lack sufficient training examples. In essence, the VLM’s contextual knowledge helps regularize the HOI model, reducing overfitting and improving generalization.

Following prior query-based HOI detection methods[16, 18, 49], we utilize the Hungarian matching algorithm to associate predicted triplets with ground truth triplets. As illustrated in Figure 3, our network consists of a DETR-based encoder and three parallel decoder branches with task-specific queries $Q^{p}$, $Q^{a}$, and $Q^{o}$ for human, object, and interaction prediction respectively. We select BLIP[21] as the VLM to compute image-text matching scores because it has higher performance relative to other methods in terms of computational efficiency. In principle, any sufficiently large VLM can be utilized and continue to improve with more compute.

In addition to the contrastive image-text matching loss $\mathcal{L}_{\text{ITM}}$, we use a standard HOI detection loss $\mathcal{L}_{\text{HOI}}$ defined as:

Where $\mathcal{L_{\text{L1}}}$ and $\mathcal{L_{\text{GIoU}}}$ are regression losses for predicting human and object bounding boxes, $\mathcal{L_{\text{oc}}}$ is a classification loss for detecting object categories, $\mathcal{L_{\text{ic}}}$ is a verb classification loss for interaction predictions, and $\lambda_{i}$ is weighting hyper-parameter.

We train our model end-to-end by jointly optimizing the HOI detection loss $\mathcal{L}_{\text{{HOI}}}$ and the image-text matching knowledge distillation loss $\mathcal{L}_{\text{ITM}}$:

The VLM is kept frozen during training to provide fixed similarity computations. At inference time, we simply feed an image into our trained model to generate the detected HOI triplets. The VLM and image-text matching loss are only used during training for knowledge transfer and are not needed at test time. Therefore, the number of learnable parameters are same as the baseline method.

Our model uses a ResNet-50 CNN backbone followed by a 6-layer transformer encoder, with $L=6$ parallel prediction branches. We set $N=64$ queries for HICO-DET and $N=100$ for V-COCO following prior work[49, 18]. The loss weights $\lambda_{i}$ are set to 2.5, 1, 1, 1 for $\mathcal{L}_{\text{L1}}$, $\mathcal{L}_{\text{GIoU}}$, $\mathcal{L}_{\text{oc}}$, and $\mathcal{L}_{\text{ic}}$ respectively. We initialize from a DETR model[5] pretrained on COCO[27] and optimize using AdamW[29] with weight decay 1e-4. The CNN backbone has a learning rate of 1e-5 while all other components use 1e-4, trained for 100 epochs. For V-COCO, the CNN weights are frozen to prevent overfitting and the learning rate is reduced to 4e-5. All experiments use a batch size of 4 on 4 RTX 3090ti GPUs. These optimized hyperparameters enable efficient end-to-end training of our method.

We conduct experiments on two standard HOI detection benchmarks: HICO-DET[14] and V-COCO[13].

HICO-DET consists of 47,776 images, with 38,118 for training and 9,658 for testing. It contains 600 HOI categories composed of 80 object classes and 117 action verbs.

V-COCO is a subset of 10,396 COCO images, split into 5,400 train and 4,964 test images. It has 29 action classes including 4 body motions without object interactions, and the same 80 objects as HICO-DET. In total, V-COCO has 263 unique HOI triplets.

Following the evaluation protocol from[49, 18], we use mean Average Precision (mAP) as the metric for measuring HOI detection performance. A predicted triplet is considered a true positive if: 1) the detected human and object boxes have over 50% IOU with ground truth, and 2) the predicted interaction categories match the labels.

On HICO-DET, we report mAP on the full 600 classes, 138 rare classes, and 462 non-rare classes. For V-COCO, we evaluate on S1 with 29 actions including body motions, and S2 with 25 actions excluding no-object categories. By benchmarking on these diverse splits, we comprehensively analyze our method’s HOI detection capabilities.

We extensively evaluate our VLM-HOI method against state-of-the-art approaches on two standard HOI detection benchmarks - HICO-DET[14] and V-COCO[13].

On HICO-DET (Table 1), our proposed knowledge distillation method improves upon GEN-VLKT[26], which also utilizes external knowledge but only achieves 33.75% and 36.78% mAP on default and known object settings. In contrast, our VLM-HOI obtains 33.64% and 36.88% mAP, highlighting the benefits of our image-text matching objective.

Analyzing the long-tail distribution, the proposed VLM-HOI demonstrates consistent gains over GEN-VLKT, with improvements of 0.62% and 0.55% mAP on rare interactions. This validates our approach of transforming visual detections into grounded text for optimized knowledge transfer from VLMs.

Similarly, on V-COCO (Table 2), the proposed method obtains 69.5% and 72.1% mAP on Scenario 1 and 2, surpassing all previous methods. Here, VLM-HOI outperforms GEN-VLKT by large margins of 7.1% AP on Scenario 1 and 7.7% AP on Scenario 2. This highlights the benefits of our image-text matching objective for richer knowledge transfer.

Notably, we improve over MUREN[18], which uses the same transformer backbone as our method. This shows that the performance gains come from our proposed VLM knowledge distillation, rather than just model architecture.

Effects of the Margin: A critical hyperparameter in our image-text contrastive loss is the positive sample margin $\alpha$. This margin controls the lower bound on the similarity scores between positive text prompts and the image. Intuitively, it determines how close we want to pull the positive grounded sentence representations to the image representation.

If $\alpha$ is too small or zero, the loss will not effectively separate positive and negative samples, as the lower bound on positive scores is too low. On the other hand, if $\alpha$ is too large, it can overly restrict and dominate the loss landscape.

Table 4 shows the impact of $\alpha$ by evaluating models trained with different values. As shown in Figure 1, the typical similarity scores between positive triplets and the image range from 1 to 2. Setting $\alpha=0$ gives a mAP drop of 1% compared to $\alpha=1$, indicating that a zero margin fails to sufficiently separate distributions.

Increasing $\alpha=2$ also decreases performance by 0.6% mAP, suggesting that too large of a margin overly restricts the optimization. The best balance is achieved with $\alpha=1$, which aligns well with the expected positive sample score distribution.

The positive margin hyperparameter should be tuned to match the anticipated similarity score range, in order to enable effective optimization. This provides a principle for setting this important variable when applying our image-text contrastive loss to new domains beyond HOI detection.

As shown in Table 4, using just the interaction verb as the prompt (“is holding”) decreases performance by 1.2% mAP compared to the full sentence. This demonstrates that grounding the textual prompt with the detected human/object classes provides useful context and regularization for the VLM.

We also experiment with a prompt containing just the human and object (“A person tennis racket”). This achieves a 0.8% lower mAP than the full prompt, showing that including the interaction verb is important for capturing the HOI semantics.

Overall, the full grounded sentence prompt leads to the best knowledge transfer from the VLM to our HOI detection model. The human, object, and verb provide complementary contextual cues that, when combined, enable optimized distillation of the VLM’s visual-linguistic knowledge.

A core component of our approach is the large pre-trained VLM which increases the computational requirements. As shown in Table 5, we use BLIP with 361M parameters due to hardware constraints, compared to 69M parameters for the baseline HOI network.

The additional VLM computations result in longer training times compared to the baseline - around 2.4$\times$ in our experiments. However, we found that our method requires fewer training epochs to converge, likely due to the beneficial regularization and optimized initialization from the VLM distillation.

At inference time, our model has the same efficiency as the baseline since the VLM is only used during training for knowledge transfer. We only need to perform a single forward pass through the learned HOI network.