Compositional Learning in Transformer-Based Human-Object Interaction Detection

Previous few-shot and zero-shot HOI detection methods can be divided into two groups. One group of methods [3, 15, 10] adopt vision-language joint learning to augment visual features with linguistic knowledge. Peyre et al. [3] learn vision-language embeddings of visual phrases and generate embeddings for unseen triplets via analogies between similar relations. Liu et al. [15] construct an knowledge graph with visual-semantic embeddings to encode multi-level relations among objects, actions and HOIs. Liao et al. [10] transfers visual-linguistic knowledge from CLIP [16] to enhance HOI understanding. The other group of methods re-compose samples to generalize knowledge to rare and unseen HOI classes. Bansal et al. [11] propose a functional generalization module where object word embeddings are replaced with those of functionally similar objects during training. Hou et al. [12, 13, 14] propose several methods to generate new interaction samples. In VCL [12], they concatenate object and verb features across HOI instances. In FCL [13], they propose to generate fabricated object features containing noise and verb features. In ATL [14], verb features re-defined as affordance features are concatenated with object features from HOI datasets and object detection datasets.

Transformer [9] has been successfully applied in a diversity of computer vision tasks, e.g. DETR [17] in object detection, which inspires early transformer-based HOI detection methods [6, 7]. In QPIC [6], a transformer encoder aggregates image-wide context, and a transformer decoder transforms a set of queries into embeddings, each directly capturing one human-object pair. In HOTR [7], two decoders respectively generate instance and interaction representations, and each interaction representation associates itself with corresponding humans and objects with HO Pointers. Some works [18] exploit deformable attention [19] to process multi-scale feature maps for fine-grained HOI detection. Recently, some works [8, 20] seek to combine advantages of two-stage and one-stage frameworks. Zhang et al. [8] propose two cascade disentangling decoders, each focusing on respective subtasks, i.e. object detection and interaction classification. Zhang et al. [20] propose a two-stage framework composed of a DETR as object detector and a transformer-based interaction head to pair humans and objects and predict action classes.

Fig. 2 illustrates the overall pipeline of our method which is mainly based on CDN [8] and VCL [12]. Given a pair of input images, the backbone and a shared encoder first extract their global features. Then the human-object pair decoder and the interaction decoder transform a set of learnable queries into corresponding representations. A set of feed-forward networks (FFNs) process human-object pair representations to predict human and object bounding boxes and object categories, and human-object pair representations are concatenated with interaction representations to predict action classes. For compositional learning, we select representations that generate the best predictions matched with ground truths, and concatenate them across different HOI instances to produce new training samples. Labels of re-composed samples are also constructed using labels of original samples. Verb-object compositions that doesn’t belong to HOI categories defined by datasets are considered infeasible and removed from re-composed labels.

We choose Cascade Disentangling Network (CDN) [8] as the baseline model, based on which we implement our compositional learning method because of its effectiveness and code availability. We briefly review its architecture and introduce our modification to it for sample re-composition.

Two decoders take $X_{s}$, $E_{pos}$ and a set of queries $Q\in\mathbb{R}^{N_{q}\times C_{q}}$ as input, apply self-attention on $Q$, conduct multi-head co-attention between $Q$ and $X_{s}$, and output updated queries denoted as representations $R\in\mathbb{R}^{N_{q}\times C_{q}}$. $N_{q}$ stands for the number of queries. HO-PD $D_{p}$ decodes features of human-object pairs, while the interaction decoder $D_{i}$ captures interaction context. $D_{p}$ and $D_{i}$ are cascaded by initializing the interaction queries $Q_{i}$ with HO pair representations $R_{p}$, so that $Q_{p}$ are guided by prior knowledge in $R_{p}$ to learn the corresponding action classes for each HO pair query. The generating process of representations can be described as:

A group of FFNs $F$ process the representations to predict HOI triplets. $R_{p}$ produce human-object pairs predictions denoted as $Y_{p}=\{(b^{h}_{n},b^{o}_{n},o_{n}),n\in\{1,2,...,N_{q}\}\}$. $b^{h}_{n}\in\mathbb{R}^{4}$, $b^{o}_{n}\in\mathbb{R}^{4}$ and $o_{n}\in\mathbb{R}^{N_{o}}$ stand for the human bounding box, the object bounding box and the object class probability distribution, respectively. $N_{o}$ is the number of object classes. $R_{i}$ predict action classes of human-object pairs, denoted as $Y_{i}=\{a_{n},n\in\{1,2,...,N_{q}\}\}$. $a_{n}\in\mathbb{R}^{N_{a}}$ is the action class probability distribution, and $N_{a}$ is the number of action classes. Composed of HO pair predictions and interaction predictions, HOI predictions are denoted as $Y=\{(b^{h}_{n},b^{o}_{n},o_{n},a_{n}),n\in\{1,2,...,N_{q}\}\}$. The prediction process is formulated as:

Inspired by Visual Compositional Learning (VCL) [12], we propose our compositional learning method consisting of re-composition of features and re-composition of labels. In our work, sample re-composition is conducted within one pair of randomly selected input images due to hardware limitation, but it’s also easier for us to explain our method clearly. If one image contains multiple HOI instances, we also re-compose samples across different instances within the image following the same process.

HO pair representations $R^{\ast}_{p}$ and interaction representations $R^{\ast}_{i}$ from different images are concatenated as new feature samples, and fed into the interaction classifier to predict action classes, which can be described as:

Taking $R^{\ast}_{p1}$ from $I_{1}$ and $R^{\ast}_{i2}$ from $I_{2}$ as example, every representation in $R^{\ast}_{p1}$ is concatenated with every representation in $R^{\ast}_{i2}$. So the shape of re-composed representations $R^{\ast}_{p1}\otimes R^{\ast}_{i2}$ is $(N_{gt1}\times N_{gt2},2\times C_{q})$, and $Y^{\ast}_{i12}=\{a^{\ast}_{n},n\in\{1,2,...,N_{gt1}\times N_{gt2}\}\}$ contains $N_{gt1}\times N_{gt2}$ verb predictions. Combining human-object predictions of original samples, the HOI predictions of re-composed samples are denoted as $Y^{\ast}_{12}=\{(b^{h1}_{n},b^{o1}_{n},o^{1}_{n},a^{\ast}_{n}),n\in\{1,2,...,N_{gt1}\times N_{gt2}\}\}$.

In VCL [12], object features from object bounding box regions are re-concatenated with verb features from union regions of human-object pairs. We think this way of re-composition fails to involve enough human features and global context, which are included in HO pair representations. So we believe our feature re-composition method can enhance the generalization of higher-dimensional semantic knowledge.

Taking HO pair labels from $I_{1}$ and action labels from $I_{2}$ as example, we pair each HO pair label in $\hat{Y}_{p1}$ with every action label in $\hat{Y}_{i2}$, and check the feasibility of compositions. An object-verb composition that doesn’t belong to HOI categories defined by the dataset is considered infeasible, even if it may be rational (e.g. “couch” and “wear”, “suitcase” and “sit on” in Fig. 2). By setting the value of infeasible actions in the action label vector to zero, infeasible compositions are removed. If all compositions between the object label and all action labels are infeasible, an all-zero action label vector is kept for the object label instead of removing the whole HOI label, because the HO pair label is still useful for instance detection. The re-composed HOI labels are denoted as $\hat{Y}^{\ast}_{12}=\{(\hat{b}^{h1}_{n},\hat{b}^{o1}_{n},\hat{o}^{1}_{n},\hat{a}^{2}_{n}),n\in\{1,2,...,N_{cp}\}\}$, where $N_{cp}$ stands for the number of kept re-composed labels.

CDN [8] provides PyTorch implementations of three variant architectures: CDN-S (small), CDN-B (base) and CDN-L (large). For quick validation, we implement our method on CDN-S, which consists of a ResNet-50 as backbone, a 6-layer encoder and two 3-layer decoders. Box regression FFNs have 3 linear layers with ReLU, while object and action classfiers are single-layer FFNs. The number of queries $N_{q}$ is set to 64 for HICO-Det and 100 for V-COCO. The number of channels of each query $C_{q}$ is set to 256. Following  [6], weight coefficients $\lambda_{b}$, $\lambda_{u}$, $\lambda_{o}$, $\lambda_{a}$ are set to 2.5, 1, 1 and 1, respectively. Same as [8], we initialize the network with pre-trained parameters of DETR [17], and use AdamW with a weight decay of $10^{-4}$ for optimization. We first train the whole model for 90 epochs with a learning rate of $10^{-4}$, decreasing by 10 times at 60th epoch. Then we fine-tune the cascade decoders together with FFNs for 10 epochs with a learning rate of $10^{-5}$. We also use the same dynamic re-weighting strategy and post-processing of pairwise non-maximal suppression as [8] proposes. All experiments are conducted on 2 RTX 2080 Ti GPUs with the batch size of 2.

We first validate the effectiveness of our method compared with baseline models. Experimental results are demonstrated in Table I and Table II, where Baseline, Baseline^∗ and Compo denote the original CDN-S, the modified baseline mentioned in Section III-B and our sample re-composition method, respectively.

As Table I shows, Baseline^∗ which concatenates representations of HO pairs and interactions to predict action classes outperforms the original baseline on HICO-Det, especially improving the mAP on rare classes by a large margin. Applying sample re-composition on the basis of Baseline^∗, our method achieves the best performance with $\rho=0.75$ on three category sets of HICO-Det. Note that our best model significantly increases the mAP on rare classes from 23.35 to 24.62 compared with Baseline. This proves the effectiveness of our method in eliminating the long-tailed distribution problem, which is also shown by qualitative results in Fig. 3.

According to Table II, our method also performs better compared with baselines on V-COCO, achieving peak performance with $\rho=0.9$. We assume that the difference of best $\rho$ values on two datasets may result from the number of feasible labels. On HICO-Det with a larger range of object and action categories, the re-composition goes beyond defined HOI classes far more easily than on V-COCO, leading to less feasible HOI labels in re-composed samples. Therefore, re-composed samples of HICO-Det contains less knowledge than those of V-COCO, and a slightly larger weight for re-composed samples becomes necessary.

We compare our method with existing state-of-the-art few-shot learning methods and transformer-based methods of HOI detection. For fair comparison, we give priority to implementations that adopt Resnet-50 as the backbone. From Table III we can see our best model outperforms all listed previous methods on HICO-Det, especially exceeding the state-of-the-art mAP on rare classes by a large margin. As Table IV shows, our best model also achieves competitive performance on V-COCO. We believe our method can outperform state-of-the-art methods if implemented on a larger network architecture, i.e. CDN-B or CDN-L.