Discovering Human-Object Interaction Concepts via Self-Compositional Learning

HOI understanding [23] is of great importance for visual relationship reasoning [61] and action understanding [5, 67]. Different approaches have been investigated for HOI understanding from various aspects, including HOI detection [7, 39, 40, 68, 34, 9, 71, 55, 65], HOI recognition [8, 33, 30], video HOI [13, 32], compositional action recognition [42], 3D scene reconstruction [66, 12], video generation [44], and object affordance reasoning [17, 28]. Recently, compositional approaches (e.g., VCL [27]) have been intensively proposed for HOI understanding using the structural characteristic [33, 27, 44, 38, 28]. Meanwhile, DETR-based methods (e.g., Qpic [55]) achieve superior performance on HOI detection. However, these approaches mainly consider the perception of known HOI concepts, and pay no attention to HOI concept discovery. To fulfill the gap between learning on known and unknown concepts, a novel task, i.e. , HOI concept discovery, is explored in this paper. Currently, zero-shot HOI detection also attracts massive interests from the community [53, 2, 48, 27, 29]. However, those approaches merely consider known concepts and are unable to discover HOI concepts. Some HOI approaches [48, 2, 59, 58] expand the known concepts via leveraging language priors. However, that is limited to existing knowledge and can not discover concepts that never appear in the language prior knowledge. HOI concept discovery is able to address the problem, and enable unknown HOI concept detection.

The notation of affordance is formally introduced in  [20], where object affordances are usually those action possibilities that are perceivable by an actor [45, 20, 21]. Noticeably, the action possibilities of an object also indicate the HOI concepts related to the object. Therefore, object affordance can also represent the existence of HOI concepts. Recent object affordance approaches mainly focus on the pixel-level affordance learning from human interaction demonstration [36, 18, 17, 25, 43, 15, 64]. Yao et al. [63] present a weakly supervised approach to discover object functionalities from HOI data in the musical instrument environment. Zhu et al. [70] introduce to reason affordances in knowledge-based representation. Recent approaches propose to generalize HOI detection to unseen HOIs via functionality generalization [2] or analogies [48]. However those approaches focus on HOI detection, ignoring object affordance recognition. Specifically, Hou et al. [28] introduce an affordance transfer learning (ATL) framework to enable HOI model to not only detect interactions but also recognize object affordances. Inspired by this, we further develop a self-compositional learning framework to facilitate the object affordance recognition with HOI model to discover novel HOI concepts for downstream HOI tasks.

Semi-supervised learning is a learning paradigm for constructing models that use both labeled and unlabeled data [62]. There are a wide variety of Deep Semi-Supervised Learning methods, such as Generative Networks [35, 54], Graph-Based methods [57, 22], Pseudo-Labeling methods [37, 60, 26]. HOI concept discovery shares a similar characteristic to semi-supervised learning approaches. HOI concept discovery has instances of labeled HOI concepts, but no instances of unknown concepts. We thus compose HOI representations for unknown concepts according to [50]. With composite HOIs, concept discovery and object affordance recognition can be treated as PU learning [14]. Moreover, HOI concept discovery requires to discriminate whether the combinations (possible HOI concepts) are reasonable and existing. Considering each value of the concept confidences also represents the possibility of the composite HOI, we construct pseudo labels [37, 50] for composite features from the concept confidence matrix, and optimize the composite HOIs in an end-to-end way.

HOI concept discovery aims to discover novel HOI concepts/categories using HOI instances from existing known HOI categories. Given a set of verb categories $\mathcal{V}$ and a set of object categories $\mathcal{O}$, let $\mathcal{S}=\mathcal{V}\times\mathcal{O}$ indicate the set of all possible verb-object combinations. Let $\mathcal{S}^{k}$, $\mathcal{S}^{u}$, and $\mathcal{S}^{o}$ denote three disjoint sets, known HOI concepts, unknown HOI concepts, and invalid concepts (or impossible verb-object combinations), respectively. That is, we have $\mathcal{S}^{k}\cap\mathcal{S}^{u}=\varnothing$ and $\mathcal{S}^{k}\cup\mathcal{S}^{u}=\mathcal{S}$ if $\mathcal{S}^{o}=\varnothing$. Let $\mathcal{T}=\{(h_{i},c_{i})\}_{i=1}^{L}$ indicate the training dataset, where $h_{i}$ is a HOI instance (i.e. , verb-object visual representation pair), $c_{i}\in\mathcal{S}^{k}$ indicates the label of the $i$-th HOI instance and $L$ is the total number of HOI instance.

We would also like to clarify the difference between the notations of “unknown HOI categories” and “unseen HOI categories” in current HOI approaches as follows. Let $\mathcal{S}^{z}$ indicate the set of “unseen HOI categories” and we then have $\mathcal{S}^{z}\subseteq\mathcal{S}^{k}$. Specifically, “unseen HOI category” indicates that the HOI concept is known but no corresponding HOI instances can be observed in the training data. Current HOI methods usually assume that unseen HOI categories $\mathcal{S}^{z}$ are known HOI categories via the prior knowledge [53, 33, 48, 2, 27]. Therefore, existing HOI methods can not directly detect/recognize HOIs with unknown HOI concepts. HOI concept discovery aims to find $\mathcal{S}^{u}$ from the existing HOI instances in $\mathcal{T}$ with only known HOI concepts in $\mathcal{S}^{k}$.

Inspired by the compositional nature of HOI, i.e. , each HOI consists of a verb and an object, visual compositional learning has been intensively explored for HOI detection by combining visual verb and object representations [33, 27, 29, 28]. Let $\mathbf{h}_{i}=\langle\mathbf{x}_{v_{i}},\mathbf{x}_{o_{i}}\rangle$ indicate a HOI instance, where $\mathbf{x}_{v_{i}}$ and $\mathbf{x}_{o_{i}}$ denote the verb and object representations, respectively. The HOI compositional learning then aims to achieve the following objective,

where $g_{h}$ indicates the HOI classifier, $\mathbf{x}_{v_{i}}$ and $\mathbf{x}_{o_{i}}$ indicate the real verb-object representation pair (i.e. , annotated HOI pair in dataset), $\langle\widetilde{\mathbf{x}}_{v_{i}},\widetilde{\mathbf{x}}_{o_{i}}\rangle$ indicates the composite verb-object pair. Specifically, $\widetilde{\mathbf{x}}_{o_{i}}$ can be obtained from either real HOIs [27], fabricated objects or language embedding [29, 2, 48], or external object datasets [28], while $\widetilde{\mathbf{x}}_{v_{i}}$ can be from real HOIs (annotated verb-object pair) and language embeddings [33, 48]. As a result, when composite HOIs are similar to real HOIs, we are then able to augment HOI training samples in a compositional manner. However, current compositional approaches for HOI detection [27, 28] simply remove the composite HOI instances out of the label space, which may also remove a large number of feasible HOIs (e.g., “ride zebra” as shown Figure 2). Furthermore, the compositional approach can not only augment the training data for HOI recognition, but also provide a method to determinate whether $\widetilde{\mathbf{x}}_{v_{i}}$ and $\widetilde{\mathbf{x}}_{o_{i}}$ are combinable to form a new HOI or not [28], i.e. , discovering the HOI concepts.

In this subsection, we introduce the proposed self-compositional learning framework for HOI concept discovery as follows. As shown in Figure 2, the main HOI concept discovery framework falls into the popular two-stage HOI detection framework [27]. Specifically, we compose novel HOI samples from pair-wise images to optimize the typical HOI branch (annotated HOIs), compositional branch (the composite HOIs out of the label space are removed [27, 28]) and the new concept discovery branch (all composite HOIs are used). The main challenge of HOI concept discovery is the lack of instances for unknown HOI concepts, but we can infer to discover new concepts according to the shared verbs and objects. Specifically, we find that the affordance transfer learning [28] can be used for not only the object affordance recognition but also the HOI concept discovery, and we thus first introduce the affordance-based method as a baseline as follows.

Datasets. We extend two popular HOI detection datasets, HICO-DET [7] and V-COCO [24], to evaluate the performance of different methods for HOI concept discovery. Specifically, we first manually annotate all the possible verb-object combinations on HICO-DET (117 verbs and 80 objects) and V-COCO (24 verbs and 80 objects). As a result, we obtain 1,681 concepts on HICO-DET and 401 concepts on V-COCO, i.e. , 1,681 of 9,360 verb-object combinations on HICO-DET and 401 of 1,920 verb-object combinations on V-COCO are reasonable. Besides, 600 of 1,681 HOI concepts on HICO-DET and 222 of 401 HOI concepts on V-COCO are known according to existing annotations. Thus, the HOI concept discovery task requires to discover the other 1,081 concepts on HICO-DET and 179 concepts on V-COCO. See more details about the annotation process, the statistics of annotations, and the novel HOI concepts in Appendix B.

Evaluation Metrics. HOI concept discovery aims to discover all reasonable combinations between verbs and objects according to existing HOI training samples. We report the performance by using the average precision (AP) for concept discovery and mean AP (or mAP) for object affordance recognition. For HOI detection, we also report the performance using mAP. We follow [28] to evaluate object affordance recognition with HOI model on COCO validation 2017 [41], Object 365 validation [52], HICO-DET test set [7] and Novel Objects from Object 365 [52].

We implement the proposed method with TensorFlow [1]. During training, we have two HOI images (randomly selected) in each mini-batch and we follow  [19] to augment ground truth boxes via random crop and random shift. We use a modified HOI compositional learning framework, i.e. , we directly predict the verb classes and optimize the composite HOIs using SCL. Following  [27, 29], the overall loss function is defined as $\mathcal{L}=\lambda_{1}\mathcal{L}_{h}+\lambda_{2}\mathcal{L}_{c}+\lambda_{3}\mathcal{L}_{d}$, where $\lambda_{1}=2$, $\lambda_{2}=0.5$, $\lambda_{3}=0.5$ on HICO-DET, and $\lambda_{1}=0.5$, $\lambda_{2}=0.5$, $\lambda_{3}=0.5$ on V-COCO, respectively. Following [29], we also include a sigmoid loss for verb representation and the loss weight is $0.3$ on HICO-DET. For self-training, we remove the composite HOIs when its corresponding concept confidence is 0, i.e. , the concept confidence has not been updated. If not stated, the backbone is ResNet-101. The Classifier is a two-layer MLP. We train the model for 3.0M iterations on HICO-DET and 300K iterations on HOI-COCO with an initial learning rate of 0.01. For zero-shot HOI detection, we keep human and objects with the score larger than 0.3 and 0.1 on HICO-DET, respectively. See more ablation studies (e.g., hyper-parameters, modules) in Appendix D. Experiments are conducted using a single Tesla V100 GPU (16GB), except for experiments on Qpic [55], which uses four V100 GPUs with PyTorch [46].

Baseline and Methods. We perform experiments to evaluate the effectiveness of our proposed method for HOI concept discovery. For a fair comparison, we build several baselines and methods as follows,

• •

  Random: we randomly generate the concept confidence to evaluate the performance.

• •

  Affordance: discover concepts via affordance prediction [28] as described in Sec 3.3.1.

• •

  GAT [56]: build a graph attention network to mine the relationship among verbs during HOI detection, and discover concepts via affordance prediction.

• •

  Qpic* [55]: convert verb and object predictions of  [55] to concept confidence similar as online discovery.

• •

  Qpic* [55] +SCL: utilize concept confidence to update verb labels, and optimize the network (Self-Training). Here, we have no composite HOIs.

Please refer to the Appendix for more details, comparisons (e.g., re-training, language embedding), and qualitative discovered concepts with analysis.

Results Comparison. Table 1 shows affordance prediction is capable of HOI concept discovery since affordance transfer learning [28] also transfers affordances to novel objects. Affordance prediction achieves 24.38% mAP on HICO-DET and 21.36% mAP on V-COCO, respectively, significantly better than the random baseline. With graph attention network, the performance is further improved a bit. Noticeably, [28] completely ignores the possibility of HOI concept discovery via affordance prediction. Due to the strong ability of verb and object prediction, Qpic achieves 27.42% on HICO-DET, better than affordance prediction. However, Qpic has poor performance on V-COCO. The inference process of affordance prediction for concept discovery is time-consuming (over 8 hours with one GPU). Thus we devise an efficient online concept discovery method which directly predicts all concept confidences. Specifically, the online concept discovery method (SCL$-$) achieves 22.25% mAP on HICO-DET, which is slightly worse than the result of affordance prediction. On V-COCO, the online concept discovery method improves the performance of concept discovery by 3.98% compared to the affordance prediction. The main reason for the above observation might be due to that V-COCO is a small dataset and the HOI model can easily overfit known concepts on V-COCO. Particularly, SCL significantly improves the performance of HOI concept discovery from 22.36% to 33.58% on HICO-DET and from 24.89% to 28.77% on V-COCO, respectively. We find we can also utilize self-training to improve concept discovery on Qpic [55] (ResNet-50) though the improvement is limited, which might be because verbs and objects are entangled with Qpic. Lastly, we meanwhile find SCL largely improves concept discovery of known concepts on both HICO-DET and V-COCO.

Following [28] that has discussed average precision (AP) is more robust for evaluating object affordance, we evaluate object affordance recognition with AP on HICO-DET. Table 2 illustrates SCL largely improves SCL$-$ (without self-training) by over 9% on Val2017, Object365, HICO-DET under the same training iterations. SCL requires more iterations to converge, and SCL greatly improves previous methods on all datasets with 3M iterations (Please refer to Appendix D.2 for convergence analysis). Noticeably, SCL directly predicts verb rather than HOI categories, and removes the spatial branch. Thus, SCL without self-training (SCL$-$) is a bit worse than ATL. Previous approaches ignore the unknown affordance recognition. We use the released models of  [28] to evaluate the results on novel affordance recognition. Here, affordances of novel classes (annotated by hand [28]) are the same in the two settings. We find SCL improves the performance considerably by over 10% on Val2017 and HICO-DET.

HOI concept discovery enables zero-shot HOI detection with unknown concepts by first discovering unknown concepts and then performing HOI detection. The experimental results of HOI detection with unknown concepts are shown in Table 3. We follow [27] to evaluate HOI detection with 120 unknown concepts in two settings: rare first selection and non-rare first selection, i.e. , we select 120 unknown concepts from head and tail classes respectively. Different from [27, 29] where the existence of unseen categories is known and the HOI samples for unseen categories are composed during optimization, HOI detection with unknown concepts does not know the existence of unseen categories. Therefore, we select top-$K$ concepts according to the confidence score during inference to evaluate the performance of HOI detection with unknown concepts (that is also zero-shot) in the default mode [7].

As shown in Table 3, with more selected unknown concepts according to concept confidence, the proposed approach further improves the performance on unseen categories on both rare first and non-rare first settings. Specifically, it demonstrates a large difference between rare first unknown concepts HOI detection and non-rare first unknown concepts HOI detection in Table 3. Considering that the factors (verbs and objects) of rare-first unknown concepts are rare in the training set [29], the recall is very low and thus degrades the performance on unknown categories. However, with concept discovery, the results with top 120 concepts on unknown categories are improved by relatively 34.52% (absolutely 0.58%) on rare first unknown concepts setting and by relatively 20.31% (absolutely 1.19%) on non-rare first setting, respectively. with more concepts, the performance on unknown categories is also increasingly improved.

We also utilize the discovered concept confidences with SCL to evaluate HOI detection with unknown concepts on Qpic [55]. For a fair comparison, we use the same concept confidences to SCL. Without concept discovery, the performance of Qpic [55] degrades to 0 on Unseen categories though Qpic significantly improves zero-shot HOI detection. Lastly, we show zero-shot HOI detection (the unseen categories are known) in Table 3 (Those rows where $K$ is all). We find that SCL significantly improves Qpic, and forms a new state-of-the-art on zero-shot setting though we merely use ResNet-50 as backbone in Qpic. We consider SCL improves the detection of rare classes (include unseen categories in rare first and seen categories in non-rare first) via stating the distribution of verb and object. See Appendix F for more analysis, e.g., SCL improves Qpic particularly for rare categories on Full HICO-DET.

Figure 3 illustrates the Grad-CAM under different methods. We find the proposed SCL focus on the details of objects and small objects, while the baseline and VCL mainly highlight the region of human and the interaction region, e.g., SCL highlights the details of the motorbike, particularly the front-wheel (last row). Besides, SCL also helps the model via emphasizing the learning of small objects (e.g., frisbee and bottle in the last two columns), while previous works ignore the small objects. This demonstrates SCL facilitates affordance recognition and HOI concept discovery via exploring more details of objects. A similar trend can be observed in Appedix G (Qpic+SCL).