Affordance Transfer Learning for Human-Object Interaction Detection

Human-object interaction (HOI) detection are receiving increasing attention from the community [3]. HOI detection aims to not only detect object and human in an image, but also reason the relationships between human and objects. Since Gputa et al.[12] presented a Human-Object Interaction approach, massive traditional methods [40, 41] were introduced for HOI recognition using spatial relation [12], pose [40], human part [41] in the early. Recently, Chao et al.[4] introduced a large HOI recognition data HICO [4] and a challenging HOI detection dataset HICO-DET [3] for HOI understanding. Meanwhile, Gupta et al.[13] introduced the V-COCO dataset, which mainly focuses on the grounding of verbs and their semantic roles.

Currently, there are a large number of HOI detection approaches [8, 22, 39, 36, 32, 2, 28, 35, 33, 43, 19] improving the HOI benchmark [3, 13, 17]. According to the target, current methods can be categoried into two-stage HOI detection, which contains common HOI detection [8, 22, 34, 36, 25] and few-and zero-shot HOI detection [32, 2, 38, 17, 28, 19, 35, 26], and one-stage HOI detection [23, 37, 18]. Recently, Hou et al.[17] propose a visual compositional learning framework to compose novel HOI samples between pair-wise HOI images for low-and zero-shot HOI detection. However, VCL [17] can not compose human-novel-object interactions and ignores the possibility of affordance recognition with HOI model. We introduce a novel framework, Affordance Transfer Learning, to transfer the affordance representation to novel objects via composing affordance and novel object representations, and thus enable the detection of HOIs with novel objects.

James J. Gibson defined the word affordance in [10]. Object affordances are those action possibilities that are perceiveable by an actor [27, 10, 15], that is also the possibilities of Human-Object Interactions. In the early, Kjellstr et al.[20] investigated learning the affordances of objects from human demonstration. Yao & Li et al.[42] presented a weak supervised approach to discover object funtionalities from HOI data in the environment where the person is interacting with musical instruments. Fouhey et al.[6] introduced an approach to estimate functional surfaces by observing human actions. Recently, Fang et al.[5] introduces Demo2Vec to learn interaction region and action label from online video. Differently, we demonstrate to learn a shared affordance representation with HOI model, and recognize object affordance via classifying the composite HOIs of shared affordance representations from existing HOIs and the target object representation.

The motivation of affordance transfer learning is to transfer the affordance to novel objects for exploring unseen HOIs, \ie, combining the affordance representations and novel object representations. The affordance transfer learning meanwhile enables HOI network to recognize object affordance. Similar to [8, 39, 14, 33], we utilize the popular two-stage HOI detection framework: 1) we first detect the objects in a given image using a common object detector (\eg, Faster-R-CNN [29]); and 2) we then construct HOIs from object and affordance representations to perform HOI classification. The main framework of our affordance transfer learning is illustrated in Figure 2, which consists of three branches, spatial HOI, real HOI, and composite HOI. Inspired by [8, 22, 17], we utilize the spatial HOI branch to further improve the HOI detection performance. Specifically, the spatial pattern representation consists of two $64\times 64$ binary feature maps to indicate human and object relative positions, \ie, the pixels within the human (or object) bounding box are assigned the value 1. Both real HOIs and composite HOIs are constructed from object/affordance features and share the same HOI classifier [30], while the difference between them is that the objects in the composite HOIs are extracted from additional object image datasets. Furthermore, by transferring the shared affordance representations extracted from HOI samples to novel objects, we are also able to use the HOI detection model to recognize the affordance of novel objects.

The proposed affordance transfer learning first composes novel HOI samples between object representations from additional object images (\eg, images from COCO dataset [24]) and decoupled affordance representation as illustrated in Figure 2, and then generalize the affordance representation to novel objects via jointly optimizing the network with the composite HOIs. With the additional objects, the affordance transfer learning effectively decouples the affordance representation from the scenes, and then enables the composition of affordance and novel objects to recognize the affordance of novel objects. In this subsection, we introduce how to efficiently compose new HOIs and remove invalid HOIs for affordance transfer learning.

Efficient HOI Composition. The label assignment for the composite HOIs can be integrated from the verb label (or the affordance label) and the object label. The object label $\widetilde{l}_{o}$ is provided by the object datasets, and we obtain the verb label $l_{v}$ by decoupling the HOI label. Similar to [17], we decouple the HOI label space into a verb-HOI co-occurrence matrix $\mathbf{A}_{v}\in R^{N_{v}\times C}$ a the object-HOI co-occurrence matrix $\mathbf{A}_{o}\in R^{N_{o}\times C}$, where $N_{v}$, $N_{o}$, and $C$ indicate the numbers of verbs, objects and HOI categories, respectively. Here, both $\mathbf{A}_{v}$ and $\mathbf{A}_{o}$ are binary matrix. Given the one-hot HOI label $y\in R^{C}$, we then obtain the verb label $l_{v}$ from $y\mathbf{A}_{v}^{T}$. To compose a new HOI by the object $\widetilde{l}_{o}$ and verb $l_{v}$, we assign the label to the composite HOI as follows,

where $\&$ indicates the element-wise “and” logical operation. Both $\widetilde{l}_{o}\mathbf{A}_{o}$ and $l_{v}\mathbf{A}_{v}$ are binary vectors (\ieHOI labels), which represent all possible HOIs corresponding to $\widetilde{l}_{o}$ and $l_{v}$, respectively. Therefore, the intersection between $\widetilde{l}_{o}\mathbf{A}_{o}$ and $l_{v}\mathbf{A}_{v}$ indicates the label of the verb-object pair $\left\langle l_{v},\widetilde{l}_{o}\right\rangle$, \ie, the label of the composite HOI.

Invalid HOI Elimination. With additional object datasets, we can compose a large number of new types of HOI samples by combining object and affordance features. Considering a variety of object categories, there is nevertheless some invalid HOIs (\eg, “ride orange”), \ie, the invalid HOIs are out the space of ground truth HOI labels. Furthermore, the same verb might have different meanings in different scenes [9, 43], while the verbs in current HOI dataset (\eg, HICO-DET) mainly represents action (affordance) and are usually not ambiguous [9]. Meanwhile, a variety of recent HOI detection methods do not distinguish the same affordance among different HOIs [32, 2, 14, 19, 38, 28]. To this end, we also equally treat the same affordance from different HOIs for the evaluation of the transfer affordance learning. Following [17], we simply remove those HOIs, which is out of the HOI label space (\eg, “ride dog” in HICO-DET) as illustrated in the right part of Figure 2. In addition, the one-hot labels of those invalid HOIs are all zeros according to Equation 1. That is, we can easily remove those composite HOIs according to the one-hot labels.

In this subsection, we introduce how to infer the object affordance during the testing phase. Considering that we jointly optimize the decoupled components (\ie, object features and affordance features from object and HOI images) in HOI samples and novel object samples with affordance transfer learning, the proposed method thus is able to distinguish whether a novel object is combinable or not with a specific affordance (\ie, valid HOIs). Therefore, we design a simple yet effective object affordance recognition method using the HOI detection model. Specifically, we first build an affordance feature bank as follows.

Affordance Feature Bank. We construct the affordance feature bank from HOI datasets (\egHICO-DET and HOI-COCO). In order to reduce storage space and computation, we randomly choose a maximun of $M$ instances for each affordance in HICO-DET. In our experiment, $M$ is 100. Then, we extract the features of those affordances to construct an off-the-shelf affordance feature bank.

Given an object feature extracted from the object image, we combine it with all affordances in the feature bank to obtain a set of HOIs. As illustrated in Figure 3, we obtain all HOI predictions from the HOI classifier. After that, we are able to convert all HOI predictions to affordance predictions according to the HOI-verb co-occurrence matrix $\mathbf{A}_{v}$. Specifically, we remove the predicted affordances whose label is not the same as the corresponding affordance labels in the feature bank. As a result, we obtain a list of affordances with many repeated elements. Let $F_{i}$ denotes the frequency (count) of the affordance $i$ and $S_{i}$ indicates the number of affordance (or verb) $i$ in the feature bank, we evaluate the probability of the affordance $i$ as $\frac{F_{i}}{S_{i}}$.

During the training stage, we train the proposed method with an unique loss $L_{ATL}$ for transferring the affordance to novel objects. Meanwhile, similar to [8], we also incorporate spatial pattern loss $L_{sp}$ to optimize the learning of the HOI spatial pattern representation. In addition, an HOI loss $L_{hoi}$ is used for HOI classification. Lastly, the overall training loss function is defined as follows,

where $\lambda_{1}$ and $\lambda_{2}$ are two hyper-parameters to balance different losses. Both the feature extractors and the HOI detection modules are jointly trained in an end-to-end manner. $\mathcal{L}_{hoi\_sp}$, $\mathcal{L}_{hoi}$, $\mathcal{L}_{ATL}$ are binary cross entropy losses.

During the testing stage, affordance transfer learning module is not necessary. Similar to [8, 22, 17], we predict the final HOIs with spatial HOI predictions and verb-object HOI predictions. Formally, given a human-object bounding box pair ($b_{h}$, $b_{o}$), we predict the score $S^{c}_{h,o}$ as $s_{h}\cdot s_{o}\cdot s^{c}_{hoi}\cdot s^{c}_{sp}$ for each HOI category $c\in 1,...,C$, where $C$ denotes the total number of possible HOI types, $s_{h}$ and $s_{o}$ are the human and object detection scores respectively, $s^{c}_{sp}$ represents the spatial HOI prediction score and $s^{c}_{hoi}$ is the verb-object HOI prediction score.

HICO-DET [3] dataset consists of 38,118 images in the training set and 9,658 test images over 600 types of interactions (80 object categories in COCO dataset and 117 unique verbs) with over 90,000 HOI instances.

HOI-COCO is built from the V-COCO dataset [13], which contains 10,346 images with 16,199 person instances. Each annotated person in V-COCO has binary labels for 26 different actions. V-COCO mainly focuses on verb recognition, and has limited object categories (only two). Thus we construct a new benchmark HOI-COCO for the evaluation of verb-object pairs as follows. We use 21 actions from all 26 actions in V-COCO (\ie, five non-interaction actions, “walk”, “run, “smile”, “stand” and “point ”, are removed). As a result, we build HOI-COCO benchmark with 222 HOI categories over 21 verbs and 80 objects. Meanwhile, we use the same train/val split in V-COCO for HOI-COCO. Similar to HICO-DET [3], we evaluate the performance on HOI-COCO under three different settings: Full (222 types), Rare (97 types), and NonRare (115 types). The HOI type in Rare category contains less than 10 training instances, and the distribution of HOI categories is long-tailed.

COCO [24] dataset is a widely-used benchmark for common object detection with 80 different object classes. Considering that both HICO-DET [3] and HOI-COCO consist of the same object label sets to COCO, we thus directly incorporate the COCO dataset as the additional object dataset in our experiments.

Object365 [31] is a recently proposed large-scale common object detection dataset with 365 object categories. The domain of Object365 is different from COCO [24]. In detail, we select objects that are labeled as COCO classes from Object365 validation dataset to evaluate the affordance recognition of objects on new domain. Meanwhile, we choose 12 new types of objects and label manually the affordance of those objects according to the HICO-DET and HOI-COCO, respectively. Those objects are used to evaluate affordance recognition on new types of objects. See more details in supplementary materials.

Evaluation Metrics. We follow the standard evaluation metric [8, 39] and report mean average precision for HICO-DET dataset [3] and HOI-COCO. A prediction is a true positive only when the detected human and object bounding boxes have IoUs larger than 0.5 with reference to ground truth, and the HOI category is accurately predicted. Object affordance recognition is a multiple label classification problem (\iean object usually has multiple affordances). Thus, we compare Precision, Recall and F1-Score for evaluating object affordance recognition.

For HICO-DET, similar to recent methods [2, 23, 37, 17], we use the object detector fine-tuned on HICO-DET, \iethe detector provided in [17]. For HOI-COCO, we directly use the object detector pre-trained on COCO. Besides, all HOI classifiers consist of two fully-connected layers with 1024 hidden units. To compare with recent methods on HICO-DET, we use two object images in each mini-batch. On HOI-COCO, we only use one object image for evaluation. Besides, we also use an auxiliary verb loss [16] to improve our baseline. During training, following [8, 22, 17], we augment the ground truth boxes via random crop and random shift. During inference, we keep human and objects with the score larger than 0.3 and 0.1 on HICO-DET respectively. Following [17], we set $\lambda_{1}=2$, $\lambda_{2}=0.5$ on HICO-DET, and $\lambda_{1}=0.5$, $\lambda_{2}=0.5$ on HOI-COCO, respectively. To prevent composite interactions from dominating the training of the model, we keep the number of composite interactions not more than the number of objects in each mini-batch by randomly sampling composite HOIs. We train the model for 1.2M iterations on HICO-DET dataset and 300K iterations on HOI-COCO with an initial learning rare of 0.01. For object affordance recognition, we use the actions of each HOI dataset as affordances and remove the “no interaction” categories on HICO-DET dataset. We keep the object affordance predictions if the affordance score is large than 0.5. All experiments are conducted on a single Tesla V100 GPU with TensorFlow [1].

HICO-DET. We report the performance on three different settings: Full (600 categories), Rare (138 categories) and NonRare (462 categories) in “Default” and “Known” modes on HICO-DET. As shown in Table 1, the proposed method outperforms recent state-of-the-art methods among all categories. Furthermore, with better object detection results provided in [7], the performance of ATL dramatically increases to 28.53%. Meanwhile, we find ATL is more effective on Rare category. Specifically, when using the objects from the training set of HICO-DET, the proposed method is similar to VCL [17] as shown in Table 1. ATL also improves the baseline effectively based on One-Stage method. Here, the baseline is the model without compositional learning. Details of One-Stage method is provided in supplementary materials.

HOI-COCO. We find the proposed method has similar performance to VCL when using HOI-COCO as the source of object images in Table 2. Here, we evaluate the performance of VCL on HOI-COCO dataset using the official code from [17]. When using the COCO object dataset, the proposed method significantly improves the performance, especially on Rare categories, \eg, over 1.5% than VCL and 2.9% than the baseline, respectively. Meanwhile, the proposed method also gives a larger improvement than baseline in NonRare category comparing with VCL, suggesting that ATL also increases the diversity of HOIs via composing new samples. Furthermore, when using both HICO-DET and COCO to provide object images, we further improve the performance to 25.29%.

The proposed affordance transfer learning enables the detection of HOIs with novel objects due to the mechanism of composing HOI samples of unseen classes. Therefore, we evaluate the proposed method for zero-shot HOI detection on HICO-DET [3]. We report the performance on two settings: 1) Unseen Composition and 2) Novel Object. Specifically, Unseen Composition means there are unseen HOIs in the test but the verbs and objects of the unseen HOIs exist in training data, while the objects of unseen HOIs in novel object HOI detection do not exist in training data. For compositional zero-shot learning, we follow [17] to evaluate on rare-first unseen HOIs (firstly select tail HOIs in HICO-DET as unseen data) and non-rare first unseen HOIs (firstly select head HOIs in HICO-DET as unseen data). We evaluate zero-shot HOI detection on three categories: Unseen (120 categories), Seen (480 categories) and Full (600 categories). For novel object HOI detection, similar to [2], we choose 100 unseen categories (includes 12 unseen objects) and 500 seen categories. We choose the object detector provided in [17] to compare fairly with [17].

Compositional Zero-Shot HOI Detection. In Table 3, we find our approach effectively improves the non-rare first zero-shot HOI detection. Meanwhile, our approach achieves better result on seen category in rare first zero-shot HOI detection. Particularly, the affordances in tail part of HOIs are usually rare, the composite samples of tail HOIs with additional objects are much less than that of head HOIs. Therefore, our approach achieves even worse result on unseen category.

Novel Object HOI Detection. Table 3 demonstrates that transferring affordance representation to novel objects effectively facilitates the detection of unseen HOIs with novel objects. Here we use the network without affordance transfer learning as our baseline. We find using HICO-DET (remove HOIs with unseen objects) as object images even degrades the performance on unseen categories compared to the baseline because we compose massive seen HOI samples but not unseen HOI samples with HICO-DET. Besides, similar to [2], we use an generic object detector to enable HOI detection with novel object, which provides a strong baseline. While we only use the boxes of the detector (not use the object label predicted by detector), the performances of baseline and ATL (HICO-DET) on unseen category decrease to 0. However, ATL (COCO) still achieves 5.05% on unseen category.

Table 4 shows ATL significantly improves the baseline by over 40 % in F1 score among all datasets with COCO categories, and by over 30% on novel object classes on HOI-COCO. On HICO-DET, ATL improves the baseline by nearly 10% among all categories in datasets with COCO categories, and by around 5% on novel object classes. Those experiments indicate that the affordance transfer learning via composing novel HOIs effectively disentangles the affordance and object representations from the scenes and endows the HOI network with the ability of affordance recognition. Noticeably, ATL^ZS (COCO), that composing interactions from affordances and novel objects, largely improves the baseline model ATL^ZS (HICO) on the 12 novel classes among all evaluation datasets.

With the object images from the HOI dataset, our method is similar to VCL [17] on HOI-COCO dataset, because both two methods compose HOI samples between two images. On HICO-DET, the proposed method has a better performance than VCL [17], while the two methods have similar performance on HOI detection, as shown in Table 1. We find Recall, Precision and F1-Score are sensitive to the threshold. Thus, we further illustrates the Mean average Precision results of all models in Supplementary Materials.

The number of object images in each batch. Table 5 shows ATL achieves best performance with 2 object images. We think more object images increase the diversity of object features and balance the object distribution. However, too much object images also hampers the performance.

Object detector. Due to the domain shift between HICO-DET and COCO, COCO detector usually achieves worse result. we thus use the same fine-tuned object detector as [17]. Table 6 illustrates better detected object boxes improves the performance largely. Meanwhile, we find ATL is apparently sensitive to worse boxes. Under worse object detector (\ieCOCO detector), ATL does not improve the result. It might be because composing affordance features and object features from additional images results in poor generalization to worse boxes. When we transfer affordance representation to objects from a large number of additional images via composing novel HOI samples, we improve the scene generalization (\iethe model generalizes to novel scenes) of the affordance representation learning, while degrading the generalization to worse object boxes on HICO-DET test set. The object affordance recognition in Table 4 illustrates the scene generalization of affordance and object representations. Noticeably, worse object detector largely hampers HOI detection in two-stage method. Thus, it is necessary to utilize better object detector for evaluating HOI detection, and ATL further improves HOI detection effectively with better object detector.

Domain difference. From the large performance gap between different object detectors in Table 6, we find the HICO-DET dataset has a different domain to COCO. Table 7 shows with the same number of object instances, COCO dataset improves the performance larger than HICO-DET dataset due to the domain difference on HOI-COCO. There is a similar trend in Table 1 and Table 2. With the same COCO dataset, our method facilitates HOI detection on HOI-COCO dataset better than that on HICO-DET.

Affordance comparison with object detection results. Our method can also be applied to detected boxes of an object detector. For a robust comparison, we directly compare ATL with the object affordance result converted from object detection results according to the object affordance annotation (\iethe ground truth affordances of an object category) on HICO-DET test set. Here we use the detected box of a COCO pretrained Faster-RCNN. We train our model on HOI-COCO dataset and COCO (2014) dataset, which has a same training set to COCO pretrained Faster-RCNN. Figure 4 illustrates ATL achieves better affordance recognition results among different confidences. Meanwhile, ATL has better performance than object affordance detection when the confidence of detected box is lower.

We demonstrate the result of exploring unseen HOIs with novel objects in Figure 5. We find the baseline can not recognize the object at all, while the proposed method effectively detects the HOI with unseen objects.