Human Interaction Recognition Framework based on Interacting Body Part Attention

The recognition in RGB video is the most representative and traditional study and has a wide range of applications [5, 9, 19, 20, 21, 22, 23, 24]. Conventional methods for human activity recognition typically use hand-crafted features to represent the video. Many studies emphasized the importance of spatio-temporal local features. The BoW-based paradigm, with 3D XYT volume, is one of the most popular approaches [1, 2, 3, 4, 5]. The key attributes of interacting people were captured using preset motion attributes, such as interactive phrases [25], discriminative key components [26], action attributes [27], or poselets [28]. Kong et al. [25] proposed multiple interactive phrases as the latent mid-level feature to represent human interaction from individual actions. Sefidgar et al. [26] represented the interaction by a set of key temporal moments and the spatial structures they entail. Despite these studies, human interaction has a subtle difference in detailed movements, such as between a punch and a pat, which is too small to be described by a specific number of phrases.

For the past few years, the deep learning-based video representation methods, such as 3D CNN [6], two-stream CNN [7], and multi-stream CNN [8] have shown their effectiveness in video representation by overcoming the problem-dependent limitation of the hand-crafted features. Ibrahim et al. [29] captured the temporal dynamics of the whole activity based on the dynamics of the individual people. A combination of multi-level representations is modeled with a temporal link. Shu et al. [30] proposed a hierarchical long short-term concurrent memory (H-LSTCM) to effectively address the problem of human interaction recognition with multiple persons by learning the dynamic inter-related representation over time. They aggregated the inter-related memory from individuals by capturing the concurrently long-term inter-related dynamics among multiple persons rather than the dynamics of individuals. Tang et al. [31] proposed coherence constrained graph LSTM (CCG-LSTM) to learn the discriminative representation of a group activity. The CCG-LSTM modeled the motion of individuals relevant to the whole activity, while suppressing the irrelevant motions. Shu et al. [32] proposed concurrence-aware long short-term sub-memories to model the long-term inter-related dynamics between two interacting people. Lee et al. [33] proposed a human activity representation method, based on the co-occurrence of individual action, to predict partially observed interaction with the aid of a pre-trained CNN. Mahmood et al. [34] segmented full-body silhouettes and identified key body points to extract features that had distinct characteristics. Deng et al. [35] utilized CNN to obtain a single person action label and a group activity label and explore the relation between the actions of all individuals. However, because most studies focused on modeling individuals of the relationships in the scene, attention to subtle movements of individual human behavior and the other person’s detailed reactions was relatively less considered.

With the advance of RGB-D sensors, such as Microsoft Kinect, Asus Xtion, and Intel RealSense, action recognition using 3D skeleton sequences has received significant attention and many advanced approaches have been proposed during the past few years [9, 13, 14, 12, 10, 11]. Liu et al. [13] proposed a novel kernel enhanced bag of semantic moving words (BSW) approach to represent the dynamic property of skeleton trajectories. A structured multi-feature representation is composited by aggregating BSW with the geometric feature of skeleton data. In other work, a novel HDS-SP descriptor that consists of both spatial and temporal information from specific viewpoints was proposed by Liu et al. [14]. The HDS-SP projects 3D trajectories on specific planes and creates reasonable histograms. The most suitable viewpoint is sought out through both local search and particle swarm optimization. In other recent studies of skeleton-based human action recognition, attention-based human action recognition was attempted, owing to greater accessibility of individual body parts. Liu et al. [12] proposed a new class of long short-term memory (LSTM) [36], which is capable of selectively focusing on the informative joints in each frame, using a global context memory cell. Song et al. [10] proposed an end-to-end framework with two types of attention modules, a spatial attention module and temporal attention module, which allocate different attention to different joints and frames, respectively.

Recently, a graph neural network (GCN)-based approach received significant attention for skeleton representation, owing to the kinematic dependency between the joints of the human body. Shi et al. [11] proposed a directed graph neural network to extract the dependencies of joints, bones, and their relationships, for the final action recognition task. Yang et al. [37] achieved outstanding performance, based on a feedback graph convolutional network, which adopted a feedback mechanism to the GCN. Multi-stage temporal sampling and a dense connections-based feedback graph convolutional block is proposed to introduce feedback connections into the GCN. Although a human skeleton can provide sophisticated information about human behavior, to maximize the effectiveness of a skeleton-based modeling method, accurate and delicate skeleton joint information is essential. However, most of cost efficient RGB-D sensors are currently limited to indoor applications in close distance. The imperfect joint state, such as missing joints or jittering joints, is often observed when the joints are estimated from RGB video in the wild.

With the success of the new era of deep learning-based approaches, some stuides have achieved excellent results in extracting the human body joints from RGB videos through pose estimation [15, 16, 38, 17, 18, 39]. Toshev et al. [18] proposed the first deep neural network-based pose estimation method, called DeepPose. They formulated the pose estimation problem to a CNN-based regression problem toward body joints. Wei et al. [16] proposed a convolutional pose machine (CPM), which consists of an image feature computation module, followed by a prediction module. The CPM can handle long-range dependencies between variables in a structured prediction task by designing a sequential architecture, composed of convolutional networks that directly operate on a belief map from the previous stage. Sun et al. [15] proposed the HRNet model, which has achieved great performance by maintaining a high-resolution representation throughout the process. Cao et al. [17] successfully estimated body joints using part affinity fields (PAFs). The proposed PAF can efficiently detect the 2D pose of multiple people in real-time with high accuracy, regardless of the number of people, using the bottom-up approach. The proposed method uses a nonparametric representation to learn to associate body parts with individuals in the image. In this paper, we utilized OpenPose method [17] to estimate initial human body joint from video, to take advantage of its high accuracy and real-time performance, which are essential properties for practical real-world applications.

Because pose estimation and action recognition are two closely related problems, there are some studies that have performed both tasks simultaneously. Luvizon et al. [40] proposed a multi-task deep learning approach and performed joint 2D and 3D pose estimation from still images and human action recognition in a single framework. Nie et al. [41] proposed an AND-OR graph-based action recognition approach, which exploited the hierarchical part composition analysis. Even though the end-to-end approach has advantages regarding task optimization, it has limited extensibility to videos in varying real-world environments, such as activity scenarios. Furthermore, an approach to research involving interactions, rather than single human actions, is methodologically distinct.

From a given video clip, to estimate a multi-person body joint, we first utilize a publicly available pre-trained object detection model to obtain the bounding box of the object; we used the Faster-RCNN [42] with the Inception-resnet-v2 network [43]. The detection result provides $(x,y)$ coordinates with the height and width of each object. We also perform initial pose estimation using a publicly available real-time multi-person 2D pose estimation method, based on bottom-up approach, i.e., OpenPose [17]. OpenPose provides the first combined body and foot key-point detector, which enables us to obtain joints even though the links between joints are disconnected.

The initial pose estimation result consists of eighteen joints for each human object, from $0$ to $17$. The composition of the joints of the estimated pose is shown in Fig. 3(a). We first select fifteen joints, out of nineteen, to generate five body parts. For each human subject, we convert each joint $j^{ori}_{i}=\{j^{ori}_{0},...,j^{ori}_{17}\}$ to $j_{i}=\{j_{0},...,j_{14}\}$, as illustrated in Fig. 3(b). Additionally, the coordinate of the center of the joints between $j^{ori}_{8}$ and $j^{ori}_{11}$ is designated as joint $j_{14}$, for utilization of the torso information; this is referred to as the hip. The five body parts consist of three different joints, from $j_{0}$ to $j_{14}$, defined as follows: right arm $p_{1}$=($j_{2}$, $j_{3}$, $j_{4}$), left arm $p_{2}=$($j_{5}$, $j_{6}$, $j_{7}$), right leg $p_{3}=$($j_{8}$, $j_{9}$, $j_{10}$), left leg $p_{4}=$($j_{11}$, $j_{12}$, $j_{13}$), and torso $p_{5}=$($j_{0}$, $j_{1}$, $j_{14}$). Each number denotes the joint index and their 2D coordinates are defined as $p_{pi,pj,x,y}$, where $pi$ denotes the body part index and $pj$ denotes the joint index in each body part.

Although the pose estimation shows promising results, it often fails to detect body joints still at an individual frame-level. We performed two additional procedures to increase the usability and dependability of body joints to manage the noisy pose estimation. To complement the occasional absence of joint information, if the previous $l$ frames have failed to estimate a joint, we restore the missing joint by linearly interpolating the coordinate between the current joint at $t$ and the last successfully detected joint at the $t-l$ frame. We also utilized the bounding box from the human detection to filter out some residual, or unsuccessful, body joint estimation results, the majority of which are the partial detection of a human object, due to occlusion. First, both the head and torso of each object must be included in the bounding box. If an estimation failure occurs about the head position $j^{ori}_{0}$, the average value of the non-failure joint among {$j^{ori}_{14},j^{ori}_{15},j^{ori}_{16},j^{ori}_{17}$} is used as the head position, $j_{0}$. By performing two additional post-processing steps, we obtained the stability and reliability of the body joints from real-time multi-person pose estimation results.

Motion and posture feature: To express the overall motion and posture of an individual human behavior, we extract two types of motion features and two types of posture features based on body part status. At each time step, for fifteen body joints of five body parts, the average value of the 2D coordinate difference between each previous point and each current point of the sequence is used to calculate the motion velocity $v_{p}^{t}$ and acceleration $\hat{v}_{p}^{t}$. This is a simple, but effective, way to express the explicit motion of the body parts and joints by changing the position of each joint. We also calculate the inner angle within the body parts to represent the relative position of the joint inside the body part. The angle of each part $\theta_{in}$ is calculated using the following equation:

We also calculate the angle between body parts using Eq. (1). The outer angle $\theta_{out}$ denotes the angle between the connected body part, which is calculated using the following joint indexes, as the input in each frame: $p^{out}_{1}$=($j_{1}$, $j_{2}$, $j_{3}$), $p^{out}_{2}=$($j_{1}$, $j_{5}$, $j_{6}$), $p^{out}_{3}=$($j_{1}$, $j_{8}$, $j_{9}$), $p^{out}_{4}=$($j_{1}$, $j_{11}$, $j_{12}$). The outer angle represents the overall shape of the body part. $\theta_{in}$ and $\theta_{out}$ are simple; however, they effectively express scale-invariant human posture information.

Local image patch feature: To consider the local image where interactions actively occurred, the $(n{\times}n)$-size local image patch features are extracted from each joint of the potential major interaction area. This is motivated by the fact that most human interactions are performed using their hand or foot. We chose the following five joints from each part, which are expected to have mainly contact when interaction occurs: $j_{0}$ (head position), $j_{4}$ (right hand), $j_{7}$ (left hand), $j_{10}$ (right foot), and $j_{13}$ (left foot). The input image patches are extracted where $Img(j,o)=([j_{i,x}-n/2:j_{i,x}+n/2],[j_{i,y}-n/2:j_{i,y}+n/2])$, when the center is in position $j_{i,x,y}$. The feature representation for a specific joint $j_{\alpha}$ is the activation value of the last fully connected layer of the convolutional network $f_{s}$, with parameters $\Theta$, taking a local image patch $Img$ as the input, at the joint position $j_{\alpha}$ of object $o$:

In this section, we present the body joint based interacting body part attention mechanism. The fundamental idea of interacting part-based attention is based on the assumption that the body parts composing each behavior will have different importance when human interaction occurs. For example, if two people shake hands, person #1 will reach for person #2 and person #2 will similarly reach for person #1. The hands of the two persons get closer, touch, and then become distant from each other. However, if person #1 punches person #2, person #1 reaches out to person #2’s head and person #2 will be pushed back without motion toward person #1. If person #1 performs a push action, person #2’s response will look similar to a punch; however, person #1 will reach out to person #2’s torso with two hands. These subtle differences between similar behaviors are maximized to generate discriminative features by assigning weight to interacting body parts.

We attempt to selectively focus on both the body part of the person who leads the interaction and the body part of the other person that is most involved. The different weights are multiplied by the terms of each part of each person. To express the movement of each part, the spatial difference for $t$ time segments is defined as follows:

where $d$ denotes the relative distance between each pair of body parts of the interacting persons. The interacting body part attention between each person is calculated as follows:

The body part velocity $v_{p}^{t}$ and body part acceleration $\hat{v}_{p}^{t}$ are multiplied by the part weight $\Lambda_{p,t}$ to determine the weight of each part:

A movement that is actively participating in the interaction, which is also called the interacting body part, is emphasized by representing individual human motion, using the weighted velocity of the body part $wv_{p}^{t}$ and the acceleration of the body part $w\hat{v}_{p}^{t}$.

The image patch feature vector from each joint is also multiplied by the weight. Since an interacting body part with high weight significant to the interaction, ${\bf{pf}}_{j_{\alpha}}^{t}$ also gives a high weight to the joint-based image feature extracted from the position of the body part as follows:

Each individual object’s behavior with the interacting body part attention is represented by the combination of the following five variables of the five body parts: two posture variables $\theta_{in}$, $\theta_{out}$, two weighted motions variable $wv_{p}^{t}$, $w\hat{v}_{j}^{t}$, and one weighted local image feature ${\bf{wf}}^{t}$.

Apart from the body joint-based feature and local image patch feature extraction, we also performed full-body image-based activity descriptor generation to consider the overall appearance change with co-occurrence of multi-person behaviors. Extracting a feature vector from a full-body image through CNN has proved its robustness in human activity recognition [32, 33, 29, 35]. Because the occasional failure of body joint extraction may negatively affect the interaction representation, the imperfectness of local features is compensated by a full-body image descriptor. The full-body image descriptor generation method was inspired from the sub-volume co-occurrence matrix descriptor for human interaction prediction [33].

From the detected object region, we first extract the activation from the last fully connected layer of the inception-resnet-v2 network to obtain a feature vector of an object image. Subsequently, we generate a sub-volume for each object $\mathbf{f}$${}_{oi}^{t}=[p,\delta{x},\delta{y}]$, where $p$ denotes the average of the image representation feature vector in a sub-volume. A series of frame-level image feature vectors of object $oi$, at time $t$ for $l$ consecutive frames, are averaged into a single vector $\mathbf{f}_{oi}^{t}$. Second, $K$-means clustering is performed on the training video to generate codewords $\{w_{k}\}_{k=1}^{K}$, where $k$ denotes the number of clusters. Each of the sub-volume features $\mathbf{f}_{oi}^{t}$ is assigned to the corresponding cluster $w_{k}$, following the BoW paradigm. The index of the corresponding cluster $k_{oi}^{t}$ is the codeword index, which is also the index of the row and column of the descriptor. Finally, we construct a descriptor using sub-volume features. From each sub-volume of an object $v_{oi}^{t}=(\mathbf{f},x,y,k)$, we first measure the spatial distance between sub-volumes $oi$ and $oj$ as follows:

The overall spatial distance between sub-volume $oi$ and the other $oj$ in segment $t$ for $\#pairs$, where $oj\neq oi$, is aggregated as follows:

The difference in distance between sub-volume $oi$ and $oj$ to the global motion activation represents the participation ratio of the pair in segment $t$. The feature scoring function, based on sub-volume clustering, is calculated as follows:

After computing all required values between all sub-volumes, we finally construct the full-body image descriptor, as follows:

where $N$ is the normalization term. The value of the sub-volume between $oi$ and $oj$ is assigned to the full-body image descriptor using the corresponding cluster index, $k_{oi}^{t}$ and $k_{oj}^{t}$, of each sub-volume. Each of the descriptors is generated for every non-overlapped time step. Therefore, the descriptor is cumulatively constructed from the beginning.

In this section, we generate the combined vectors after the three-stream feature generation. From an individual object, we extract five local image features ${\bf{wf}}^{t}$, which represent the local image appearance. We select a feature vector ${\bf{wf}}^{t}_{mp}$ from the most activate body part, where the $\Lambda_{p}$ is maximized. The local image patch feature is concatenated with the motion feature and posture feature, as a combined vector of individual $\#m$, ${\bf{cf}}_{m}=[{\bf{wf}}_{mp},\theta_{in},\theta_{out},wv_{p},w\hat{v}_{j}]$. The representation of individual $\#m$ was the fed into the sub-LSTM $\#m$ to model the temporal dynamics of each person. At each time step, the hidden unit of the sub-LSTM $\#m$, $h^{t}_{m}$ is concatenated with a full body image co-occurrence descriptor $M^{t}$ as ${\bf{ff}}^{t}=[h^{t}_{1},h^{t}_{2},...,h^{t}_{m},M^{t}]$. Finally, the concatenated vector ${\bf{ff}}^{t}$ is used as the input to the LSTM and the activity classification task is the output of the LSTM after $t$ segments.

We validated the effectiveness of the proposed method by comparing with the state-of-the-art methods on the (1) BIT-Interaction dataset [44]; (2) UT-Interaction dataset [45]; (3) VIRAT 1.0 Ground dataset and VIRAT 2.0 Ground dataset [46]. The first two datasets are common and widely used in human-human interaction recognition research. The VIRAT 1.0 and VIRAT 2.0 Ground dataset consists of human-vehicle interaction tasks, which we chose with the aim of exhibiting the extensibility of our work.

The BIT-Interaction dataset used in the experimental evaluation consists of eight classes of human interactions: bow, boxing, handshake, high-five, hug, kick, pat, and push. Each class contains 50 clips. The videos were captured in a very realistic environment, including partial occlusion, movement, complex background, varying sizes, view-point changes, and lighting changes. For the evaluation, the training set is composed of 34 videos per class (272 in total), and the remaining 16 videos per class (128 in total) are used as the test set following in [25, 4, 47, 33, 32]. Fig. 4(a)-4(b) shows an example snapshot of the BIT-Interaction dataset. In 4(a), the lower half of the body of the person bowing on the left is occluded by a parked vehicle. There is a vending machine in the background in 4(b). Both images show the environmental difficulties in analyzing the human activity.

The UT-Interaction dataset used in the experimental evaluation consists of six classes of human interactions: push, kick, hug, point, punch, and handshake. The dataset is composed of two sets of videos; UT-set #1 and UT-set #2, which were captured in different environments. Each class contains 10 clips for each set. We performed a leave-one-out cross-validation for the evaluation of each set, following competing methods [3, 28, 47, 25, 48, 32, 33]. The set #1 videos were captured in a parking lot background (Fig. 4(c)-4(d)). However, the backgrounds in set #2 of the UT-Interaction dataset (Fig. 4(e)-4(f)) consisted of grass and shaking twigs, which may add noise to local patches of interest points.

VIRAT 1.0 and VIRAT 2.0 Ground datasets used in the experimental evaluation, consist of six classes of human-object interactions: a person opening a vehicle trunk (OAT), a person closing a vehicle trunk (CAT), a person loading an object to a vehicle (LAV), a person unloading an object from a vehicle (UAV), a person getting into a vehicle (GIV), and a person getting out of a vehicle (GOV). The VIRAT 1.0 Ground dataset and VIRAT 2.0 Ground dataset consist of approximately 3 hours and 8 hours of video in several parking lot backgrounds, respectively. Fig. 4(g) demonstrates that the acting person is not fully observable until he gets out of the vehicle in the shadow. The carrying object is fully occluded by the acting person in 4(h), which can disturb the understanding of the context of opening a vehicle trunk. For the evaluation, half of the video clips were used as the training set, and the remaining half were used as a testing set following previous studies [49, 50, 51, 33, 52].

In this experiment, the initial pose estimation of human objects over all video frames was done using OpenPose [17]. The bounding boxes of person are detected by Faster-RCNN [42] with the Inception-resent-v2 network [43]. To extract image patch features from joints and full body image features, we used the weight of the pre-trained Inception-resnet-v2 network, as a backbone network. The learning rate was set to $0.5\times 10^{-4}$ and the lambda_loss set to $0.5\times 10^{-3}$. The length of the time step was set to 20 and the minibatch of size 32. We set the $K$ values of the full-body feature clustering is set to 20. The number of memory cell nodes of the sub-LSTMs and the second LSTM was set to 200 and 625, respectively. The weights and biases of the LSTMs are initialized with random values having normal distributions. We used Adam optimizer [53] for all experiments. We implemented all modules in this framework using Tensorflow [54], and an NVIDIA Titan XP was used to run the experiments.

In these experiments, we additionally conduct two baseline methods to show the effect of each feature stream on the proposed framework as follows:

baseline #1: This baseline only used body joint-based features, with attention from the joint-based feature stream and local image patch feature stream. The interacting body part attention is applied to this baseline. The concatenated features at all time steps are only pooled from the LSTM of an individual object.

baseline #2: This baseline used the full-body feature, without the body joint-based feature and interacting body part attention. The co-occurrence descriptor of the image-based full-body feature stream was directly fed into final LSTM. This baseline is similar to that of the sub-volume co-occurrence matrix [33].

The experimental results on the BIT-Interaction dataset are compared with the latest studies, i.e. dynamic BoW [3], mixture of training video segment sparse coding (MSSC) [55], multiple temporal scale support vector machine (MTSSVM) [56], max-margin action prediction machine (MMAPM) [4], Lan et al. [57], Liu et al. [27], Kong et al. [25, 47], Donahue et al. [58], sub-volume co-occurrence matrix (SCM) [33], concurrence-aware long short-term sub-memories (Co-LSTM) [32], and hierarchical long short-term concurrent memory(H-LSTCM) [30]. We also compared the proposed method with two baseline performances.

Table 1 shows the classification results of the proposed method for a quantitative comparison with competing methods. The proposed method achieved higher recognition accuracy on average than all baseline methods and most of the competing approaches. In this experiment, the very recent H-LSTCM [30] shows 0.8% higher accuracy owing to the dynamic inter-related representation of the Co-LSTM unit, but the proposed method also achieved a comparable accuracy with remarkable representation power itself. Specifically, the proposed method with all three feature streams achieved a higher accuracy than baseline #1 and baseline #2, by combining implicit and explicit representation. Thus, the body joint-based representation with attention complements the lack of high-level information on the low-level image feature representation.It is better to use the high-level information than to utilize the low-level image features alone. The comprehensive consideration of individual human motions, co-occurrence of overall appearance, and local importance representation let achieve robust human interaction recognition.

UT-set #1: We compared the recognition accuracy of the proposed method with a competing state-of-the-art method, including some traditional methods and two baselines, i.e. dynamic BoW [3], Lan et al. [59], MSSC [55], MMAPM [4], Wang et al. [51], Donahue et al. [58], SCM [33], Co-LSTM [32], H-LSTCM [30], Mahmood et al. [34], and Slimani et al. [48].

The comparison of the interaction recognition performance between the proposed method with two baselines and competing methods, on the UT-set #1, is shown in Table 2. It can be seen that the combination of interacting body part attention and co-occurrence descriptor could improve the performance, which prove that specific information from different types of features can complement each other. In this experiment, the proposed method achieved 93.22% recognition accuracy, which is slightly lower than the competing state-of-the-art methods [50, 4, 32, 30]. We believe that the reason for very high performances in UT-set #1 is the superiority of the competing methods as well as the environmental simplicity. The UT-set #1 is recorded against a gray asphalt background, which allows for the clear identification of the appearance of a person, without noise. Here, it is noted that Wang et al. [51] adopt deep context feature on the event neighborhood, the size of which requires manual definition. Thus, all the competing methods achieved high performance, with ideal and rich appearance information, while the proposed method experienced difficulty in extracting body joints from some frames, owing to their occlusion.

UT-set #2: We separately compared the recognition accuracy of the proposed method on the UT-set #2, for a fair comparison. Table 3 shows the performance of competing state-of-the-art methods with some traditional methods and two baselines, i.e., dynamic BoW [3], Lan et al. [59], MSSC [55], SC [55], MMAPM [4], SCM [33], Mahmood et al. [34], and Slimani et al. [48].

We can see that different from UT-set #1, the proposed method outperformed all the competing methods with 91.33% recognition accuracy on UT-set #2. The proposed method exhibits outstanding recognition performance considering that the accuracy of the competing methods on UT-set #2 were decreased approximately 2%-15% on average compare to UT-set #1. The recognition performances of dynamic BoW [3], MSSC [55], MMAPM [4], Mahmood et al. [34], Slimani et al. [48] decreased from 85.00%, 83.33%, 95.00%, 83.50%, and 90.00% to 70.00%, 81.67%, 86.67%, 72.50%, and 83.90%, respectively. This is because UT-set #2 has more complicated background than UT-set #1; thus, the proposed method of using human structural features through joint estimation performs better than the competing methods. We believe that the proposed method is highly effective, considering the complexities of the environmental change in real-world scenarios.

We also evaluate the proposed framework on the VIRAT 1.0 Ground dataset and VIRAT 2.0 Ground dataset, to show the extensibility of our method. We compared the performance with Zhu et al. [49], Bayesian network (BN) [50], a deep hierarchical context model (DHCM) [51], SCM [33], and WSCF [52]. Because these two datasets consist of human-object interaction tasks, each human and object (mainly vehicle or box) are treated as separate objects. Here, we should note that in this experiment, the object does not have a body joint; the center of the detected object is set to the coordinates of the interacting joint. In the body joint extraction procedure, the failure of the head and torso estimation of non-human objects was exceptionally handled in this experiment.

Table 4 lists the recognition accuracies of human-vehicle interactions, using VIRAT 1.0 Ground dataset and VIRAT 2.0 Ground dataset. The proposed method outperformed all the competing methods, with recognition accuraccy of 79.83% and 81.50%, respectively.

In both datasets, the baseline #1 shows a lower performance. Owing to the characteristics of the type of interaction in the dataset, body joint extraction failures occur very frequently when occlusion occurs with the vehicle. Moreover, the vehicle has no reaction to human action at the joints. The absence of mutual interacting body parts also makes it difficult to use a local image patch and body joint-based feature. However, the proposed method shows a much higher performance than baselines #1 and #2, considering the full-body image together. The representation power of the full-body image feature is sufficient to describe the behavior of the acting objects. Moreover, the local image patch feature, extracted from the joint of the acting object, provides information about the directly interacting part of the vehicle, such as the trunk or the door. We can observe the recognition results of the VIRAT 1.0 and VIRAT 2.0 Ground dataset for each class in Fig. 5 and Fig. 6, respectively. We can see that the baseline #1 result does not show the worst accuracies on some interactions, despite the difficulty in body joint extraction. We conduct a similar experiment on the VIRAT 2.0 Ground dataset. As expected, the proposed method achieved a better performance than other methods. Even though, for the same reasons mentioned for the VIRAT 1.0 Ground dataset, baseline #1 shows a lower accuracy, the propose framework achieved the best performance by complementary modeling of human activity with full-body co-occurrence with interacting body part attention.