Figure 2 gives an overview of the proposed CAGNet, which consists of three components including a base-model, a TOGN and a CAR module. Given an input image and the detected human bodies as RoIs, the base-model takes a backbone CNN to extract features from the input, which are then processed by a RoIAlign module [13] to generate local features for each individual. Afterwards the local features are processed by one FC layer to generate base features as inputs to TOGN. Our TOGN graph (Section 4.2) includes two types of variable nodes (circles): one type is the $y$ node to represent the action category of the associated person, the other type is the $z$ node to represent the existence of interactive relation between a pair of people. The graph also includes a series of factor nodes (squares) in order to capture two types of third-order dependencies, respectively encoded by the $(y_{i},y_{j},z_{i,j})$ triplets (blue factor nodes) and the $(z_{u,v},z_{v,w},z_{u,w})$ triplets (the yellow factor node). We take the base features to initialize TOGN, and perform feature updating by passing messages between factor nodes and variable nodes such that rich contextual information could be embedded. Though TOGN is able to learn rich contextual representations to facilitate the HIU task, the labeling and grouping consistencies among variables are not explicitly modeled. To alleviate this, we introduce a CAR module, which essentially conducts a deductive reasoning leveraging the oracles presented in Section 4.3. In practice the reasoning is implemented via solving a surrogate mean-field inference with differentiable high-order energy functions, which allows end-to-end learning of all modules within our CAGNet with GPU acceleration (Section 4.4).

We now elaborate our TOGN for HIU in order capture two categories of third-order dependencies among action and interactive-relation variables.

Third-Order Factor Graph Formally, we define the factor graph as $\mathcal{G}=(\mathcal{V},\mathcal{F},\mathcal{E})$, where $\mathcal{V}$ is the set of variable nodes, $\mathcal{F}$ is the set of factor nodes, and $\mathcal{E}$ is the set of edges. The node set is split into two disjoint subsets: $\mathcal{V}=\mathcal{V}^{y}\cup\mathcal{V}^{z},\mathcal{V}^{y}\cap\mathcal{V}^{z}=\emptyset$. Specifically, $\mathcal{V}^{y}=\{1,\cdots,n\}$, and $\mathcal{V}^{z}=\{n+1,\cdots,n+\tbinom{n}{2}\}$. For each node $i\in\mathcal{V}^{y}$, a variable $y_{i}\in\mathcal{Y}$ is associated with it to represent the action category of the $i$-th individual. Let $g(u,v)$ be a function:

For each node $k\in\mathcal{V}^{z}$, a variable $z_{u,v}\in\{0,1\}$ is associated with it to represent if the pair of people $(u,v)$ are interacting ($z_{u,v}=1$) or not ($z_{u,v}=0$), where $k=g(u,v)$.

To encode different relations, we create two groups of factor nodes. The first group is

which is taken to implicitly model the correlations among $y_{i}$, $y_{j}$ and $z_{i,j}$ based on their base features. Intuitively, action labels $(y_{i},y_{j})$ are highly correlated when the associated people are interacting (taking the kicking interaction in Figure 2 as an example), while this correlation vanishes if they are not interacting (e.g., Person 2 and Person 3 in Figure 2).

The second group of factors is defined as

which is leveraged to implicitly model the correlations among $z_{r,s}$, $z_{s,t}$ and $z_{r,t}$ for each triplet of people $(r,s,t)$. With such factors, we encourage the model to learn representations for the prediction of consistent interactive relations for each triplet. Fortunately, higher-order consistencies can be guaranteed if all third-order consistencies are satisfied (detailed in Section 4.3).

In summary, the factor node set is $\mathcal{F}=\mathcal{F}^{y}\cup\mathcal{F}^{z}$. Given $\mathcal{F}$ and $\mathcal{V}$, the edge set $\mathcal{E}$ is set up by connecting variable nodes with factor nodes. Specifically, for each factor node $c=(i,j,k)\in\mathcal{F}$, we put three edges $(c,i)$, $(c,j)$ and $(c,k)$ into $\mathcal{E}$, which finalizes the construction of the TOGN graph.

Initial Node Feature For each node $i\in\mathcal{V}^{y}$, let $\mathbf{\phi}_{i}$ be the base feature extracted from the bounding box region of the $i$-th person using the base-model. For each $(u,v)\in\mathcal{V}^{y}\times\mathcal{V}^{y}$, $u<v$, let $j=g(u,v)\in\mathcal{V}^{z}$. We concatenate $\mathbf{\phi}_{u}$ and $\mathbf{\phi}_{v}$, and use the concatenation as the base feature (denoted by $\mathbf{\phi}_{j}$) for the variable node $j$. In order to compute the initial node features, we apply to the base features the linear transformations:

which project the original features into $\mathbb{R}^{D_{1}}$ space:

Initial Factor Feature The factor features are computed based on node features. For each factor node $c=(i,j,g(i,j))\in\mathcal{F}^{y}$, the initial factor feature $\mathbf{g}^{1}_{c}\in\mathbb{R}^{D_{1}}$ is computed with:

For each $d=(g(r,s),g(s,t),g(r,t))\in\mathcal{F}^{z}$, the associated factor feature $\mathbf{g}_{d}^{1}\in\mathbb{R}^{D_{1}}$ is obtained using:

Edge Feature For each edge $e=(q,p)\in\mathcal{E}$, the related feature $\mathbf{t}_{e}\in\mathbb{R}^{H}$ is given by:

where $p\in\mathcal{V},q\in\mathcal{F}$ and ${\rm FC^{e}}$ maps the concatenated feature vector to $\mathbb{R}^{H}$ space.

Taking as inputs the factor graph and the initial features, the first TOGN layer performs feature updating with the method described in Section 3. Afterwards we take the updated features as inputs to the next TOGN layer (which shares the factor graph and the feature updating algorithm with the first TOGN layer, but using different model parameters), and perform feature updating again. Empirically, we find that TOGN with 10 such layers works well for HIU. Finally, we compute the classification scores for individual actions and pairwise interactive relations using

where $\mathbf{f}_{i}^{*}$, $\mathbf{f}_{j}^{*}$ are updated node features output by the last TOGN layer, $\alpha$ and $\beta$ are linear functions which compute classification scores for individual actions ($\theta_{i}\in\mathbb{R}^{|\mathcal{Y}|}$) and pairwise interactive relations ($\theta_{j}\in\{0,1\}$), respectively.

To resolve possible inconsistencies (recall Figure 1) incurred by using consistency-unaware models like CNN, PGNN and TOGN, we first present two deductive reasoning bias for human interaction understanding:

• •

  The compatibility oracle: For any pair of interacting (denoted by $\leftrightarrow$) people $A$ and $B$, their action categories must be compatible (denoted by $\odot$). In logical words, this rule is represented by $A\leftrightarrow B\Rightarrow y_{A}\odot y_{B}$.

• •

  The transitivity oracle: Considering the interactive relations among a triplet of people $(A,B,C)$, we have $(A\leftrightarrow B)~{}\&~{}(B\leftrightarrow C)\Rightarrow(A\leftrightarrow C)$.

Typical compatible examples include (handshake, handshake), (pass, receive) and (punch, fall), and typical incompatible examples are (handshake, hug), (punch, pass), (highfive, handshake). Instead of predesignating such compatibility, which might change across datasets, our CAR module is able to learn them directly from data. Examples obey or violate the transitivity are shown in Figure 3. Though this oracle only considers triplets of people, it is straightforward to prove that the higher-order transitivity associated with an arbitrary number of people is simply a conclusion of the third-order transitivity. Intuitively, by enforcing the transitivity across all triplets, participants in the scene are split into different groups, such that individuals in the identical group are interacting with each other, while people in different groups have no interaction.

With such oracles, predictions of the TOGN described in Section 4.2 could be refined by applying the traditional logical reasoning algorithms like resolution. However, embedding such reasoning into deep learning frameworks directly is highly challenging. As a workaround, our reasoning approach first embeds the knowledge into an energy function defined by

where $\theta_{j,k}(z_{j,k})=\theta_{g(j,k)}(z_{j,k})$. The data terms $-\theta_{i}$ and $-\theta_{j,k}$ (computed by Equations (11) and (12)) are utilized to penalize particular $y$-label and $z$-label assignments respectively based on the learned deep representations. The functions $K^{\mathcal{C}}$ and $K^{\mathcal{T}}$ are the so-called $P^{n}$-Potts models [19] defined by

Here $\Gamma$ is a set $\{(1,1,0),(1,0,1),(0,1,1)\}$ that includes all cases violating the transitivity oracle, $\lambda^{\mathcal{C}}(y_{j},y_{k})$ and $\lambda^{\mathcal{T}}$ are penalties incurred by predictions which violate the compatibility and transitivity oracles. It is easy to check that when $\lambda^{\mathcal{C}}$ and $\lambda^{\mathcal{T}}$ are sufficiently large, minimizing the energy (13) delivers desirable $\mathbf{y}$ and $\mathbf{z}$ predictions which satisfy the compatibility and transitivity oracles. In this paper, instead of predesignating suitable $\lambda^{\mathcal{C}}$ and $\lambda^{\mathcal{T}}$ values, we learn them from data in conjunction with other parameters of CAGNet.

Mean-Field Inference Minimizing (13) is NP-complete. Here we derive an efficient mean-field inference algorithm by first approximating the joint distribution $P(\mathbf{y},\mathbf{z}|\mathbf{x})\propto\exp(-E(\mathbf{y},\mathbf{z};\mathbf{x}))$ with a product of independent marginal distributions:

Then we derive the mean-field updates of all marginal distributions using the techniques described in [36], which gives

where $Z_{i}$ is a normalization constant. The marginal distributions on $z$ variables are

where $\mathbbm{1}(\cdot)$ is an indicator function (gives $1$ if the testing condition holds, and $0$ otherwise), $t\in\{1,2,\ldots,T\}$, $Z_{k,l}$ is a normalization constant. We initialize the marginal distributions $Q_{i}^{0}(y_{i})$, $Q_{k,l}^{0}(z_{k,l})$ by applying the softmax function to the scores output by the graph network. The inference is summarized by Algorithm 1. Note that we can perform the updates of all expectations (Equation (17) and (19)) and marginal probabilities (Equation (18) and (20)) in parallel, which yields very efficient inference.

As mentioned, Algorithm 1 is a surrogate of the consistency-aware reasoning task taking the two oracles as its knowledge-base. This algorithm actually forms the last layer of our CAGNet, which outputs updated action scores $\check{\theta}_{i}~{}\forall i\in V^{y}$ and interactive scores $\check{\theta}_{j,k}$, $\forall l\in\mathcal{V}^{z}$ and $g(j,k)=l$. Our experimental results in Section 5 demonstrate that such updated scores are able to deliver more consistent HIU predictions.

The mean-field inference algorithm allows the back-propagation of the error signals $\frac{\partial{Loss}}{\partial{Q}}$ to all parameters of CAGNet (including that of the base-model, the TOGN and the CAR module), which enables the joint training of all parameters from scratch. In practice, we resort to a two-stage training due to the limitation of computational resources. The first stage learns a base-model with the backbone CNN initialized by a model pre-trained on ImageNet. The second stage trains the TOGN, $\lambda^{C}(y_{j},y_{k})$ and $\lambda^{T}$ jointly with fixed backbone-parameters. We train all models using the identical cross-entropy losses computed on both $\mathbf{y}$ and $\mathbf{z}$ predictions.

Implementation Details Our implementation is based on PyTorch deep learning toolbox and a workstation with three pieces of NVIDIA GeForce GTX 1080 Ti GPU. We test several backbone CNNs including VGG19 [33], ResNet 50 [14] and Inception V3 [35]. We use the official implementation of RoIAlign by PyTorch, which outputs feature maps with a size of $5\times 5\times 1056$ (using Inception V3). We add dropout (the ratio is 0.3) followed by a layer-normalization to every FC layer of CAGNet except for the ones computing final classification scores. For the mean-field inference we set $\lambda^{\mathcal{C}}(y_{j},y_{k})=0.5$ and $\lambda^{\mathcal{T}}=0.1$ for initialization. We adopt mini-batch SGD with Adam to learn the network parameters, and train all models in 200 epochs. We augment training data with random combinations of scaling, cropping, horizontal flipping and color jittering. For the scaling and flipping operations, the bounding boxes are scaled and flipped as well.

Evaluation Metric Since the numbers of instances across different classes are significantly imbalanced, we use multiple metrics including F1-score, overall accuracy and mean IoU for evaluation. Specifically, we calculate the macro-averaged-F1 scores on $\mathbf{y}$ and $\mathbf{z}$ predictions respectively (using the f1_score function in sklearn package), and present the mean of the two F1 scores. Likewise, overall accuracy calculates the mean of the action-classification accuracy and the interactive-relation-classification accuracy. To obtain mean IoU, we first compute IoU value on each class, then average all IoU values. We first analyze the capabilities of different components in the proposed CAGNet, using results provided by Table 1.

Choice of Backbone-CNN. Here we evaluate base models (see Figure 2) taking different backbone CNNs to extract image features. We test three popular backbones: VGG19 [33], Inception V3 [35] and ResNet50 [14], and the results correspond to the first three rows (from top to bottom) in Table 1. Inception V3 performs notably better than other backbones on all benchmarks. The reason might be that Inception V3 is able to learn multi-scale feature representations, which stacks into a feature pyramid to better capture the appearance of human actions. Hence we use Inception V3 as the backbone for all subsequent experiments.

Power of the TOGN Remember that the proposed TOGN takes the features extracted by Inception V3 as inputs, and learn third-order dependencies among structured variables. Overall, the proposed TOGN outperforms all basemodels on the three benchmarks under all considered metrics. In particular, TOGN results are moderately higher than the best basemodel (i.e. Inception V3) on UT, and are significantly better than the basemodel on BIT (91.26 vs. 87.84 on F1) and TVHI (90.41 vs. 83.0 on F1), thanks to the rich contextual representations learned by our TOGN. Note that the performance gain on UT is much less compared with results on other benchmarks. This is probably because human interactions in UT (each video contains just two individuals, and the background is clear) are simpler than BIT and TVHI, and the performance on this dataset tends to be saturated.

Effect of the CAR Module Here we compare four models: 1) Base model + CAR that consists of the base-model (Inception V3 as backbone) followed by the proposed CAR module; 2) TOGN+${\rm CAR^{C}}$ is the proposed CAGNet without taking the transitivity oracle into consideration; 3) TOGN+${\rm CAR^{T}}$ is the proposed CAGNet without taking the compatibility oracle into consideration; 4) TOGN+${\rm CAR^{CT}}$ actually is our full CAGNet. Here we can draw two conclusions based on the results in Table 1. First, the incorporation of the CAR module (Base model + CAR and TOGN+${\rm CAR^{CT}}$) boosts HIU performance (2.42, 2.78 and 4.62 points better than TOGN on TVHI in terms of F1, Accuracy and mean IoU), which validates the significance of exploiting consistency-aware-reasoning. Second, both oracles are critical to achieve the best results. Though incorporating either the compatibility (TOGN+${\rm CAR^{C}}$) or the transitivity oracle (TOGN+${\rm CAR^{T}}$) already performs better than TOGN, TOGN with both oracles (TOGN+${\rm CAR^{CT}}$) performs notably better than using each of them separately, which suggests that these two oracles complement each other for HIU.

We consider three recent approaches. Joint + AS [39] first extracts motion features of individual actions with backbone CNN. Afterwards the deep and contextual features of human interactions are fused by structured SVM. This method is able to predict $\mathbf{y}$ and $\mathbf{z}$ in a joint manner. QP + CCCP [41] takes a structured model to represent the correlations between $\mathbf{y}$ and $\mathbf{z}$ variables as well. It also developed an inference algorithm (namely QP + CCCP) to solve the related inference problem. GN [42] is a recent state-of-the-art for recognizing collective human activities. This model is empowered by both the representative ability of deep CNNs and the attention mechanism of PGNN. Note that GN does not yield $\mathbf{z}$ predictions. We fix this with two solutions. First, we just set $z_{i,j}=1$ if the learned relation value is greater than $0.5$ (see Equation (2) in [42]), otherwise we set $z_{i,j}=0$ (this solution does not introduce new parameters). Second, we attach a head to the tail of GN to make $z$ predictions, and train parameters of this head with cross-entropy loss. We call such a solution the Modified GN. We find that such a straightforward modification offers a boost of performance on HIU compared against the original GN (see Table 2 to Table 4). The reason is that Modified GN is trained with additional supervision on interactive relations, which guides the network to learn more useful representations for the prediction of $z$.

For fair comparison, all methods take Inception V3 as the backbone to extract image features. Results on three datasets are provided in Table 2, Table 3 and Table 4. We can see that CAGNet outperforms modified GN and shallow structured models ( Joint + AS and QP + CCCP) significantly on all evaluated benchmarks. Compared with CAGNet which is able to model higher-order relations, modified GN is only able to model pairwise interactive relations, hence it is consistency-unaware. Consequently GN and modified GN perform much worse than CAGNet. Albeit sharing the same feature extractor (Inception V3) with CAGNet, Joint + AS and QP + CCCP learn human interactive relations via shallow structured models without incorporating higher-order contextual dependencies and consistency-aware reasoning, hence their performances are much worse than our CAGNet.

To provide a qualitative analysis of different models, we visualize a few predictions in Figure 4. Though the predicted action labels are inconsistent or the predicted interactive relations violate the transitivity oracle using either the Base-model or the modified GN, thanks to our proposed TOGN and CAR module, CAGNet is able to make perfect predictions, at least on these examples.