Asynchronous Interaction Aggregation for Action Detection

To model interactions in video, we need to find correctly what the queried person is interacted with. Previous works such as [36] calculate the interactions among all the pixels in feature map. Being computational expensive, these brute-force methods struggle to learn interactions among pixels due to the limited size of video dataset. Thus we go down to consider how to obtain concentrated interacted features. We observe that persons are always interacting with concrete objects and other persons. Therefore, we extract object and person embedding as the instance level feature. In addition, video frames are always highly correlated, thus we keep the long-term person features as the memory features.

Instance level features are cropped from the video features. Since computing the whole long video is impossible, we split it to consecutive short video clips $[v_{1},v_{2},\dots,v_{T}]$. The $d$-dimensional features of the $t^{th}$ clip $v_{t}$ is extracted using a video backbone model: $f_{t}=\mathcal{F}(v_{t},\phi_{\mathcal{F}})$ where $\phi_{\mathcal{F}}$ is the parameters.

A detector is applied on the middle frame of $v_{t}$ to get person boxes and object boxes. Based on the detected bounding boxes, we apply RoIAlign to crop the person and object features out from extracted features $f_{t}$. The person and object features in $v_{t}$ are denoted respectively as $P_{t}$ and $O_{t}$.

One clip is only a short session and misses the temporal global semantics. In order to model the temporal interaction, we keep tracks of memory features. The memory features consist of person features in consecutive clips: $M_{t}=[P_{t-L},\dots,P_{t},\dots,P_{t+L}]$, where $(2L+1)$ is the size of clip-wise reception field. In practice, a certain number of persons are sampled from each neighbor clip.

The three features above have semantic meaning and contain concentrated information to recognize actions. With these three features, we are now able to model semantic interactions explicitly.

How do we leverage these extracted features? For a target person, there are multiple detected objects and persons. The main challenge is how to correctly pay more attention to the objects or the persons that the target person is interacted with. In this section, we introduce first our Interaction Block that can adaptively model each type of interactions in a uniform structure. Then we describe our Interaction Aggregation (IA) structure that aggregates multiple interactions.

The hierarchical IA structure consists of multiple interaction blocks. Each of them is tailored for a single type of interactions. The interaction blocks are deep nested with other blocks to efficiently integrate different interactions for higher level features and more precise attentions.

-P-Block: P-Block models person-person interaction in the same clip. It is helpful for recognizing actions like listening and talking. Since the query input is already the person features or the enhanced person features, we take the key/value input the same as the query input.

-O-Block: In O-Block, we aim to distill person-object interactions such as pushing and carrying an object. Our key/value input is the detected object features $O_{t}$. In the case where detected objects are too many, we sample based on detection scores. Figure 3a is an illustration of O-Block.

-M-Block: Some actions have strong logical connections along the temporal dimension like opening and closing. We model this type of interaction as temporal interactions. To operate this type, we take memory features $M_{t}$ as key/value input of an M-Block.

-Parallel IA: A naive approach is to model different interactions separately and merge them at last. As displayed in Figure 4a., each branch follows similar structure to [12] that treats one type of interactions without the knowledge of other interactions. We argue that the parallel structure struggles to find interaction precisely. We illustrate the attention of the last P-Block in Figure 4c. by displaying the output of the softmax layer for different persons. As we can see, the target person is apparently watching and listening to the man in red. However, the P block pays similar attention to two men.

-Serial IA: The knowledge across different interactions is helpful for recognizing interactions. We propose the serial IA to aggregate different types of interactions. As shown in Figure 3b., different types of interaction blocks are stacked in sequence. The queried features are enhanced in one interaction block and then passed to an interaction block of a different type. Figure 4f. and 4g. demonstrate the advantage of serial IA: The first P block can not differ the importance of the man in left and the man in middle. After gaining knowledge from O-block and M-block, the second P-block is able to pay more attention to man in left who is talking to the target person. Comparing to the attention in parallel IA (Figure 4c.), our serial IA is better in finding interactions.

-Dense Serial IA: In above structures, the connections between interaction blocks are totally manually designed and the input of an interaction block is simply the output of another one. We expect the model to further learn which interaction features to take by itself. With this in mind, we propose the Dense Serial IA extension. In Dense Serial IA, each interaction block takes all the outputs of previous blocks and aggregates them using a learnable weight. Formally, the query of the $i^{th}$ block can be represent as

where $\odot$ denotes the element-wise multiplication, $\mathbf{C}$ is the set of indices of previous blocks, $W_{j}$ is a learnable $d$-dimenional vector normalized with a Softmax function among $\mathbf{C}$, $E_{t,j}$ is the enhanced output features from the $j^{th}$ block. Dense Serial IA is illustrated in Figure 3c.

Long-term memory features can provide useful temporal semantics to aid recognizing actions. Imagine a scene where a person opens the bottle cap, drinks water, and finally closes the cap, it could be hard to detect opening and closing with subtle movements. But knowing the context of drinking water, things get much easier.

Inspired by [38], our algorithm is composed of a memory component, the memory pool $\Omega$ and two basic operations, READ and WRITE. The memory pool $\Omega$ records memory features. Each feature $\hat{P}_{t}^{(i)}$ in this pool is an estimated value and tagged with a loss value $\delta_{t}^{(i)}$. This loss value $\delta_{t}^{(i)}$ logs the convergence state of the whole network. Two basic operations are invoked at each iteration of training:

-READ: At the beginning of each iteration, given a video clip $v_{t}^{(i)}$ from the $i^{th}$ video, memory features around the target clip are read from the memory pool $\Omega$, which is $[\hat{P}_{t-L}^{(i)},\dots,\hat{P}_{t-1}^{(i)}]$ and $[\hat{P}_{t+1}^{(i)},\dots,\hat{P}_{t+L}^{(i)}]$ specifically.

-WRITE: At the end of each iteration, personal features for the target clip $P_{t}^{(i)}$ are written back to the memory pool $\Omega$ as estimated memory features $\hat{P}_{t}^{(i)}$, tagged with current loss value.

-Reweighting: The features we READ are written at different training steps. Therefore, some early written features are extracted from the model whose parameters are much different with current ones. Therefore, we impact a penalty factor $w_{t^{\prime}}^{(i)}$ to discard badly estimated features. We design a simple yet effective way to compute such penalty factor by using loss tag. The difference between the loss tag $\delta_{t^{\prime}}^{(i)}$ and current loss value is expressed as,

which should be very close to 1 when the difference is small. As the network converges, the estimated features in the memory pool are expected to be closer and closer to the precise features and $w_{t^{\prime}}^{(i)}$ approaches to 1.

As shown in Figure 5, the consumption of our algorithm has no obvious increase in both GPU memory and computation as the length of memory features grows, and thus we can use long enough memory features on current common devices. With dynamic updating, the asynchronous memory features can be better exploited than frozen ones.

Finally, we compare our results on AVA with previous methods in Table 2. We show our results on both AVA v2.1 and v2.2. Our method surpass all previous works on both versions.

The AVA v2.2 dataset, is the newer benchmark used in ActivityNet challenge 2019 [8]. On the validation set, our method reports a new state-of-the-art 33.11 mAP with one single model, which outperforms the strong baseline SlowFast networks by 3.7 mAP. On the test split, we train our model on both training and validation splits and use a relative longer scheduler due to more data. With an ensemble of three models with different learning rates and aggregation structures, our method achieves better performance than the winning entry of AVA action detection challenge 2019 (an ensemble with 7 SlowFast networks). The per category results for our method and SlowFast baseline is illustrated in Figure 6. We can observe the performance gain for each category, especially for those who contain interactions with video context.

We note that we use a new larger Kinetics-700 for pre-training. As shown in Table 2, the backbone we implement in this work has a similar performance to the official model pre-trained with Kinetics-600. This comparison proves that the very most part of performance gain is from our method.

Since the bounding boxes are annotated in frame level, we choose to use a ResNet50-C2D proposed in [36] as our video backbone model. To receive single image as input, we remove all temporal poolings in it. We pre-train it on the Kinetics-400 dataset. Other settings are basically the same as AVA experiments. More implementation details are provided in Supplementary Material.

Table 3 shows the result on UCF101-24 test split in terms of Frame-mAP with 0.5 IOU threshold. As we can see in the table, AIA achieves 3.3% mAP gain comparing to our C2D baseline. Moreover, with only a lightweight 2D backbone (other works could use some 3D backbone), our method still achieves very competitive results.

For both verb model and noun model, we use the extracted segment features (global average pooling of $f_{t}$) as query input for IA structure. Hand features and object features are cropped and then fed into IA to model person-person and person-object interactions. For verb model, the memory features are the segment features. For noun model, the memory features are the object features extracted from object detector feature map, thus the AMU algorithm is only applied to the verb model. Other experimental details different from AVA setting are provided in Supplementary Material.

We observe from Table 4 a significant gain for all three tasks. All the variants of AIA outperform the strong baseline SlowFast. Among them, the dense serial IA achieves the best performance for the verbs test, leading to 3.2% improvement on top-1 score. The serial IA results in 4.9% for the nouns test and 3.6% for the action test.