NVIDIA-UNIBZ Submission for EPIC-KITCHENS-100 Action Anticipation Challenge 2022

To exploit the effective information in space-time structure, we proposed space-time decomposition attention – a light-weighted and computation-efficient attention, which integrates spatial and temporal operators from separated branches as shown in Figure 2. To define spatial and temporal branch operators, spatial filter was introduced to recognize the relevant spatial information by the max and mean pooled features of inputs:

$$f_{\mathcal{X}}(X)=\text{sigmoid}(\theta_{\mathcal{X}}*[X_{max},X_{avg}]+b_{\mathcal{X}})$$

(1)

where $*$ is convolution, $X_{avg},X_{max}$ are channel mean and max pooled, $\theta_{\mathcal{X}}$ and $b_{\mathcal{X}}$ are convolution kernels and biases.

The general higher-order recurrent network [6, 7, 11] is with the following form:

$$h_{t}=f(x_{t},\phi(h_{t-1:t-S})),$$

(2)

where the hidden state at time $t$, $h_{t}$, is computed by the cell function $f$ on input $x_{t}$ and $S$ orders states $h_{t-1:t-S}$ aggregated by the function $\phi$.

The HORST cell can be viewed as instantiating $\phi$ with space-time decomposition attention and maintain the previous states $h_{t-1:t-S}$ in an internal queue by the first-in first out update policy. The overall design is shown in Figure 1. At each step $t$, we process the video frame by a 2D-CNN backbone to obtain the feature map and encode it to the intermediate representation. Such representation is served as query and cross-reference from the historical states via the space-time decomposition attention. The attention output is then pushed to the queue while releasing the oldest state. Cell output finally propagate to the classifier. More details are found in [8] and we build our HORST models for this competition based on the codebase published at https://github.com/CorcovadoMing/HORST.

MPNNEL translates the anticipation problem into a message passing scheme, producing a graph-structured space-time representation. The connectivity of the graph structure is inferred from the input at each time step. The readout function is called when the prediction is required at any timestep. The proposed model utilizes only multi-head self-attention for information routing between vertices. The overall architecture definition is illustrated in Figure 4. Note that the resulting spatial graph is either bi-directed, when an adjacency matrix $A$ is provided, or else it is un-directed.

Without any prior knowledge, we assume each vertex in the graph can be accessible by any other vertices. In this case the scaled dot-product in the self-attention computes the pairwise similarity of all vertices from the inputs can be viewed as an implicit edge estimation. This can be extended by optionally providing the edge estimation explicitly by one of following strategies, also shown in Figure 3:

• •

  Template Bank (TB), which forms the estimation of edge connections by soft-fusing a set of learnable templates using weights computed from the frame input.

• •

  Class Token Projection (CTP), which performs the outer-product of class tokens to construct the edge estimation. The class tokens are supervised from provided verb and noun labels.

More details are found in [9] and we build our MPNNEL models for this competition based on the codebase published at https://github.com/CorcovadoMing/MPNNEL.

We trained all of our models by having them experience four learning phases, where they are (i) warmup phase; (ii) ordinary training; (iii) finetune; and (iv) finetune with joint validation set. The demonstration is shown in Figure 5.

The details of different training phases are:

• •

  Warmup Phase: The model is end-to-end trainable on the target dataset. The model can access the actual action frames beyond the anticipation limitation only in this training phase.

• •

  Ordinary Phase: The model is trained with backbone freeze, and the action frames are not accessible in this and following training stages.

• •

  Finetune Phase: The model is trained with backbone freeze, under the lower learning rate, and with the class weightings adjusted in the loss functions.

• •

  Finetune with joint validation set: The model is trained with backbone freeze, under the lower learning rate, and with the class weightings adjusted. Additionally, the validation samples are joint together in the supervised learning.

The warmup phase is targeted to build a strong feature extractor for the competition, the backbone model is able to receive gradients and the action frames are allowed to be observable exclusively in this phase. The ordinary phase focus on the anticipation task and trains the HORST and MPNN architectures with the feature extractor kept frozen. The finetune phases learn to distinguish the hard samples and tailed cases by class weighting adjustments, also with the validation set jointly in the last training. Every phases are resumed from its previous step and each model training experiences the complete learning rate scheduling.

We adjusted the class weightings of the cross-entropy loss for individual verbs, nouns, and actions during the fine-tuning stages. The adjustment is based on the label frequency summarized from the training set. Note the action distribution defined in EPIC-Kitchen-100 is composed of joint probability of verbs and nouns, however, the label frequency of the action class could be different than the individual frequency belonging to verbs and nouns. Therefore, we empirically found this adjustment brings additional regularization to the model learning and results in noticeable gains on the validation up to 4% improvements.

We prepared each input modality as follow: RGB frames are resized to 224x224 and the pixel values are scaled from [0, 256] to [-1, 1]. The Flow modality came with the two maps described for horizontal and vertical optical changes, we stacked the two maps in channel dimension and resized them to 224x224 with pixel intensity scaled to [-1, 1]. As inspired from [3], the Obj modality is formed by summarizing the object detection confidences from the officially provided object features, and discards the location information of the bounding boxes. Masked-RGB is the modality which multiplies the masking, extracted from a pretrained MaskedRCNN, with the RGB input.

The RandAugment [1] is applied for RGB, Masked-RGB, and Flow inputs. The video clip for training and inference are all sampled at 4 FPS (i.e., step 0.25s), as inherited from RU-LSTM baseline [3]. Each sample contains 14 sequential frames during training from 3.5s to 0.25s before action starts. However the last 3 frames are strictly not allowed to access in this competition since the anticipation time set to 1s. The total length of the inference context in our models are 2.5s (observed from 3.5s to 1s).

We trained our model using batch size 32 on 4 $\times$ NVIDIA A100 GPUs. AdaBelief [13] in combination with the look-ahead optimizer [12] is adopted. Weight decay is set to 0.001. The learning rate is set to 1e-4 and decreased to 1e-6 for warmup and ordinary training, and 1e-5 decreased to 1e-7 for finetune phases. The learning rate scheduling uses FlatCosine, which keeps the initial learning rate for the first 75% of total epochs and switches to cosine schedule for the last 25% epochs (see also Figure 5). The total epochs for warmup and ordinary training are set to 50, and 20 for finetune phases.

Unlike other modalities which are in an spatial-temporal structure, the Obj modality is presented as a temporal sequence of frame vectors. Each such vector represents the frame-level object scores computed from an object detection pretrained model. We modified HORST and MPNNEL models for supporting the 1D object vector representation, by replacing the 2D Convolution in HORST with the fully-connected layer; and by replacing the object entities with learnable vectors multiplied by corresponding object scores to defined the vertices in MPNNEL. Some models apply on Flow modality by snippets, as suggested in [3], where the previous 5 sequential Flow features are stacked.

For all of our models we considered the Swin Transformer (i.e., base configuration, Swin-B) [4], and ConvNeXt [5] as backbones. We showed validation results of each representative category in Table 1. Note the validation results reported in Table 1 are before training with the joint validation set, in order to keep the numbers meaningful.

We manually selected the strong models from each individual variant, and tried to balance between HORST and MPNNEL instances to maintain the diversity among the ensembled models. Our best submission, 19.61% overall accuracy, was achieved by an ensemble of in total 54 models. Those models consisted of 30 RGB models, 10 Flow models, 8 Obj models, and 6 Masked-RGB models.

Table 2 reports on the trajectory we stepped to our highest score submission. Averaging the prediction scores in the HORST family resulted in 17.47% overall test accuracy, and 18.19% in the MPNNEL family. Combining both HORST and MPNNEL further improved the score significantly, to 19.52%, indicating some degree of complementarity of the two recurrent models. We also empirically found that emphasizing the prediction scores of all RGB models can have additional performance gains. In our best submission we weighted all RGB models by a factor 1.2x higher than other modalities.