Semantic Role Aware Correlation Transformer
for Text to Video Retrieval

Text description internally has a hierarchical structure. For example, the whole sentence can describe global contexts; nouns and verbs can define object and actions. We use an off-the-shelf toolkit [20] for semantic role labelling to parse these components and the semantic relationship between them in a graph structure. Verb nodes are directly connected with the global node by showing temporal relations of various actions. Noun nodes are connected with verb nodes by defining the objects. $r_{ij}$ edges show the semantic relationship between verb nodes $i$ and object nodes $j$. For graph representation, we follow the method proposed by Chen et al. [3]. This approach, which is shown in Eq. 1, uses factorized weights in GCN, where $W_{t}$ is a transformation matrix and shared among all relationship types. $W_{t}$ denotes a unique matrix for different semantic roles. $g_{a}$ and $g_{b}$ denote node embeddings which can be for sentences, verbs or objects. $\beta$ is the outcome after attention is applied at the nodes.

Parsing videos into hierarchical semantic features is more challenging than texts since it includes detection, action segmentation and so on. We extract spatial context features $F_{S}$ with 2D CNNs by ResNet-152 [21], pre-trained on Imagenet [22]. The temporal context features $F_{T}$ are extracted with 3D CNNs by ResNeXt-101 [23] pre-trained on Kinetics [24], then we perform temporal max-pooling on feature maps. We extract object context features $F_{O}$ by using Faster R-CNN [25], pre-trained on MS COCO [26], and the backbone is ResNet-101 [21]. We follow the settings of Zhu et al. [17] by extracting the features at 1 FPS after RoI-pooling happens. Confidence threshold is set as 0.4, while each frame could contain up to ten boxes. Dimension for all three expert embedding is 2048.

To capture the modality-complement and specific information, we adapt a transformer with attention modules [12, 13, 14, 27]. For example, to generate spatial context-specific feature $s_{e}$, the temporal context features $F_{T}$, and object context features $F_{O}$ are concatenated in feature dimension and fed into an encoder with a multi-head self-attention layer and a feed-forward linear layer FF, as shown in Eq. LABEL:encoder.

The attention layers are multi-headed as Vaswani et al. [12]’s dot-product attention, and each layer also follows layer normalization and residual connection, as shown in Eq. 3. All the W matrices are trainable parameters.

Eq. LABEL:decoder shows the formula of generating spatial context-complement feature. The spatial context feature $F_{S}$ is first fed into the self-attention layer to encode into $z_{s}$, then with cross-modal attention is conditioned on $s_{e}$ and $z_{s}$ to generate complement feature $c_{e}$. Next, it is encoded through feed-forward linear layer FF and together with the modality-specific feature $s_{e}$ to deliver final embeddings for the spatial context $E_{S}$. ’Norm’ refers to layer normalization.

The same process is applied to generate the final embeddings $E_{T}$ and $E_{O}$ for temporal context feature $F_{T}$ and object context feature $F_{O}$, respectively, for matching.

We utilize cosine similarity to calculate the cross-modal matching score for each level by corresponding visual and textual embeddings.

Then, we average the similarities and utilize contrastive ranking loss as a training objective. Our aim is to have such positive pair $(v_{p},c_{p})$ pushed away from the negative pairs $(v_{p},c_{n})$ and $(v_{n},c_{p})$ than a set margin. $v$ and $c$ refer to the video and textual representation, respectively. This matching applies to three levels of embeddings. $\Delta$ represents the pre-defined margin, which is for training with contrastive loss.

Dataset. We evaluate our model on YouCook2 [29]. It is a video dataset on cooking gathered from YouTube. The videos have a diverse number of cooking styles and methods. It includes 89 recipe types and 14k video clips correlated by imperative English captions defining the action. Note that since annotations for the test set is not published yet, we evaluate the task on validation clips which is around 3.5k totally.

Evaluation metrics. The task is retrieving video clips based on text queries. We evaluate our model using common metrics on recall at various sets and median rank. R@1, R@5, R@10 gives the number of accurately retrieved clips in the ranking list’s top associated rates. MedR gives the median rank of correct clips in the ranking list. While higher is better for recall metrics, lower is better for the median rank.

Training. We use Glove embeddings [30] by setting the word embedding size as 300 for the text encoding. We adopt the text encoding approach of [3] to disentangle the text embeddings. Graph convolutions have two layers, and it outputs features with 1024 dimensions. A linear layer is applied to each visual expert embeddings to transform their dimensions into 1024. We use a cross-modal attention mechanism for each level in visual encoding. For training, the margin is set to $\Delta=0.2$; the epoch is 100 for each experiment. The mini-batch size is 32.

Table 1 shows the comparison with SOTA methods. While ’Visual Backbone’ in the table only refers to 3D CNNs Features, ResNet-based models are also used in all the models as 2D CNNs backbone. We focus on training only with YouCook2 dataset; however, we also add the methods when using pre-training for reference. ’FT’ denotes fine-tuning on YouCook2 dataset. We see that the accelerator type, which directly defines the maximum batch size, and pre-training usage affects the result sharply. There are even differences with the same model when training on different accelerators, epoch, and batch sizes; which can be found on corresponding papers. For example, the MIL-NCE method reaches 50% of its accuracy when trained with less batch size and epochs. Our method surpasses the HGR method [3] with a high margin for all metrics. We also outperform the other two SOTA methods: Miech et al. [6], and HGLMM [28], in terms of the first two parameters. We think that modality-specific and modality-complement features improve accuracy at R@1 and R@5, which are more demanding and useful for real-world applications.

Since we adopt our model from HGR [3], we implement extensive ablation to show the improvement in our approach on the visual encoding part. We aim to find how the feature expert combination affects the result. Our model falls short when we feed three levels with only 2D features. The same decrease continues with the concatenated 2D and 3D features at dimension zero as well as the concatenated 2D and 3D features at dimension one. However, when we feed them with 2D, 3D, and RoI features respectively, while the HGR model shows a slight decrease, our model reaches a better result with a high margin. This confirms our insight that inter-modal correlation can be exploited with our proposed cross-modal attention mechanism to achieve better results.