Exploiting Semantic Role Contextualized Video Features
for Multi-Instance Text-Video Retrieval
EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022

With the rise of videos uploaded by users via social media channels, cross-modal retrieval of video data and natural language descriptions has gained popularity. The goal of video-to-text retrieval, given a query action segment, is to rank captions in a gallery set so that those with a higher rank are more semantically related to the video action. Text-to-video retrieval, on the other hand, ranks videos based on a query caption.

While most methods [8, 10, 3] use one joint embedding space to align video and text features, recent methods [1, 15] use multiple embedding spaces to match video features into the noun and verb embedding spaces along with the textual features but did not consider their interactions. Moreover, it is relatively easy to parse the textual features into verb and noun levels since an off-the-shelf toolkit could be used. However, mapping a video feature into the object and action levels is still challenging, which corresponds to noun and verb levels in text.

Inspired by [11, 12, 5], we implement self-attentions to exploit visual features on top of the baseline, JPoSE [15] by leveraging the contexts from nouns and verbs of the text query with details in the following section. While we outperform the strong baseline in normalized Discounted Cumulative Gain (nDCG), which is more beneficial for semantic similarity, we fall short in mean Average Precision (mAP), a traditional technique for binary relevance. Our approach is ranked third for nDCG, while it is ranked fourth for mAP. We also analyze various failure examples to save the time of the following researchers on this task.

We follow the baseline work [15]: we create a pair of functions that map videos and texts into a joint embedding space, in which embeddings for matched texts and videos should be close together, and embeddings for mismatched texts and videos should be far apart, given a video and a query text. A suitable embedding space should also ensure that related videos/texts stay close together.

With this motivation, we first parse caption into the noun $t_{i}^{1}$ and verb $t_{i}^{2}$ levels, followed by linear layers. We utilize linear layers to embed corresponding video features $v_{i}^{1}$, $v_{i}^{2}$ and use a self-attention layer to exploit contextualized features. Then, we concatenate textual and visual features to compute the distance between these representations, $\hat{v_{i}}$ and $\hat{t_{i}}$. L1 and L2 refer to triplet losses. The more details of the loss functions are in Eq. 1 and baseline paper [15] and the architecture details are in Fig. 1

In Eq. 1, the first two rows refer to cross-modal losses, and the last two rows indicate within-modal losses. $\theta$ function denotes two fully connected layers. $\delta$ function signifies two linear layers and one self-attention layer. $m$ refers to the constant margin, while $d$ is the distance function. While $i$ refers to the selected video, $j$ and $k$ denote positive and negative samples, respectively.

Eq. LABEL:encoder shows that the visual features $V_{i}$ are fed into the self-attention layer to encode into $z_{s}$. Then, a feed-forward layer FF outputs the final contextualized appearance feature. Normalization of the layer is done under the Norm function.

For multi-headed attention layers, we follow [13], as formulated in Eq. 3. All of the W matrices are learned during the training procedure. Since it is a self-attention layer, query Q is the same as key K and value V. After each attention layer, layer normalization and the residual connection are implemented.

In this report, we propose an approach to exploit contextualized video features via self-attentions and disentangling them into multiple embedding spaces. It also parses text into corresponding embedding spaces, and then the similarity between representations is calculated via triplet loss. While our strategy outperforms the strong baseline in normalized Discounted Cumulative Gain (nDCG), a semantic similarity measurement, it falls short in mean Average Precision (mAP), a standard measure of binary relevance. For nDCG, our proposal is ranked third, and for mAP, it is ranked fourth. We plan to exploit each video feature separately via novel fusion methods as well as utilize domain-specific features such as hand-object relations for future work.