Winning the CVPR’2022 AQTC Challenge:
A Two-stage Function-centric Approach

Generally speaking, in order to calculate text similarity, it is important to represents the text as numerical features that can be calculated directly, which could be categorized into two groups: string-based method and corpus-based method.

String-based methods aim to measure similarity between two text strings based on string sequences or character composition, including character-based methods [14, 21] and phrase-based methods [11, 15]. Different from string-based methods, corpus-based methods leverage the textual feature or co-occurrence probability to calculate the text similarity at the corpus level, which are usually achieved in three ways: bag-of-words model like Term Frequency–Inverse Document Frequency (TF-IDF) [20], distributed representation methods like Word2Vec [18] and BERT [10], and matrix factorization methods like Latent Semantic Analysis (LSA) [8] and Latent Dirichlet Allocation (LDA) [5].

VQA is a task to answer questions based on image [4] or video [17], which could be roughly divided into attention based methods [2, 25] and bilinear pooling based approaches [13, 16, 26]. [2] developed different attention modules to adaptively attend on the relevant image regions based on the question representation. [13] proposed to employ the compact bilinear pooling methods to combine the visual and linguistic features. However, these tasks mainly focus on the third-person perspective, while the AQTC task concentrates on the egocentric scenes.

Given an instructional video $V$, the video’s corresponding textual script $S$, the user’s question $Q$, the set of candidate answers $A^{i}_{j}$ where $A^{i}_{j}$ denotes the $j$-th potential answer in the $i$-th step, we target at select one correct answer in each step. Specifically, to grounding question $Q$ in the instructional video $V$ and script $S$, we first segment $V$ and $S$ into function set $\{f_{1},f_{2},\dots,f_{n}\}$ and then matching $Q$ with related functions. Note that each function $f$ consists of function-clip $f^{v}$ and function-para $f^{t}$. Afterwards, taking the weighted function set $\{f_{1},f_{2},\dots,f_{n}\}$, the question $Q$ and the candidate answer $A^{i}_{j}$ as input, we formulate the multi-step QA as a classification task in a supervised way.

We now turn to explain the technical details for the Question2Function Module. Since the instructional videos are used to guide user or AI assistant in a step-by-step manner, we first segment both script and video into individual functions, instead of sentences or frames, to insure the completeness of each step. Meanwhile, it is critical to ground the specific question with the related function as the correct answer often co-occurs with the corresponding function.

Specifically, we first segment the script $S$ into the textual function-paras $f^{t}$ according to the pre-defined schema (see details in our project’s repo), and then divide the corresponding video $V$ into the visual function-clips $f^{v}$ via the aligned script timestamp. In this way, the instructional video and script are divided into the functions set$\{f_{1},f_{2},\dots,f_{n}\}$, and each function $f$ not only contains textual description $f^{t}$ but also includes visual guidance $f^{v}$.

We further match the specific question with the function set based on text similarity. Due to the small volume of the dataset and the corresponding functions are not highly semantic similar with the given question, we calculate the similarity score between $Q$ and $\{f_{1},f_{2},\dots,f_{n}\}$ via TF-IDF model [19] instead of deep learning based methods. The ablation result in Sec.4.2 indicates that the traditional statistical based TF-IDF method behaves much better than the deep learning based approach.

After we determined the related function thanks to the former module, we now turn to formulate the following multi-step QA as a classification task. Since the candidate answers are given as textual action descriptions and visual button images, we need to predict the correct action in each step well as the corresponding button according to the historical steps.

Input Features. For the text embedding, we encode the function-para, question and candidate answers into $E_{f}^{t}$, $E_{q}^{t}$ and $E_{a}^{t}$ via XL-Net [24], which performs much better than the Bert [9] backbone as shown in Tab.1, since the XL-Net is good at processing long context. For the visual part, we encode the frames of function-clip and the button image in candidate answers into $E_{f}^{v}$ and $E_{a}^{v}$ via vision transformer (ViT) [12] following  [23].

Steps Network and Prediction Head. Same as baseline [23], we use GRU [6] to leverage the historical steps and predict the final score for each answer via a two-layer MLP followed by softmax activation.

Loss Function. By taking the embeddings of question $E_{q}^{t}$, candidate answer $E_{a}=\{E_{a}^{t},E_{a}^{v}\}$ and the weighted function set $E_{f}=\{E_{f}^{t},E_{f}^{v}\}$ given by the former module as input, we formulate the following multi-step QA as a classification task as follows.

where $\hat{y_{i}}$ = Pred_head(Steps_network($E_{f}$, $E_{q}^{t}$, $E_{a}$)) and $y_{i}$ represents the ground truth.

Considering the changing views within the same instructional video and the occlusion of buttons, it is really challenging to link the button’s image to its function. Therefore, we tried to use finger detection [1] in the video as additional information in the Function2Answer module (see details in our project’s repo). However, since the strong assumption and the cumulative error introduced by the detection module, the performance of this purely-inference method is poor as shown in Tab.2.

Following  [23], we trained our model on the training set containing 80 instructional videos and evaluated the model performance on the testing set which contains 20 instructional videos. We use the same evaluation metrics in [23], i.e., Recall@k, Mean rank (MR) and Mean reciprocal rank (MRR).

For fair comparison, we train our model as well as ablation experiments under the same parameters setting (Adam optimizer with the learning rate $1\times 10^{-4}$, maximum 100 training epochs and 16 batchsize) and report the best evaluation metrics epoch on the testing set.

In this section, we compare the performance of possible solutions in each module separately.

Specifically, in the Question2Function module, we evaluate the different segment methods (sentence-centric v.s. function-centric) as well as the grounding approaches (cross attention v.s. TF-IDF) as shown in Tab.1. Compared with baseline using raw settings, we observe that the adjustment of the text encoder(BERT $\rightarrow$ XL-Net) and optimizer (SGD $\rightarrow$ Adam) has a significant impact on performance (30.2$R@1$ $\rightarrow$ 39.3$R@1$). Meanwhile, the function-centric segment approach performs better than the sentence-centric on most of the metrics, which demonstrates the significance of the step completeness. For grounding approach, the TF-IDF model performs much better than the cross-attention mechanism (39.3$R@1$ $\rightarrow$ 44.4$R@1$ on sentence-centric and 38.5$R@1$ $\rightarrow$ 45.2$R@1$ on function-centric). We also show an case (see Fig.2) between the two different grounding approaches, which reveals that TF-IDF model can help us find the related functions effectively.

For Function2Answer module, we evaluate the impact of visual and textual part of functions and answers via extensive ablation studies. As we can see, the visual guidance of the function helps the model choose the correct answer significantly (41.0$R@1$ $\rightarrow$ 45.2$R@1$). However, the inclusion of the visual part of answer brings little gains on metrics (43.3$R@1$ $\rightarrow$ 45.2$R@1$), which reveals that the link of button image to its function should be further investigated.