Answering from Sure to Uncertain: Uncertainty-Aware Curriculum Learning for Video Question Answering

Video question answering (VideoQA) is the generalization of visual question answering [28, 29, 30, 31, 32] from image domain to the video domain. It requires temporal reasoning over a sequence of events in videos, and various techniques are exploited, such as the attention mechanism [33, 34], graph neural networks [2, 35, 36], memory networks [5, 4], and hierarchical structures [37, 13]. For example, a dual-LSTM-based approach with both spatial and temporal attention is proposed in [38]. For example, MASN [2] models each object as a graph node and captures the spatial and temporal dependencies of all objects with graph neural networks. HQGA [13] is developed to model the video as a conditional graph hierarchy to align with the multi-granular nature of questions. , which achieves great results on MSVD and MSRVTT [39]. Besides the efforts in improving the model structure, many works develop new frameworks or methodologies for this task [3, 40]. For example, invariant grounding is exploited in [3] for VideoQA, which aims to find the question-critical scenes whose causal relations with answers are invariant. The atemporal probe (ATP) [40] is presented to degrade the video-language task to image-level understanding, which provides a stronger baseline on the performance of image-level understanding in the video-language setting than random frames. Recently, large-scale video-text pretraining has shown great power in promoting the performance of multimodal video understanding [41, 42, 43]. For instance, MERLOT [41], VIOLET [42], and All-in-One [43] attain the state-of-the-art performance on several VideoQA benchmarks. However, they require large-scale data and computational resources for training. In this work, we follow [2, 13, 40, 3] and make comparisons only to the methods without pretraining to show the effectiveness of modeling approaches instead of that of more training data. Despite the progress made in VideoQA, existing works do not consider the impact of the order of training samples or the uncertainty in the data. In this work, we propose a new training framework for VideoQA concerning appropriate difficulty scheduling based on uncertainty.

Curriculum learning (CL) [18] emulates the human learning process by starting with easier tasks and gradually progressing to more challenging ones. Two central components of CL are the difficulty measure and the training scheduler. In the case of self-paced CL (SPL, where difficulty is measured during training), the loss function is often used as the difficulty measure [19, 20, 21, 22]. Initially, during training, samples with higher losses are excluded from optimization. As training advances, the threshold is gradually increased to incorporate more complex data into the optimization process. However, relying solely on loss might not accurately represent the inherent difficulty of data, as difficulty is an intrinsic attribute of samples and should be independent of ground truth labels. To overcome this limitation, we propose employing uncertainty as the difficulty measure for SPL. To the best of our knowledge, [44] is the only work that also uses uncertainty for CL. However, our method is essentially different from it: 1) We derive uncertainty by probabilistic modeling, while it obtains data uncertainty by a pretrained language model (predefined CL); 2) We perform CL by re-weighting the data, while it adopts baby step [18] to arrange data; 3) We focus on VideoQA, while it addresses neural machine translation. The pretrained model and the training scheduler based on baby step make [44] more complex to implement than ours.

In Bayesian modeling, there exist two main types of uncertainty: model (epistemic) uncertainty and data (aleatoric) uncertainty [45, 46]. Specifically, model uncertainty accounts for uncertainty in the model parameters and comes from our ignorance about which model generated the data. This type of uncertainty can be reduced by giving more training data. As for data uncertainty, it captures the inherent noise in our observations such as blur in images and videos. This type of uncertainty is an inherent characteristic of data and cannot be alleviated with more collected data. A great number of methods exploit data uncertainty and achieve considerable improvements in various tasks [47, 48, 49, 50]. For example, DeNet [51] is proposed to resolve query uncertainty and label uncertainty in temporal grounding, where a decoupling module and a de-bias mechanism are designed for the probabilistic language encoding and diverse temporal regression. UGPT [52] is presented for complex action recognition, where the attention scores in Transformer are modeled as probabilistic variables to capture the complex and long-term interaction of actions. Besides, uncertainty has also been applied to CL [53, 44]. For instance, in the case of [44], they incorporated data and model uncertainty into the CL process for neural machine translation, focusing on pre-computed data uncertainty to facilitate a baby-step-based CL approach. However, this approach introduces complexity due to the need for prior data uncertainty computation. In contrast, our method stands out by dynamically learning uncertainty during training, which is then utilized to adjust sample weights, leading to a more straightforward and practical implementation. Moreover, the work presented in [53] utilized snippet-level uncertainty to assign varying weights to different snippets in the context of weakly-supervised temporal action localization. This was accomplished through the lens of evidential deep learning [54, 55], with a focus on addressing intra-action variation. Our approach, however, distinguishes itself by introducing probabilistic modeling for uncertainty quantification, with the primary objective of enhancing the generalization capacity of VideoQA models. More importantly, our framework is designed to be a versatile tool applicable to diverse VideoQA models and to improve their performance.

A remaining challenge for our UCL is the uncertainty quantification. To tackle this challenge, we propose probabilistic modeling for VideoQA. Specifically, we consider VideoQA as a stochastic computation graph, where the input nodes are the video $V$ and the question $Q$. Following video encoding (parameterized by $\phi$) and question encoding (parameterized by $\psi$), we obtain the stochastic video representation $M$ and the stochastic question representation $N$. The distribution of the answer $Y$ can be deduced by variational inference:

Since the integral over $(m,n)$ is intractable, we sample $m$ and $n$ from $q_{\phi}(m|V)$ and $q_{\psi}(n|Q)$, respectively, for $K$ times to approximate the predictive distribution of $Y$, i.e.,

where $m_{k}\sim q_{\phi}(m|V)$ and $n_{k}\sim q_{\psi}(n|Q)$. In practice, we use the reparameterization trick [56] to make the sampling differentiable. $p_{\theta}(y|m_{k},n_{k})$ can be specified for classification and regression as follows,

where $g_{k}=g(m_{k},n_{k};\theta)\in\mathbb{R}^{C}$ is the predicted logits ($C$ is the number of classes), and $\mu_{k}=\mu(m_{k},n_{k};\theta)$ and $\sigma_{k}^{2}=\sigma^{2}(m_{k},n_{k};\theta)$ are the predicted expectation and variance.

The proposed stochastic computation graph for VideoQA is optimized by maximizing the evidence lower bound (ELBO) that is derived as follows¹¹1The derivation can be found in Appendix.,

where $D_{\mathrm{KL}}\left(\cdot||\cdot\right)$ is the Kullback–Leibler (KL) divergence. The KL divergence can also be regarded as regularization to $M$ and $N$, preventing them from degradating to deterministic representations. Since $p(V)$ and $p(Q)$ are irrelevant to optimization, once we apply normal Gaussian prior to $M$ and $N$ ($q_{\phi}(m|V)=\mathcal{N}(\mu_{m},\sigma^{2}_{m}),q_{\psi}(n|Q)=\mathcal{N}(\mu_{n},\sigma^{2}_{n})$), the objective can be re-written and approximated as follows,

where $\alpha$ is a hyper-parameter. For classification and regression, $\log p_{\theta}(y|m_{k},n_{k})$ is specified as

where $p^{k}_{c^{*}}$ is the predictive probability for the correct class $c^{*}$, and $y$ is the ground truth value.

We then define two types of uncertainty for difficulty quantification based on our probabilistic modeling: feature uncertainty and predictive uncertainty. Feature uncertainty measures inherent uncertainty in data such as noise, blur and occlusion in videos. This type of uncertainty is computed based on the variance of $p_{\theta}(y|m_{k},n_{k})$ across different sampling results. Concretely, the feature uncertainty for classification is defined as the variance of predicted logits, while the feature uncertainty for regression is computed as the variance of predicted expectations, i.e.,

where $\bar{g}=\frac{1}{K}\sum_{k=1}^{K}g_{k}\in\mathbb{R}^{C}$ and $\bar{\mu}=\frac{1}{K}\sum_{k=1}^{K}\mu_{k}\in\mathbb{R}$ are the average of predicted logits and that of predicted expectations, respectively, and the power in Eq. 9 is performed element-wisely. Feature uncertainty essentially measures the difference among the predictions from different sampled features. That is to say, if the variances of feature $m$ and $n$ are zeros (i.e., there is no uncertainty in them), the computed feature uncertainty would be zero because the random sampling degrades to a deterministic process. As for the predictive uncertainty, it measures the confidence of the model in the final outputs. Concretely, this type of uncertainty for classification is defined as the entropy of the output distribution, while for regression, we use the predicted variance as the predictive uncertainty, i.e.,

where $p_{c}=\frac{1}{K}\sum_{k=1}^{K}p^{k}_{c}$ is the predicted probability of class $c$. A lower predictive uncertainty means the model has more confidence in its prediction. The proposed feature uncertainty and predictive uncertainty depict the inherent characteristics of data and are agnostic to ground truth, which are more reasonable for the difficulty measurement in SPL.

Distinguishing itself from aleatoric and epistemic uncertainty, our uncertainty serves a distinct purpose: our feature uncertainty assesses the uncertainty linked to high-level video features, while predictive uncertainty conveys the model’s confidence in its predictions. In contrast, aleatoric uncertainty quantifies uncertainty within observations, while epistemic uncertainty pertains to uncertainty in model parameters. Importantly, it’s worth noting that aleatoric and epistemic uncertainty can also be incorporated into our UCL framework. We substantiate this further by presenting a comparison of various uncertainties in the supplementary.

Despite that the above two types of uncertainty can measure the difficulty of data, they cannot be directly applied to SPL because their ranges are intractable, which makes it difficult to determine suitable schedulers for SPL (similar to the design of $\lambda(e)$ in the original SPL). To address this issue, we assume they are Gaussian distributed and normalize them to the standard Gaussian distribution, i.e.,

where $U$ represents $U_{F}$ or $U_{P}$, and $\mathrm{E}[U]$ and $\mathrm{Var}[U]$ are the mean and the variance of the uncertainty $U$, respectively. Consequently, $\bar{U}_{F},\bar{U}_{P}\sim\mathcal{N}(0,1)$ and they are statistically bounded. Since the mean and variance for the whole dataset are intractable, we use batch normalization [57] as a practical means to estimate the normalized uncertainty. In this work, we exploit the two types of normalized uncertainty for SPL. To maintain simplicity, we still use $U$ to denote the normalized $U_{F}$ or $U_{P}$. Fig. 1 demonstrates the overview of our framework.

Our uncertainty-aware curriculum learning framework is designed to be versatile and adaptable, fitting seamlessly into various model structures. Alg. 1 outlines the pseudo code for our uncertainty-aware curriculum learning applied to VideoQA. It’s worth noting that the algorithm encompasses both classification-based (CB) VideoQA and regression-based (RB) VideoQA. For CB VideoQA, retain the blue lines (the tops of Line 11, 13, 14, 16) and disregard the red ones (the bottoms of Line 11, 13, 14, 16). Conversely, for RB VideoQA, follow the opposite steps. Furthermore, we have chosen predictive uncertainty for CB VideoQA and feature uncertainty for RB VideoQA in the examples provided. It’s important to understand that these uncertainties are interchangeable. The KL divergence regularization for the feature distributions within the loss is omitted here for the sake of simplicity.

The proposed framework is agnostic to the essential modules in VideoQA models (i.e., video encoder, question encoder, video-question interaction module, and answer decoder). Therefore, it can be applied to existing VideoQA methods as a plug-and-play method. In this section, we show how a VideoQA model can be adapted into our framework. Specifically, we choose MASN [2] for our purpose because 1) it is a typical VideoQA model with clear VideoQA modules; 2) it uses various types of visual features that may contain more uncertainty in the hidden space.

MASN exploits four types of visual features, i.e., the global/local appearance/motion feature. In this paper, we extract these features as described in [2]. Concretely, for a video of $T$ frames, we can obtain the global appearance features $\left\{\bm{g}_{a}^{t}\right\}_{t=1}^{T}$, global motion features $\left\{\bm{g}_{m}^{t}\right\}_{t=1}^{T}$, local appearance features $\left\{\bm{l}_{a}^{it}\right\}_{i=1,t=1}^{N,T}$, and local motion features $\left\{\bm{l}_{m}^{it}\right\}_{i=1,t=1}^{N,T}$, where $\bm{g}_{a}^{t}\in\mathbb{R}^{1024}$, $\bm{g}_{m}^{t},\bm{l}_{a}^{it},\bm{l}_{m}^{it}\in\mathbb{R}^{2048}$ and $N$ is the number of objects in each frame.

There are two parallel streams in the video encoder of MASN for appearance and motion feature encoding, respectively. Since the two streams are identical in structure, we elaborate on them without specifying appearance or motion. The visual encoder is essentially a graph convolution network (GCN) [58] where the object features are modeled as a graph. Specifically, the graph node $\bm{n}^{it}\in\mathbb{R}^{d}$ ($d$ is the dimension of the node representations) is computed as follows,

where $\bm{b}^{it}\in\mathbb{R}^{4}$ is the coordinate of the bounding box of the $i$-th object in the $t$-th frame, $\bm{s}(t)$ is the sinusoidal function for time-step encoding [59], $[\cdot]$ represents concatenation, and $\bm{W}_{l},\bm{W}_{g},\bm{W}_{n}$ are parameters. For simplicity, we flatten the two indexes of $\bm{n}^{it}$ and denote them as $\left\{\bm{n}^{z}\right\}_{z=1}^{Z}$, where $Z=NT$ is the number of all objects.

We combine all the nodes $\left\{\bm{n}^{z}\right\}_{z=1}^{Z}$ to form a object matrix $\bm{N}\in\mathbb{R}^{Z\times d}$. The adjacent matrix for the graph is computed based on the dot-product of the projected nodes, i.e,

where $\bm{W}_{1},\bm{W}_{2}$ are parameters. The visual features are then encoded by a GCN, i.e., $\bm{M}=\mathrm{GCN}(\bm{N},\bm{A})$, where $\bm{M}\in\mathbb{R}^{Z\times d}$ contains interaction information among objects.

In this work, we encode the video to a probabilistic representation, which is defined as the combination of all encoded objects $\mathcal{R}=\left\{\bm{r}^{z}\right\}_{z=1}^{Z}$. $\bm{r}^{z}$ is the encoded $z$-th object in the video, which is modeled as a multivariate Gaussian distribution, i.e., $\bm{r}^{z}\sim\mathcal{N}(\bm{\mu}^{z},\bm{\sigma}^{z}\bm{I}_{d})$, where $\bm{\mu}^{z},\bm{\sigma}^{z}\in\mathbb{R}^{d}$ is the expectation and the variance, and $\bm{I}_{d}$ is the $d$-order identity matrix (we assume the dimensions in $\bm{r}^{z}$ are independent). The expectation and variance are obtained from $\bm{M}$ with a two-layer perceptron (MLP) as follows,

where $\bm{M}^{z}\in\mathbb{R}^{d}$ is the $z$-th row in $\bm{M}$. In practice, we use the reparameterization trick to sample $K$ sets of visual representations $\left\{\mathcal{R}_{k}\right\}_{k=1}^{K}$, where $\mathcal{R}_{k}=\left\{\bm{r}_{k}^{z}\right\}_{z=1}^{Z}$ and $\bm{r}_{k}^{z}=\bm{\mu}^{z}+\bm{\sigma}^{z}\odot\bm{\epsilon}_{k}^{z}$ ($\bm{\epsilon}_{k}^{z}\sim\mathcal{N}(\bm{0},\bm{I}_{d})$, and $\odot$ represents element-wise multiplication). Note that the process above is parallel for both appearance and motion encoding, and thus two types of visual representations are obtained. $\left\{\mathcal{R}_{k}\right\}_{k=1}^{K}$ is then exploited for video-question interaction.

The question encoder comprises word embedding and LSTM [60]. Specifically, the words are transformed into 300D vectors using GloVe [61]. Subsequently, a linear projection and LSTM are utilized to process the word sequence. To summarize, the LSTM encodes each question into a sequence of hidden states. We do not apply probabilistic modeling to question encoding as our prior experiments show it brings no improvement for MASN.

After obtaining the appearance/motion representation and the question encoding, appearance/motion-question interaction is applied, by which the appearance/motion-question interacted representation is obtained as follows. For the sake of simplicity, we will not explicitly denote the sampling index $k$ since the processing is the same across the $K$ sets of visual features. Upon obtaining $\mathcal{R}=\left\{\bm{r}^{z}\right\}_{z=1}^{Z}$ (applicable to either appearance or motion features) and question encoding $\left\{\bm{w}^{j}\right\}_{j=1}^{J}$ ($\bm{w}^{j}\in\mathbb{R}^{d}$ and $J$ represents the number of words in the question), we combine each of these sets into a matrix. These matrices are represented as $\bm{R}\in\mathbb{R}^{Z\times d}$ and $\bm{W}\in\mathbb{R}^{J\times d}$, respectively. The video-question interaction is performed using the bilinear attention network (BAN) [14] as follows,

where $\bm{B}_{0}=\bm{W}$, $\bm{1}=[1,1,\cdots,1]^{\mathrm{T}}\in\mathbb{R}^{J}$, and $\bm{C}_{i}$ is the attention map. Four blocks of BAN are applied, and the final result is the video-question interacted representation, which is denoted as $\bm{Q}\in\mathbb{R}^{J\times d}$.

The above stream is applied to appearance features and motion features in parallel. Then two types of video-question representation, $\bm{Q}_{a}$ for appearance and $\bm{Q}_{m}$ for motion, are obtained. $\bm{Q}_{a}$ and $\bm{Q}_{m}$ are fused with the motion-appearance-centered attention. Specifically, three types of attention are computed as follows,

where $\bm{U}=[\bm{Q}_{a};\bm{Q}_{m}]\in\mathbb{R}^{2J\times d}$, and the function $\mathrm{Attn}(\cdot,\cdot,\cdot)$ refers to scaled dot-product attention [59], taking as inputs query, key, and value. Then, a residual connection [62] and a LayerNorm [63] are further applied as follows,

where $x\in\{a,m,mix\}$, and $\bm{Z}_{a}/\bm{Z}_{m}/\bm{Z}_{mix}\in\mathbb{R}^{2J\times d}$ is the appearance-centered/motion-centered/mix attention. We then fused the results with the guidance of questions as follows,

where $\bm{z}_{x}\in\mathbb{R}^{d}$ is the sum of $\bm{Z}_{x}$ along the first dimension. Finally, the video-question feature is computed by aggregating $\bm{O}\in\mathbb{R}^{2J\times d}$ along the first dimension as follows,

where $\bm{O}^{j}\in\mathbb{R}^{d}$ is the $j$-th row of $\bm{O}$, and $\mathrm{MLP}(\cdot)$ projects $\bm{O}^{j}$ to a scalar. $\bm{s}\in\mathbb{R}^{d}$ is the fused feature for answer prediction. Given that we have $K$ sets of appearance/motion representations, we compute $K$ features $\left\{\bm{s}_{k}\right\}_{k=1}^{K}$.

The answer decoder varies according to the task at hand. For the regression task, the answer is predicted by projecting $\bm{s}$ to two scalars (predicted expectation and variance). For the classification task, the answer distribution is computed by projecting $\bm{s}$ to logits and applying Softmax to the logits. For the multi-choice task, we first concatenate the question and each answer, and then model VideoQA as binary classification. The prediction is the answer with the highest estimated probability of being correct. Importantly, $\left\{\bm{s}_{k}\right\}_{k=1}^{K}$ yields $K$ predictions, and the ultimate outcome is their average. The optimization follows our uncertainty-aware CL.

In the probabilistic modeling, the visual representations are sampled five times during training and ten times during testing. As for the CL scheduler $\lambda(e)$, we employ a linear²²2Concave/convex functions yield similar results to the linear one after tuning hyperparameters. function with increasing values over epochs, ranging from 3 to 7. The model is trained with a learning rate of $1\times 10^{-4}$ and a batch size of 32, utilizing the Adam optimizer [64]. All experiments are conducted using PyTorch with NVIDIA A100 GPUs.

We evaluate our methods on four datasets: TGIF-QA [38], NExT-QA [65], MSVD-QA [39], and MSRVTT-QA [39]. Specifically, TGIF-QA consists of four sub-tasks: Count, FrameQA, Action, and Transition. For instance, Count is formulated as regression, FrameQA involves open-ended tasks (multi-class classification), while Action and Transition are multi-choice tasks (binary classification for each option). NExT-QA is a multi-choice dataset encompassing description, chronology, and causality. MSVD-QA and MSRVTT-QA are open-ended VideoQA datasets. Regarding evaluation metrics, we employ mean square error (MSE) for Count and accuracy for the other sub-tasks and datasets.

In this section, we compare our model with the existing methods. As we discussed in Related Work, we follow HQGA [13], IGV [3], and ATP [40], and compare only with the methods without large-scale video-text pretraining for fair comparisons. We carefully selected the compared methods based on specific criteria: 1) the state of the art in recent years such as MASN [2], IGV [3], and HQGA [13]; 2) the methods that report results on the corresponding datasets in the original paper. In summary, we compare the best and result-available methods on each dataset.