PolySmart @ TRECVid 2024 Medical Video Question Answering

For each video $V_{i}$ in the corpus, we use the YouTube API to extract its transcript, denoted as $T_{i}$. This transcript represents the spoken content within the video.

To enhance the query’s richness and handle medical terminology, we generate an additional embedding based on a GPT-4-enhanced query answer $q_{\text{GPT-4}}$. The prompt for GPT-4 is:

"You act as a medical or a health helper. Given a list of medical or health-related how-to questions, output the instructions step by step."

We utilize a sentence transformer model [9, 10] to encode both the query and the transcripts into vector representations in a semantic embedding space. Given a query $q$ and a video transcript $T_{i}$, we compute the embeddings as follows:

$$\mathbf{q}_{\text{orig}}=\text{SentenceTransformer}(q)$$

$$\mathbf{t}_{i}=\text{SentenceTransformer}(T_{i})$$

$$\mathbf{q}_{\text{GPT-4}}=\text{SentenceTransformer}(q_{\text{GPT-4}})$$

We compute the cosine similarity between the embeddings of each video transcript $\mathbf{t}_{i}$ and both the original query embedding $\mathbf{q}_{\text{orig}}$ and the GPT-4-enhanced query embedding $\mathbf{q}_{\text{GPT-4}}$. The cosine similarity for each query embedding with respect to $T_{i}$ is defined as:

$$\text{Sim}(\mathbf{q}_{\text{orig}},\mathbf{t}_{i})=\frac{\mathbf{q}_{\text{orig}}\cdot\mathbf{t}_{i}}{\|\mathbf{q}_{\text{orig}}\|\|\mathbf{t}_{i}\|}$$

$$\text{Sim}(\mathbf{q}_{\text{GPT-4}},\mathbf{t}_{i})=\frac{\mathbf{q}_{\text{GPT-4}}\cdot\mathbf{t}_{i}}{\|\mathbf{q}_{\text{GPT-4}}\|\|\mathbf{t}_{i}\|}$$

The final similarity score for each video $V_{i}$ is the maximum of the two similarity values:

$$\text{Sim}_{\text{final}}(q,T_{i})=\max\left(\text{Sim}(\mathbf{q}_{\text{orig}},\mathbf{t}_{i}),\text{Sim}(\mathbf{q}_{\text{GPT-4}},\mathbf{t}_{i})\right)$$

Based on the computed similarity scores $\text{Sim}_{\text{final}}(q,T_{i})$ for each video transcript $T_{i}$, we rank the videos and retrieve the top-10 or top-100 videos with the highest scores. These selected videos are then passed to the subsequent localization stage.

We utilize the I3D [11] model, a powerful network for video processing, to encode the visual information from each video segment. I3D operates by inflating 2D convolutional filters into 3D, allowing it to effectively capture both spatial and temporal features. This model generates a feature matrix $V\in\mathbb{R}^{k\times d}$, where $k$ is the number of video frames and $d$ represents the feature dimension. By using I3D, we gain high-quality visual representations that capture both static spatial information and dynamic motion patterns crucial for video answer localization.

For the textual modality, we employ the DeBERT [12] to process the concatenated text, including the query $Q$ and any relevant video captions $T=[Q,T_{1},...,T_{r}]$. This produces a feature vector $T\in\mathbb{R}^{n\times d}$, where $n$ is the length of the concatenated text tokens, representing the textual information in alignment with the video content.

The MCL approach applies Context Query Attention (CQA) [13] to merge visual and textual features effectively. CQA leverages two attention mechanisms: query-to-context and context-to-query, facilitating deeper semantic interaction between the modalities. This fusion step results in enhanced feature representations, enabling better alignment between the video content and the query.

To determine the start and end of the visual answer, MCL utilizes two predictors: a Visual Predictor and a Textual Predictor. The Visual Predictor employs LSTMs followed by feedforward networks to identify key time points in the video, while the Textual Predictor, based on the structure of question-answering networks, predicts relevant time spans from textual features. This dual predictor setup ensures robust answer localization by leveraging insights from both modalities.

A critical component of MCL is the Adaptive Knowledge Transfer Module. To synchronize the predictions of the Visual and Textual Predictors, we introduce a Lookup Table that facilitates cross-modal knowledge alignment. By mapping predicted time spans from one modality to another, the Lookup Table ensures consistent understanding across the predictors.

MCL employs a One-Way Dynamic Loss Adjustment mechanism from previous work [8], dynamically adjusting the knowledge transfer between modalities based on prediction alignment using an Intersection over Union (IoU) criterion. This process stops gradient flow between predictors, enabling them to independently refine their learning while optimizing for overall consistency. The final loss function combines contributions from each predictor and includes additional loss terms from cross-modal transfer. The total loss is defined as:

$$\text{Loss}=\text{Loss}_{\text{Visual}}+\text{Loss}_{\text{Textual}}+\text{Loss}_{\text{Transfer}}^{\text{Visual}}+\text{Loss}_{\text{Transfer}}^{\text{Textual}}$$

In our experiments, the model demonstrates strong coverage in retrieving relevant segments within medical videos, though it sometimes lacks precision in identifying exact answer boundaries (Figure 3).

In our experiments on Query-Focused Instructional Step Captioning (QFISC), we found that using LLaVA NEXT 32B alone did not yield sufficiently structured instructional captions. However, by incorporating GPT-4 to refine and segment the response, we achieved significantly more coherent and detailed step-by-step captions.