SparrowVQE: Visual Question Explanation for Course Content Understanding

The data collection process involves three key stages: the collection of slide images, transcriptions, and the formulation of questions and answers. Fig. 2 shows the workflow of these stages, which start with the conversion of slides into images, formatting videos to audio, then getting the transcripts, and ending with the generation of question-answer pairs. We carefully examined each data collection stage to provide clarity on the methods and their significance in the context of machine learning analysis¹¹1The slides are the property of the teacher and the university. For the privacy issue, it cannot be tested with closed-source large language models, like ChatGPT..

After rigorous testing of these ten tools, we found that professional platforms vastly outperformed Python-based models. The output from Wav2Vec and similar Python tools was fundamentally flawed, making the creation of coherent sentences unattainable. While professional tools mostly produced accurate transcripts, some errors occurred. These mistakes did not affect the grammar but led to factually wrong sentences, which could change the text’s intended meaning. After detailed analysis and review, Deepgram [35] emerges as the superior option with distinguished and exceptional accuracy.

The initial poor quality of the recordings presented a significant obstacle due to background noise and distance from the speaker to the microphone, which often made the audio difficult to understand. Therefore, despite time constraints, careful manual review and adjusting transcript contents to its original meanings was essential to ensure the training quality. For instance, reviewing a two-hour video could take five times that amount of time or more, underscoring the labor-intensive nature of this phase.

We name our dataset as MLVQE. This section provides a comprehensive statistical overview of our MLVQE dataset, offering insights into its composition and variability that are essential for the proposed method.

Dataset Overview. Tab. III shows the detailed composition of the dataset, highlighting the predominant presence of 9,416 question-answer pairs. These are accompanied by 110,407 words of transcripts that provide detailed textual documentation of the lectures. The dataset also includes 885 images, with each image directly linked to a specific lecture slide and its corresponding transcript.

The Structure of the Dataset. The dataset consists primarily of lecture slides in a visual format. Recorded as images at a resolution of 960 by 540 pixels, these slides encapsulate the most important topic-related information from each lecture. They provide a visual overview and textual reference and convey the lecture’s overarching narrative. The categorization of our textual dataset reflects the variety of questions or tasks it encompasses. According to the ”Category Distribution” detailed in Tab. IV, the dataset has several distinct categories, including closed-ended questions: “closed_qa”, information extraction: “information_extraction”, open-ended questions: “open_qa”, among others. The “closed_qa” category contains the largest number of entries, totaling 2,776, whereas the “creative_writing” category has the smallest, with only 326 entries.

Distribution Data per Week. Tab. V provides the weekly distribution of our dataset. The fluctuation in question-answer pairs, ranging from 1,210 in the third week to a minimum of 135 in the sixteenth week, reflects the dynamic nature of the course curriculum, with a total of 9,416 pairs. Likewise, the steady count of weekly transcripts and images, which sum up to 885, highlights the equilibrium in the presentation of visual and textual materials in the course content. Weeks 12 and 13 are holiday breaks without any courses. Hence, it is a 14-week course.

Distribution of Transcript Words and Question and Answers (QA) Pairs. Fig. 3 illustrates the distribution of word counts in transcripts, highlighting the mean and maximum string lengths. Similarly, Fig. 4 and Fig. 5 present the distributions of word counts in questions and answers, respectively, elucidating the typical and extreme lengths of the text being analyzed. These visual representations help underscore the variability and scope of the dataset’s content, from concise questions to detailed transcripts.

Tab. VI offers a detailed statistical analysis of the word counts for questions, answers, and transcripts. It reveals that, on average, questions contain 7.6 words, answers comprise approximately 15.85 words, and transcripts average 130.07 words. The data also shows the range of content counts within the dataset, with maximum word counts reaching 22 for questions, 98 for answers, and 1,077 for transcripts.

Comparative Lexical Analysis of MLVQA Datasets. The creation of the MLVQE dataset took five students for around six months, which is characterized by an average question length of 7.60 words and an unprecedented average answer length of 15.85 words, showcasing an unprecedented level of lexical richness and complexity. Intriguingly, for inquiries such as ’Can you explain this slide?’, the average answer length escalates to 37.10 words, highlighting the dataset’s capacity for detailed and comprehensive explanations. This stands in stark contrast to the Visual Genome dataset [40], where the average question and answer lengths are merely 5.70 and 1.80 words, respectively. In comparison, datasets like CLEVR [41] and VQAv2 [17] exhibit significantly longer questions, averaging 18.38 and 6.12 words, with answers typically spanning one to three words, thereby accentuating the pronounced distinctions. GQA [42], with its average question length of 10 words, further illustrates the variability in lexical brevity across datasets. However, the significantly verbose nature of our dataset, especially in terms of answer length, not only broadens the spectrum of language usage but also introduces a complex layer to the training of VQA models, necessitating advanced language processing capabilities. This depth and breadth of linguistic expression in our dataset challenge the development of robust models capable of nuanced understanding and generation, thereby significantly advancing the VQA field.

Most of the VQA models are open-ended and cannot answer domain-specific questions; they are mostly fine-tuned on downstream tasks, and in most cases, they do not answer well; it depends on how well they were trained, the data quality, and the structure. Our custom dataset MLVQE has two major aspects, transcripts, and slides, which help answer the question, ”Can you explain the slide?”. With the help of transcripts, the models can predict answers like an instructor of the course. In addition, the slide and question-answer pairs are conversation-based data that help the model generate conversations.

We propose the SparrowVQE, a model innovatively trained in three sequential stages to adeptly integrate multimodal data, specifically slide-text pairs, addressing a significant gap in educational technology. This structured training enhances its capability in visual question answering and educational content synthesis, promising a transformative impact on dynamic learning environments.

SparrowVQE connects the frozen image encoder using a trainable module of two-layer MLP [43, 44] to a casual language model. The overall workflow is shown in Fig. 6. The pre-trained language model is Phi2 [45], which is a 2.7 billion-parameter language model that demonstrates outstanding reasoning and language understanding capabilities, showcasing state-of-the-art performance among base language models with less than 13 billion parameters, and the vision encoder is Sigmoid loss for Language-ImagePre-training (SigLIP) [28]. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. This custom 3B parameter model tailored for specialized educational settings incorporates slide-text pairs from machine learning courses.

Model consists of multimodal pre-training (slide image and transcripts feature alignment), instruction tuning of the pre-trained model with transcripts and QA pairs, and domain fine-tuning of slide, image, and QA pairs.

Machine Learning Concept Alignment Data. We adopted and expanded the training pipeline and datasets from the LLaVA-v1.5 [46] framework. Our methodology worked through three training stages, each designed to enhance our model’s multi-modal capabilities on our MLVQE dataset. During the initial phase, Multimodal Pretraining, we employed a subset of slide and transcript pairs to train a concept alignment adapter. The adapter was developed to create a vector compatibility interface between a frozen pre-trained vision encoder (SigLIP) and a frozen Large Language Model (Phi-2), enabling a symbiotic processing of visual and textual information. In the subsequent phase, Multimodal Instruction Tuning involved the refinement of the adapter with more image and text pairs constituting different classes, giving the model a wider range of understanding between the images and the text pairs. This instruction tuning utilized other benchmark multimodal VQA instruction data alongside our MLVQE formatted datasets to boost the model’s proficiency in complex multimodal instructions. Later, the domain adaptation was carried out from the instruction checkpoint to give the model a more robust understanding between the slide and QA pairs; details about each stage are provided in the consecutive sections.

Stage 1: Multimodal Pretraining. During this initial phase, we leverage the subset of slide-transcript pairs from MLVQE dataset to pre-train the Machine learning concept alignment adapter. This adapter facilitates a trainable weight matrix between the visual and textual representations by minimizing the feature alignment loss. We minimized the Mean Squared Error (MSE) loss in this stage between the projected visual and textual feature vectors to enhance the model’s comprehension of the domain-specific nuances in the machine learning lectures as in Eq. (1):

where $f(\cdot)$ and $g(\cdot)$ are the transformation functions for visual and textual features, respectively, $v_{i}$ and $t_{i}$ are the visual and textual features of the $i^{th}$ sample and $N$ is the total number of samples.

Stage 2: Multimodal Instruction Tuning. In the second stage, the model undergoes further refinement using a comprehensive set of multimodal instruction data, including the LLaVA-Instruct-150K [26], slide transcript pairs, and additional image-text pairs from diverse sources. This stage employs a Cross-Entropy Loss function to optimize the model’s performance on a wide range of VQA tasks, effectively enhancing its understanding of complex instructions and the ability to generate coherent responses, illustrated in Eq. (2):

where $O$ is the total number of observations, $M$ is the number of classes, $y_{o,c}$ is a binary indicator of whether class $c$ is the correct classification for observation $o$, and $p_{o,c}$ is the predicted probability of observation $o$ being of class $c$.

Stage 3: Multimodal Domain Finetuning using PEFT and LoRA

In the final stage, SparrowVQE leverages parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) techniques to refine its predictive capabilities for visual question explanation. PEFT aims to reduce the computational and memory requirements of fine-tuning large language models by introducing a small set of trainable parameters. The original model parameters remain frozen, and fine-tuning is performed on the added parameters. By employing PEFT and LoRA techniques in stage 3, SparrowVQE can leverage the benefits of efficient fine-tuning while maintaining or enhancing the model’s performance on the VQE task.

The model undergoes a specialized fine-tuning process employing Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA). The original model parameters, denoted by $\theta$, are enhanced with an additive update $\Delta_{\phi}(\theta)$ derived from the PEFT parameters ${\phi}$ resulting in the updated model parameters $\theta^{\prime}$. This PEFT function computes updates informed by both the current model state $\theta$ and the newly introduced PEFT parameters ${\phi}$.

LoRA, a variant of PEFT, optimizes the fine-tuning by introducing low-rank matrices B and A, which are smaller in size yet capture the essence of the update needed for the weights W in the model. These matrices are multiplied to produce the update ${BA}$, which is then added to the original weight matrix ${W}$ to produce the updated weight matrix ${W^{\prime}}$. This low-rank update not only makes the fine-tuning process more parameter-efficient but also helps in retaining the original model’s structure while allowing for significant adaptations. Using LoRA is particularly effective as it balances the fine-tuning’s specificity and efficiency, making it a powerful tool for domain-specific adaptations.

Let $\theta$ be the original model parameters and $\phi$ be the added PEFT parameters. The updated model parameters $\theta^{\prime}$ can be computed as:

where $\Delta_{\phi}(\theta)$ is a PEFT function that computes the updates based on $\theta$ and $\phi$. LoRA is a specific implementation of PEFT that introduces low-rank matrices to adapt the original model. It decomposes the weight update into two low-rank matrices, resulting in a more efficient and effective fine-tuning process.

Let $W$ be a weight matrix in the original model. The updated weight matrix $W^{\prime}$ in LoRA can be expressed as:

where $B$ and $A$ are low-rank matrices, and their product $BA$ represents the weight update.

Upon completing stage 3 training, we must integrate the newly trained parameters with the foundational model established in stage 2. This integration will enhance the model’s capability to discern the nuanced differences between slides and question-answer pairs, significantly improving its performance. As a result, the model will achieve precise predictions for the MLVQE dataset, demonstrating a comprehensive understanding of the domain.

This comprehensive training methodology, distributed across three strategic stages, significantly elevates SparrowVQE’s capability to process and interpret multimodal data, ensuring its adept at handling the complexities of machine learning education content. The training strategy, as shown in Tab. VII, goes beyond simple fine-tuning. Typically, models are trained to answer questions based on a dataset, but our goal was to emulate the teaching style of a professor. Our three-stage method successfully enhanced the model’s performance on our MLVQE dataset.

We evaluate the following seven metrics on our developed MLVQE dataset. ROUGE[49] assesses the similarity between predicted and ground truth answers in VQA tasks, focusing on unigrams, bigrams, and longest subsequences. BLEU [50] measures the precision of n-gram overlaps in predicted answers. METEOR [51] evaluates answers for linguistic variations like synonyms and paraphrases, aligning with human judgment. CIDEr [52] uses TF-IDF [53] to gauge the relevance and uniqueness of VQA predictions compared to the ground truth. COSINE [54] score calculates the cosine of the angle between the vectors of the two answers in a high-dimensional space. The higher these metrics, the better the model is.

Implementation details. We train our SparrowVQE using 8 A100 80GB GPU with a three-stage training scheme. Across all three stages of training, the computation time is 5, 9, and 3 hours, respectively. We used the following parameters: a learning rate (LR) scheduler employing cosine decay, an LR warmup ratio set to 0.03, an AdamW optimizer, and a weight decay of 0.0. Tab. VIII shows other parameters. We utilized the first 12 weeks as the training set (8,681 QA pairs) and the remaining two weeks (735 QA pairs) as the test dataset.

Fig. 7 presents the comparison of predicted results of the remarkable questions, “Can you explain this slide?”. The results show the superiority of our model in the capacity of slide information explanation. Fig. 8 shows the comparison of predicted results from different models based on two proposed questions from respective lecture slides. Among the predicted answers of all models, our SparrowVQE model produced the answers closest to the ground truth answers, showing the outstanding performance of our model compared to others. The results on the training set and the test set in Tab. IX and Tab. X show that our SparrowVQE model outperforms all other models, including BLIP [55], Pix2Struct [27], LLaVA [26], and LLM-Blender [56], on all metrics of the answer-predicting task and reference coherence. In Tab. XI, we compare the performance of our model with LLaVA-1.5 [46], LLaVA-Phi [45], MobileVLM [57], MC-LLaVA-3b [58], and TinyGPTV [59] on multiple datasets and compare their performance. It turns out that the SparrowVQE model not only stands out from other models on the question-answering performance but also presents better robustness on different benchmark datasets.