\name: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models

Learner and Optimizer. We denote any LLM-based model as $f(x;\phi)$ where $x$ represents the input data, and $\phi$ denotes the natural language instructions for $f$ to follow, which is considered as learnable parameters in our verbalized learning framework. Specifically, $\mathbf{Q}$ contains parameters to be learned in \name. As depicted in Fig. 2, in each iteration $t$, the learner agent $f_{\text{learner}}^{(t)}$ is modeled by the frozen VLM $f_{\text{VLM}}(\cdot)$ used for VAD with a specific prompt template $\theta$ that guide $f_{\text{VLM}}(\cdot)$ to conduct a learning task by pondering on current guiding questions $\mathbf{Q}_{t}$. We denote the learner agent as $f_{\text{learner}}^{(t)}(x)=f_{\text{VLM}}(x;(\theta,\mathbf{Q}_{t}))$, where $x$ is the input in a learning task, and $\mathbf{Q}_{t}$, the learnable guiding questions applied in each iteration $t$, constitutes the core parameters that distinguish the learner between iterations. Meanwhile, we introduce an optimizer $f_{\text{opt}}^{(t)}$ to assess the quality of the predictions of the learner and to optimize $\mathbf{Q}_{t}$. W.l.o.g., we use the same frozen VLM $f_{\rm VLM}$ to model the optimizer. As demonstrated in Fig. 2, we provide another specific prompt template $\psi$ for the learner to follow to optimize $\mathbf{Q}_{t}$, so we denote the optimizer agent as $f_{\text{opt}}^{(t)}(z)=f_{\text{VLM}}(z;(\psi,\mathbf{Q}_{t}))$, where $z$ is its input and $\psi$ is the instruction to improve $\mathbf{Q}_{t}$. It is important to note that $f_{\text{learner}}^{(t)}\neq f_{\text{opt}}^{(t)}$ because $f_{\text{learner}}^{(t)}$ follows $(\theta,\mathbf{Q}_{t})$ to conduct a learning task, while $f_{\text{opt}}^{(t)}$ follows $(\psi,\mathbf{Q}_{t})$ to refine $\mathbf{Q}_{t}$.

Learning Task for $f_{\rm learner}$. The learner executes the “forward pass” and outputs a prediction. Recall that we only use the original coarsely labeled information for training. Thus, we design a binary classification task for $f_{\rm learner}$, which accounts for the temporal nature of video data, the sparsity of anomalies, and the weak supervision in VAD datasets. In this task, the job of the learner $f_{\rm learner}$ is to produce a binary classification prediction $\hat{Y}^{(j)}$ to determine whether there is an anomaly in the video based on the sampled frames $\tilde{V}^{(j)}$. As shown in Fig. 2, we explain the task in natural language in the “Model Description” section in $\theta$. Guiding questions $\mathbf{Q}_{t}$ are inserted in the “Prompt Questions” section in $\theta$ to elicit reasoning of the VLM. This template design is based on the prompt structures used in VML, with targeted modifications to help the learner effectively address this WS learning task. Due to the space limit, please refer to the Appendix for detailed information on $\theta$. Given $\theta$ and a sampled frame set $\tilde{V}^{(j)}$, the learner will output a prediction as

where $\hat{Y}^{(j)}=1$ if the learner thinks there is an anomaly after skimming across the sampled frames $\tilde{V}^{(j)}$ and reasoning through the guiding questions $\mathbf{Q}_{t}$, and otherwise, $\hat{Y}_{i}=0$.

Optimization Step in $f_{\rm opt}$. The optimizer executes the “backward pass” to update the questions $\mathbf{Q}_{t}$ via a mini-batch (batch size is $n$). Suppose the visual input in a batch is $V_{\rm batch}=[\tilde{V}^{(1)}_{\rm batch},\cdots,\tilde{V}^{(n)}_{\rm batch}]$ and the corresponding ground truths are $Y_{\rm batch}=[Y^{(1)}_{\rm batch},\cdots,Y^{(n)}_{\rm batch}]$. The learner generates prediction as $\hat{Y}_{\rm batch}=[\hat{Y}^{(1)}_{\rm batch},\cdots,\hat{Y}^{(n)}_{\rm batch}]$ with the current questions $\mathbf{Q}_{t}$ by Eq. (1). The optimizer will output a new set of questions $\mathbf{Q}_{t+1}$ by following the prompt $\psi$ with batched data. We denote the optimization step as

where $\mathbf{Q}_{t+1}$ is a new set of guiding questions constructed from $f_{\rm opt}^{(t)}$ owing to its text generation and instruction following abilities after reading $\psi$. Due to space constraints, please refer to the Appendix for information about $\psi$. As shown in Algorithm 1 in the Appendix, we will repeat Eq. (1) and Eq. (2) for $P$ iterations to optimize $\mathbf{Q}$. We denote the one with the largest validation accuracy as $\mathbf{Q^{*}}$.

Given $\mathbf{Q}^{*}$, \name yields fine-grained anomaly score $\hat{Y}$ for a test video $V$ via a coarse-to-fine process shown in Fig. 3.

We measure the relevance between different video segments by the cosine similarity of their feature representations [24], extracted by a pretrained vision feature extractor $g$, e.g., ImageBind [11]. For the $u$-th segment $V_{u}$, its similarity with any segment $V_{w}$ ($1\leq w\leq h)$ is ${\rm sim}(u,w)={\rm cos}\left(\frac{e_{u}\cdot e_{w}}{||e_{u}||\cdot||e_{w}||}\right)$, where ${\rm cos}$ denotes the cosine function, and $e_{u}=g(V_{u})$ and $e_{w}=g(V_{w})$ represent their features. Let $\kappa_{u}=[\kappa_{u}^{(1)},\dots,\kappa_{u}^{(K)}]$ denote the indices of the top-$K$ segments similar to $V_{u}$. We refine the anomaly score by

where $\bar{y}_{u}$ is an ensemble of initial scores of top-$K$ video segments relevant to $V_{u}$. Here, the initial score of each retrieved segment is weighted by a factor derived from the cosine similarity and normalized by the Softmax function (with $\tau$ as the temperature hyperparameter). Accordingly, scenes with greater similarity are assigned higher weights, making the ensemble score a more comprehensive reflection of anomalies with the video context. By applying Eq. (4) for all segments, we obtain $\bar{Y}=[\bar{y}_{1},\dots,\bar{y}_{h}]$.

• •

  UCF-Crime dataset is collected from real-world surveillance videos (128-hour long in total), covering crime-related anomalies including abuse, arrest, arson, assault, burglary, explosion, fighting, road accident, robbery, shoplifting, shooting, stealing, and vandalism. The training set has 1610 videos (810 abnormal ones and 800 normal ones), while the test set has 290 videos (140 abnormal ones and 150 normal ones). The total number of test frames is over 1 million (1,111,808), and abnormal frames account for 7.92%. The average duration of a test video is 2.13 minutes, which is relatively long compared to common video datasets and serves as a benchmark.

• •

  XD-Violence is another representative large-scale (217-hour long in total) VAD dataset with 6 anomaly categories, i.e., abuse, car accident, explosion, fighting, riot, and shooting, which defines anomalous events as the ones related to violence. This dataset is collected from movies and YouTube videos. It has 3954 training videos and 800 test videos (500 abnormal ones and 300 normal ones). The total number of test frames is over 2 million (2,335,801), and abnormal frames account for 23.07%. The average duration of a test video is 1.62 minutes.

• •

  WS learning methods [45, 46, 44, 38, 21, 7, 17, 35, 54, 60, 10, 53, 19] usually use task-specific learning models with pretrained weights such as C3D [39], I3D [5], VideoSwin [27], ResNet [15], and ResNext [48] to extract feature for each video segment. Based on that, they form the training of classifiers, which output predictions after the feature extractors, as a multiple instance learning task, regarding the segments containing anomaly scenes as positive bags and the others as negative bags to handle the lack of frame-level annotations and the uncertainty of the anomaly locations in the video. Such learning objectives can fully use the only available video-level label information and effectively improve the discriminative ability of the classifiers in the network. However, the trained neural networks from these methods operate on highly abstract features that are hard for humans to interpret.

• •

  Unsupervised learning methods [37, 40, 36, 41, 14, 28] improve the discriminative ability of the models regarding anomalies and normality without any knowledge of the video label. Note that we include one-class learning [14, 28, 41, 40] methods in this category. Unsupervised methods mostly learn reconstruction models from unlabeled data and use reconstruction errors to distinguish normal and abnormal video frames. Another common strategy [37, 36] is introducing pseudo-labels for unlabeled data and using this information to train discriminative models for VAD. Still, these methods cannot produce explainable results for VAD due to the structure gap.

For explainable approaches, we use LAVAD [55], Holmes-VAD [58], and VADor [29] as representatives of Pipeline 1 and Pipeline 2 shown in Fig. 1. It should be noted that [49] does not report performance on UCF-Crime and XD-Violence. Additionally, we include zero-shot (ZS) VAD by frozen VLMs designed by  [55] as baselines.

We address Q1 by empirically comparing \name to existing VAD methods. First, in Table 2, \name achieves the highest AUC among explainable VAD methods on UCF-Crime, outperforming Holmes-VAD and VADor (without instruction tuning, as reported in their papers) in a fair comparison. Importantly, unlike these methods, \name does not need to modify the model parameters, demonstrating its suitability to directly adapt VLM to the VAD task with minimal training requirements. Moreover, \name surpasses LAVAD by $6\%$ in AUC on UCF-Crime, uniquely integrating both description and reasoning capabilities in VAD. Compared to non-explainable methods, \name achieves AUC performance that is comparable to one of the top-performing methods, CLIP-TSA, on UCF-Crime, while offering the additional advantage of explainable predictions.

Similar advantages are also observed in Table 3 for XD-Violence. Considering multiple factors, including performance, training efficiency, system integration, and explainability, \name stands out as a promising pipeline for VLMs in VAD.

We perform necessary ablation studies on UCF-Crime to answer both Q2 and Q3 for a comprehensive evaluation.

The results are shown in Table 5. If the batch size $n$ is 1 with $S=$ 16, the learned questions cannot be generalized due to the limited video sample in the batch which leads to a suboptimal AUC, and it takes longer to train for \name. Meanwhile, if we set $n$ as large numbers like 4 or 8 (with $S=$ 4 or $S=$ 2), the learned questions are suboptimal too because relatively few sampled frames generally lack the temporality for the optimizer to look into the details and conceive good questions. Thus, setting $n$ to 2 and $S$ to 8 is in default in this paper, which strikes the balance between training efficiency and effectiveness.

First, we apply \name to InternVL2-40B, a larger model in the InternVL2 family compared to InternVL2-8B. As shown in Table 10, InternVL2-40B achieves effective AUC performance, slightly exceeding that of InternVL2-8B, indicating that verbalized learning in \name enables models of various scales to identify a $\mathbf{Q}$ suitable for their reasoning capabilities. Additionally, We also evaluate the transferability of $\mathbf{Q}$ across different scales and and observe an interesting phenomenon: the $\mathbf{Q}$ learned by InternVL2-8B remains effective for InternVL2-40B, but not vice versa. This is likely because the $\mathbf{Q}$ learned by the smaller model is readily interpretable by the larger model, whereas the $\mathbf{Q}$ derived from the larger model is more complex in syntactic structure and does not align well with the reasoning framework of the smaller model. Secondly, we select a different VLM, Qwen2-VL-7B [43], as the backbone for \name. As shown in Table 10, while the AUC achieved with Qwen2-VL-7B is lower than that with InternVL2-8B, the verbalized learning in \name remains effective, allowing it to outperform notable baselines such as LAVAD [55]. However, a notable gap exists when transferring $\mathbf{Q}$ across different model architectures in Table 10. Developing a universal $\mathbf{Q}$ that can effectively elicit reasoning capabilities across various VLM structures would be an promising direction for future research. Lastly, we observe that the transferability of $\mathbf{Q}$ depends on the training dataset. From Table 10, we observe that transferring $\mathbf{Q}$ learned from UCF-Crime to XD-Violence results in a smaller performance drop compared to the reverse case. This suggests the source dataset is crucial to the transferability of $\mathbf{Q}$ across datasets.

To illustrate how \name performs video anomaly detection, we take one video for a qualitative demonstration of the explainability brought by the learned $\mathbf{Q}$, as shown in Fig. 5. Please refer to the Appendix for more qualitative examples if interested. The main anomaly in this video is that a man tries to steal money from the washing machines in a laundromat and is arrested after being found by the police. In Fig. 5, we provide the guiding questions learned by \name and take 6 main video segments (each with 2 sampled frames and their time indices are given in Fig. 6) as examples to demonstrate the explanation produced by \name. From every given answer, the frozen VLM with \name-learned questions is able to explain the scene by closely following the detailed anomaly characterization of the five learned guiding questions. For example, in the second scene, one question in $\mathbf{Q}^{*}$ states that “Are there any people in the video who are not in their typical positions or engaging in activities that are not consistent with their usual behavior”, and it successfully triggers the reasoning abilities from the frozen VLM. The VLM then accurately describes the abnormal event and explains why it is regarded as an anomaly under the cue from the question.

Moreover, owing to the proposed coarse-to-fine detection strategy in testing, the anomaly score dynamics shown in Fig. 6 well represents the actual real-time anomaly level in this video and gradually increases to nearly 1 when the man is being arrested. This result verifies that \name allows VLMs to effectively identify anomalies with a holistic model, reducing the manpower and computational overhead for explainable VAD.

More interestingly, we want to highlight one more advantage of \name. That is, \name allows humans to further interact with VLMs because it retains the general question-answering ability of pretrained VLMs. This is because \name does not require finetuning of the VLM backbone weights. Although finetuning VLMs with parameter-efficient methods like [16, 32, 26] is easy and computationally tractable, instruction-tuned models still inevitably lose the flexibility to handle general questions (due to catastrophic forgetting), as they are trained to respond to certain queries with fixed answer styles. In contrast, as shown in Fig. 7, the learned $\mathbf{Q}^{*}$ can steer reasoning in a frozen VLM while still being able to allow the VLM to answer open-ended (like follow-up or counterfactual) questions, which is an important ability lost in instruction tuning-based models.