Overlooked Video Classification in Weakly Supervised Video Anomaly Detection

The transformer first appeared in 2017 in a paper entitled “Attention Is All You Need” [24]. It is a very successful Natural Language Processing (NLP) model and has become one of the break-through innovations in recent years. Since then, the transformer has been extended to almost every field of machine learning, including image classification [5], object detection [1], and video understanding [8] and many more. The transformer pay attention to every element of a sequence of input data in self-attention and extracts the traces of the entire data set.

Following the success of the transformer - a one directional model, the Bi-directional Encoder Representations from Transformers (BERT) [4] is a bidirectional self-attention model, which has also been a big success in many downstream NLP tasks. The bidirectional property enables BERT to fuse the contextual information from both directions. Moreover, BERT introduces challenging unsupervised pre-training tasks which leads to useful representations for many tasks. BERT is introduced in [9] for video action recognition and achieves SOTA performance on two major action recognition datasets, UCF-101 [20] and JHMDB-51 [12]. We are inspired by the BERT, particularly its application on action recognition [9].

In [9], the BERT is used as a late pooling function to replace the previous widely used global average pooling (GAP). The input to the BERT is the internal CNN feature map usually taken before the GAP and FC layers. To preserve the positional information, a learned positional encoding is added to the extracted features. In order to perform classification, additional classification token $x_{cls}$ is appended as in [4]. The classification is implemented with the corresponding classification vector $y_{cls}$, which is sent to FC layers for classification prediction.

The general single head self-attention model of BERT is formulated as:

where $x_{i}$ values are the input input vector including positional encoding; $i$ indicates the index of the target output temporal position; $j$ denotes all possible combinations; and $N(x)$ is a normalization term. The function $g()$ is the linear projection inside the self-attention mechanism of BERT, whereas function $f()$ denotes the similarity between $x_{i}$ and $x_{j}$ as $f(x_{i},x_{j})=softmax_{j}(\theta(x_{i})^{T}\phi(x_{j}))$, where the functions $\theta(\textperiodcentered)$ and $\phi(\textperiodcentered)$ are linear projections. The learnable functions $g(\textperiodcentered)$, $\theta(\textperiodcentered)$ and $\phi(\textperiodcentered)$ try to project the feature embedding vectors to a better space where the attention mechanism works more efficiently. The outputs of $g(\textperiodcentered)$, $\theta(\textperiodcentered)$ and $\phi(\textperiodcentered)$ functions are usually called value, query and key [24] . $PFFN$ is position-wise feed-forward network (PFFN) applied to all positions separately and identically as $PFFN(x)=W_{2}GELU(W_{1}x+b_{1})+b_{2}$, where $GELU(\textperiodcentered)$ is the Gaussian Error Linear Unit (GELU) activation function [24]. The classification vector $y_{cls}$ takes similar form as for $y_{i}$,

In our work, since our focus is to study the impact of the video classification, our main goal is to use the learned classification embedding $y_{cls}$ which aggregates the temporal features of a video into a single feature for video classification. The input vector $x_{i}$’s are the feature vectors of the segments. The learned better subspace for feature representation ${y_{i}}$’s are a plus if they work better than the original features $x_{i}$’s in anomaly MIL. However, we have a strong motivation why we do not have to use it for application purpose - to have a solution in inference mode that has low complexity as the original MIL framework [21].

Shown in Figure 1 are the block diagram of our proposed anomaly detection with explicit video classification. We call this proposed solution MIL-BERT.

Figure 1(a) is the training pipeline. First, given a video, frames are extracted into snippets of 16 frames. A pretrained 3D CNN backbone network is used to extract CNN feature. In the diagram we demonstrate a I3D network [3], but other networks, C3D [23] and newer X3D [6], or MoViNet [11] can also be used. This backbone network keeps frozen and does not participate our training. The output snippet features are denotes as $f_{i},j=1,2,...,N$, where N is the number of snippets of the video. These features are divided equally into 32 segments $x_{i},i=1,2,...,32$. This function is defined as,

where $seg()$ stands for the segmentation on snippets. Its inverse function is denoted as $seg^{-1}()$.

The segmented features $x_{i}$ are sent to the BERT as input temporal features. The output features $y_{i}$ in a different sub-space and the classification feature $y_{cls}$ are output after all the bidirectional attention mechanisms in BERT. The standard MIL anomaly detection framework can take the feature $x_{i}$ or $y_{i}$ as input. We propose to send the video classification $\hat{y}$ out of the BERT to the MIL as an input. This similar idea is also used in [14].

We keep using the MIL ranking loss function with the smoothness and sparsity term as in [21]. We add the video classification binary cross entropy (BCE) loss onto it. So the overall loss function is defined as,

where the subscript $a$ and ${n}$ denotes anomalous and normal video, $v$ is an input feature instance, which can be $x_{i}$, ${y_{i}}$, $f_{i}$ in our work. $B_{a}$ and $B_{n}$ are the bags of segments in the abnormal and normal video, $s(.)$ is the predicted anomaly scoring function in range of 0 and 1. The function $max$ is taken over all instances in a bag. It is expected that in the positive bag, the highest-scored instance is a true abnormal segment. The highest-scored instance in the negative bag is the one most similar to the positive bag, but is actually a negative instance. This makes the negative instance a hard one and therefore benefits the discriminability in the model training. In the BCE loss part, we only keep part of the standard form since $y_{a}=1,y_{n}=0$, $p$ is the scoring function of the video classifier. The smoothness and sparsity terms are still used, even though they are not shown in this equation.

In Figure 1(a) we have the $y_{i}$’s on a dashed line as an optional input to the MIL block. We will study how the this new feature work in anomaly detection.

In the training of BERT and the downstream MIL block, the input features are always segmented into 32 segments. This is required by both the standard implementation of the BERT and the MIL block. However, this is not required in the testing or inference mode.

In Figure 1(b) a testing pipeline is shown, where the video classification score $p(\hat{y}_{cls})$ is combine with the MIL snippet scores $s(v_{i})$. So the final snippet anomaly score is,

This is called a score correction method in [14]. We conceive this idea before we find the work in [14]. However, [14] uses the feature after the transformer, while in our work, we use the original feature in the MIL block.

In Figure 1(b) we use the original feature $f_{i}$ as input to the MIL block in an online mode, where features can go into the MIL block for processing as they become available. We can also use $x_{i}$ in an offline mode, where all the video features are divided into 32 segments before they go into the MIL block. This online mode is used in many previous work’s implementation. However, the BERT block requires to have 32 segment features, so Figure 1(b) can only actually work in an offline mode.

In Figure 1(c) we show a simplified testing mode, where the video classification score is not used. In this mode, the BERT is used in the training process but is not used at all at testing time. This make this model very attractive since it has very low complexity.

The proposed method can work alone, or combine with other anomaly detection methods. We use the RTFM [22] as an example. In this case, the MIL ranking loss is replaced with the feature magnitude based ranking loss of RTFM, and our BCE loss is balanced with RTFM’s BCE loss. So the total loss function is defined as,

where the BERT BCE loss is defined in last line of Equation (4), and the RTFM BCE loss and ranking loass can be found in [22]. This proposed solution is called RTFM-BERT.

In RTFM, the BCE loss is defined on the top-k snippets whose feature magnitudes are the largest in a video. They call this classifier a snippet classifier. Since these top-k snippets are selected per video, and their anomaly scores are averaged to represent the video, this score actually represent the whole video class. After doing some analysis we find that this is one of the key contributions of RTFM. When we try to remove this BCE loss, or replace with a snippet score ranking loss as in [21], the performance of RTFM becomes a lot worse. So the authors of RTFM may not realize that they have put the power of video classification implicitly in their solution.

From Equation (5) we can see that the video classification score ($p(\hat{y}_{cls})$) helps the snippet prediction score $s(v_{i})$. For an anomalous snippet, the video classification score does not have effect since if it is not used, then the snippet score is simply $s(v_{i})$. For a normal snippet, when the video classification score is small (near 0), then the snippet prediction score is suppressed further smaller. This helps to reduce the chance that a normal snippet is mistakenly classified as an anomalous one.

In addition, even if the video classification score is not used in Equation (5), the video classification also helps the anomaly detection implicitly. In Equation (4), the video classification is on the BERT output classification vector $\hat{y}_{cls}$, which is a function of the input features $v_{i}$. The video classification is correct if the MIL selects the correct max-scored instance in the anomalous and normal bags. So this explicit video classification helps the MIL to select the correct instances.

We use three anomaly detection datasets: UCF-Crime [21], ShanghaiTech [16], and newly released XDViolence [29]. We do most of our ablation study on UCF-Crime.

UCF-Crime [21] is a large-scale anomaly detection dataset that contains 1900 untrimmed videos with a total duration of 128 hours from real-world street and indoor surveillance cameras. UCF-Crime consists of complicated and diverse backgrounds. Both training and testing sets contain the same number of normal and abnormal videos. The data set covers 13 classes of anomalies in 1,610 training videos with video-level labels and 290 test videos with frame-level labels.

ShanghaiTech [16] is a medium-scale dataset from a fixed street video surveillance camera. It has 13 different background scenes and 437 videos, including 307 normal videos and 130 anomaly videos. The original dataset is a popular benchmark for the anomaly detection task that assumes the availability of normal training data. Zhong et al. [30] reorganised the data set by selecting a subset of anomalous testing videos into training data to build a weakly supervised training set, so that both training and testing sets cover all 13 background scenes.

XD-Violence [29] is a recently proposed large-scale multiscene anomaly detection data set, collected from real life movies, online videos, sport streaming, surveillance cameras and CCTVs. The total duration of this data set is over 217 hours, containing 4754 untrimmed videos with video-level labels in the training set and frame-level labels in the testing set. It is currently the largest publicly available video anomaly detection data set.

We follow previous work [21], [22], [14] to use the frame-level area under the ROC curve (AUC) as the evaluation metrics on the UCF-Crime and the Shanghai Tech datasets. Following [29], we use average precision (AP) as the evaluation metric on the the XD-Violence dataset.

Please note that there are two ways to evaluate the AUC or AP performance on the testing dataset. Since the video features are first in snippet, then divided to 32 segments, there are also two ways to do the testing, where the snippet or segment features are used as input to the MIL block, and their performance differs. We use the better of these two to benchmark previous and our work, unless specified otherwise. In the first way where snippet feature is used, the MIL score is mapped back to frame simply by repeating the score 16 times. In the second way where the segment feature is used, we first do an inverse segmentation $seg_{-1}$ as in Equation (3) where each segment’s score is mapped back to the original snippets, then every snippet’s score is repeated 16 times.

We implement the BERT and MIL in PyTorch, the BERT code is borrowed from [9]. The default BERT uses 2 layers and 8 attention heads. An initial learning rate of 1E-4 is used, and the training runs 100 epochs. After that some manual fine tune may be used. Two dataset iterators, one for the abnormal data and the other for the normal data, are used. This way, the pairing of abnormal and normal data is random, even when the numbers of abnormal and normal samples are different. We use the Adam optimizer.

For the video snippet feature, we use two different I3D network. In [22], the I3D with Resnet50 [25], whose feature dimension is 2048, is used. In [29] and [19], the I3D with Resnet18 [2], whose feature dimension is 1024 is used. We use two sets of UCF-Crime and ShanghaiTech pre-generated features we find online, one used in [19] and the other used in in [22]. The first one has both RGB and optical flow (simply called Flow hereafter) without multiple-crop augmentation, the other one has RGB only with 10-crop augmentation. On the UCF-Crime, we call the first feature set UCF-Crime, and the second on UCF-Crime-RTFM. For the XD-Violence dataset, we use the 5-crop augmented RGB feature generated by the author of the dataset [29].

For the evaluation of the combination of our BERT video classification on top of RTFM, we use the code base of RTFM and add BERT into it. As an ablation study, we add one FC layer at the input of the snippet feature, when the feature dimension is 2048. With this change the number of total model parameter is reduced significantly while the impact on performance is negligible.

Since we use two sets of pre-generated feature sets on UCF-Crime and ShanghaiTech, we first want to confirm the consistency of these two feature sets. In addition to different modality and crop, the first feature set is L2 normalized, while the second feature set is not.

In this experiment we test the standard MIL, RGB only, and 1-crop only. The feature set [19] is already L2-normalized and divided into 32 segments. For the RTFM [22] feature set, we test both L2-norm and non-L2-norm. In RTFM design, L2 is not used, while L2 normalization is suggested in original MIL work [21]. We use the 32-segments in the testing mode. The results are listed in Table 1. We see that the results are all very close to each other. On the RTFM feature set, non- L2-norm performs slightly better. So in our following comparison, we always use the best result of every method. From this prestudy, we see that these two different feature sets give about same performance.

In this study we use the UCF-Crime feature set pre-generated in [19]. We use 32 segments in the testing mode. These features do not use multiple cropping, and are L2-normalized.

We first check the video classification only to look at how accurate it is. For this purpose all MIL functions are turned off. The accuracy results for the RGB, Flow, and RGB+Flow are 83.45%,85.52%, 90.00%, respectively.

After that we turn the MIl functions back on, and the MIL-BERT is trained end to end. For ablation study purpose, we test a two-step training: first train the video classifier, then freeze BERT and train MIL only. The performance is same as the end to end training. The experiment results are listed in Table 2. Note that in this table, AUC uses the segment score defined in Equation (5), while AUC-2 do not use the video classification score $p(\hat{y}_{cls}$) in Equation (5). The AUC and AUC-2 represent the testing mode in Figure 1(b) and 1(c), respectively. We find that the RGB is the weakest modality in UCF-Crime, the Flow is better, and RGB+Flow gives the best performance. We observe that our best AUC with RGB+Flow modality is better than the SOTA result.

We repeat the test on ShanghaiTech. Again we use the feature set pre-generated in [19]. We use 32 segments in the testing mode, one-crop, and L2-normalized features. The experiment results are listed in Table 2. The trend is very similar as on UCF-Crime. The RGB+Flow gives the best AUC of 97.54%, exceeding the SOTA result already.

We choose to test RTFM-BERT on UCF-Crime [21] and XD-Violence [29]. We add the BERT video classification into the RTFM [22]. As one ablation study, we add a FC layer at the input of RTFM to reduce the feature dimension from 2048 to 1024 on the UCF-Crime. The dimension of XD-Violence is 1024 already.

As in RTFM [22], we only use RGB modality, 10-crop, non-L2-normalized pre-generated feature set on UCF-Crime and RGB modality, 5-crop, non-L2-normalized pre-generated feature set on XD-Violence. Snippet features are used in the testing mode. We follow the same implementation details of RTFM. The results are listed in Table 3. Note that in this table, AP and AUC uses the snippet score defined in Equation (5), while AP-2 and AUC-2 do not use the video classification score $p(\hat{y}_{cls}$) in Equation (5). The AP-2 and AUC-2 represent the testing mode in Figure 1(c).

From the results, we observe that the BERT video classification does not bring much performance improvement on UCF-Crime. Furthermore, the two APs are very close to each other. $\beta=0.5$ gives better results than $\beta=0.7$.

However, on the XD-Violence dataset, the BERT video classification brings surprising performance improvement. This is perhaps due to different attribute of dataset and is left for further study.

Similar to [14], where a transformer is used for both video classification and feature refinement, we test using BERT refined feature in MIL block. We use the same setting as in MIL-BERT on UCF-Crime using RGB+Flow modality. The result is listed in Table 4. From the results we notice that the performance of using $y_{i}$ is much worse. So this feature is not refined, but deteriorated. No need to mention that, we want an online testing mode shown in Figure 1(c) that offers a low complexity solution. The authors of [14] might not have realized this effect. We predict that if they implement the transformer as a video classifier only, their performance might be even better.

In the previous experiments, we also demonstrate that the transformer based video classifier works very closely to the BERT video classifier. As an ablation study, we implement a simpler LSTM based video classifier with two layers and hidden layer dimenstion same as input dimension. Only the feature $x_{i}$ is used in this classifier. The result is put in the third panel of Table 3. From the result we observe that its performance is almost identical to the BERT classifier, even though its complexity is a lot smaller (about 1/4) than the BERT. This proves the key contribution of this work, as the title states, the overlooked power of video classification is now implemented.

We compare our best results on UCF-Crime, ShanghaiTech and XD-Violence with SOTA results in the literaturem as shown in Table 5. We show our best AUC or AP result on each dataset. Note that we only include best results published in recent a few years. Older results a lot worse than the SOTA results, including those of the unsupervised ones, are not included.

From the results, we achieve new SOTA results on all three datasets. Please note that on UCF-Crime and ShanghaiTech, we use RGB+Flow modality while many previous works use RGB modality only [22], [14]. On the XD-Violence, we use RGB modality and achieve AP %82.10, a 3.51% jump from previous %78.59 [29] which uses both video and audio signals.

Anomaly score curves of some examples videos are shown Figure 2. We see that the BERT video classification work very effectively to press down scores for normal snippets. It may improve or degrade the scores for anomalous snippets. With this video classification, the decision threshold is lowered. The overall impact is reflected in the AUC values.