Abnormal Behavior Detection Based on Target Analysis

The first step is to detect every targets in the video. We use the well-known Yolo v3 [10] algorithm for object detection. Each frame of video is inputted into the Yolo v3 to obtain the label, confidence score, and coordinates of the targets. We only keep the label with highest confidence score.

In the anomaly detection, the training set usually only contains the normal samples, thus we don’t know which labels correspond to the anomalies. According to the definition of the anomalies, an event is usually considered as an anomaly if it is not very likely to occur in the video. Therefore, if the label of the object appears in the training set, high object confidence means the low probability of anomalies. Otherwise, high object confidence means that the target is more likely to be an anomaly. Considering that the network may obtain some false detections in the object label of the training set, we only retain the object label with high confidence.

We take the training set of the anomaly detection dataset as the input of the Yolo v3, and get the label and confidence of each target. We choose the label whose confidence is higher than a threshold $\alpha$ to store in a list denoted as $label\_list$. The abnormal score $Sco_{obj}$ of each target is defined as:

$$Sco_{obj}=\left\{\begin{array}[]{l}\!-\!Conf_{obj}\hskip 19.91684ptif\hskip 2.84544ptObj_{label}\hskip 2.84544ptin\hskip 2.84544ptlabel\_list\\ Conf_{obj}\hskip 22.76228ptotherwise\end{array}\right.\$$

(1)

where $Obj_{label}$ and $Conf_{object}$ are the label and the confidence of the target respectively. The $Sco_{obj}$ of all targets in the test set are stored into a list denoted as $Sco_{obj}\_list$, and then we normalize each abnormal score to [0,1].

$$Sco_{obj}=\frac{{Sco_{obj}-\min(Sco_{obj}\_list)}}{{\max(Sco_{obj}\_list)-\min(Sco_{obj}\_list)}}$$

(2)

The difficulty for action recognition in the anomaly detection field is how to solve the problems of multi-target and multi-action recognition in a surveillance video. We propose a action recognition module to solve this problem. Based on the object detection, we use the well-known KCF [11] to track the target. Each target is tracked in $K$ frames, so we can get the the coordinates of the continuous $K$-frame of the target. After clipping, we get a short RGB video only containing the target. At the same time, the optical flow is extracted by using the coordinates of each target through previous frame and current frame of the video and obtain the optical flow video. Taking RGB video and optical flow video as the input of the action recognition network to obtain the action category and confidence of the target, thus we can solve the multi-target and multi-action detection problem in anomaly detection.

As for the action recognition network, we borrow the idea from the TSN algorithm [12]. The TSN algorithm combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. The video used in the TSN algorithm usually contains much redundant information. However, in the anomaly detection tasks, the video used for the action recognition is a short video obtained from the target tracking, which contains less redundant information. Therefore, we remove the sparse temporal sampling strategy.

The process of the action branch is shown in Fig.3. We use a two-stream ConvNet architecture which incorporates spatial and temporal networks for action recognition. The ConvNet used in our work is BNInception network [13]. We send each RGB image and optical flow image of the short video into the spatial network and the temporal network respectively and the category scores of different frames are averaged. Finally the category scores of the RGB and optical flow branches are fused by a weight. The weight and the training process is similar to those in the TSN algorithm.

Similar to the object branch, if the action category of the target has appeared in the training set, high action confidence means low probability of anomalies. We take the video frame sequence of each target tracked in the training set as the input of the action recognition network, and get the action category and confidence. After that, we choose the action category whose confidence is higher than $\beta$ to store in a list denoted as $action\_list$. The action abnormal score $Sco_{act}$ of each target is defined as:

$$Sco_{act}=\left\{\begin{array}[]{l}-Conf_{act}\hskip 11.38092ptif\hskip 2.84544ptAct_{label}\hskip 2.84544ptin\hskip 2.84544ptaction\_list\\ Conf_{act}\hskip 19.91684ptotherwise\end{array}\right.\$$

(3)

where $Act_{label}$ and $Conf_{act}$ are the action category and the confidence of the target respectively. The action abnormal score $Sco_{act}$ of all targets in the test set are stored into a list denoted as $Sco_{act}\_list$, and then we normalize each abnormal score to [0,1].

$$Sco_{act}=\frac{{Sco_{act}-\min(Sco_{act}\_list)}}{{\max(Sco_{act}\_list)-\min(Sco_{act}\_list)}}$$

(4)

We use Histogram of Magnitude Optical Flow (HMOF) features as the motion features. We extract the HMOF features of each target in the video and then send them to the auto-encoder network for reconstruction in order to further expand the difference between normal and abnormal features. All features of the training set are used to train the Gaussian Mixture Model (GMM) Classifiers, then we use the trained classifier to test the features of the testing set. Each feature get a score after passing through the classifier. The score is denoted as $Sco_{mot}$. Finally, the motion abnormal score $Sco_{mot}$ of all targets in the test set are stored into a list denoted as $Sco_{mot}\_list$, and each abnormal score is normalized to [0, 1].

$$Sco_{mot}=\frac{{\max(Sco_{mot}\_list)-Sco_{mot}}}{{\max(Sco_{mot}\_list)-\min(Sco_{mot}\_list)}}$$

(5)

The larger the $Sco_{mot}$, the less possibility the motion feature of the target match the distribution of normal motion features, so the target is more likely to be an anomaly.

After getting the abnormal scores of the three branches, we fuse these abnormal scores to get the final abnormal score. However, different branches play a different role in various scenes. For example, in the UMN dataset, the scene is more concerned with the panic running of the crowd, the weights of the motion and action branches should be higher than the object branch. Thus, the abnormal score of each branch should be weighted by a factor, and the final abnormal score of the target is determined by the three weighted abnormal scores.

In addition, the anomaly detection is different from other tasks. In practical applications, the monitoring system notifies the staff to handle the abnormality after detecting the abnormality. A few false detections only slightly burden the staff, but a few missing detections mean that the abnormal events cannot be handled in time, which may eventually lead to serious consequences. Considering the practical significance of anomaly detection, we select the maximum value of the abnormal scores from the weighted branches as the final fusing score denoted as $Sco_{fuse}$:

$$\max({w_{obj}}*Sco_{obj},{w_{act}}*Sco_{act},{w_{mot}}*Sco_{mot})$$

(6)

where the $w_{obj}$, $w_{act}$, $w_{mot}$ are the weight of the object, action and motion branch respectively. The $Sco_{fuse}$ of all targets in the test set are stored into a list denoted as $Sco_{fuse}\_list$, and then we normalize the final abnormal score $Sco_{fuse}$ of each target to [0,1].

$$Sco_{fuse}=\frac{{Sco_{fuse}-\min(Sco_{fuse}\_list)}}{{\max(Sco_{fuse}\_list)-\min(Sco_{fuse}\_list)}}$$

(7)

If the $Sco_{fus}$ is greater than the threshold $\partial$, it is considered an anomaly.

$$object=\left\{\begin{array}[]{l}abnoraml\hskip 19.91684ptif\hskip 2.84544ptSco_{fus}>\partial\\ normal\hskip 28.45274ptotherwise\end{array}\right.$$

(8)

Because the UMN dataset does not have pixel-level annotation information, we only evaluated it on the frame-level.

The hyperparameters are empirically set as follows: ${\rm{w}}_{object}$, ${\rm{w}}_{action}$, ${\rm{w}}_{object}$ are set to 1, 1.5, 1.5. $\alpha$ is 0.95, and $\beta$ is 0.99. The threshold $\vartheta$ of HMOF is 1.8, the interval $n$ is 8, and the number $K$ of the tracking frames is 5.

Table.1 shows the performance comparison between the proposed algorithm and other state-of-the-art algorithms. The object branch’s performance is very poor, since the anomalies are panic people, and the object branch is difficult to detect the anomaly. The motion branch is sensitive to the panic people and has better performance. The performance of the proposed method after fusion is not reduced, which indicates the effectiveness of the method. Compared with other state-of-the-art algorithms, the proposed algorithm has the best performance in the UMN dataset.

The UCSD Ped2 dataset is the standard benchmark for anomaly detection, and have different scenes. The hyperparameters are empirically set as follows: ${\rm{w}}_{object}$, ${\rm{w}}_{action}$, ${\rm{w}}_{object}$ are set to 1, $\alpha$ is 0.95, and $\beta$ is 0.99. The threshold $\vartheta$ of HMOF is 2.4, the interval $n$ is 8, and the number $K$ of tracking frames is 5.

Fig.4 shows the ROC curves on the UCSD Ped2 dataset and the EER comparison with state-of-the-art method is shown in Table2. From Fig.4 and Table.2, we can see that:

1. Object, motion and action branches’s performance have their own limitations, but the performance of the fusion is better than the respective branches in frame-level and pixel-level.

2. The performance of our method is significantly improved compared with MT-FRCN. The combined performance EER decreased by 11.6% at the frame-level, from 17.1% to 5.5%, and decreased by 4.9% at the pixel-level, from 19.4% to 14.5%.

3. Compared with other state-of-the-art algorithms, EER of our method has reached the lowest at frame-level and pixel-level respectively.