RANDOMSEMO: NORMALITY LEARNING OF MOVING OBJECTS
FOR VIDEO ANOMALY DETECTION

We propose RandomSEMO, a transformation method that eliminates the details of the foregrounds by randomly erasing the superpixels of the moving objects. The purpose of RandomSEMO is to let AE be trained to extract prototypical normal features of the foreground objects than simply learning to predict frames well.

To apply RandomSEMO, we first localize the moving objects. Given a sequence of frames $\mathbf{F_{t-N_{f}}}$, $\mathbf{F_{t-N_{f}+1}}$, $\mathbf{F_{t-N_{f}+2}}$, $\dots$, $\mathbf{F_{t-1}}$, $\mathbf{P_{t}}$, we compute the optical flow between each two successive frames (Fig. 1 (2)). Then, we generate moving object masks $\mathbf{MoMask}$ (Fig. 1 (3)) by thresholding the magnitude of the optical flow. Next, we apply SLIC [17] on the masked regions of the former frames to generate superpixel maps (Fig. 1 (4)) with maximum $N_{sp}$ components. Finally, we randomly erase the superpixels by replacing the values with zero (Fig. 1 (5)). Each superpixel is erased with probability $T_{sp}$.

By utilizing RandomSEMO, the network is forced to learn the appearance and motion features of the moving objects to infer the future frame with the randomly erased information, as seen in Fig. 1 (b). This solves the aforementioned excessive predicting power problem of the previous methods because our method makes the model to focus on the normal feature learning, rather than simply implying the upcoming frame with the cues of the input frames. Also, the AE effectively extracts normal features because RandomSEMO nudges the model to pay attention to the moving objects which are more potentially abnormal than the background.

To maximize the effect of RandomSEMO, we optimize our model with the newly proposed MOLoss. MOLoss is a weighted sum of a moving object weighted L1 loss $L_{m}$ and SSIM loss [18] $L_{s}$ obtained between $\mathbf{P^{\prime}_{t}}$ and $\mathbf{P_{t}}$. $L_{m}$ amplifies loss on the pixels near the moving objects by using a Moving Object Weight Map ($\mathbf{MOWM}$). Fig. 3 shows the example and generating process of MOWM. We sum up all the $\mathbf{MoMask}$ obtained during RandomSEMO to generate a summed binary mask ($\mathbf{SBM}$) (Fig. 3 (b)), which represents the moving object region of the input frame cuboid $C^{\prime}_{t}$. Therefore, $\mathbf{SBM}$ is expressed as:

$$\mathbf{SBM_{t}}=min\left(\sum_{k=0}^{t-1}\mathbf{MoMask_{k}},255\right)\vspace{-0.2cm}$$

(1)

Then, we apply a distance function to $\mathbf{SBM}$ to generate $\mathbf{MOWM}$, which demonstrates the inverse-distance from each pixel to the foreground. Thereby, the values near the foreground pixel are large and vice versa. $\mathbf{MOWM}$ is defined as:

$$\mathbf{MOWM^{t}}=255-\begin{cases}min\sqrt{(p_{x}-q_{x})^{2}+(p_{y}-q_{y})^{2}}&,p\in\mathbf{SBM}_{bg}\\ \hskip 56.9055pt0&,p\in\mathbf{SBM}_{fg}\end{cases},$$

(2)

where $p$ and $q$ indicate each pixel in $\mathbf{SBM_{t}}$ and its nearest foreground pixel, respectively. Evidently, the pixels near the foreground have relatively big values for the corresponding location in $\mathbf{MOWM}$. Then, $\mathbf{MOWM}$ is multiplied during L1 loss calculation, amplifying the values near the moving objects. During this stage, we add a small value $\eta$ to the tensorized $\mathbf{MOWM}$ to avoid zero values. Because $L_{m}$ is a pixel-level loss function, we combine this with $L_{s}$ to guarantee the similarity of $\mathbf{P^{\prime}_{t}}$ and $\mathbf{P_{t}}$ in the feature-level. The equations are as follows:

$$MOLoss(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})=w_{m}L_{m}(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})+w_{s}L_{s}(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})$$

(5)

The AE used in our method is trained to recognize the likelihood of the normal frames and predict the upcoming frame $\mathbf{P^{\prime}_{t}}$ from the transformed input frame cuboid $\mathbf{C^{\prime}_{t}}$ using the learned normal features. To avoid excessive generalization, we design a lightweight, yet effective AE. As demonstrated in Fig. 2, our AE is composed of a three-layer encoder and a three-layer decoder with skip connections in between to supplement the features lost during downsampling. Each encoder layer consists of 3D convolution, batch normalization, and LeakyReLU activation. After the last encoder layer, an Atrous Spatial Pyramid Pooling (ASPP) [21] layer takes place to enlarge the receptive field. The decoder layers are all made of 3D deconvolution, batch normalization, and ReLU activation.

During inference, we adopt the peak signal to noise ratio (PSNR) to estimate the abnormality of each frame. It is defined as $PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})=10\log_{10}\frac{\max(\mathbf{P^{\prime}_{t}})}{\|\mathbf{P^{\prime}_{t}}-\mathbf{P_{t}}\|_{2}^{2}/N}$, where N is the number of pixels in the frame. When $\mathbf{P_{t}}$ contains anomalies which our network has never seen during training, our network is incapable of predicting a clear $\mathbf{P^{\prime}_{t}}$, resulting in low PSNR and vice versa. Following [2, 11, 4, 7, 5, 10, 9, 8], we normalize $PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}}$) of each video clip to the range $[0,1]$ to obtain the final normality score $S_{t}$.

$$S_{t}=\frac{PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})-\min PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})}{\max PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})-\min PSNR(\mathbf{P^{\prime}_{t}},\mathbf{P_{t}})}\vspace{-0.15cm}$$

(6)

The quantitative results are shown in Table 1. We compare our network with eight VAD networks that do not require any pre-trained network—such as object detectors or optical flow networks—during inference for fair comparison. It is observed that the accuracy of our model is superior to most of the compared methods in the three datasets. Furthermore, our proposed network is capable of detecting anomalies at 127 frames per seconds (FPS), faster than most methods. Because accuracy and speed are both mandatory in the real world application, our network is more practical than other methods that show low accuracy or slow speed.

Fig. 5 demonstrates the output and difference map of our network compared to those of FastAno [9]. The difference map shows that our network more precisely localizes the abnormal object. Only the biker is saturated in our proposed model whereas the background pixels near the biker is unnecessarily highlighted in FastAno [9], mirroring the effect of our moving object aware RandomSEMO and MOLoss. Furthermore, the difference between foreground and background is more pronounced in our results.

We found the optimal value of $T_{sp}$ by changing $T_{sp}$ from $0.1$ to $0.6$. Table 2 and Fig. 4 show the result of the experiments conducted on Avenue [1] and Ped2 [3]. For Avenue [1], the accuracy is highest when $T_{sp}$ is $0.3$. For Ped2 [3], the model trained with $T_{sp}=0.2$ shows the best performance, and it is also as much accurate when trained with $T_{sp}=0.3$. Furthermore, it is observed that both low $T_{sp}$ leads to relatively low performance because only a few superpixels are obscured, practically having no significant difference from the original image. Similarly, the accuracy drops when $T_{sp}$ is high because too much information is lost.

We conducted ablation studies to validate the contribution of MOLoss. The results are compared in Table 3. We experimented by changing the loss function to $L_{1}$ loss, SSIM loss, and a weighted sum of $L_{1}$ and SSIM where the weights are equivalent to those of the MOLoss. From the results, it is observed that using MOLoss strongly boosts the performance, especially when compared to using a single $L_{1}$ loss.