Audio-visual Representation Learning for
Anomaly Events Detection in Crowds

The recent anomaly detection methods can be divided into two main categories: local analysis and global analysis. As for local analysis methods, the locations where abnormal events occur are detected. The typical local analysis methods include [30, 31, 32, 33], etc. These methods detect the abnormal objects, such as trucks or running people in crowds and localize the position where anomalies occur. Liao et al. [34] use video descriptors to detect the fighting event in video frames. The other category, global analysis, usually analyze the whole video clips and predict whether there are abnormal events in the video clips. The global analysis methods have three typical schemes: trajectory points based, optical flow based, and classification based. [35, 36, 37] utilize the object trajectories extracted from crowd video clips for anomaly detection. Helbing et al. [38] proposed is a typical trajectory points based method named social force model for crowd interaction description. Inspired by [38], [35] and [37] introduce the social force model into the social events analysis in videos. Cui et al. [36] attempt to cluster trajectories by Fuzzy C-Means Clustering [39] and predict the category of input trajectory through clustering. Du et al. [33] propose a change detector named DSFA, and it utilizes two symmetric streams and slow feature analysis module to acquire better performance on changes in remote sensing imagery.

Because the definition of abnormal events in statistic based methods is ambiguous and inchoate, the VSD [40] and UCF-Crime [41] are proposed for abnormal events detection in crowd scenes, containing 7 and 13 different categories about anomalies respectively. The video clips in these two datasets are gained from movies and videos on internet. [42] is a typical method for video classification with VSD benchmark. In the meantime, [43] estimates the crowd emotion for abnormal behavior analysis and understanding on dataset in [44]. The datasets mentioned above define the abnormal events in crowd detailedly, leading the abnormal detection task to the video classification task. Therefore, we deploy the recent video classification methods into the anomaly detection and receive excellent results.

The joint audio-visual multi-modal learning attempts to the learn the representation from visual and auditory modalities for special tasks. The early researches on this field concentrate on the speech recognition task [45]. The motivation of these works is that the visual modality provides auxiliary information for audio analysis with noisy. A typical research on audio-visual multi-modal learning is to model the movement of face and mouth through the videos and corresponding audio signals [46, 47]. Similarly, the multi-modal learning is introduced into other tasks [48, 49].

In recent years, the audio-visual multi-modal learning is utilized in more general scenes. Owens et al. [50, 51] try to transfer the knowledge from audio learning to visual learning. Arandjelovic et al. [52] analyze the video through audio-visual relations. Meanwhile, this work is employed to sound localization [53, 54] and audio-visual separation [55].

This paper is inspired by these audio-visual multi-modal learning methods and we attempt to introduce the audio modality into the traditional visual-based abnormal detection methods. The anomaly detection in crowd scenes benefits from special ambient sounds in some events.

Compared with the images, videos have inherent temporal correlation, and video data have extra time dimension. In order to extract the temporal information, 3D convolutional filter, a 2D convolutional filter with time dimension, is proposed and introduced into the vision task on videos. Thus, the video representation learning module in AVRL framework uses 3D-ResNet [28] to extract the visual representation with temporal correlation among video frames.

Here, the input videos with $l$ frames are sampled to a fixed length $T$ ($T=50$ in this paper) with a fixed sample interval $\lfloor l/T\rfloor$. This fixed length frame sequence $\mathcal{V}\in\mathbb{R}^{C\times T\times H\times W}$ is passed to the visual representation learning module $\mathcal{F}_{vid}$. The visual representation $f_{v}$ is extracted with the equation:

where $v_{feat}\in\mathbb{R}^{C\times 1\times 1}$ is the finally output of 3D-ResNet-34, and $C$ is the number of channels ($C_{v}=512$ in 3D-ResNet).

In vanilla 3D-ResNet, the representation $v_{feat}$ is mapped to logits for events classification, achieving the abnormal events detection. In our proposed AVRL framework, this video representation do not use for classification directly, it will be merged with audio representation to realize the multi-modal learning system.

With the development of deep learning, there are some CNN-based methods [29] have applied on audio analysis successful. The CNN-based method are usually utilized to the image data due to the characteristic of convolutional layer. However, the audio signals from videos only has one-dimension, and a audio signal preprocess is necessary for the CNN-based audio representation learning.

In this paper, Log Mel Spectrogram (LMS) for audio preprocess is used in audio representation learning module, because the LMS features from audio is used in CNN-based audio task generally. According to the VGGish [29], the input audio signal from video is first resampled with 16kHz, and then the time-frequency map of resampled audio signal is gain through a Short Time Fourier Transform (STFT) with Hann window, and the details of preprocess are same as VGGish. The LMS feature $\mathcal{A}\in\mathbb{R}^{H\times W}$ has two-dimensiom, and the CNN-based methods can easily extract the hidden feature from audio signal. In AVRL, we use VGGish as audio representation learning module $\mathcal{F}_{aud}$. The audio representation extraction is defined as:

where $f_{a}\in\mathbb{R}^{C_{a}\times 1}$ is the audio representation, and $C_{a}=512$ in VGGish.

The traditional abnormal events detection methods only analyze the feature from videos, but some anomaly is not noticeable in crowd scene. Supposing that two people are fighting in a congested crowd, this fight event is not obvious in vision due to the dense crowd, but if the people in fight are roaring or crying, the fight event will be judged easily. Under this circumstance, the fusion of audio and visual representation has the advantages for events detection in crowd analysis. To detect abnormal events effectively, we fuse the audio and visual representation by audio-visual representation fusion module. In order to select the better fusion methods, we design several fusion strategy and compared their performances in Section IV-E. Finally, we select a simple and effective fusion method for AVRL framework.

Based on the visual/audio representation learning above, we fuse the representations through feature concatenation, and then pass this multi-modal feature into classification network $\mathcal{F}_{cls}$ to detect whether abnormal events occur in the video. The formula is shown here:

where $p\in\mathbb{R}^{N\times 1\times 1}$ is the event category prediction, $N$ is the number of categories in SHADE, $Cat$ denotes the feature concatenate manipulation. This simple fusion module achieve the better performance in comparison with other fusion strategy we designed, and the details of different fusion strategy is shown in Section IV-E. The classification network $\mathcal{F}_{cls}$ in audio-visual representation fusion module is a simple fully connected network.

Intuitively, the audio-visual representation fusion module combines the visual representation and audio representation, and extracts the comprehensive feature with fully connected network for abnormal events classification. With the auxiliary information from audio, the audio-visual combined representation is more discriminating compared with traditional visual methods. Although the fusion module of AVRL framework is quite simple, the experiment results illustrate that this simple strategy is effective in abnormal events detection, and even surpasses more complex fusion methods.

Anomaly events detection in crowd scenes is a video classification task. In order to find the anomaly events, the model should judge whether the action in video is belong to abnormal events, such as shoot, scatter, fight, etc. Given a video with category $y$ and a predict category $p$, we selected Cross Entropy Loss, a typical loss function for classification task. The loss function $\mathcal{L}$ is defined as:

where $M$ is the mini-batch size, $N$ denotes the number of categories.

We select the 3D-ResNet-34 pre-trained on UCF-101 [56] as the backbone for visual representation learning on video data. 3D-ResNet is a widely used deep learning model for video vision task. 3D-ResNet replace the traditional 2D convolutional layer in ResNet with 3D convolutional layer, introducing the temporal correlation in CNN. 3D convolutional filter adds an extra length dimension in 2D convolutional filter.

To be specific, suppose $F^{i,j}_{x,y,z}$ is the value at position $(x,y,z)$ on $j^{th}$ feature map from the output of $i^{th}$ 3D convolutional layer, and the input feature of $i^{th}$ layer has $C$ channels. The 3D convolutional filter has $H$ height, $W$ width, and $L$ length. The 3D convolution is shown as follow:

where $w^{ij}$, and $b^{ij}$ are the weight and bias of convolutional filter respectively.

In addition, we select a common audio analysis CNN, VGGish pre-trained on AudioSet [57], to learn the audio representation. The PCA postprocess of VGGish is removed from our audio representation learning module. Due to the discrepancy of audio length in SHADE dataset, the LMS features have different shapes. We impose Global Average Pooling (GAP) along channel on the output of audio representation, so that the audio representation $f_{a}$ has a unified size $C_{a}\times 1$.

The fused audio-visual representation is passed to a fully connected network for event categories classification. We design a simple fully connected network $\mathcal{F}_{cls}$, and the architecture of $\mathcal{F}_{cls}$ is illustrated in Table I:

Abnormal events detection can be considered as a video classification task, so that the classification accuracy is the core metric for this task. The Top-1 Accuracy is gauged to compare the performances among different methods:

where i indicates the $i^{th}$ category, and $N$ is the total number of categories in SHADE dataset. $M_{i}$ is the total number of samples belong to the $i^{th}$ category. $p_{i}$ and $y_{i}$ is the prediction and ground-truth of category label respectively. And the indicator function $\mathbbm{1}\{\cdot,\cdot\}$ is defined as Eq. 6. In the experiments, we gauge the Top-1 Accuracy on every category separately.

Due to the lack of audio collection recorders on current surveillance cameras, the recent abnormal events detection/classification datasets under real world only contain visual information without ambient sounds. Under this condition, the traditional action classification dataset or abnormal events detection datasets are not suitable for our AVRL framework, so that we choose a synthetic dataset: SHADE [7] which is generated in the video game named Grand Theft Auto V (GTA5), including 2149 videos (879,932 frames with the size of $1920\times 1080$). The videos in SHADE are from nine categories, arrest, chase, fight, knowkdown, run, shoot, scatter, normal type 1, and normal type 2. Every category contains about 200 videos with different weather conditions (rain, foggy, sunny, etc.) and different happening time. There are 1701 videos (about 80%) for training, and the remaining 448 videos (about 20%) for validation or testing. Meanwhile, we extract the audio sounds from videos for the training of audio representation learning module.

In this part, some experimental settings and details are explained. Both the spatial transformations and temporal transformations are imposed on the input video. The input frames are first resized to the size of $240\times 240$, and a random horizontal flip with a probability of 0.5. The temporal transformations sample $50$ frames from the whole frame sequence with a fixed step. The audio signal only preprocess to gain the LMS feature without any other transformations.

In the training phase, we select Stochastic Gradient Descent (SGD) method as the optimizer. The momentum of SGD optimizer is 0.9, and the weight decay is 0.001. The learning rate is set as $1e-2$ at the beginning, and the learning rate is updated automatically through minimize the validation loss. From the experiment results, the final learning rate is $1e-7$. The batch size in training is set as 12, and we train our model 80 epochs. The experiments run on two NVIDIA GTX 1080TI GPUs.

Table II illustrates the comparisons among listed models. Specially, we compare the performance with several visual-based video classification models, such as MLP, LSTM [58], LRCN [59], 3D ResNet [28], and N3D ResNet [7]. The comparison demonstrates that the audio information improves the classification accuracy significantly. The model introducing 3D convolutional layer, e.g. 3D ResNet and N3D ResNet, achieves the better performance among visual-based methods. Compared to the state-of-the-art method, N3D ResNet, on SHADE dataset, the proposed AVRL framework achieve higher Top-1 accuracy on majority of the categories, and only two categories, run and chase, are lower than N3D ResNet. Our AVRL embeds the audio representation learning module into the visual-based method 3D ResNet with an extremely simple fusion module, reaching the notable advance on Top-1 accuracy for events classification. The events with special ambient sounds, such as shoot, scatter, arrest, and knokdown, have the remarkable increase, which means the AVRL framework learns the discriminating feature in audio signals and aids the classification effectively. Meanwhile, only a simple concatenation method to fuse the audio and visual representations can gain a great improvement, showing the superiority of the multi-modal learning in abnormal events detection.

In Section III, we only introduce the concatenation fusion strategy. Actually, we design several different fusion strategy and compare the performance, and we choose the concatenation fusion strategy (Concat) finally. Besides the Concat strategy, the other fusion strategies are concatenation with layer normalization (Concat w/ LN), concatenation with unified magnitude (Concat w/ UM), addition fusion strategy (Add), and low-level feature fusion strategy (LF).

From the experiments, we find that the audio representation has a lower magnitude than video representation. The average visual representation is tenfold to that of audio representation. We attempt to introduce the layer normalization and a scale factor $S$ respectively to alleviate this divergence, acquiring the Concat w/ LN and Concat w/ UM strategies. Meanwhile, to find out the suitable fusion strategy, we design different prevalent feature fusion strategies, Add and LF, to select a better fusion strategy.

The details of several fusion strategy mentioned above are shown in Fig. 3. First, the Concat ((a) in Fig. 3) directly concatenates the visual and audio representation along channel dimension. The Concat w/ LN introduces the Layer Normalization after feature concatenation to balance the visual representation and audio representation. In addition, we adjust the magnitude by multiplying an hand-craft scale factor $S=10$ to the audio representation in Concat w/ UM. Due to the same shape of visual and audio representations, we naturally add the audio feature to visual feature, fusing the multi-modal features. The former fusion strategies all fuse the representations in the high-level feature. In contrast, we embed the audio representation in the low-level feature from visual representation. We first calculate the inherent correlation in audio representation by outer product, gaining the audio correlation map $Corr_{a}\in\mathbb{R}^{1\times 512\times 512}$. And the $Corr_{a}$ is resized to the same spatial shape with the input frames through bilinear interpolation. Then, the resized $Corr_{a}$ is stacked with all of input video frames along channel dimension.

The comparison of fusion strategy introduced above is illustrated in Table III, and the classification results are demonstrated in Fig. 4. From the comparison, the following conclusions are obtained: (1) The Concat strategy achieves relatively high performance among several fusion strategies, reaching the top place on shoot, arrest, and chase. (2) Compared to Concat, the Concat w/ LN reaches higher accuracy on run, normal1, and normal2, but the performances on events with discriminating ambient sounds (shoot and arrest) have significant decrease, which means the Concat w/ LN can not exploit the ambient sounds sufficiently. (3) The Concat w/ UM achieves better performance than Concat w/ LN, and the accuracy on run has increase, but the accuracy on chase decrease notably. We find that the Concat w/ UM strategy tends to consider the events with running people as run events, causing the increase of accuracy on run. However, the chase events contain two running people in scenes, and the Concat w/ UM confuses chase with run, leading to the decrease of accuracy on chase. (4) The Add also achieve high performances on several events, but the improvements is not higher than Concat, and there is a significant decrease of performance on chase due to the same reason as (3). Finally, the improvement of LF strategy is not obvious than other fusion strategies. In summary, with the comprehensive consideration, we select the Concat strategy in audio-video representation fusion module.

In the real world, the abnormal events usually happen in the dark environment. One of the motivations of AVRL is to aid the abnormal events detection task, especially in some degeneration scenes, such as low illumination, noise, etc. In order to stand out the robustness of AVRL, we select a subset in SHADE that only includes the dark scenes. The selected dark subset contains 147 events from 18:00 pm to 23:59 pm, 85 events from 0:00 am to 5:59 am, and 145 events occur in daytime with low illumination environments. Furthermore, the comparison between the proposed AVRL and the state-of-the-art N3D ResNet on SHADE dataset. Table IV illustrates the comparison on dark subset. From the table, we find that the proposed AVRL framework surpasses the state-of-the-art method N3D ResNet absolutely.

The nighttime events in the dark subset usually occur in quite environments, so that the event sounds are discriminative, leading to a better distinguish of anomalies. Due to this point, the classification on run and chase achieves better accuracy. And some events with notable ambient sounds, such as shoot, scatter, arrest, and knokdown, have the accuracy extremely close to 1.0. Interestingly, the AVRL does not reaches high performance on fight, run, and chase in SHADE dataset, but the performance on these categories in dark subset increases significantly. The daytime samples have noise from crowds involving chatting, bustle, etc. which do harm to the assistance from audio. Overall, the AVRL framework addresses the anomaly detection with dark scenes effectively, proving the advantage of audio-visual multi-modal learning.

The ablation study is conduct to further compare the influences from audio representation and visual representation. We consider three schemes: only visual scheme (3D ResNet), only audio (VGGish), and audio-visual (the proposed AVRL). The results of ablation study are shown in Table V.

From the Table V, the following conclusion can be obtained: (1) The 3D ResNet (only visual) achieves better accuracy than VGGish (only audio), which means the visual representation makes heavier contributions on abnormal events detection than audio representation. (2) VGGish (only audio) reaches high classification accuracy on the events with special ambient sounds, especially the shoot. (3) Although the performance of VGGish (only audio) is weak, the AVRL (audio-visual) achieves better classification accuracy by combining the audio and video representations.

In order to probe the relationship between visual representation and audio representation, we add patches randomly in videos, as shown in Fig. 5. As for temporal patch, the video frames are randomly selected under a proportion and colored by black (Fig. 5(a)). Moreover, the black patches with a proportion of the full spatial area are put to random positions on every frames (Fig. 5(b)). The comparison results are demonstrated in Fig. 6(a) and Fig. 6(b).

With the increase of patch ratio, the classification accuracy of both temporal patch and spatial patch decreases unsurprisingly. The effect of temporal patch is less than that of spatial patch, so that the accuracy of spatial patch have fallen more against temporal patch, which means that the spatial correlation in single frame provide more contributions than temporal correlation among frames for events classification. From Fig. 6, we find that the accuracy on some categories with 80% spatial patch ratio is lower than that of the method with only audio representation. This probably because the high spatial patch ratio lead to the extremely noisy distortion on content and this distortion mislead the visual representation learning, causing the impediment to events detection.