Weakly Supervised Realtime Dynamic Background Subtraction

There is a category of methods that use statistical approaches based on probability density estimation of pixel values. The most basic of these is the single Gaussian model [17]. However, this approach has limitations as a single function cannot account for all variations in pixel values. To overcome this, the Gaussian mixture model (GMM) [4] was proposed, which uses several Gaussian densities. Various improved versions of this traditional and widely used method have been presented [5, 6, 7, 8] with better results. Flux Tensor with Split Gaussian models (FTSG) [18] is a state-of-the-art method that uses flux tensor-based motion segmentation and GMM-based background modeling separately and then merges the results. Finally, it enhances the results using a multi-cue appearance comparison. However, parametric methods such as GMM and its successors are unable to handle sudden changes in a scene. To address this issue, a statistical non-parametric algorithm called KDE [9] was introduced, which uses kernel density estimation to model the probability of pixel values.

One of the main categories of methods for background modeling involves using controller parameters that update the background model based on dynamic feedback mechanisms. One such method, SuBSENSE [10], incorporates color channel intensity and spatiotemporal binary features and adjusts its parameters using pixel-wise feedback loops based on segmentation noise. PAWCS [11], a newer and more advanced method, extends the capabilities of SuBSENSE by generating a strong and persistent dictionary model based on spatiotemporal features and color. Similar to SuBSENSE, PAWCS also employs automatic feedback mechanisms to adjust its parameters. Another method, SWCD [12], combines the dynamic controllers of SuBSENSE with a sliding window approach to update background frames. Finally, CVABS [13], a recent subspace-based method, utilizes dynamic self-adjustment mechanisms like SuBSENSE and PAWCS to update the background model.

Ensemble methods have emerged as a new approach for change detection algorithms. The authors of [19, 20] have recently introduced a method named IUTIS (In Unity There Is Strength) that utilizes genetic programming (GP) to combine different algorithms and maximize their individual strengths. By selecting the best methods, combining them in various ways, and applying appropriate post-processing techniques, GP enables IUTIS to achieve high performance. The method shows promising performance by integrating several top-ranked methods evaluated on CDnet 2014 ([21]).

Several deep neural networks (NN) have been proposed in recent years for foreground segmentation, owing to the success of deep learning in computer vision. FgSegNet and its variations [22, 23, 24] are presently considered state-of-the-art based on their performance on CDnet 2014. Motion U-Net [25] is another deep NN method that requires fewer parameters than FgSegNet. BSPVGAN [26] employs Bayesian Generative Adversarial Networks (GANs) to create a background subtraction model. Another technique called Cascade CNN [27] uses a multi-resolution convolutional neural network (CNN) to segment moving objects. In DeepBS [28], a CNN is trained using patches of input images, which are then merged to reconstruct the frame. Temporal and spatial median filtering is utilized to enhance the segmentation outcomes. Another supervised approach, BSUV-Net [29, 30], is trained on some image sequences along with their spatiotemporal data augmentations, and exhibits good performance on unseen videos after training. Among the assessed methods on CDnet 2014, the aforementioned neural network techniques are ranked at the top. However, they require supervised training, which entails pixel-wise annotated ground truth, a time-consuming and impractical task in many situations.

A number of recently developed techniques, including SemanticBGS [31] and its variations RT-SBS-v1 and RT-SBS-v2 [32], integrate semantic segmentation with background subtraction methods. They employ the information from a semantic segmentation algorithm to obtain a pixel-wise probability that enhances the output of any background subtraction method. However, we cannot compare them to our method because they rely on pixel-level information as input, even though they are not trained using ground-truth labels.

In recent years, some weakly supervised methods have emerged that solely rely on image-level tags, which indicate whether a foreground object is present in the image [14, 15]. The method proposed in [15] generates a binary mask image by subtracting a background image from an input image to identify foreground regions. It then uses intermediate feature maps of a CNN to refine the foreground locations. However, image-level supervision presents a challenge due to the lack of location information in training the network. To address this, the method introduces some constraints that help to locate foreground pixels.

Another recent technique, LDB [14], adopts a tensor-based decomposition framework to represent the background as a low-rank tensor and classify the sparse noise as foreground. Additionally, it trains a two-stream neural network using an object-free video to explicitly learn the dynamic background. The dynamic background component of LDB leads to a more precise decomposition of the background and foreground, making it the current state-of-the-art method in weakly supervised moving object detection, particularly in dynamic background scenes.

Our proposed method is also based on explicit modeling of the dynamic background using a neural network. However, our framework differs from LDB in that it relies exclusively on neural networks for the segmentation of the background and foreground, rather than using a low-rank-based approach. As a result, we gain significant advantages in reducing the running time once our networks are trained. Further, LDB uses a very light CNN to model dynamic background. Instead, we use a U-Net to model the same. Because of a large number of parameters, U-Net has significant representative power, and it overfits the scene sequence to generate the dynamic background. We make use of this overfitting, because for a sequence containing moving objects the U-Net should ignore the moving objects and output only the dynamic background.

Our framework uses an autoencoder to generate static background images. Autoencoders are a type of neural network that consists of two components: an encoder and a decoder. The encoder maps the input to a compressed code, and the decoder reconstructs the input from the code, aiming to make the output as close to the input as possible [33]. Consequently, autoencoders learn a compressed and meaningful representation of the input data. This results in the removal of insignificant data and noise from the reconstructed input [34].

The autoencoder used in our method for generating the static background images is a fully connected one with dense layers, each followed by a SELU [35] activation function. The only exception is the last layer, which uses a Sigmoid activation function to limit the output values between zero and one. Fig. 2 shows the details of the architecture of the autoencoder used in our approach.

The $L_{recons}$ loss term, which is responsible for constructing the background images, is defined as follows (Eq. 1):

$$L_{recons}=\sum_{t=1}^{N}\|I_{t}-B_{t}\|_{1},$$

(1)

Here, $I_{t}$ and $B_{t}$ are the $t^{\text{th}}$ input and output of the autoencoder, respectively, and $\|.\|_{1}$ denotes the $L_{1}$-norm. We used the $L_{1}$-norm instead of the $L_{2}$-norm in $L_{recons}$ because it encourages sparsity [36].

The autoencoder can learn a low-dimensional manifold of the data distribution by applying constraints such as limiting the network’s capacity and choosing a small code size  [37]. The network can extract the most salient features of the data, and the $L_{recons}$ loss term imposes similarity between the input and reconstructed frames, allowing the autoencoder to learn a background model during optimization. This is possible because the input image sequence is temporally correlated, and the background of the images is common among them [38].

Autoencoders can be optimized in an unsupervised way and do not require labeled data. However, our second network, U-Net, requires pixel-wise labeled training data to be trained. A moving object-free sequence of input frames is used as training data. During the training phase, the autoencoder generates static background images of the training frames. The static background images are then subtracted from the input images, and after applying a threshold on them, the binary images are obtained. These binary images exhibit a dynamic background since they are extracted from training images without any foreground object. The entire process is illustrated in Fig. 1.

The second network in our proposed method is a U-Net, which was originally developed for image segmentation tasks and has shown great success in medical image analysis [16]. Its architecture resembles a U-shape and consists of two paths: a contracting path and an expansive path. The contracting path is composed of convolutional layers followed by ReLU activation functions and max-pooling layers, where the number of features gets doubled in each contracting step. The expansive path consists of up-convolutional layers for upsampling and halving the number of features, concatenations of features from the contracting path, convolutional layers, and ReLU activation functions. In our method, we used the basic U-Net architecture as described in [16].

U-Net is a network that requires pixel-wise labels for supervised training. However, in our framework, it is trained using the prepared binary images that were explained in the previous section, which makes our method weakly supervised. All we need for training is a moving object-free sequence. In other words, if we have a training sequence with frame-level tags whether the frame contains only background or not, we select only the frames with a tag value of zero as the training data. The output of U-Net is a binary image with pixel values of zero or one, where each pixel labeled as one indicates the presence of dynamic background.

Although most of the dynamic background pixels are detected by the U-Net, there is still a possibility that some of them may be missed due to the selected threshold when preparing the dynamic background ground-truth images. To address this, we use a per-pixel thresholding technique inspired by the SuBSENSE method [10] to obtain the foreground masks. This technique calculates the dynamic entropy of each pixel, represented by the dynamic entropy map $C(x)$, to detect blinking pixels. The dynamic entropy map tracks how often a pixel changes from being a foreground pixel to a background pixel, or vice versa, between consecutive frames.

The calculation of $C(x)$ is based on the XOR operator and is given by:

$$C(x)=\frac{1}{N-1}\sum_{t=2}^{N}XOR(S_{t}^{init}(x),S_{t-1}^{init}(x)),$$

(3)

Here, $x$ represents a pixel, $S_{t}^{init}$ is the binary result of the $t^{\text{th}}$ frame in the sequence after an initial segmentation, and $N$ is the total number of frames in the sequence. The initial segmentation uses $\alpha\max(F)$ as the threshold, $\max(F)$ is the maximum value of the foreground frames $F$, and $\alpha$ is a coefficient. The dynamic entropy map values, $C$, range from 0 to 1.

In the next step, we compute the pixel-wise distance thresholds using the following equation:

$$R(x)=\beta\max(F)+C(x),$$

(4)

Here, $\max(F)$ is the maximum value of the foreground frames $F$, and $\beta$ is a coefficient. The binary segmented result $S_{t}$ is obtained by applying the distance threshold $R(x)$ to the foreground $F_{t}(x)$.

Our method was implemented using the Keras deep learning framework [39]. The autoencoder architecture shown in Figure 2 consists of densely connected layers with 64, 32, 16, 4, 16, 32, and 64 units, respectively. All layers use the scaled exponential linear unit (SELU) activation function [35], except for the final layer which uses the sigmoid activation function to produce output values within the range of $[0,1]$.

The U-Net’s contracting path consists of four steps, each composed of two convolutional layers with a $3\times 3$ kernel size and ReLU activation function, followed by a $2\times 2$ max pooling operation with a stride of 2. The numbers of features are 64, 128, 256, 512, and 1024 for the top-to-bottom steps. The expansive path mirrors the contracting path but with two differences: first, features of the same contracting level are concatenated to the feature channels. Second, the max pooling operation is replaced with a transposed convolution layer with a $3\times 3$ kernel size and stride 2. Consequently, the number of features is halved in each expansive step. The final layer of the model is a convolutional layer with a $1\times 1$ kernel size and two features that construct the binary output image. Our U-Net architecture has the same design as the basic U-Net proposed in [16], except that we use convolutional and transposed convolution layers with the padding mode set to ’same’, which eliminates the need for the cropping operation in [16].

The hyper-parameters $\alpha$ and $\beta$ were set to $0.2$ and $0.08$, respectively, after conducting several trial and error experiments. We used the Adam optimization algorithm [40] with learning rates of $0.0001$ and $0.005$ for the autoencoder and U-Net, respectively. Both networks were trained for 50 epochs. During the testing phase, we achieved an average processing speed of 107 frames per second on the CDnet 2014 dataset [21] using a GeForce GTX 1080 Ti GPU.

We evaluated the effectiveness of our approach on various video datasets to demonstrate its suitability for real-world scenarios.

To evaluate the performance of our method, we utilize the F-Measure (FM) metric, which is a widely used performance indicator for moving object detection and background subtraction algorithms. The F-measure is calculated using the equation shown in (5), which combines the recall and precision scores. In order to maintain consistency with existing methods, we compute all evaluation metrics according to the definitions provided in [21].

$$\text{F-measure}=2*\frac{\text{Recall}*\text{Precision}}{\text{Recall}+\text{Precision}}$$

(5)

In Fig. 3, we present the intermediate and final qualitative results of our method’s steps on the videos. The first six rows depict the Dynamic Background category, followed by the next four rows from the Bad Weather category of the CDnet 2014 dataset. The last three rows show videos from the I2R dataset.

The first three columns display the input frame, the autoencoder-generated background, and the background-subtracted image, respectively. The fourth column exhibits the dynamic background image predicted by the U-Net. The next column displays the foreground image obtained by multiplying the background-subtracted image with the inverted version of the dynamic background image. The sixth and seventh rows showcase the initial segmented image after thresholding and the final segmented image after post-processing, respectively. The last column shows the ground-truth images.

In Fig. 3, it is evident from the 4th column that the U-Net model can efficiently capture the dynamic background motion and create a precise representation of the dynamic background image, particularly for the Dynamic Background videos. By comparing the background-subtracted image in the third column with the obtained foreground in the fifth column, it proves our method can effectively decompose the dynamic background from the foreground objects.

For the Bad Weather videos, our method is able to predict some of the dynamic background pixels. However, due to the nature of our autoencoder, it tends to absorb snow noise in the generated static background image, leading to a lack of visible snow pixels in the dynamic background image. Nevertheless, our method is still able to produce high-quality results, demonstrating its effectiveness in handling challenging weather scenarios.

Regarding the I2R dataset, our U-Net accurately predicts the dynamic background pixels in the “Campus” sequence and some of the dynamic background pixels in the “WaterSurface” sequence. However, in the “Fountain” sequence, the U-Net is not able to predict the fountain pixels in the dynamic background image since they are already detected as part of the background generated by the autoencoder. This is because the values of the fountain pixels remain constant in consecutive frames, and therefore, they are absorbed in the static background image.

The key aspect is to effectively separate the foreground pixels from the dynamic background pixels, which our framework achieves well through the use of both the autoencoder and the U-Net models. This ultimately leads to the superior performance of our method.

In this section, we present the quantitative results of our method compared to the top-performing methods on the CDnet 2014 dataset [21] listed on ChangeDetection.net website. Specifically, we chose the top 30 methods based on their average F-measure (FM) performance on the Dynamic Background and Bad Weather categories, excluding supervised methods and the ensemble method IUTIS [19]. We also included the results of the LDB weakly supervised method [14], as well as the CANDID algorithm [42], which was specifically proposed for dynamic background subtraction.

The results are shown in Table I. The second to eighth columns display the results on Dynamic Background videos, and the ninth to thirteenth columns show the results on Bad Weather videos. The methods are sorted based on their average FM on Dynamic Background videos, which is listed in the eighth column. Our method’s results are reported in the last row of the table.

As shown in Table I, our method achieves an average FM of 0.91 on Dynamic Background, outperforming all unsupervised methods and the LDB [14] method. Our method also achieves the highest FM on the “fall” video and performs the best on “fountain01” along with the FTSG method [18]. These results demonstrate the practicality of our method for dynamic background scenes with only the cost of frame-level tag training data.

On Bad Weather sequences, our method achieves an average FM of 0.89, outperforming all unsupervised top-ranked methods, but is surpassed by the LDB weakly supervised method [14], which has an FM of 0.91. Our method also achieves the best FM on the “blizzard” video, while LDB obtains the best FM on the “WetSnow” and “snowFall” sequences.

To compare our method to LDB more comprehensively, we also perform experiments on the I2R dataset and report the results in Table II. As shown, our method achieves the same average FM as LDB, but we obtain slightly better results on the “Fountain” and “WaterSurface” sequences, while LDB performs slightly better on the “Campus” sequence.

A comparison of our method and LDB on various videos shows that our method performs better in the Dynamic Background category, while LDB performs better in the Bad Weather category. For the I2R sequences, both methods achieve similar performance. It is worth noting that LDB uses a low-rank tensor decomposition and is a batch method, whereas our method is an online method achieving realtime performance where an input image is fed through the two networks to obtain the result.

We also compared our method to another weakly supervised method [15] described in the related work section. Minematsu et al. performed experiments on eight categories of the CDnet 2014 dataset [21], but only selected some of the sequences for each category. We performed the same experiments and report the results in Table III. As shown in the table, our method achieves an average FM of $0.72$, while Minematsu et al. [15] obtain an average FM of $0.66$. Our method outperforms theirs on the Bad Weather, Dynamic Background, Shadow, Thermal, and Turbulence sequences, while they achieve better performance on the Baseline, Camera Jitter, and Night Videos sequences.